Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Delta][VL] Support IncrementMetric expression #7110

Closed
dcoliversun opened this issue Sep 4, 2024 · 1 comment · Fixed by #7111
Closed

[Delta][VL] Support IncrementMetric expression #7110

dcoliversun opened this issue Sep 4, 2024 · 1 comment · Fixed by #7111
Labels
enhancement New feature or request

Comments

@dcoliversun
Copy link
Contributor

Description

Delta uses IncrementMetric to count the number of rows passing through it.

More info: https://github.com/delta-io/delta/blob/master/spark/src/main/scala/org/apache/spark/sql/delta/metric/IncrementMetric.scala

We found the operation with expression fallbacks.

== Fallback Summary ==
(4) Project: Not supported to map spark function name to substrait function name: true, class name: IncrementMetric.
(25) Project: Not supported to map spark function name to substrait function name: false, class name: IncrementMetric.

== Physical Plan ==
VeloxColumnarToRowExec (32)
+- ^ ProjectExecTransformer (30)
   +- ^ FilterExecTransformer (29)
      +- ^ InputIteratorTransformer (28)
         +- RowToVeloxColumnar (26)
            +- Project (25)
               +- VeloxColumnarToRowExec (24)
                  +- ^ ShuffledHashJoinExecTransformer FullOuter BuildLeft (22)
                     :- ^ InputIteratorTransformer (14)
                     :  +- ShuffleQueryStage (12), Statistics(sizeInBytes=<> GiB, rowCount=<>)
                     :     +- ColumnarExchange (11)
                     :        +- ^ ProjectExecTransformer (9)
                     :           +- ^ FilterExecTransformer (8)
                     :              +- ^ InputIteratorTransformer (7)
                     :                 +- RowToVeloxColumnar (5)
                     :                    +- * Project (4)
                     :                       +- VeloxColumnarToRowExec (3)
                     :                          +- ^ Scan parquet spark_catalog.<db>.<tbl>(1)
                     +- ^ InputIteratorTransformer (21)
                        +- ShuffleQueryStage (19), Statistics(sizeInBytes=<> GiB, rowCount=<>)
                           +- ColumnarExchange (18)
                              +- ^ ProjectExecTransformer (16)
                                 +- ^ Scan parquet spark_catalog.<db>.<tbl>(15)
@zhztheplayer
Copy link
Member

@dcoliversun Hi, I just bumped into the relevant code. Do you know whether we have some test cases for PR #7111 in the code base?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants