[GLUTEN-7068][CORE] Fix issue updating leaf input metrics #7067
Conversation
|
Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues? https://github.com/apache/incubator-gluten/issues Then could you also rename commit message and pull request title in the following format? See also: |
|
Run Gluten Clickhouse CI |
ivoson
force-pushed
the
fix-input-metrics
branch
from
c8c1d4c to
a3388f6
Compare
|
Run Gluten Clickhouse CI |
ivoson
changed the title
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
|
cc @zhztheplayer @ulysses-you can you please review? Thanks |
| val leavesBuffer = new ArrayBuffer[TransformSupport]() | ||
| collectTransformSupportLeaves(child, leavesBuffer) | ||
| val leavesMetricsUpdater = leavesBuffer.map(_.metricsUpdater()) |
There was a problem hiding this comment.
nit: leafBuffer and leafMetricsUpdater? As one should already know they are for leaves.
| case t: TransformSupport if t.children.forall(!_.isInstanceOf[TransformSupport]) => true | ||
| case _ => false | ||
| private def leafInputMetricsUpdater(): InputMetricsWrapper => Unit = { | ||
| def collectTransformSupportLeaves( |
There was a problem hiding this comment.
nit: Could rename to collectLeaves? To simplify code.
There was a problem hiding this comment.
Sure, will update in next iteration.
gluten-substrait/src/main/scala/org/apache/gluten/execution/WholeStageTransformer.scala
Outdated
Show resolved
Hide resolved
…oleStageTransformer.scala Co-authored-by: Hongze Zhang <[email protected]>
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
|
Run Gluten Clickhouse CI |
|
late lgtm |
What changes were proposed in this pull request?
Fix issue #7068 collecting input metrics such as bytesRead and recordsRead from file scan.
How was this patch tested?
UT added.