Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[VL] Offload TakeOrderedAndProjectExec at planning time rather than at runtime #5927

Closed
wants to merge 9 commits into from

Conversation

zhztheplayer
Copy link
Member

@zhztheplayer zhztheplayer commented May 30, 2024

Offload TakeOrderedAndProjectExec at planning time. This means TakeOrderedAndProjectExec will be replaced to Sort + Limit + Project or TopN + Project during columnar rule execution, and the replaced operators can be displayed on UI and by SQL explain.

Examples:

Single Top N

Plan before:

VeloxColumnarToRowExec (53)
   +- TakeOrderedAndProjectExecTransformer (52)
      +- ^ RegularHashAggregateExecTransformer (50)
         +- ^ InputIteratorTransformer (49)
            +- ^ InputAdapter (48)
               +- ^ ShuffleQueryStage (47)
...

After:

VeloxColumnarToRowExec (54)
   +- ^ ProjectExecTransformer (52)
      +- ^ TopNTransformer (51)
         +- ^ RegularHashAggregateExecTransformer (50)
            +- ^ InputIteratorTransformer (49)
               +- ^ InputAdapter (48)
                  +- ^ ShuffleQueryStage (47)
...

Local + Global Top N

Before:

VeloxColumnarToRowExec (39)
   +- TakeOrderedAndProjectExecTransformer (38)
      +- ^ ProjectExecTransformer (36)
         +- ^ RegularHashAggregateExecTransformer (35)
            +- ^ RegularHashAggregateExecTransformer (34)
               +- ^ ProjectExecTransformer (33)
                  +- ^ ShuffledHashJoinExecTransformer Inner (32)
                     :- ^ InputIteratorTransformer (23)
                     :  +- ^ InputAdapter (22)
                     :     +- ^ ShuffleQueryStage (21), Statistics(X)
...

After:

VeloxColumnarToRowExec (45)
   +- ^ ProjectExecTransformer (43)
      +- ^ TopNTransformer (42)
         +- ^ InputIteratorTransformer (41)
            +- ^ InputAdapter (40)
               +- ^ ColumnarExchange (39)
                  +- ^ TopNTransformer (37)
                     +- ^ ProjectExecTransformer (36)
                        +- ^ RegularHashAggregateExecTransformer (35)
                           +- ^ RegularHashAggregateExecTransformer (34)
                              +- ^ ProjectExecTransformer (33)
                                 +- ^ ShuffledHashJoinExecTransformer Inner (32)
                                    :- ^ InputIteratorTransformer (23)
                                    :  +- ^ InputAdapter (22)
                                    :     +- ^ ShuffleQueryStage (21), Statistics(X)
...

@zhztheplayer zhztheplayer changed the title [VL] Offload TakeOrderedAndProjectExec at planning time rather than at runtime [CORE] Offload TakeOrderedAndProjectExec at planning time rather than at runtime May 30, 2024
@apache apache deleted a comment from github-actions bot May 30, 2024
@apache apache deleted a comment from github-actions bot May 30, 2024
Copy link

Run Gluten Clickhouse CI

5 similar comments
Copy link

Run Gluten Clickhouse CI

Copy link

Run Gluten Clickhouse CI

Copy link

Run Gluten Clickhouse CI

Copy link

Run Gluten Clickhouse CI

Copy link

Run Gluten Clickhouse CI

Copy link

Run Gluten Clickhouse CI

Copy link

Run Gluten Clickhouse CI

fixup

fixup

fixup
@zhztheplayer zhztheplayer changed the title [CORE] Offload TakeOrderedAndProjectExec at planning time rather than at runtime [VL] Offload TakeOrderedAndProjectExec at planning time rather than at runtime May 31, 2024
Copy link

Run Gluten Clickhouse CI

Copy link

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

@github-actions github-actions bot added the stale stale label Jul 16, 2024
Copy link

This PR was auto-closed because it has been stalled for 10 days with no activity. Please feel free to reopen if it is still valid. Thanks.

@github-actions github-actions bot closed this Jul 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
stale stale
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant