Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Stop copying LogicalPlan and Exprs in EliminateNestedUnion #10296

Closed
alamb opened this issue Apr 29, 2024 · 1 comment · Fixed by #10319
Closed

Stop copying LogicalPlan and Exprs in EliminateNestedUnion #10296

alamb opened this issue Apr 29, 2024 · 1 comment · Fixed by #10319
Assignees
@emgeee
Labels
good first issue Good for newcomers

Comments

@alamb
Copy link
Contributor

alamb commented Apr 29, 2024

Is your feature request related to a problem or challenge?

Part of #9637

As part of making the planner faster, we are updating the optimizer passes to avoid copying LogicalPlan and Expr (see #9637)

Describe the solution you'd like

I would like to reduce the amount of copying in this pass (even though it doesn't appear in current profiling)

Describe alternatives you've considered

Apply the model from @Lordworms in #10166 to this pass
2. Update OptimizerRule::supports_rewrite` to return true

  1. Update OptimizerRule to use rewrite
  2. Update the pass itself to not copy the LogicalPlan (ideally using the TreeNode API) - it is implemented for LogicalPlan (API) and Expr (API)

Other examples: #10218

Additional context

I think this will be a pretty simple refactor with a good existing pattern, so mark this as good first issue
@alamb alamb added the good first issue Good for newcomers label Apr 29, 2024
@alamb alamb mentioned this issue Apr 29, 2024
Closed
31 tasks
@emgeee
Copy link
Contributor

emgeee commented Apr 30, 2024

take

emgeee added a commit to emgeee/datafusion that referenced this issue Apr 30, 2024
@emgeee
First pass of apache#10296
22609aa
emgeee added a commit to emgeee/datafusion that referenced this issue Apr 30, 2024
@emgeee
First pass of apache#10296
d90f277
emgeee added a commit to emgeee/datafusion that referenced this issue Apr 30, 2024
@emgeee
First pass of apache#10296
d592275
@emgeee emgeee mentioned this issue Apr 30, 2024
Merged
alamb added a commit that referenced this issue May 2, 2024
@emgeee @alamb
Stop copying LogicalPlan and Exprs in EliminateNestedUnion (#10319)
d4da80b 
* First pass of #10296

* Update datafusion/optimizer/src/eliminate_nested_union.rs

Co-authored-by: Andrew Lamb <[email protected]>

* fix formatting

* return original expr instead of clone()

---------

Co-authored-by: Andrew Lamb <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@emgeee emgeee

Labels
good first issue Good for newcomers
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Stop copying LogicalPlan and Exprs in EliminateNestedUnion emgeee/datafusion
2 participants
@alamb @emgeee