-
Notifications
You must be signed in to change notification settings - Fork 164
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
perf: Apply DataFusion's projection pushdown rule #907
base: main
Are you sure you want to change the base?
Conversation
impl Default for PhysicalPlanner { | ||
fn default() -> Self { | ||
let session_ctx = Arc::new(SessionContext::new()); | ||
let execution_props = ExecutionProps::new(); | ||
Self { | ||
exec_context_id: TEST_EXEC_CONTEXT_ID, | ||
execution_props, | ||
session_ctx, | ||
} | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This code was only used in tests so I moved it into the test module
Ok(Arc::new(CopyExec { | ||
input: new_input, | ||
input: Arc::clone(&children[0]), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Adding the optimizer highlighted that the previous implementation of CopyExec::with_new_children
was incorrect.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @andygrove
Makes sense to me. Would you mind to share how much is the performance benefit?
let state = SessionStateBuilder::new() | ||
.with_config(session_config) | ||
.with_runtime_env(Arc::new(runtime)) | ||
.with_default_features() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
default features needed to use the planner?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let me see if I can remove that
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Removing this causes test failures. We were previously calling the following method to create the context and this also enabled default features.
pub fn new_with_config_rt(config: SessionConfig, runtime: Arc<RuntimeEnv>) -> Self {
let state = SessionStateBuilder::new()
.with_config(config)
.with_runtime_env(runtime)
.with_default_features()
.build();
Self::new_with_state(state)
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
since we using a planner only perhaps this should be enough
.with_query_planner(SessionStateDefaults::default_expr_planners())
?
I don't expect it to have much impact, but I am running some benchmarks now. I will post results later today. |
The benchmark results are not very exciting, and the improvements could just be noise. However, there is some correlation with the improvements in q18 and q21 that were noted in apache/datafusion#9236 (comment). |
.with_config(session_config) | ||
.with_runtime_env(Arc::new(runtime)) | ||
.with_default_features() | ||
.with_physical_optimizer_rules(vec![Arc::new(ProjectionPushdown::new())]) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I wonder if other rules can be considered?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Possibly. We are starting out with a physical plan that is already optimized by Spark though, so many optimizations have already been applied. We also are running queries in single partitions within a distributed cluster so we cannot leverage anything that uses RepartitionExec
.
It makes sense. As the projection happens after join in HashJoin operator, it looks more like an early projection from the upper Projection operator. I don't expect this could get some performance on the join. |
Which issue does this PR close?
Closes #908
Rationale for this change
Improve performance (and reduce memory overhead) of
HashJoinExec
by pushing down projection into the join leveraging a rule that already exists in DataFusion.Here is an example from TPC-DS q3.
Before
Note the two instances of
ProjectionExec
and noprojection
displayed in theHashJoinExec
.After
Now there is only one
ProjectionExec
and a projection of 4 columns has been pushed into theHashJoinExec
. I suspect that we could remove the remainingProjectionExec
with some additional work (in a separate PR).What changes are included in this PR?
Enable optimizer with projection pushdown rule, which can potentially fuse the projection with other operators.
How are these changes tested?
Existing tests.