Add LogicalPlanStats to logical plan nodes
#13618
Conversation
github-actions
bot
added
documentation
|
cc @alamb , @berkaysynnada , @findepi I haven't created any issue for this change yet, but will do if we think it is worth doing such a huge breaking change. |
I love the idea in principle (vide #12604) There are however some important questions
Those observations lead to conclusions on the design what should and shouldn't be part of a plan (logical or physical):
To me, the best way to address these various needs would be
|
|
Yeah, I see your point. I think this all comes down to if we want to / have the resources to implement a new IR plan structure and refactor the existing analyzer / optimizer rules. (And it can be also a breaking change to projects that have their own rules...) Or, we want to / can adjust the existing logical plan to incorporate the above ideas without much API breaking changes and so keep the existing optimizer and the exsisting logical to physical plan conversion. |
|
The LogicalPlan loose typing, overly rich Expr syntax and close ties to the syntax tree are an ongoing maintenance tax we continue to pay with every new change, every new feature, so I do hope we accumulate enough decidedness to commence to a new IR. I personally would be very interested in working on this, but this also requires buy-in from other project maintainers. |
This is quite cool 🤯
I don't know if we have the resources to do this. I know I don't have the bandwidth to help drive it forward but there are now quite a few other high bandwidth maintainers who might be able to do so. My personal focus for the next few months is likely on making DataFusion more stable for existing systems, which is likely not exacly aligned with making major changes. However I think internal refactoring is possible (we did it with function representation --> all to udfs) it just needs sustained coding and organizational effort |
|
The pre-computed bitset to know what is in each subtree is a very neat idea |
|
I am cleaning up the review queue and this PR has a pile of conflicts. I think it is a neat POC type approach, but I don't think we are planning to proceed as is -- though please correct me if I got that wrong |
|
Hi @alamb, sure we can close this PR. If I get the above comments right, there will be a new IR plan structure and the current optimizer rules will be rewritten to work on that. Once DataFusion reached that point we can revisit this PR to improve traversal speed of optimizer rules. However, the idea in this PR is not specific to optimizer rules. It can improve analyzer rules (I think we want to keep them using The problem with |
Maybe that would make more sense. I just feel like our iteration code has gotten pretty intense lately (and it is already getting pretty hard to understand). I understand the need for optimization, but I think we also need to balance it with maintainability / readability |
|
Atually I just realized after reading @jayzhan211's comment that |
|
I didn't notice this change, it looks quite similar to what I proposed. What is the current challenge faced to? It seems like HUGE breaking change? |
|
Yeah, adding a new field to |
|
I quickly scan the change. I think we don't require to modify pub struct Projection {
/// The list of expressions
pub expr: Vec<Expr>,
/// The incoming logical plan
pub input: Arc<LogicalPlan>,
/// The schema description of the output
pub schema: DFSchemaRef,
// Any kind of computed information place here
properties: LogicalPlanProperties
} I guess this reduces about half the change. And given the approach in the above, we can also change the plan one by one (I guess), so we can iterate the change in a smaller and easy review one. Some plans such as |
|
Yeah, most likely we don't need The property ( |
Which issue does this PR close?
This a proof of concept PR to improve performance of
TreeNodetraversals. The main purpose of this PR is to demonstrate a promising but API breaking optimization.Rationale for this change
TreeNodetraversal APIs are crucial parts of query plan analysis and optimzation. This PR explores the idea of storing some pre-calculated statistics/properties of nodes (or subtrees) inside the nodes during creation and automatically update the values during transfomations.This PR focuses on logical plan building blocks (
LogicalPlan,Expr) and only one particular optimization that stores a bitset pattern in each node to describe the subtree content, but additional attributes/properties can be added in follow-up PRs like:Expr::get_type()(LargeORlist overflows the stack #9375 (comment), Proposal: introduced typed expressions, separate AST and IR #12604)and the idea can be extended to physical trees as well.
What changes are included in this PR?
To store the pre-calculated statistics/properties
LogicalPlanStatsstruct fields are added to eachLogicalPlanandExprnodes:This might look redundant but most likely this is the least intrusive way to the existing code. Pattern matching against
LogicalPlanandExprjust need to add a new param, while new enum constructor methods can be defined as lowercase version of the enum items (if there is no conflict with existing methods). The constructor methods calculate theLogicalPlanStatsfields based on main content of the node.Please note that even this approach requites quite a lot of API breaking changes in the codebase, but all are mechanical. The following table summarizes the before/after code to construct the enum items. (I just used the
_prefix to avoid conflicts, but we can come up with better names.)Expr::AliasExpr::alias_qualifiedExpr::ColumnExpr::columnExpr::ScalarVariableExpr::scalar_variableExpr::LiteralExpr::literalExpr::BinaryExprExpr::binary_exprExpr::LikeExpr::_likeExpr::SimilarToExpr::similar_toExpr::NotExpr::_notExpr::IsNotNullExpr::_is_not_nullExpr::IsNullExpr::_is_nullExpr::IsTrueExpr::_is_trueExpr::IsFalseExpr::_is_falseExpr::IsUnknownExpr::_is_unknownExpr::IsNotTrueExpr::_is_not_trueExpr::IsNotFalseExpr::_is_not_falseExpr::IsNotUnknownExpr::_is_not_unknownExpr::NegativeExpr::negativeExpr::BetweenExpr::_betweenExpr::CaseExpr::caseExpr::CastExpr::castExpr::TryCastExpr::ScalarFunctionExpr::scalar_functionExpr::AggregateFunctionExpr::aggregate_functionExpr::WindowFunctionExpr::window_functionExpr::InListExpr::_in_listExpr::ExistsExpr::existsExpr::InSubqueryExpr::in_subqueryExpr::ScalarSubqueryExpr::scalar_subqueryExpr::WildcardExpr::wildcardExpr::GroupingSetExpr::grouping_setExpr::PlaceholderExpr::placeholderExpr::OuterReferenceColumnExpr::outer_reference_columnExpr::UnnestExpr::unnestLogicalPlan::ProjectionLogicalPlan::projectionLogicalPlan::FilterLogicalPlan::filterLogicalPlan::WindowLogicalPlan::windowLogicalPlan::AggregateLogicalPlan::aggregateLogicalPlan::SortLogicalPlan::sortLogicalPlan::JoinLogicalPlan::joinLogicalPlan::RepartitionLogicalPlan::repartitionLogicalPlan::UnionLogicalPlan::unionLogicalPlan::TableScanLogicalPlan::table_ccanLogicalPlan::EmptyRelationLogicalPlan::empty_relationLogicalPlan::SubqueryLogicalPlan::subqueryLogicalPlan::SubqueryAliasLogicalPlan::subquery_aliasLogicalPlan::LimitLogicalPlan::limitLogicalPlan::StatementLogicalPlan::statementLogicalPlan::ValuesLogicalPlan::valuesLogicalPlan::ExplainLogicalPlan::explainLogicalPlan::AnalyzeLogicalPlan::analyzeLogicalPlan::ExtensionLogicalPlan::extensionLogicalPlan::DistinctLogicalPlan::distinctLogicalPlan::DmlLogicalPlan::dmlLogicalPlan::DdlLogicalPlan::ddlLogicalPlan::CopyLogicalPlan::copyLogicalPlan::DescribeTableLogicalPlan::describe_tableLogicalPlan::UnnestLogicalPlan::unnestLogicalPlan::RecursiveQueryLogicalPlan::recursive_queryFor the above mentioned pattern based optimization the PR defines a
LogicalPlanPatternenum that contains all possible node kinds of a logical plan:A bitset of
LogicalPlanPatternenum is added to theLogicalPlanStatsstruct to reflect the content of the node's subtree. The implementation could use any kind of bitset, but https://docs.rs/enumset/latest/enumset/ looks like a good candidate.For example here are a few
Expritem constructors:While maintaining the bitset during tree transformations comes with some costs, with the bitset we can speed up
LogicalPlanandExprtraversals significantly. For example if we have a traversal that does something withExpr::BinaryExprnodes only:then we can check the presence of
Expr::BinaryExprin a subtree and simply skip traversing subtrees without theLogicalPlanPattern::ExprBinaryExprpattern:I modified some of the traversal functions in this PR to demonstrate that the optimization brings significant performance improvement to
sql_planner:Most likely further improvements could be achievable by using the new patterns in more traversals, but let's leave it to follow-up PRs.
To sum up this PR contains 3 initial commits:
Expr::Wildcardto incorporate its fields to a namedWildcardstruct. This makesExpr::Wildcardsimilar to otherExpritems.LogicalPlanStatscontainer to allLogicalPlanandExprenum items.LogicalPlanStatsis just an empty struct in this commit and all the changes are mechanical.LogicalPlanStatsand adjusts some of the logical plan traversals to utilize it.Are these changes tested?
Yes, with existing tests.
Are there any user-facing changes?
No.