-
Notifications
You must be signed in to change notification settings - Fork 447
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[CORE] Add custom cost evaluator for optimize buildSide of shuffled hash join #6143
Conversation
Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues? https://github.com/apache/incubator-gluten/issues Then could you also rename commit message and pull request title in the following format?
See also: |
Run Gluten Clickhouse CI |
cc @zhztheplayer @ulysses-you thank you. |
"If true and gluten enabled, use " + | ||
"org.apache.spark.sql.execution.adaptive.GlutenCostEvaluator as custom cost " + | ||
"evaluator class, else follow the configuration " + | ||
"org.apache.spark.sql.execution.adaptive.GlutenCostEvaluator.") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
org.apache.spark.sql.execution.adaptive.GlutenCostEvaluator
this should be corrected
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My mistake, fixed, thank you.
Run Gluten Clickhouse CI |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks!
case class GlutenCostEvaluator() extends CostEvaluator with SQLConfHelper { | ||
override def evaluateCost(plan: SparkPlan): Cost = { | ||
val forceOptimizeSkewedJoin = conf.getConf(SQLConf.ADAPTIVE_FORCE_OPTIMIZE_SKEWED_JOIN) | ||
new GlutenCost(SimpleCostEvaluator(forceOptimizeSkewedJoin), plan) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if (GlutenConfig.getConf.enableGluten) {
GlutenCost(SimpleCostEvaluator(forceOptimizeSkewedJoin), plan)
} else {
SimpleCostEvaluator(forceOptimizeSkewedJoin)
}
To make it as a query level config.
Run Gluten Clickhouse CI |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm if tests pass
Thank you for review. @ulysses-you @zhztheplayer |
What changes were proposed in this pull request?
Split from #6093 .
Add custom cost evaluator for use the smaller table to build hashmap in shuffled hash join.
How was this patch tested?