Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CORE] Add expression blacklist #3985

Merged
merged 1 commit into from
Dec 12, 2023
Merged

Conversation

ulysses-you
Copy link
Contributor

@ulysses-you ulysses-you commented Dec 8, 2023

What changes were proposed in this pull request?

This pr adds a new config spark.gluten.expression.blacklist to fast skip transform expression. It can be used when we find bug for expression or do some debug.

How was this patch tested?

add test

Copy link

github-actions bot commented Dec 8, 2023

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/oap-project/gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

Copy link

github-actions bot commented Dec 8, 2023

Run Gluten Clickhouse CI

Copy link

Run Gluten Clickhouse CI

Copy link

Run Gluten Clickhouse CI

Copy link

Run Gluten Clickhouse CI

Copy link

Run Gluten Clickhouse CI

Copy link
Contributor

@PHILO-HE PHILO-HE left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your PR! Could you please update Configuration.md?

@@ -1252,6 +1261,12 @@ object GlutenConfig {
.stringConf
.createWithDefaultString("")

val EXPRESSION_BLACK_LIST =
buildStaticConf("spark.gluten.expression.backlist")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo: blacklist

Copy link

Run Gluten Clickhouse CI

Copy link

Run Gluten Clickhouse CI

@ulysses-you
Copy link
Contributor Author

ulysses-you commented Dec 12, 2023

thank you @PHILO-HE for the reminder, addressed.

Copy link
Contributor

@PHILO-HE PHILO-HE left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! cc @baibaichen, please note this change.

@ulysses-you ulysses-you merged commit 8c6f164 into apache:main Dec 12, 2023
18 checks passed
@ulysses-you ulysses-you deleted the blacklist branch December 12, 2023 12:03
@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_3985_time.csv log/native_master_12_11_2023_71ec720d4_time.csv difference percentage
q1 34.04 32.10 -1.946 94.28%
q2 25.21 24.79 -0.424 98.32%
q3 37.66 38.42 0.759 102.02%
q4 37.45 39.90 2.457 106.56%
q5 71.78 72.41 0.627 100.87%
q6 6.90 7.02 0.115 101.67%
q7 85.25 85.38 0.124 100.15%
q8 85.81 86.65 0.843 100.98%
q9 121.25 126.76 5.504 104.54%
q10 44.11 46.79 2.683 106.08%
q11 21.81 20.34 -1.467 93.27%
q12 26.81 26.95 0.144 100.54%
q13 46.94 45.56 -1.381 97.06%
q14 15.54 18.31 2.770 117.83%
q15 28.96 27.78 -1.183 95.91%
q16 15.44 15.83 0.388 102.51%
q17 104.01 101.57 -2.437 97.66%
q18 152.41 151.54 -0.873 99.43%
q19 12.79 12.91 0.112 100.88%
q20 28.04 26.84 -1.205 95.70%
q21 226.45 228.06 1.610 100.71%
q22 14.14 13.81 -0.329 97.68%
total 1242.81 1249.71 6.892 100.55%

@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_master_12_12_2023_time.csv log/native_master_12_11_2023_71ec720d4_time.csv difference percentage
q1 33.75 32.10 -1.649 95.11%
q2 24.88 24.79 -0.093 99.63%
q3 37.71 38.42 0.715 101.90%
q4 37.86 39.90 2.047 105.41%
q5 72.83 72.41 -0.425 99.42%
q6 7.05 7.02 -0.036 99.49%
q7 86.50 85.38 -1.119 98.71%
q8 86.24 86.65 0.409 100.47%
q9 125.50 126.76 1.253 101.00%
q10 44.68 46.79 2.110 104.72%
q11 20.21 20.34 0.128 100.63%
q12 25.06 26.95 1.890 107.54%
q13 46.58 45.56 -1.014 97.82%
q14 19.99 18.31 -1.682 91.59%
q15 27.55 27.78 0.230 100.84%
q16 15.86 15.83 -0.032 99.80%
q17 103.27 101.57 -1.697 98.36%
q18 151.33 151.54 0.210 100.14%
q19 13.89 12.91 -0.985 92.91%
q20 28.68 26.84 -1.847 93.56%
q21 229.12 228.06 -1.061 99.54%
q22 13.95 13.81 -0.142 98.99%
total 1252.49 1249.71 -2.790 99.78%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants