Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CORE] Simplify WholeStageTransformer and BroadcastBuildSideRDD #3574

Merged
merged 2 commits into from
Nov 1, 2023

Conversation

ulysses-you
Copy link
Contributor

What changes were proposed in this pull request?

In WholeStageTransformer, the columnarInputRDDs would contain BroadcastBuildSideRDD, but the dependency and partition of BroadcastBuildSideRDD is meaningless. BroadcastBuildSideRDD should only be used to hold the broadcast value and generate iterator for hash join.

How was this patch tested?

PASS CI

@github-actions
Copy link

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/oap-project/gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

@github-actions
Copy link

Run Gluten Clickhouse CI

@ulysses-you
Copy link
Contributor Author

/Benchmark Velox

@github-actions
Copy link

Run Gluten Clickhouse CI

@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_3574_time.csv log/native_master_10_30_2023_f4c419e1c_time.csv difference percentage
q1 35.85 34.74 -1.110 96.90%
q2 25.01 25.08 0.069 100.28%
q3 40.20 40.40 0.196 100.49%
q4 37.27 37.88 0.609 101.63%
q5 70.02 70.91 0.887 101.27%
q6 7.35 9.39 2.046 127.84%
q7 87.97 85.52 -2.450 97.22%
q8 85.76 83.86 -1.902 97.78%
q9 119.36 118.87 -0.485 99.59%
q10 54.03 53.35 -0.687 98.73%
q11 19.94 20.07 0.134 100.67%
q12 27.57 26.35 -1.223 95.57%
q13 48.82 49.81 0.982 102.01%
q14 19.82 19.49 -0.330 98.33%
q15 33.22 35.98 2.757 108.30%
q16 15.98 16.33 0.351 102.19%
q17 102.45 100.70 -1.753 98.29%
q18 148.13 147.38 -0.744 99.50%
q19 17.18 16.78 -0.395 97.70%
q20 31.39 31.64 0.244 100.78%
q21 224.65 225.78 1.131 100.50%
q22 13.28 13.13 -0.146 98.90%
total 1265.26 1263.44 -1.821 99.86%

@ulysses-you
Copy link
Contributor Author

/Benchmark Velox

@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_3574_time.csv log/native_master_10_30_2023_f4c419e1c_time.csv difference percentage
q1 34.72 34.74 0.018 100.05%
q2 23.15 25.08 1.928 108.33%
q3 40.59 40.40 -0.190 99.53%
q4 37.35 37.88 0.529 101.42%
q5 71.31 70.91 -0.408 99.43%
q6 8.86 9.39 0.529 105.97%
q7 89.39 85.52 -3.867 95.67%
q8 88.39 83.86 -4.529 94.88%
q9 121.16 118.87 -2.291 98.11%
q10 53.44 53.35 -0.097 99.82%
q11 19.54 20.07 0.537 102.75%
q12 26.82 26.35 -0.475 98.23%
q13 48.69 49.81 1.112 102.28%
q14 18.03 19.49 1.453 108.05%
q15 33.14 35.98 2.838 108.56%
q16 16.25 16.33 0.082 100.50%
q17 101.98 100.70 -1.288 98.74%
q18 147.09 147.38 0.292 100.20%
q19 16.72 16.78 0.069 100.41%
q20 31.22 31.64 0.421 101.35%
q21 222.42 225.78 3.362 101.51%
q22 13.25 13.13 -0.117 99.12%
total 1263.53 1263.44 -0.092 99.99%

@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_3574_time.csv log/native_master_10_30_2023_f4c419e1c_time.csv difference percentage
q1 33.35 34.74 1.387 104.16%
q2 24.66 25.08 0.418 101.69%
q3 40.05 40.40 0.348 100.87%
q4 38.07 37.88 -0.191 99.50%
q5 71.27 70.91 -0.364 99.49%
q6 7.37 9.39 2.023 127.44%
q7 88.19 85.52 -2.672 96.97%
q8 86.46 83.86 -2.607 96.98%
q9 119.62 118.87 -0.751 99.37%
q10 51.98 53.35 1.367 102.63%
q11 19.57 20.07 0.500 102.56%
q12 27.55 26.35 -1.206 95.62%
q13 48.50 49.81 1.303 102.69%
q14 18.27 19.49 1.213 106.64%
q15 34.87 35.98 1.109 103.18%
q16 15.94 16.33 0.398 102.50%
q17 100.64 100.70 0.057 100.06%
q18 148.53 147.38 -1.147 99.23%
q19 16.80 16.78 -0.014 99.92%
q20 31.42 31.64 0.218 100.69%
q21 223.52 225.78 2.267 101.01%
q22 13.41 13.13 -0.284 97.89%
total 1260.07 1263.44 3.373 100.27%

@ulysses-you
Copy link
Contributor Author

cc @zhouyuan @PHILO-HE @marin-ma @zzcclp thank you

Copy link
Contributor

@zzcclp zzcclp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ulysses-you ulysses-you merged commit 0c59a5d into apache:main Nov 1, 2023
14 checks passed
@ulysses-you ulysses-you deleted the broadcast branch November 1, 2023 05:38
@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_3574_time.csv log/native_master_10_31_2023_d0ec9310f_time.csv difference percentage
q1 34.26 34.66 0.397 101.16%
q2 24.89 24.70 -0.195 99.22%
q3 39.92 40.22 0.309 100.77%
q4 37.49 36.02 -1.463 96.10%
q5 71.59 71.14 -0.447 99.38%
q6 8.79 7.39 -1.401 84.07%
q7 86.93 88.05 1.121 101.29%
q8 87.16 86.81 -0.349 99.60%
q9 124.81 122.50 -2.318 98.14%
q10 54.61 53.84 -0.768 98.59%
q11 20.09 19.59 -0.494 97.54%
q12 27.81 27.64 -0.167 99.40%
q13 48.61 50.31 1.696 103.49%
q14 19.15 19.33 0.177 100.92%
q15 33.47 34.74 1.270 103.79%
q16 16.54 16.30 -0.242 98.54%
q17 103.23 102.95 -0.280 99.73%
q18 148.80 147.95 -0.851 99.43%
q19 16.85 16.80 -0.054 99.68%
q20 30.29 34.80 4.519 114.92%
q21 223.96 225.79 1.832 100.82%
q22 13.38 13.35 -0.026 99.80%
total 1272.63 1274.89 2.265 100.18%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants