Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[VL] Refactor split buffer allocation #3177

Merged
merged 3 commits into from
Nov 9, 2023

Conversation

marin-ma
Copy link
Contributor

@marin-ma marin-ma commented Sep 18, 2023

Extract the duplicated logic in both resizePartitionBuffer and allocatePartitionBuffer.
In prealloc partition buffer, use resizePartitionBuffer instead of allocatePartitionBuffer if need to preserve buffer data for the next split, and remove the resize branch in allocate PartitionBuffer.

@github-actions
Copy link

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/oap-project/gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

@github-actions
Copy link

Run Gluten Clickhouse CI

@marin-ma marin-ma force-pushed the refactor-split-buffer branch from 56bf7bd to b5f4579 Compare September 18, 2023 04:55
@github-actions
Copy link

Run Gluten Clickhouse CI

@marin-ma marin-ma force-pushed the refactor-split-buffer branch from b5f4579 to cc29bd2 Compare September 18, 2023 05:08
@github-actions
Copy link

Run Gluten Clickhouse CI

1 similar comment
@github-actions
Copy link

Run Gluten Clickhouse CI

@marin-ma marin-ma force-pushed the refactor-split-buffer branch from 501666c to 2b17f93 Compare September 18, 2023 09:40
@github-actions
Copy link

Run Gluten Clickhouse CI

1 similar comment
@github-actions
Copy link

Run Gluten Clickhouse CI

@marin-ma marin-ma force-pushed the refactor-split-buffer branch from 2b17f93 to ea89f45 Compare September 18, 2023 10:31
@github-actions
Copy link

Run Gluten Clickhouse CI

@marin-ma marin-ma force-pushed the refactor-split-buffer branch 2 times, most recently from e84cf0f to bd3cd5a Compare September 22, 2023 04:29
@github-actions
Copy link

Run Gluten Clickhouse CI

@FelixYBW FelixYBW mentioned this pull request Oct 17, 2023
14 tasks
@marin-ma marin-ma force-pushed the refactor-split-buffer branch from bd3cd5a to ac0ba92 Compare November 6, 2023 04:50
@marin-ma marin-ma marked this pull request as ready for review November 6, 2023 04:51
@marin-ma
Copy link
Contributor Author

marin-ma commented Nov 6, 2023

/Benchmark Velox

@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_3177_time.csv log/native_master_11_05_2023_16e287450_time.csv difference percentage
q1 33.52 34.85 1.335 103.98%
q2 24.95 25.12 0.163 100.65%
q3 38.48 40.11 1.632 104.24%
q4 36.47 37.09 0.620 101.70%
q5 71.42 71.44 0.025 100.04%
q6 7.96 9.10 1.144 114.37%
q7 81.97 87.61 5.646 106.89%
q8 85.79 85.22 -0.578 99.33%
q9 123.35 121.12 -2.226 98.20%
q10 49.98 51.83 1.851 103.70%
q11 19.32 19.68 0.354 101.83%
q12 24.33 28.01 3.680 115.12%
q13 47.61 48.61 1.002 102.10%
q14 17.28 19.14 1.861 110.77%
q15 28.57 33.03 4.459 115.61%
q16 16.06 16.10 0.043 100.27%
q17 101.96 102.08 0.120 100.12%
q18 148.64 149.04 0.403 100.27%
q19 15.85 18.25 2.401 115.15%
q20 33.34 30.91 -2.424 92.73%
q21 224.53 222.95 -1.586 99.29%
q22 13.44 13.47 0.032 100.24%
total 1244.81 1264.76 19.956 101.60%

@marin-ma
Copy link
Contributor Author

marin-ma commented Nov 6, 2023

/Benchmark Velox

@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_3177_time.csv log/native_master_11_05_2023_16e287450_time.csv difference percentage
q1 32.95 34.85 1.903 105.78%
q2 24.69 25.12 0.427 101.73%
q3 38.27 40.11 1.834 104.79%
q4 37.43 37.09 -0.341 99.09%
q5 69.50 71.44 1.942 102.79%
q6 7.62 9.10 1.475 119.35%
q7 84.45 87.61 3.165 103.75%
q8 87.78 85.22 -2.566 97.08%
q9 119.44 121.12 1.680 101.41%
q10 53.15 51.83 -1.321 97.51%
q11 20.56 19.68 -0.881 95.71%
q12 23.61 28.01 4.408 118.68%
q13 49.45 48.61 -0.835 98.31%
q14 18.64 19.14 0.498 102.67%
q15 29.81 33.03 3.217 110.79%
q16 16.52 16.10 -0.419 97.46%
q17 102.64 102.08 -0.565 99.45%
q18 147.36 149.04 1.678 101.14%
q19 15.03 18.25 3.217 121.40%
q20 31.04 30.91 -0.125 99.60%
q21 225.14 222.95 -2.197 99.02%
q22 14.26 13.47 -0.788 94.48%
total 1249.36 1264.76 15.406 101.23%

@marin-ma
Copy link
Contributor Author

marin-ma commented Nov 7, 2023

@FelixYBW Could you help to review? Thanks!

@marin-ma
Copy link
Contributor Author

marin-ma commented Nov 8, 2023

/Benchmark Velox

@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_3177_time.csv log/native_master_11_07_2023_e3eff1d8f_time.csv difference percentage
q1 34.55 34.38 -0.170 99.51%
q2 24.79 25.03 0.238 100.96%
q3 38.33 38.14 -0.185 99.52%
q4 37.03 37.57 0.544 101.47%
q5 71.21 71.50 0.296 100.42%
q6 7.67 6.26 -1.407 81.65%
q7 83.14 82.22 -0.914 98.90%
q8 83.57 86.95 3.380 104.05%
q9 122.68 119.81 -2.875 97.66%
q10 50.81 51.26 0.452 100.89%
q11 19.49 19.73 0.246 101.26%
q12 28.30 24.39 -3.913 86.17%
q13 48.65 50.30 1.645 103.38%
q14 20.32 17.67 -2.649 86.96%
q15 31.63 30.35 -1.278 95.96%
q16 16.10 16.20 0.099 100.61%
q17 101.14 101.51 0.369 100.36%
q18 148.25 148.26 0.012 100.01%
q19 15.56 16.17 0.608 103.91%
q20 29.88 30.31 0.435 101.46%
q21 222.53 224.88 2.352 101.06%
q22 13.19 14.08 0.897 106.80%
total 1248.80 1246.98 -1.817 99.85%

Copy link
Contributor

@FelixYBW FelixYBW left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your work!

@FelixYBW FelixYBW merged commit 5ee4e34 into apache:main Nov 9, 2023
16 checks passed
@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_3177_time.csv log/native_master_11_08_2023_29b58997a_time.csv difference percentage
q1 34.52 34.11 -0.407 98.82%
q2 25.21 25.13 -0.076 99.70%
q3 38.83 38.62 -0.215 99.45%
q4 36.84 36.07 -0.766 97.92%
q5 72.32 71.84 -0.475 99.34%
q6 8.08 7.59 -0.486 93.99%
q7 86.65 85.82 -0.829 99.04%
q8 86.55 86.97 0.423 100.49%
q9 123.53 121.86 -1.669 98.65%
q10 52.72 54.80 2.081 103.95%
q11 19.77 20.59 0.821 104.15%
q12 27.08 24.63 -2.444 90.97%
q13 48.95 49.95 1.002 102.05%
q14 18.34 16.99 -1.348 92.65%
q15 30.53 30.97 0.443 101.45%
q16 16.66 16.51 -0.149 99.11%
q17 102.15 102.40 0.245 100.24%
q18 147.19 148.28 1.090 100.74%
q19 14.83 15.13 0.303 102.04%
q20 29.96 29.97 0.014 100.05%
q21 225.18 222.20 -2.976 98.68%
q22 14.14 13.55 -0.588 95.84%
total 1260.01 1254.01 -6.005 99.52%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants