Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[VL] Avoid shrinking the binary buffer which triggers spill #3663

Merged
merged 2 commits into from
Nov 27, 2023

Conversation

marin-ma
Copy link
Contributor

@marin-ma marin-ma commented Nov 9, 2023

If spill is triggered by resizing binary partition buffer, should avoid shrinking itself during reclaiming memory from shrinking partition buffers. Because shrinking the buffer can possibly invalid the original underlying ptr.

Copy link

github-actions bot commented Nov 9, 2023

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/oap-project/gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

@marin-ma
Copy link
Contributor Author

/Benchmark Velox

@marin-ma marin-ma marked this pull request as ready for review November 10, 2023 01:04
@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_3663_time.csv log/native_master_11_09_2023_745ecb383_time.csv difference percentage
q1 32.85 33.80 0.945 102.87%
q2 25.05 24.95 -0.100 99.60%
q3 36.84 38.87 2.028 105.50%
q4 36.00 36.96 0.955 102.65%
q5 71.23 69.94 -1.285 98.20%
q6 7.98 7.87 -0.104 98.69%
q7 85.86 85.40 -0.459 99.47%
q8 86.64 88.61 1.966 102.27%
q9 122.99 121.14 -1.855 98.49%
q10 51.59 54.35 2.761 105.35%
q11 20.39 19.52 -0.873 95.72%
q12 26.57 24.93 -1.639 93.83%
q13 49.15 50.92 1.770 103.60%
q14 17.78 18.17 0.384 102.16%
q15 29.82 30.79 0.967 103.24%
q16 16.30 16.06 -0.239 98.53%
q17 101.52 101.51 -0.006 99.99%
q18 148.96 148.23 -0.734 99.51%
q19 15.43 14.95 -0.485 96.85%
q20 29.86 31.24 1.380 104.62%
q21 223.06 223.30 0.243 100.11%
q22 13.41 14.24 0.830 106.19%
total 1249.29 1255.74 6.449 100.52%

@marin-ma marin-ma force-pushed the fix-shrink-in-split branch from 2263eea to e02d697 Compare November 20, 2023 09:34
@marin-ma marin-ma force-pushed the fix-shrink-in-split branch from e02d697 to 1985b27 Compare November 20, 2023 10:02
@marin-ma
Copy link
Contributor Author

/Benchmark Velox

@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_3663_time.csv log/native_master_11_19_2023_5f5d18abe_time.csv difference percentage
q1 34.24 34.50 0.262 100.77%
q2 24.85 24.57 -0.283 98.86%
q3 37.55 37.14 -0.409 98.91%
q4 37.35 36.74 -0.610 98.37%
q5 70.67 70.93 0.259 100.37%
q6 5.43 7.09 1.656 130.48%
q7 84.49 83.62 -0.875 98.96%
q8 84.57 85.87 1.302 101.54%
q9 121.71 126.22 4.503 103.70%
q10 45.42 46.75 1.336 102.94%
q11 19.99 19.85 -0.138 99.31%
q12 23.11 25.57 2.462 110.65%
q13 47.18 45.26 -1.915 95.94%
q14 19.28 18.39 -0.893 95.37%
q15 28.92 28.36 -0.559 98.07%
q16 16.14 15.85 -0.287 98.22%
q17 101.92 100.27 -1.648 98.38%
q18 148.94 147.67 -1.268 99.15%
q19 12.97 13.00 0.028 100.22%
q20 28.16 26.75 -1.414 94.98%
q21 220.74 222.79 2.051 100.93%
q22 13.90 13.20 -0.700 94.96%
total 1227.51 1230.37 2.857 100.23%

@marin-ma
Copy link
Contributor Author

@jackylee-ch @zhztheplayer Could you help to review? Thanks!

Copy link
Contributor

@zhouyuan zhouyuan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@zhouyuan zhouyuan merged commit a192737 into apache:main Nov 27, 2023
15 checks passed
@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_3663_time.csv log/native_master_11_26_2023_f456a7e86_time.csv difference percentage
q1 33.13 34.38 1.248 103.77%
q2 24.87 24.62 -0.244 99.02%
q3 37.66 37.74 0.087 100.23%
q4 36.86 36.67 -0.189 99.49%
q5 71.39 68.82 -2.570 96.40%
q6 5.49 7.12 1.626 129.62%
q7 83.64 82.01 -1.632 98.05%
q8 84.93 87.72 2.793 103.29%
q9 127.66 120.79 -6.864 94.62%
q10 46.49 44.22 -2.278 95.10%
q11 20.31 19.96 -0.356 98.25%
q12 27.91 28.56 0.651 102.33%
q13 46.63 46.61 -0.029 99.94%
q14 17.91 14.35 -3.564 80.10%
q15 28.92 29.15 0.226 100.78%
q16 15.38 15.41 0.030 100.19%
q17 104.12 100.98 -3.143 96.98%
q18 148.99 145.61 -3.386 97.73%
q19 14.00 12.88 -1.120 92.00%
q20 28.02 27.79 -0.225 99.20%
q21 225.71 223.56 -2.145 99.05%
q22 14.02 13.03 -0.990 92.94%
total 1244.06 1221.98 -22.073 98.23%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants