Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GLUTEN-6864][VL] Set a Velox gflag to allow growing buffer created in another Velox task #6932

Merged
merged 1 commit into from
Aug 20, 2024

Conversation

zhztheplayer
Copy link
Member

Rely on Velox rebase #6928.

Fixes #6864.

Copy link

#6864

@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCDS SF2000 with Velox backend, for reference only ====

query log/native_6932_time.csv log/native_master_08_12_2024_cc2b3cc8b0_time.csv difference percentage
q1 14.04 13.83 -0.209 98.51%
q2 14.40 13.84 -0.559 96.12%
q3 2.53 4.70 2.168 185.73%
q4 71.80 70.09 -1.710 97.62%
q5 9.05 7.73 -1.319 85.42%
q6 3.91 2.09 -1.820 53.43%
q7 5.06 6.11 1.042 120.59%
q8 3.35 3.23 -0.124 96.31%
q9 26.27 23.99 -2.285 91.30%
q10 9.56 9.87 0.312 103.26%
q11 37.74 37.80 0.060 100.16%
q12 1.43 1.38 -0.044 96.92%
q13 6.46 6.56 0.099 101.54%
q14a 45.66 45.77 0.115 100.25%
q14b 43.99 41.64 -2.342 94.68%
q15 2.78 2.68 -0.098 96.46%
q16 45.47 45.91 0.443 100.97%
q17 4.98 4.92 -0.060 98.80%
q18 6.74 6.73 -0.010 99.85%
q19 2.16 2.06 -0.107 95.06%
q20 1.43 2.93 1.499 204.64%
q21 1.04 1.16 0.115 110.98%
q22 7.90 7.89 -0.010 99.88%
q23a 104.51 105.11 0.597 100.57%
q23b 128.74 127.87 -0.874 99.32%
q24a 112.88 95.94 -16.945 84.99%
q24b 110.65 88.46 -22.195 79.94%
q25 4.17 4.20 0.030 100.72%
q26 5.92 3.04 -2.881 51.35%
q27 4.47 4.01 -0.461 89.69%
q28 30.05 32.37 2.325 107.74%
q29 9.46 14.59 5.128 154.21%
q30 5.06 4.91 -0.152 97.01%
q31 7.32 6.94 -0.383 94.77%
q32 1.24 1.10 -0.133 89.27%
q33 4.37 4.69 0.323 107.39%
q34 3.92 4.14 0.217 105.52%
q35 7.77 9.37 1.599 120.57%
q36 4.72 4.80 0.080 101.70%
q37 4.64 4.86 0.220 104.75%
q38 14.76 14.80 0.047 100.32%
q39a 3.33 3.10 -0.225 93.23%
q39b 3.06 3.41 0.346 111.31%
q40 3.73 4.00 0.265 107.09%
q41 0.64 0.63 -0.008 98.81%
q42 1.16 1.01 -0.159 86.37%
q43 4.73 4.58 -0.153 96.75%
q44 10.02 10.11 0.086 100.86%
q45 3.38 3.23 -0.154 95.45%
q46 4.04 3.72 -0.316 92.16%
q47 19.09 17.85 -1.243 93.49%
q48 5.30 5.09 -0.206 96.12%
q49 8.71 8.68 -0.031 99.64%
q50 21.42 21.78 0.359 101.68%
q51 10.16 10.02 -0.139 98.63%
q52 1.12 1.01 -0.118 89.48%
q53 2.39 2.47 0.075 103.11%
q54 3.97 4.04 0.064 101.61%
q55 1.07 1.13 0.061 105.72%
q56 4.24 4.24 0.002 100.04%
q57 11.26 10.66 -0.599 94.68%
q58 3.49 2.42 -1.070 69.38%
q59 10.96 10.69 -0.275 97.49%
q60 3.99 4.00 0.003 100.07%
q61 4.05 4.00 -0.051 98.73%
q62 4.45 4.12 -0.334 92.50%
q63 2.25 2.41 0.162 107.17%
q64 60.61 59.94 -0.672 98.89%
q65 17.20 18.02 0.826 104.80%
q66 4.04 3.90 -0.143 96.45%
q67 410.17 398.51 -11.654 97.16%
q68 3.85 3.62 -0.220 94.27%
q69 5.28 6.98 1.695 132.07%
q70 13.12 10.84 -2.273 82.67%
q71 2.36 2.40 0.038 101.59%
q72 218.41 218.15 -0.263 99.88%
q73 2.31 2.39 0.079 103.43%
q74 23.26 22.54 -0.718 96.91%
q75 27.55 26.42 -1.121 95.93%
q76 11.60 11.47 -0.132 98.86%
q77 2.21 2.43 0.220 109.96%
q78 50.31 49.86 -0.453 99.10%
q79 4.03 4.03 -0.008 99.80%
q80 12.25 12.22 -0.031 99.75%
q81 5.02 4.87 -0.145 97.11%
q82 7.57 6.87 -0.702 90.73%
q83 1.67 1.54 -0.127 92.38%
q84 2.75 2.75 0.003 100.12%
q85 6.92 7.48 0.558 108.06%
q86 4.09 3.92 -0.168 95.90%
q87 13.95 13.11 -0.843 93.96%
q88 21.47 21.46 -0.019 99.91%
q89 3.58 3.73 0.148 104.12%
q90 3.49 3.37 -0.122 96.50%
q91 2.44 2.42 -0.025 98.98%
q92 1.26 1.28 0.022 101.72%
q93 40.42 43.12 2.694 106.66%
q94 24.64 25.43 0.789 103.20%
q9 90.25 89.74 -0.515 99.43%
q5 2.64 2.57 -0.073 97.24%
q96 17.57 17.67 0.100 100.57%
q97 1.93 1.87 -0.057 97.05%
q98 11.04 10.09 -0.947 91.42%
q99 11.04 10.09 -0.947 91.42%
total 2215.74 2159.52 -56.222 97.46%

Copy link
Contributor

@zhouyuan zhouyuan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCDS SF2000 with Velox backend, for reference only ====

query log/native_6932_time.csv log/native_master_08_12_2024_cc2b3cc8b0_time.csv difference percentage
q1 14.28 13.83 -0.445 96.88%
q2 13.94 13.84 -0.092 99.34%
q3 4.48 4.70 0.212 104.73%
q4 71.89 70.09 -1.791 97.51%
q5 9.53 7.73 -1.807 81.05%
q6 4.00 2.09 -1.916 52.15%
q7 4.83 6.11 1.271 126.28%
q8 4.86 3.23 -1.636 66.35%
q9 24.25 23.99 -0.261 98.92%
q10 9.67 9.87 0.210 102.17%
q11 38.43 37.80 -0.628 98.37%
q12 1.48 1.38 -0.094 93.63%
q13 7.74 6.56 -1.176 84.80%
q14a 44.65 45.77 1.121 102.51%
q14b 43.25 41.64 -1.605 96.29%
q15 2.59 2.68 0.094 103.62%
q16 46.77 45.91 -0.859 98.16%
q17 5.18 4.92 -0.259 95.00%
q18 7.08 6.73 -0.353 95.01%
q19 2.33 2.06 -0.274 88.24%
q20 1.60 2.93 1.333 183.44%
q21 1.22 1.16 -0.065 94.72%
q22 7.77 7.89 0.123 101.59%
q23a 108.43 105.11 -3.321 96.94%
q23b 130.50 127.87 -2.630 97.98%
q24a 113.76 95.94 -17.826 84.33%
q24b 110.66 88.46 -22.199 79.94%
q25 4.09 4.20 0.106 102.59%
q26 3.11 3.04 -0.066 97.88%
q27 3.80 4.01 0.214 105.64%
q28 32.75 32.37 -0.377 98.85%
q29 12.54 14.59 2.045 116.31%
q30 4.95 4.91 -0.042 99.15%
q31 7.49 6.94 -0.549 92.66%
q32 1.30 1.10 -0.197 84.88%
q33 4.75 4.69 -0.059 98.77%
q34 3.83 4.14 0.314 108.20%
q35 9.17 9.37 0.206 102.25%
q36 4.76 4.80 0.042 100.87%
q37 4.64 4.86 0.217 104.68%
q38 14.22 14.80 0.585 104.12%
q39a 3.32 3.10 -0.218 93.42%
q39b 2.86 3.41 0.546 119.09%
q40 3.90 4.00 0.099 102.55%
q41 0.64 0.63 -0.003 99.45%
q42 0.97 1.01 0.037 103.85%
q43 4.38 4.58 0.197 104.49%
q44 9.99 10.11 0.114 101.14%
q45 3.16 3.23 0.065 102.07%
q46 3.78 3.72 -0.059 98.43%
q47 19.18 17.85 -1.332 93.06%
q48 5.31 5.09 -0.216 95.93%
q49 8.58 8.68 0.099 101.15%
q50 21.87 21.78 -0.086 99.61%
q51 10.32 10.02 -0.299 97.11%
q52 1.08 1.01 -0.070 93.54%
q53 2.46 2.47 0.009 100.38%
q54 3.89 4.04 0.148 103.81%
q55 1.03 1.13 0.102 109.90%
q56 4.08 4.24 0.164 104.02%
q57 10.61 10.66 0.054 100.51%
q58 2.43 2.42 -0.001 99.95%
q59 11.30 10.69 -0.613 94.57%
q60 4.19 4.00 -0.194 95.36%
q61 3.92 4.00 0.078 102.00%
q62 4.61 4.12 -0.489 89.39%
q63 2.36 2.41 0.051 102.15%
q64 61.95 59.94 -2.012 96.75%
q65 17.95 18.02 0.069 100.38%
q66 3.98 3.90 -0.081 97.96%
q67 389.25 398.51 9.262 102.38%
q68 4.12 3.62 -0.496 87.96%
q69 5.01 6.98 1.973 139.42%
q70 11.98 10.84 -1.132 90.55%
q71 2.57 2.40 -0.165 93.55%
q72 216.61 218.15 1.540 100.71%
q73 2.47 2.39 -0.082 96.69%
q74 24.65 22.54 -2.110 91.44%
q75 26.71 26.42 -0.285 98.93%
q76 11.69 11.47 -0.213 98.18%
q77 2.23 2.43 0.198 108.88%
q78 49.68 49.86 0.177 100.36%
q79 3.79 4.03 0.232 106.11%
q80 12.31 12.22 -0.086 99.30%
q81 5.06 4.87 -0.184 96.35%
q82 7.04 6.87 -0.165 97.65%
q83 1.74 1.54 -0.201 88.46%
q84 2.80 2.75 -0.045 98.38%
q85 7.13 7.48 0.352 104.94%
q86 4.04 3.92 -0.115 97.16%
q87 13.54 13.11 -0.428 96.84%
q88 21.34 21.46 0.120 100.56%
q89 3.61 3.73 0.125 103.47%
q90 3.00 3.37 0.372 112.42%
q91 2.57 2.42 -0.158 93.87%
q92 1.30 1.28 -0.018 98.60%
q93 41.07 43.12 2.044 104.98%
q94 24.15 25.43 1.279 105.29%
q9 90.32 89.74 -0.579 99.36%
q5 2.82 2.57 -0.248 91.19%
q96 17.47 17.67 0.205 101.18%
q97 2.03 1.87 -0.164 91.91%
q98 11.25 10.09 -1.155 89.73%
q99 11.25 10.09 -1.155 89.73%
total 2205.94 2159.52 -46.425 97.90%

@zhztheplayer zhztheplayer merged commit 11a236d into apache:main Aug 20, 2024
43 checks passed
@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCDS SF2000 with Velox backend, for reference only ====

query log/native_master_08_20_2024_time.csv log/native_master_08_12_2024_cc2b3cc8b0_time.csv difference percentage
q1 14.25 13.83 -0.420 97.05%
q2 14.04 13.84 -0.200 98.58%
q3 4.56 4.70 0.137 103.01%
q4 71.77 70.09 -1.675 97.67%
q5 9.38 7.73 -1.652 82.39%
q6 3.59 2.09 -1.498 58.23%
q7 4.78 6.11 1.330 127.85%
q8 5.32 3.23 -2.097 60.60%
q9 24.50 23.99 -0.511 97.91%
q10 9.92 9.87 -0.050 99.50%
q11 38.51 37.80 -0.708 98.16%
q12 1.38 1.38 0.003 100.20%
q13 6.51 6.56 0.053 100.81%
q14a 46.89 45.77 -1.114 97.62%
q14b 43.62 41.64 -1.979 95.46%
q15 2.50 2.68 0.185 107.43%
q16 46.35 45.91 -0.444 99.04%
q17 5.00 4.92 -0.085 98.30%
q18 8.62 6.73 -1.887 78.10%
q19 2.11 2.06 -0.053 97.47%
q20 1.51 2.93 1.421 194.06%
q21 1.07 1.16 0.094 108.85%
q22 7.77 7.89 0.122 101.57%
q23a 104.27 105.11 0.832 100.80%
q23b 126.59 127.87 1.277 101.01%
q24a 105.12 95.94 -9.183 91.26%
q24b 118.20 88.46 -29.747 74.83%
q25 4.31 4.20 -0.115 97.33%
q26 3.13 3.04 -0.084 97.31%
q27 4.68 4.01 -0.671 85.68%
q28 34.42 32.37 -2.047 94.05%
q29 12.40 14.59 2.191 117.67%
q30 5.82 4.91 -0.909 84.38%
q31 7.03 6.94 -0.094 98.67%
q32 1.20 1.10 -0.094 92.16%
q33 4.34 4.69 0.353 108.13%
q34 3.75 4.14 0.390 110.41%
q35 7.92 9.37 1.449 118.28%
q36 4.77 4.80 0.032 100.66%
q37 4.91 4.86 -0.050 98.98%
q38 13.48 14.80 1.327 109.84%
q39a 3.11 3.10 -0.008 99.74%
q39b 2.88 3.41 0.530 118.41%
q40 3.92 4.00 0.075 101.91%
q41 0.67 0.63 -0.039 94.19%
q42 0.96 1.01 0.048 104.98%
q43 4.65 4.58 -0.073 98.42%
q44 10.08 10.11 0.025 100.25%
q45 3.35 3.23 -0.127 96.22%
q46 3.63 3.72 0.092 102.54%
q47 18.61 17.85 -0.757 95.93%
q48 5.28 5.09 -0.186 96.48%
q49 9.01 8.68 -0.328 96.36%
q50 21.62 21.78 0.162 100.75%
q51 9.97 10.02 0.050 100.50%
q52 1.06 1.01 -0.051 95.19%
q53 2.47 2.47 -0.000 100.00%
q54 3.88 4.04 0.153 103.95%
q55 1.42 1.13 -0.289 79.64%
q56 4.35 4.24 -0.111 97.45%
q57 11.39 10.66 -0.722 93.66%
q58 2.45 2.42 -0.027 98.88%
q59 11.01 10.69 -0.318 97.11%
q60 4.11 4.00 -0.119 97.11%
q61 4.15 4.00 -0.149 96.42%
q62 4.59 4.12 -0.472 89.72%
q63 2.45 2.41 -0.040 98.35%
q64 60.37 59.94 -0.434 99.28%
q65 18.96 18.02 -0.940 95.04%
q66 4.13 3.90 -0.232 94.38%
q67 381.80 398.51 16.711 104.38%
q68 3.65 3.62 -0.025 99.32%
q69 5.47 6.98 1.509 127.59%
q70 11.77 10.84 -0.923 92.16%
q71 2.54 2.40 -0.139 94.52%
q72 215.14 218.15 3.006 101.40%
q73 2.36 2.39 0.029 101.23%
q74 23.43 22.54 -0.889 96.20%
q75 27.04 26.42 -0.619 97.71%
q76 11.08 11.47 0.389 103.51%
q77 2.19 2.43 0.242 111.06%
q78 52.27 49.86 -2.408 95.39%
q79 3.79 4.03 0.233 106.14%
q80 12.45 12.22 -0.229 98.16%
q81 4.86 4.87 0.014 100.29%
q82 7.45 6.87 -0.581 92.20%
q83 1.64 1.54 -0.104 93.66%
q84 2.91 2.75 -0.157 94.62%
q85 7.50 7.48 -0.018 99.76%
q86 4.16 3.92 -0.234 94.38%
q87 13.78 13.11 -0.668 95.15%
q88 21.90 21.46 -0.448 97.95%
q89 3.78 3.73 -0.053 98.61%
q90 3.50 3.37 -0.131 96.26%
q91 2.41 2.42 0.005 100.19%
q92 1.26 1.28 0.016 101.28%
q93 39.13 43.12 3.988 110.19%
q94 25.60 25.43 -0.170 99.34%
q9 90.71 89.74 -0.972 98.93%
q5 2.73 2.57 -0.162 94.07%
q96 17.33 17.67 0.337 101.95%
q97 1.94 1.87 -0.076 96.10%
q98 10.59 10.09 -0.492 95.35%
q99 10.59 10.09 -0.492 95.35%
total 2192.99 2159.52 -33.477 98.47%

@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_master_08_20_2024_time.csv log/native_master_08_19_2024_45d629e1d7_time.csv difference percentage
q1 40.11 39.36 -0.742 98.15%
q2 30.06 30.00 -0.058 99.81%
q3 51.48 53.30 1.813 103.52%
q4 42.79 42.63 -0.158 99.63%
q5 104.90 104.83 -0.064 99.94%
q6 13.17 12.68 -0.491 96.27%
q7 116.23 117.27 1.042 100.90%
q8 114.60 115.19 0.593 100.52%
q9 170.51 169.64 -0.874 99.49%
q10 66.72 64.81 -1.912 97.13%
q11 27.42 26.35 -1.065 96.12%
q12 29.40 32.85 3.445 111.72%
q13 51.67 51.31 -0.356 99.31%
q14 25.66 27.14 1.483 105.78%
q15 55.42 54.69 -0.731 98.68%
q16 18.68 17.77 -0.906 95.15%
q17 129.49 130.71 1.216 100.94%
q18 196.79 197.75 0.959 100.49%
q19 26.43 30.51 4.076 115.42%
q20 43.34 39.19 -4.154 90.42%
q21 375.24 384.57 9.335 102.49%
q22 15.35 15.48 0.135 100.88%
total 1745.45 1758.03 12.586 100.72%

sharkdtu pushed a commit to sharkdtu/gluten that referenced this pull request Nov 11, 2024
@lyy-pineapple
Copy link
Contributor

@zhztheplayer hi, should this patch be merged into version 1.2.1?

@zhztheplayer
Copy link
Member Author

@zhztheplayer hi, should this patch be merged into version 1.2.1?

Thanks for reminder. This needs to be backported to 1.2.

@wForget
Copy link
Member

wForget commented Dec 16, 2024

@zhztheplayer hi, should this patch be merged into version 1.2.1?

Thanks for reminder. This needs to be backported to 1.2.

Gluten 1.2.1 seems to miss the fix on velox: facebookincubator/velox#10768

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[VL] Error during spilling: The current running driver and the request driver must be from the same task
5 participants