Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[GLUTEN-5944][CH] Fallback to run delta vacuum command #5945

Merged
merged 1 commit into from
Jun 1, 2024
Merged

[GLUTEN-5944][CH] Fallback to run delta vacuum command #5945

merged 1 commit into from
Jun 1, 2024

Conversation

zzcclp
Copy link
Contributor

@zzcclp zzcclp commented May 31, 2024

What changes were proposed in this pull request?

Fallback to run delta vacuum command:
When AQE is on, now gluten CH backend + Delta ran delta vacuum command failed, fallback to run it first.

Close #5944.

(Fixes: #5944)

How was this patch tested?

(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)

(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

@github-actions GitHub Actions
Copy link

#5944

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@zzcclp
[GLUTEN-5944][CH] Fallback to run delta vacuum command
0839021 
Fallback to run delta vacuum command:
When AQE is on, now gluten CH backend + Delta ran delta vacuum command failed, fallback to run it first.

Close apache#5944.
@github-actions GitHub Actions
Copy link

github-actions bot commented Jun 1, 2024

Run Gluten Clickhouse CI

baibaichen
baibaichen approved these changes Jun 1, 2024
View reviewed changes
Copy link
Contributor

@baibaichen baibaichen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@baibaichen baibaichen merged commit 26ff58d into apache:main Jun 1, 2024
7 checks passed
@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCDS SF2000 with Velox backend, for reference only ====

query log/native_master_06_01_2024_time.csv log/native_master_05_31_2024_91f54072c_time.csv difference percentage
q1 14.16 14.94 0.783 105.53%
q2 17.32 14.82 -2.496 85.59%
q3 4.23 4.62 0.391 109.23%
q4 62.42 67.68 5.255 108.42%
q5 8.45 7.90 -0.551 93.48%
q6 2.41 2.44 0.026 101.06%
q7 6.10 6.18 0.080 101.32%
q8 3.78 3.17 -0.611 83.84%
q9 17.19 16.81 -0.385 97.76%
q10 9.81 9.84 0.037 100.38%
q11 38.60 35.17 -3.424 91.13%
q12 1.31 1.39 0.087 106.70%
q13 5.67 5.75 0.071 101.25%
q14a 42.32 44.79 2.470 105.84%
q14b 39.36 39.22 -0.134 99.66%
q15 2.44 3.66 1.217 149.83%
q16 40.80 40.67 -0.131 99.68%
q17 5.77 5.52 -0.251 95.65%
q18 6.45 6.22 -0.236 96.34%
q19 3.43 3.22 -0.202 94.09%
q20 2.33 1.32 -1.004 56.90%
q21 2.45 1.12 -1.328 45.85%
q22 8.31 7.62 -0.689 91.71%
q23a 81.70 79.69 -2.016 97.53%
q23b 102.20 100.28 -1.924 98.12%
q24a 78.76 81.43 2.668 103.39%
q24b 69.30 72.54 3.238 104.67%
q25 4.11 5.93 1.823 144.36%
q26 2.50 2.81 0.302 112.06%
q27 2.97 2.78 -0.191 93.57%
q28 19.95 19.83 -0.117 99.41%
q29 12.00 6.69 -5.312 55.72%
q30 6.50 4.14 -2.354 63.77%
q31 6.00 5.99 -0.015 99.74%
q32 1.09 1.07 -0.020 98.16%
q33 5.82 5.40 -0.416 92.85%
q34 5.20 4.82 -0.385 92.60%
q35 6.79 7.30 0.512 107.55%
q36 3.22 2.87 -0.347 89.23%
q37 4.00 3.88 -0.113 97.17%
q38 12.01 11.86 -0.158 98.69%
q39a 3.21 3.17 -0.042 98.68%
q39b 3.10 3.23 0.127 104.09%
q40 3.63 6.46 2.836 178.21%
q41 0.58 0.69 0.115 119.99%
q42 0.79 0.95 0.153 119.24%
q43 7.27 3.64 -3.630 50.06%
q44 6.96 8.60 1.643 123.63%
q45 3.49 4.16 0.670 119.19%
q46 3.01 3.21 0.196 106.49%
q47 14.69 18.42 3.733 125.41%
q48 4.56 4.55 -0.011 99.75%
q49 7.26 7.21 -0.050 99.31%
q50 21.36 25.40 4.046 118.94%
q51 8.61 8.55 -0.056 99.35%
q52 0.96 1.03 0.074 107.67%
q53 1.78 1.71 -0.071 96.03%
q54 3.16 3.23 0.068 102.16%
q55 0.96 3.22 2.263 335.60%
q56 4.24 4.67 0.433 110.21%
q57 8.71 13.54 4.829 155.44%
q58 3.57 3.27 -0.302 91.54%
q59 14.26 14.39 0.129 100.90%
q60 4.79 4.93 0.137 102.87%
q61 5.24 5.36 0.118 102.26%
q62 3.88 4.25 0.369 109.51%
q63 4.22 1.86 -2.359 44.13%
q64 48.72 50.29 1.563 103.21%
q65 13.75 13.52 -0.230 98.33%
q66 3.07 3.09 0.020 100.66%
q67 350.80 350.41 -0.395 99.89%
q68 3.58 3.74 0.163 104.55%
q69 10.14 6.84 -3.294 67.51%
q70 7.25 10.41 3.155 143.50%
q71 2.18 2.40 0.216 109.90%
q72 191.37 185.26 -6.113 96.81%
q73 1.98 2.02 0.044 102.25%
q74 24.03 21.00 -3.035 87.37%
q75 21.31 24.50 3.191 114.97%
q76 7.08 7.05 -0.034 99.52%
q77 1.84 1.73 -0.103 94.38%
q78 37.87 37.69 -0.174 99.54%
q79 3.36 3.45 0.098 102.91%
q80 11.54 10.80 -0.746 93.54%
q81 4.61 5.79 1.182 125.66%
q82 6.58 6.27 -0.311 95.27%
q83 1.39 1.62 0.228 116.38%
q84 2.90 3.08 0.177 106.12%
q85 6.40 6.63 0.230 103.59%
q86 6.03 3.24 -2.788 53.74%
q87 12.43 15.54 3.106 124.98%
q88 16.37 17.73 1.364 108.33%
q89 2.69 2.65 -0.043 98.40%
q90 3.17 3.17 0.001 100.02%
q91 2.42 2.51 0.094 103.90%
q92 1.18 1.23 0.054 104.61%
q93 29.71 28.61 -1.095 96.31%
q94 21.53 21.45 -0.077 99.64%
q9 81.38 79.82 -1.557 98.09%
q5 2.36 2.32 -0.042 98.21%
q96 11.94 11.89 -0.046 99.62%
q97 2.04 1.98 -0.058 97.14%
q98 11.81 9.30 -2.511 78.74%
q99 11.81 9.30 -2.511 78.74%
total 1880.32 1882.13 1.803 100.10%

@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_master_06_01_2024_time.csv log/native_master_05_31_2024_91f54072c_time.csv difference percentage
q1 35.38 35.76 0.378 101.07%
q2 22.20 23.75 1.546 106.96%
q3 36.68 38.56 1.880 105.12%
q4 32.82 32.20 -0.618 98.12%
q5 68.48 68.66 0.182 100.27%
q6 5.88 10.43 4.550 177.38%
q7 82.83 78.68 -4.152 94.99%
q8 86.15 85.75 -0.410 99.52%
q9 118.21 117.45 -0.754 99.36%
q10 46.10 44.10 -2.002 95.66%
q11 21.35 19.80 -1.550 92.74%
q12 26.36 28.68 2.320 108.80%
q13 39.95 39.50 -0.448 98.88%
q14 21.07 17.71 -3.357 84.06%
q15 33.06 31.55 -1.511 95.43%
q16 13.63 14.19 0.566 104.15%
q17 104.25 104.15 -0.096 99.91%
q18 144.23 143.01 -1.222 99.15%
q19 13.61 13.50 -0.107 99.22%
q20 28.24 28.61 0.365 101.29%
q21 257.75 259.62 1.872 100.73%
q22 12.08 11.97 -0.108 99.10%
total 1250.29 1247.61 -2.676 99.79%

@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCDS SF2000 with Velox backend, for reference only ====

query log/native_master_06_02_2024_time.csv log/native_master_06_01_2024_26ff58d3b_time.csv difference percentage
q1 15.08 14.16 -0.916 93.93%
q2 14.95 17.32 2.363 115.80%
q3 4.43 4.23 -0.193 95.64%
q4 66.37 62.42 -3.942 94.06%
q5 7.96 8.45 0.491 106.17%
q6 5.68 2.41 -3.265 42.49%
q7 6.79 6.10 -0.686 89.89%
q8 5.59 3.78 -1.814 67.58%
q9 15.99 17.19 1.201 107.51%
q10 9.07 9.81 0.734 108.09%
q11 37.12 38.60 1.481 103.99%
q12 1.43 1.31 -0.122 91.45%
q13 5.75 5.67 -0.074 98.71%
q14a 41.43 42.32 0.886 102.14%
q14b 40.12 39.36 -0.766 98.09%
q15 3.62 2.44 -1.174 67.53%
q16 38.44 40.80 2.368 106.16%
q17 4.44 5.77 1.333 130.04%
q18 7.46 6.45 -1.011 86.45%
q19 2.35 3.43 1.071 145.48%
q20 1.70 2.33 0.630 137.12%
q21 2.29 2.45 0.160 107.00%
q22 9.07 8.31 -0.757 91.65%
q23a 80.11 81.70 1.592 101.99%
q23b 102.92 102.20 -0.720 99.30%
q24a 80.51 78.76 -1.742 97.84%
q24b 81.81 69.30 -12.508 84.71%
q25 8.82 4.11 -4.708 46.61%
q26 2.74 2.50 -0.235 91.41%
q27 2.74 2.97 0.225 108.21%
q28 21.13 19.95 -1.175 94.44%
q29 6.67 12.00 5.323 179.76%
q30 4.04 6.50 2.457 160.82%
q31 6.14 6.00 -0.134 97.81%
q32 0.95 1.09 0.142 114.98%
q33 4.64 5.82 1.181 125.46%
q34 4.94 5.20 0.265 105.37%
q35 8.67 6.79 -1.887 78.25%
q36 3.15 3.22 0.070 102.23%
q37 3.84 4.00 0.160 104.16%
q38 12.51 12.01 -0.498 96.02%
q39a 3.33 3.21 -0.115 96.53%
q39b 2.77 3.10 0.337 112.19%
q40 3.74 3.63 -0.112 97.02%
q41 0.61 0.58 -0.033 94.51%
q42 0.83 0.79 -0.041 95.08%
q43 3.53 7.27 3.743 206.13%
q44 6.60 6.96 0.359 105.44%
q45 3.31 3.49 0.179 105.41%
q46 3.08 3.01 -0.063 97.94%
q47 15.91 14.69 -1.222 92.32%
q48 4.45 4.56 0.107 102.40%
q49 7.24 7.26 0.013 100.18%
q50 21.27 21.36 0.085 100.40%
q51 8.32 8.61 0.292 103.51%
q52 1.01 0.96 -0.048 95.27%
q53 1.76 1.78 0.020 101.12%
q54 3.10 3.16 0.055 101.78%
q55 1.01 0.96 -0.049 95.18%
q56 4.41 4.24 -0.170 96.15%
q57 10.97 8.71 -2.259 79.41%
q58 2.39 3.57 1.187 149.77%
q59 14.13 14.26 0.134 100.95%
q60 4.75 4.79 0.038 100.81%
q61 5.22 5.24 0.023 100.44%
q62 5.35 3.88 -1.462 72.66%
q63 1.80 4.22 2.422 234.53%
q64 48.59 48.72 0.133 100.27%
q65 13.51 13.75 0.241 101.78%
q66 2.97 3.07 0.093 103.14%
q67 361.10 350.80 -10.293 97.15%
q68 3.42 3.58 0.160 104.69%
q69 6.43 10.14 3.712 157.76%
q70 8.62 7.25 -1.370 84.11%
q71 2.18 2.18 0.007 100.34%
q72 192.52 191.37 -1.148 99.40%
q73 2.10 1.98 -0.126 94.01%
q74 21.00 24.03 3.033 114.45%
q75 24.23 21.31 -2.920 87.95%
q76 7.23 7.08 -0.144 98.01%
q77 1.85 1.84 -0.017 99.09%
q78 37.46 37.87 0.413 101.10%
q79 5.87 3.36 -2.514 57.17%
q80 10.83 11.54 0.718 106.63%
q81 4.63 4.61 -0.025 99.45%
q82 6.78 6.58 -0.201 97.04%
q83 1.38 1.39 0.015 101.10%
q84 2.85 2.90 0.045 101.58%
q85 6.55 6.40 -0.151 97.69%
q86 3.10 6.03 2.922 194.11%
q87 12.28 12.43 0.150 101.22%
q88 18.20 16.37 -1.829 89.95%
q89 5.61 2.69 -2.924 47.91%
q90 3.25 3.17 -0.073 97.76%
q91 2.59 2.42 -0.174 93.30%
q92 1.18 1.18 -0.002 99.79%
q93 28.10 29.71 1.608 105.72%
q94 21.57 21.53 -0.042 99.81%
q9 81.48 81.38 -0.107 99.87%
q5 2.38 2.36 -0.017 99.29%
q96 12.12 11.94 -0.185 98.47%
q97 2.10 2.04 -0.057 97.28%
q98 8.67 11.81 3.145 136.29%
q99 8.67 11.81 3.145 136.29%
total 1899.02 1880.32 -18.698 99.02%

@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_master_06_02_2024_time.csv log/native_master_06_01_2024_26ff58d3b_time.csv difference percentage
q1 33.06 35.38 2.320 107.02%
q2 26.68 22.20 -4.477 83.22%
q3 36.70 36.68 -0.027 99.93%
q4 35.50 32.82 -2.686 92.43%
q5 69.83 68.48 -1.350 98.07%
q6 5.99 5.88 -0.109 98.17%
q7 81.70 82.83 1.129 101.38%
q8 84.47 86.15 1.686 102.00%
q9 119.12 118.21 -0.920 99.23%
q10 45.39 46.10 0.708 101.56%
q11 20.18 21.35 1.166 105.77%
q12 25.30 26.36 1.057 104.18%
q13 37.41 39.95 2.536 106.78%
q14 20.07 21.07 1.000 104.98%
q15 30.21 33.06 2.851 109.44%
q16 14.14 13.63 -0.518 96.34%
q17 101.99 104.25 2.251 102.21%
q18 147.16 144.23 -2.927 98.01%
q19 17.14 13.61 -3.536 79.37%
q20 26.85 28.24 1.395 105.20%
q21 259.27 257.75 -1.520 99.41%
q22 12.39 12.08 -0.311 97.49%
total 1250.57 1250.29 -0.285 99.98%

loneylee pushed a commit to loneylee/kylin that referenced this pull request Jul 31, 2024
@baibaichen @loneylee
KE-43825 Upgrade gluten (240701) (#31819)
6682bee 
1. Disable gluten in analyze table
2. Gluten support delta 2.3.0
- apache/incubator-gluten#5902
- apache/incubator-gluten#5945
3. Support sum0
4. Fix build
- apache/incubator-gluten#5796
- apache/incubator-gluten#5767
5. Fix SlowQueryDetectorTest.testSparderTimeoutCancelJob due to context dirty
6. Cleanup threadlocal contexts
7. Fix GMT+8 not support
8. Fix case class test coverage
9. Add 1 gluten disabled case in analyze table
10.Remove unsupported pushdown filter:
    10.1. when we create KylinFileSourceScanExec, we didn't remove subquery filter.
    10.2. KylinFileSourceScanExec doesn't inherit from FileSourceScanExec, we miss chance to correct push down filter.
11. native support floor_datetime and ceil_datetime
12. native support kap_add_months and kap_months_between
13. native support _ymdint_between
14. native support truncate
15. native support kylin_split_part
16. native support kylin instr
pfzhan pushed a commit to Kyligence/kylin that referenced this pull request Jul 31, 2024
@baibaichen @pfzhan
KE-43825 Upgrade gluten (240701) (#31819)
5f6fe13 
1. Disable gluten in analyze table
2. Gluten support delta 2.3.0
- apache/incubator-gluten#5902
- apache/incubator-gluten#5945
3. Support sum0
4. Fix build
- apache/incubator-gluten#5796
- apache/incubator-gluten#5767
5. Fix SlowQueryDetectorTest.testSparderTimeoutCancelJob due to context dirty
6. Cleanup threadlocal contexts
7. Fix GMT+8 not support
8. Fix case class test coverage
9. Add 1 gluten disabled case in analyze table
10.Remove unsupported pushdown filter:
    10.1. when we create KylinFileSourceScanExec, we didn't remove subquery filter.
    10.2. KylinFileSourceScanExec doesn't inherit from FileSourceScanExec, we miss chance to correct push down filter.
11. native support floor_datetime and ceil_datetime
12. native support kap_add_months and kap_months_between
13. native support _ymdint_between
14. native support truncate
15. native support kylin_split_part
16. native support kylin instr
pfzhan pushed a commit to pfzhan/kylin that referenced this pull request Jul 31, 2024
@baibaichen @pfzhan
KE-43825 Upgrade gluten (240701) (#31819)
95b62c7 
1. Disable gluten in analyze table
2. Gluten support delta 2.3.0
- apache/incubator-gluten#5902
- apache/incubator-gluten#5945
3. Support sum0
4. Fix build
- apache/incubator-gluten#5796
- apache/incubator-gluten#5767
5. Fix SlowQueryDetectorTest.testSparderTimeoutCancelJob due to context dirty
6. Cleanup threadlocal contexts
7. Fix GMT+8 not support
8. Fix case class test coverage
9. Add 1 gluten disabled case in analyze table
10.Remove unsupported pushdown filter:
    10.1. when we create KylinFileSourceScanExec, we didn't remove subquery filter.
    10.2. KylinFileSourceScanExec doesn't inherit from FileSourceScanExec, we miss chance to correct push down filter.
11. native support floor_datetime and ceil_datetime
12. native support kap_add_months and kap_months_between
13. native support _ymdint_between
14. native support truncate
15. native support kylin_split_part
16. native support kylin instr
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@baibaichen baibaichen baibaichen approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[CH] Fallback to run delta vacuum command
3 participants
@zzcclp @GlutenPerfBot @baibaichen