Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[GLUTEN-7028][CH][Part-6] Introduce MergeTreeDelayedCommitProtocol #7506

Merged
merged 15 commits into from
Oct 22, 2024
Merged

[GLUTEN-7028][CH][Part-6] Introduce MergeTreeDelayedCommitProtocol #7506

merged 15 commits into from
Oct 22, 2024

Conversation

baibaichen
Copy link
Contributor

@baibaichen baibaichen commented Oct 13, 2024 •
edited
Loading

What changes were proposed in this pull request?

This PR introduces MergeTreeDelayedCommitProtocol as a bridge between native backend and FileFormatWriter. We have moved the special logic of native write from MergeTreeFileformatWriter to this class, so we can directly use FileFormatWriter in spark 3.2 and spark 3.3.

I will use MergeTreeDelayedCommitProtocol in the next PR for the one pipeline write

(Fixes: #7028)

How was this patch tested?

Use existed UTs

@github-actions GitHub Actions
Copy link

#7028

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@baibaichen baibaichen force-pushed the feature/one_pipeline branch from af540cc to 0319925 Compare October 14, 2024 03:27
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

1 similar comment
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@baibaichen baibaichen force-pushed the feature/one_pipeline branch from 9bf10f1 to 80aa1ad Compare October 15, 2024 11:09
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@baibaichen baibaichen force-pushed the feature/one_pipeline branch from 80aa1ad to bfb4a2a Compare October 15, 2024 13:51
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@baibaichen baibaichen force-pushed the feature/one_pipeline branch from bfb4a2a to 768156d Compare October 16, 2024 09:46
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@baibaichen baibaichen force-pushed the feature/one_pipeline branch from 768156d to 360f0b6 Compare October 17, 2024 12:28
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

1 similar comment
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@baibaichen baibaichen force-pushed the feature/one_pipeline branch from 223dbd1 to 2e271a8 Compare October 18, 2024 15:37
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@baibaichen baibaichen force-pushed the feature/one_pipeline branch from 2e271a8 to 9ea7d2f Compare October 21, 2024 07:09
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

1 similar comment
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@baibaichen baibaichen force-pushed the feature/one_pipeline branch from 7af8250 to c110840 Compare October 21, 2024 13:59
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

2 similar comments
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@baibaichen baibaichen changed the title [GLUTEN-7028][CH][Part-6] [GLUTEN-7028][CH][Part-6] Introduce MergeTreeFileCommitProtocol Oct 22, 2024
@baibaichen baibaichen marked this pull request as ready for review October 22, 2024 07:52
@baibaichen baibaichen force-pushed the feature/one_pipeline branch from 39d53df to 4c46451 Compare October 22, 2024 08:01
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@baibaichen
DEFAULT_USE_DATASOURCE_V2
720793f 
(cherry picked from commit 59f974a)
@baibaichen
Remove unnecessary parameters
1872250 
(cherry picked from commit 69a160a)
@baibaichen baibaichen force-pushed the feature/one_pipeline branch from 4c46451 to 728f59f Compare October 22, 2024 11:09
@github-actions GitHub Actions
Copy link

Run Gluten Clickhouse CI

@baibaichen baibaichen mentioned this pull request Oct 22, 2024
Closed
11 tasks
@baibaichen baibaichen changed the title [GLUTEN-7028][CH][Part-6] Introduce MergeTreeFileCommitProtocol [GLUTEN-7028][CH][Part-6] Introduce MergeTreeDelayedCommitProtocol Oct 22, 2024
@baibaichen baibaichen merged commit 19090da into apache:main Oct 22, 2024
11 checks passed
@baibaichen baibaichen deleted the feature/one_pipeline branch October 22, 2024 14:39
@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCDS SF2000 with Velox backend, for reference only ====

query log/native_master_10_22_2024_time.csv log/native_master_10_21_2024_ec6e6f7803_time.csv difference percentage
q1 14.43 14.15 -0.281 98.05%
q2 13.43 16.57 3.138 123.36%
q3 5.31 3.08 -2.231 57.96%
q4 72.37 71.97 -0.399 99.45%
q5 9.41 10.56 1.150 112.22%
q6 2.35 3.70 1.350 157.41%
q7 7.80 6.72 -1.077 86.19%
q8 5.46 3.33 -2.137 60.90%
q9 24.93 24.92 -0.017 99.93%
q10 9.73 9.62 -0.106 98.91%
q11 38.49 36.46 -2.029 94.73%
q12 1.46 1.48 0.029 101.96%
q13 6.42 6.16 -0.259 95.96%
q14a 47.96 49.58 1.626 103.39%
q14b 43.18 41.18 -2.008 95.35%
q15 2.53 2.50 -0.037 98.54%
q16 48.66 48.94 0.276 100.57%
q17 4.74 4.79 0.054 101.14%
q18 6.77 6.96 0.193 102.85%
q19 2.53 2.47 -0.054 97.86%
q20 1.48 1.47 -0.011 99.24%
q21 1.18 1.24 0.063 105.40%
q22 7.95 7.93 -0.019 99.76%
q23a 106.86 108.71 1.847 101.73%
q23b 127.88 129.62 1.741 101.36%
q24a 117.92 111.77 -6.154 94.78%
q24b 110.48 105.09 -5.392 95.12%
q25 6.25 4.13 -2.113 66.16%
q26 4.41 4.36 -0.051 98.84%
q27 4.71 5.51 0.792 116.80%
q28 31.25 34.29 3.042 109.73%
q29 9.25 9.34 0.086 100.93%
q30 4.94 4.90 -0.038 99.23%
q31 7.72 6.50 -1.220 84.20%
q32 1.14 1.18 0.048 104.26%
q33 4.35 4.30 -0.042 99.04%
q34 4.11 4.01 -0.101 97.54%
q35 8.01 7.84 -0.164 97.95%
q36 5.87 5.66 -0.208 96.46%
q37 5.08 4.98 -0.094 98.16%
q38 14.16 13.44 -0.721 94.91%
q39a 3.52 3.18 -0.338 90.41%
q39b 3.10 3.16 0.057 101.83%
q40 3.88 3.98 0.103 102.65%
q41 0.63 0.62 -0.010 98.45%
q42 0.98 0.90 -0.081 91.74%
q43 5.31 4.59 -0.722 86.41%
q44 8.63 8.57 -0.056 99.35%
q45 3.17 3.28 0.106 103.35%
q46 3.99 3.81 -0.183 95.41%
q47 17.79 17.72 -0.071 99.60%
q48 5.46 5.19 -0.269 95.07%
q49 6.54 6.59 0.047 100.72%
q50 22.41 21.54 -0.870 96.12%
q51 9.76 9.84 0.080 100.82%
q52 1.07 1.13 0.060 105.56%
q53 2.42 3.29 0.873 136.06%
q54 3.76 3.66 -0.100 97.34%
q55 1.08 1.06 -0.023 97.90%
q56 3.97 3.89 -0.081 97.97%
q57 10.58 10.60 0.025 100.24%
q58 2.61 2.46 -0.157 94.00%
q59 10.87 10.35 -0.527 95.16%
q60 4.53 4.13 -0.401 91.15%
q61 3.92 5.09 1.169 129.83%
q62 4.63 4.26 -0.369 92.04%
q63 2.38 2.53 0.142 105.95%
q64 60.91 60.65 -0.260 99.57%
q65 16.68 17.63 0.955 105.72%
q66 4.36 4.63 0.267 106.12%
q67 434.68 445.13 10.456 102.41%
q68 4.09 3.86 -0.238 94.19%
q69 7.29 5.27 -2.020 72.31%
q70 11.92 12.74 0.817 106.85%
q71 2.60 2.28 -0.314 87.91%
q72 218.28 215.89 -2.392 98.90%
q73 2.48 2.38 -0.098 96.05%
q74 22.92 22.85 -0.079 99.66%
q75 26.99 27.55 0.555 102.06%
q76 13.52 14.78 1.254 109.27%
q77 2.18 2.19 0.016 100.76%
q78 49.31 50.02 0.716 101.45%
q79 3.92 4.07 0.144 103.68%
q80 11.76 11.55 -0.210 98.22%
q81 4.61 4.56 -0.046 99.00%
q82 7.78 7.01 -0.772 90.08%
q83 1.59 1.66 0.068 104.30%
q84 2.71 2.93 0.225 108.30%
q85 7.97 8.10 0.133 101.67%
q86 4.30 4.50 0.193 104.48%
q87 15.28 15.70 0.416 102.72%
q88 18.13 18.05 -0.085 99.53%
q89 3.37 3.47 0.106 103.14%
q90 2.99 2.83 -0.157 94.76%
q91 2.38 2.34 -0.036 98.47%
q92 1.34 1.29 -0.051 96.16%
q93 42.29 42.45 0.168 100.40%
q94 27.38 25.83 -1.547 94.35%
q9 89.59 88.07 -1.520 98.30%
q5 2.97 2.60 -0.364 87.73%
q96 18.13 17.92 -0.205 98.87%
q97 1.87 1.96 0.093 104.96%
q98 11.69 11.39 -0.302 97.42%
q99 11.69 11.39 -0.302 97.42%
total 2260.23 2253.00 -7.237 99.68%

baibaichen added a commit to Kyligence/gluten that referenced this pull request Oct 26, 2024
@baibaichen
fix gtest build due to apache#7506
76caf9f
baibaichen added a commit that referenced this pull request Oct 26, 2024
@kyligence-git @baibaichen
[GLUTEN-1632][CH]Daily Update Clickhouse Version (20241026) (#7689)
f5d42f1 
* [GLUTEN-1632][CH]Daily Update Clickhouse Version (20241026)

* Fix Build due to ClickHouse/ClickHouse#71019

* fix gtest build due to #7506

---------

Co-authored-by: kyligence-git <[email protected]>
Co-authored-by: Chang Chen <[email protected]>
yikf pushed a commit to yikf/incubator-gluten that referenced this pull request Oct 28, 2024
@kyligence-git @baibaichen @yikf
[GLUTEN-1632][CH]Daily Update Clickhouse Version (20241026) (apache#7689
509a1a0 
)

* [GLUTEN-1632][CH]Daily Update Clickhouse Version (20241026)

* Fix Build due to ClickHouse/ClickHouse#71019

* fix gtest build due to apache#7506

---------

Co-authored-by: kyligence-git <[email protected]>
Co-authored-by: Chang Chen <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zzcclp zzcclp zzcclp approved these changes

Assignees
No one assigned
Labels
CLICKHOUSE
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[CH] Fully Support writing parquet and mergetree in spark 3.5.x with delta protocol
3 participants
@baibaichen @GlutenPerfBot @zzcclp