Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GLUTEN-6067][CH][Part 1] Support CH backend with Spark3.5 #6068

Merged
merged 2 commits into from
Jun 14, 2024

Conversation

zzcclp
Copy link
Contributor

@zzcclp zzcclp commented Jun 13, 2024

What changes were proposed in this pull request?

Support CH backend with Spark3.5:

  1. Upgrade Spark version to 3.5 and compile passed;
  2. Upgrade Delta version to 3.2 and compile passed;

How was this patch tested?

for task 1 and task 2, manual tests

(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

Copy link

#6067

Copy link

Run Gluten Clickhouse CI

1 similar comment
Copy link

Run Gluten Clickhouse CI

@zzcclp zzcclp force-pushed the support_spark35_delta32 branch from bf599c0 to 85d3483 Compare June 13, 2024 05:37
Copy link

Run Gluten Clickhouse CI

@zzcclp zzcclp force-pushed the support_spark35_delta32 branch from 85d3483 to 76c1227 Compare June 13, 2024 09:57
Copy link

Run Gluten Clickhouse CI

@zzcclp zzcclp force-pushed the support_spark35_delta32 branch from 76c1227 to 86269ac Compare June 13, 2024 12:03
Copy link

Run Gluten Clickhouse CI

1 similar comment
Copy link

Run Gluten Clickhouse CI

zzcclp added 2 commits June 14, 2024 09:45
Support CH backend with Spark3.5:

1. Upgrade Spark version to 3.5 and compile passed;
2. Upgrade Delta version to 3.2 and compile passed;
3. CH backend UT passed: (now only the MergeTree + Delta UT passed);
4. Parquet native write passed;
5. Gluten UT passed;
6. Support to run Gluten CH CI with Spark 3.5
@zzcclp zzcclp force-pushed the support_spark35_delta32 branch from 6871923 to ca9581b Compare June 14, 2024 01:57
Copy link

Run Gluten Clickhouse CI

@baibaichen baibaichen changed the title [GLUTEN-6067][CH] Support CH backend with Spark3.5 (Task 1 and Taks 2) [GLUTEN-6067][CH][Part 1] Support CH backend with Spark3.5 Jun 14, 2024
Copy link
Contributor

@baibaichen baibaichen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zzcclp zzcclp closed this Jun 14, 2024
@zzcclp zzcclp reopened this Jun 14, 2024
Copy link

Run Gluten Clickhouse CI

@zzcclp
Copy link
Contributor Author

zzcclp commented Jun 14, 2024

misoperation, merging this pr, all ci already passed

@zzcclp zzcclp merged commit e52bda6 into apache:main Jun 14, 2024
45 of 46 checks passed
@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_6068_time.csv log/native_master_06_13_2024_142cf0fbc_time.csv difference percentage
q1 34.22 35.57 1.350 103.95%
q2 25.37 23.78 -1.591 93.73%
q3 37.15 39.61 2.453 106.60%
q4 34.36 32.76 -1.599 95.35%
q5 68.71 67.54 -1.167 98.30%
q6 8.47 9.42 0.946 111.17%
q7 84.00 80.69 -3.302 96.07%
q8 86.52 84.77 -1.751 97.98%
q9 120.20 120.70 0.503 100.42%
q10 45.40 45.25 -0.149 99.67%
q11 20.45 19.79 -0.667 96.74%
q12 26.52 27.92 1.393 105.25%
q13 39.84 39.43 -0.407 98.98%
q14 18.95 18.98 0.039 100.20%
q15 32.82 33.04 0.227 100.69%
q16 13.41 13.85 0.440 103.28%
q17 104.75 101.57 -3.183 96.96%
q18 145.85 144.97 -0.878 99.40%
q19 15.18 13.79 -1.386 90.87%
q20 27.99 26.68 -1.309 95.32%
q21 261.57 260.83 -0.747 99.71%
q22 14.21 13.91 -0.299 97.89%
total 1265.94 1254.85 -11.086 99.12%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants