Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GLUTEN-3547][VL][CORE] Add WriteFiles into spark32 & spark33 shim #3800

Merged
merged 2 commits into from
Nov 23, 2023

Conversation

JkSelf
Copy link
Contributor

@JkSelf JkSelf commented Nov 21, 2023

What changes were proposed in this pull request?

Spark 3.4 introduces a new operator called WriteFiles. To implement the WriteTransformer in Spark 3.4, we need to move WriteFiles in the Spark 3.2 and Spark 3.3 shims to pass the compile. Additionally, this pull request removes five override classes.

How was this patch tested?

Existing unit tests

Copy link

#3547

Copy link

Run Gluten Clickhouse CI

Copy link

Run Gluten Clickhouse CI

@JkSelf JkSelf requested a review from ulysses-you November 22, 2023 01:40
@ulysses-you
Copy link
Contributor

ulysses-you commented Nov 22, 2023

shall we also remove HiveFileFormat ?

Copy link

Run Gluten Clickhouse CI

@JkSelf
Copy link
Contributor Author

JkSelf commented Nov 22, 2023

shall we also remove HiveFileFormat ?

@ulysses-you Good catch. Delete HiveFileFormat.

Copy link

Run Gluten Clickhouse CI

@JkSelf
Copy link
Contributor Author

JkSelf commented Nov 23, 2023

shall we also remove HiveFileFormat ?

@ulysses-you Good catch. Delete HiveFileFormat.

@ulysses-you One unit test failed when deleting HiveFileFormat, but it will be resolved in #3690. Therefore, I will remove HiveFileFormat in #3690. Thanks.

@ulysses-you
Copy link
Contributor

thank you @JkSelf, I'm fine with it

Copy link

Run Gluten Clickhouse CI

@JkSelf JkSelf merged commit f68a86f into apache:main Nov 23, 2023
17 checks passed
@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_3800_time.csv log/native_master_11_22_2023_16abf3966_time.csv difference percentage
q1 34.50 34.52 0.016 100.04%
q2 24.62 24.61 -0.009 99.96%
q3 37.42 37.51 0.093 100.25%
q4 36.76 36.72 -0.044 99.88%
q5 70.22 70.57 0.347 100.49%
q6 7.17 7.86 0.693 109.67%
q7 84.08 84.69 0.601 100.71%
q8 85.06 85.82 0.752 100.88%
q9 120.41 124.99 4.582 103.80%
q10 44.96 43.50 -1.460 96.75%
q11 19.93 19.62 -0.313 98.43%
q12 26.06 25.22 -0.839 96.78%
q13 46.77 45.75 -1.015 97.83%
q14 16.55 19.48 2.924 117.67%
q15 27.40 27.08 -0.318 98.84%
q16 15.81 15.52 -0.287 98.19%
q17 101.28 100.26 -1.023 98.99%
q18 148.36 147.25 -1.112 99.25%
q19 13.09 13.16 0.063 100.48%
q20 27.13 26.97 -0.161 99.41%
q21 222.89 221.08 -1.816 99.19%
q22 12.82 12.89 0.078 100.60%
total 1223.31 1225.07 1.751 100.14%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants