-
Notifications
You must be signed in to change notification settings - Fork 447
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[GLUTEN-3547][VL][CORE] Add WriteFiles into spark32 & spark33 shim #3800
Conversation
Run Gluten Clickhouse CI |
Run Gluten Clickhouse CI |
shall we also remove |
Run Gluten Clickhouse CI |
@ulysses-you Good catch. Delete |
Run Gluten Clickhouse CI |
@ulysses-you One unit test failed when deleting |
thank you @JkSelf, I'm fine with it |
Run Gluten Clickhouse CI |
===== Performance report for TPCH SF2000 with Velox backend, for reference only ====
|
What changes were proposed in this pull request?
Spark 3.4 introduces a new operator called WriteFiles. To implement the WriteTransformer in Spark 3.4, we need to move WriteFiles in the Spark 3.2 and Spark 3.3 shims to pass the compile. Additionally, this pull request removes five override classes.
How was this patch tested?
Existing unit tests