-
Notifications
You must be signed in to change notification settings - Fork 447
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[VL] Add Scala 2.13 support #6326
Conversation
Thanks for opening a pull request! Could you open an issue for this pull request on Github Issues? https://github.com/apache/incubator-gluten/issues Then could you also rename commit message and pull request title in the following format?
See also: |
Run Gluten Clickhouse CI |
Run Gluten Clickhouse CI |
@zhztheplayer Could you please review this PR? Thanks! |
Prior to Spark 4.0, Scala 2.13 was not the default Scala version, and I personally do not recommend supporting it for Spark 3.x. |
Thanks @LuciferYang, I think Scala 2.13 support is already added for CH backend, plan was to extend to VL backend as well, since the build fails when you enable FYI, the support for Scala 2.13 was added here already - #6311 cc: @zhztheplayer |
@Preetesh2110 please take a look at failing CI and maybe also add a CI option to build with Scala 2.13 |
Got |
Run Gluten Clickhouse CI |
thanks @surnaik. fixed the scala Style issues. |
Do we need to have a test for Scala 2.13 (with Spark 3.5 probably) ? https://github.com/apache/incubator-gluten/blob/main/.github/workflows/velox_docker.yml |
Run Gluten Clickhouse CI |
Thanks for the feedback. I have added a workflow @zhztheplayer. |
Run Gluten Clickhouse CI |
.github/workflows/velox_docker.yml
Outdated
- name: Build and Run unit test for Spark 3.5.1 with Scala 2.13 (other tests) | ||
run: | | ||
cd $GITHUB_WORKSPACE/ | ||
export SPARK_SCALA_VERSION=2.13 | ||
$MVN_CMD clean install -Pspark-3.5 -Pscala-2.13 -Pbackends-velox -Pceleborn -Piceberg -Pdelta -Pspark-ut -DargLine="-Dspark.test.home=$GITHUB_WORKSPACE//shims/spark35/spark_home/" -DtagsToExclude=org.apache.spark.tags.ExtendedSQLTest,org.apache.gluten.tags.UDFTest,org.apache.gluten.tags.SkipTestTags && \ | ||
$MVN_CMD test -Pspark-3.5 -Pscala-2.13 -Pbackends-velox -Piceberg -Pdelta -DtagsToExclude=None -DtagsToInclude=org.apache.gluten.tags.UDFTest |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done. Thanks @zhztheplayer
Run Gluten Clickhouse CI |
Run Gluten Clickhouse CI |
Run Gluten Clickhouse CI |
Run Gluten Clickhouse CI |
Run Gluten Clickhouse CI |
Run Gluten Clickhouse CI |
Hey @zhztheplayer thanks for your message, how can I trigger the CIs, I have created a PR in my own repo Preetesh2110#1. |
Run Gluten Clickhouse CI |
Run Gluten Clickhouse CI |
Run Gluten Clickhouse CI |
Run Gluten Clickhouse CI |
Run Gluten Clickhouse CI |
You can sync gluten's main branch to your own main branch, then open a PR targeting to your own main branch. The CI should be directly runnable. Anyway you can still update here if it's OK for you to wait for CI to be triggered. Once you have one PR merged you will not get blocked in your subsequent PRs. |
Run Gluten Clickhouse CI |
Run Gluten Clickhouse CI |
Run Gluten Clickhouse CI |
Run Gluten Clickhouse CI |
@Preetesh2110 FYI, below mvn command can fix the style issues so this will not stop CI thanks, -yuan |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM - Non binding
@zhztheplayer please take a look, thanks! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 overall. I haven't try with the feature yet but assume CI should cover the changes.
Thank your for the effort @Preetesh2110
* [VL] Add Scala 2.13 support * Fix scalaStyle issues * Fix Scala Style issues * Add Spark 3.5.1 and Scala 2.13 test in workflow * Add run-spark-test-spark35-scala213 job * Add Spark 3.5.1 and Scala 2.13 test in workflow * Fix tests failures * Fix tests failures * ScalaStyle fix * Fix SoftAffinitySuite * Fix ArrowUtil error * Fix backend-velox scala issues * Fix ColumnarArrowEvalPythonExec issues * Fix ColumnarArrowEvalPythonExec issues * Fix TestOperator.scala for style issues * Fix TestOperator.scala for style issues * Fix issues in DeltaRewriteTransformerRules.scala * DeltaRewriteTransformerRules fix * Fix style issues * Fix issues * Fix issues * Fix issues * Fix issues * Fix issues * Fix issues * Fix issues --------- Co-authored-by: Hongze Zhang <[email protected]>
* [GLUTEN-6612] Fix ParquetFileFormat issue caused by the setting of local property isNativeApplicable (#6627) * [CORE] Fix schema mismatch between ReadRelNode and LocalFilesNode (#6746) Co-authored-by: 蒋添 <[email protected]> * [UT] Test input_file_name, input_file_block_start & input_file_block_length when scan falls back (#6318) * [VL] Fix E function fallback issue (#6397) * [VL] Add Scala 2.13 support (#6326) * [VL] Add Scala 2.13 support * Fix scalaStyle issues * Fix Scala Style issues * Add Spark 3.5.1 and Scala 2.13 test in workflow * Add run-spark-test-spark35-scala213 job * Add Spark 3.5.1 and Scala 2.13 test in workflow * Fix tests failures * Fix tests failures * ScalaStyle fix * Fix SoftAffinitySuite * Fix ArrowUtil error * Fix backend-velox scala issues * Fix ColumnarArrowEvalPythonExec issues * Fix ColumnarArrowEvalPythonExec issues * Fix TestOperator.scala for style issues * Fix TestOperator.scala for style issues * Fix issues in DeltaRewriteTransformerRules.scala * DeltaRewriteTransformerRules fix * Fix style issues * Fix issues * Fix issues * Fix issues * Fix issues * Fix issues * Fix issues * Fix issues --------- Co-authored-by: Hongze Zhang <[email protected]> * [VL] Fix Alinux3 arrow build issue (#6363) * update velox docker and port PR #6363 for get_velox.sh update --------- Co-authored-by: PHILO-HE <[email protected]> Co-authored-by: jiangjiangtian <[email protected]> Co-authored-by: 蒋添 <[email protected]> Co-authored-by: 高阳阳 <[email protected]> Co-authored-by: Preetesh2110 <[email protected]> Co-authored-by: Hongze Zhang <[email protected]> Co-authored-by: Joey <[email protected]>
What changes were proposed in this pull request?
Add Scala 2.13 support
(Fixes: #6316)
How was this patch tested?
Build locally
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)