Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[VL] Add Scala 2.13 support #6326

Merged
merged 30 commits into from
Aug 10, 2024
Merged

[VL] Add Scala 2.13 support #6326

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
cab90d7
[VL] Add Scala 2.13 support
Preetesh2110 Jul 3, 2024
03bbe04
Fix scalaStyle issues
Preetesh2110 Jul 3, 2024
418f0e5
Fix Scala Style issues
Preetesh2110 Jul 10, 2024
f236d57
Add Spark 3.5.1 and Scala 2.13 test in workflow
Preetesh2110 Jul 11, 2024
c6eaa1a
Add run-spark-test-spark35-scala213 job
Preetesh2110 Jul 12, 2024
0df95dd
Merge branch 'main' into VLScala-2.13
zhztheplayer Jul 12, 2024
a0b0059
Add Spark 3.5.1 and Scala 2.13 test in workflow
Preetesh2110 Jul 11, 2024
3d40195
Fix tests failures
Preetesh2110 Jul 12, 2024
b0aac94
Fix tests failures
Preetesh2110 Jul 12, 2024
bb5e0cd
Merge branch 'main' into VLScala-2.13
Preetesh2110 Jul 12, 2024
c45ad02
ScalaStyle fix
Preetesh2110 Jul 12, 2024
1f46381
Fix SoftAffinitySuite
Preetesh2110 Jul 13, 2024
74bef54
Fix ArrowUtil error
Preetesh2110 Jul 13, 2024
97da095
Fix backend-velox scala issues
Preetesh2110 Aug 4, 2024
7b70834
Fix ColumnarArrowEvalPythonExec issues
Preetesh2110 Aug 4, 2024
55aae15
Fix ColumnarArrowEvalPythonExec issues
Preetesh2110 Aug 5, 2024
ffa6c15
Merge branch 'main' into VLScala-2.13
Preetesh2110 Aug 5, 2024
14de42a
Fix TestOperator.scala for style issues
Preetesh2110 Aug 5, 2024
8b61078
Fix TestOperator.scala for style issues
Preetesh2110 Aug 5, 2024
ab7a783
Fix issues in DeltaRewriteTransformerRules.scala
Preetesh2110 Aug 5, 2024
7edcb0b
DeltaRewriteTransformerRules fix
Preetesh2110 Aug 6, 2024
017c2b1
Fix style issues
Preetesh2110 Aug 6, 2024
b4467b0
Fix issues
Preetesh2110 Aug 6, 2024
958d71e
Fix issues
Preetesh2110 Aug 6, 2024
929bcc7
Merge branch 'main' into VLScala-2.13
Preetesh2110 Aug 6, 2024
6db7f17
Fix issues
Preetesh2110 Aug 6, 2024
da3f371
Fix issues
Preetesh2110 Aug 7, 2024
4f69e81
Fix issues
Preetesh2110 Aug 7, 2024
2235752
Fix issues
Preetesh2110 Aug 8, 2024
6fe4db3
Fix issues
Preetesh2110 Aug 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .github/workflows/velox_docker.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1089,6 +1089,12 @@ jobs:
export SPARK_SCALA_VERSION=2.12
$MVN_CMD clean install -Pspark-3.5 -Pbackends-velox -Pceleborn -Piceberg -Pdelta -Pspark-ut -DargLine="-Dspark.test.home=$GITHUB_WORKSPACE//shims/spark35/spark_home/" -DtagsToExclude=org.apache.spark.tags.ExtendedSQLTest,org.apache.gluten.tags.UDFTest,org.apache.gluten.tags.SkipTestTags && \
$MVN_CMD test -Pspark-3.5 -Pbackends-velox -Piceberg -Pdelta -DtagsToExclude=None -DtagsToInclude=org.apache.gluten.tags.UDFTest
- name: Build and Run unit test for Spark 3.5.1 with Scala 2.13 (other tests)
run: |
cd $GITHUB_WORKSPACE/
export SPARK_SCALA_VERSION=2.13
$MVN_CMD clean install -Pspark-3.5 -Pscala-2.13 -Pbackends-velox -Pceleborn -Piceberg -Pdelta -Pspark-ut -DargLine="-Dspark.test.home=$GITHUB_WORKSPACE//shims/spark35/spark_home/" -DtagsToExclude=org.apache.spark.tags.ExtendedSQLTest,org.apache.gluten.tags.UDFTest,org.apache.gluten.tags.SkipTestTags && \
$MVN_CMD test -Pspark-3.5 -Pscala-2.13 -Pbackends-velox -Piceberg -Pdelta -DtagsToExclude=None -DtagsToInclude=org.apache.gluten.tags.UDFTest
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could make job run-spark-test-spark35 much slower.

If so, probably we should add a job run-spark-test-spark35-scala213. cc @zhouyuan @PHILO-HE

Copy link
Contributor Author

@Preetesh2110 Preetesh2110 Jul 12, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Thanks @zhztheplayer

- name: Upload golden files
if: failure()
uses: actions/upload-artifact@v4
Expand Down
3 changes: 2 additions & 1 deletion backends-velox/src/main/scala/org/apache/gluten/execution/HashAggregateExecTransformer.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -396,7 +396,8 @@ abstract class HashAggregateExecTransformer(
childNodes.add(expressionNode)
}
}
exprNodes.add(getRowConstructNode(args, childNodes, newInputAttributes, aggFunc))
exprNodes.add(
getRowConstructNode(args, childNodes, newInputAttributes.toSeq, aggFunc))
case other =>
throw new GlutenNotSupportException(s"$other is not supported.")
}
Expand Down
2 changes: 1 addition & 1 deletion gluten-data/src/main/scala/org/apache/spark/sql/utils/SparkArrowUtil.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,7 @@ object SparkArrowUtil {
val dt = fromArrowField(child)
StructField(child.getName, dt, child.isNullable)
}
StructType(fields)
StructType(fields.toSeq)
case arrowType => fromArrowType(arrowType)
}
}
Expand Down
4 changes: 2 additions & 2 deletions shims/spark32/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,13 +43,13 @@
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-catalyst_2.12</artifactId>
<artifactId>spark-catalyst_${scala.binary.version}</artifactId>
<scope>provided</scope>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.12</artifactId>
<artifactId>spark-core_${scala.binary.version}</artifactId>
<scope>provided</scope>
<optional>true</optional>
</dependency>
Expand Down
4 changes: 2 additions & 2 deletions shims/spark33/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,13 +43,13 @@
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-catalyst_2.12</artifactId>
<artifactId>spark-catalyst_${scala.binary.version}</artifactId>
<scope>provided</scope>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.12</artifactId>
<artifactId>spark-core_${scala.binary.version}</artifactId>
<scope>provided</scope>
<optional>true</optional>
</dependency>
Expand Down
4 changes: 2 additions & 2 deletions shims/spark34/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,13 +43,13 @@
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-catalyst_2.12</artifactId>
<artifactId>spark-catalyst_${scala.binary.version}</artifactId>
<scope>provided</scope>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.12</artifactId>
<artifactId>spark-core_${scala.binary.version}</artifactId>
<scope>provided</scope>
<optional>true</optional>
</dependency>
Expand Down
Loading