From 1f7182418241676f2a7f85fa871cbe691fabf3e4 Mon Sep 17 00:00:00 2001 From: Qian Sun Date: Wed, 7 Aug 2024 06:24:20 +0000 Subject: [PATCH 1/2] package delta into bundle jar when specify delta profile --- docs/get-started/Velox.md | 3 +-- gluten-delta/pom.xml | 4 ---- package/pom.xml | 10 ++++++++++ 3 files changed, 11 insertions(+), 6 deletions(-) diff --git a/docs/get-started/Velox.md b/docs/get-started/Velox.md index 63239c39931e..36e3efd49e90 100644 --- a/docs/get-started/Velox.md +++ b/docs/get-started/Velox.md @@ -314,8 +314,7 @@ First of all, compile gluten-delta module by a `delta` profile, as follows: mvn clean package -Pbackends-velox -Pspark-3.3 -Pdelta -DskipTests ``` -Then, put the additional `gluten-delta-XX-SNAPSHOT.jar` to the class path (usually it's `$SPARK_HOME/jars`). -The gluten-delta jar is in `gluten-delta/target` directory. +Once built successfully, delta features will be included in gluten-velox-bundle-X jar. After the two steps, you can query delta table by gluten/velox without scan's fallback. diff --git a/gluten-delta/pom.xml b/gluten-delta/pom.xml index 253d5a8e4f4f..08e2060d0f63 100755 --- a/gluten-delta/pom.xml +++ b/gluten-delta/pom.xml @@ -62,10 +62,6 @@ test-jar test - - org.apache.spark - spark-core_${scala.binary.version} - org.apache.spark spark-core_${scala.binary.version} diff --git a/package/pom.xml b/package/pom.xml index 794118f833ce..ae1432770f8a 100644 --- a/package/pom.xml +++ b/package/pom.xml @@ -93,6 +93,16 @@ + + delta + + + org.apache.gluten + gluten-delta + ${project.version} + + + From a618cfda840539257e0fb2ad95d20261b2d24bc8 Mon Sep 17 00:00:00 2001 From: Qian Sun Date: Thu, 8 Aug 2024 01:04:07 +0000 Subject: [PATCH 2/2] update velox.md --- docs/get-started/Velox.md | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/docs/get-started/Velox.md b/docs/get-started/Velox.md index 36e3efd49e90..86d21b3a750f 100644 --- a/docs/get-started/Velox.md +++ b/docs/get-started/Velox.md @@ -314,9 +314,7 @@ First of all, compile gluten-delta module by a `delta` profile, as follows: mvn clean package -Pbackends-velox -Pspark-3.3 -Pdelta -DskipTests ``` -Once built successfully, delta features will be included in gluten-velox-bundle-X jar. - -After the two steps, you can query delta table by gluten/velox without scan's fallback. +Once built successfully, delta features will be included in gluten-velox-bundle-X jar. Then you can query delta table by gluten/velox without scan's fallback. Gluten with velox backends also support the column mapping of delta tables. About column mapping, see more [here](https://docs.delta.io/latest/delta-column-mapping.html). @@ -335,8 +333,6 @@ mvn clean package -Pbackends-velox -Pspark-3.3 -Piceberg -DskipTests Once built successfully, iceberg features will be included in gluten-velox-bundle-X jar. Then you can query iceberg table by gluten/velox without scan's fallback. -After the two steps, you can query iceberg table by gluten/velox without scan's fallback. - # Coverage Spark3.3 has 387 functions in total. ~240 are commonly used. To get the support status of all Spark built-in functions, please refer to [Velox Backend's Supported Operators & Functions](../velox-backend-support-progress.md).