Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GLUTEN-6737]package delta into bundle jar when specify delta profile #6738

Merged
merged 2 commits into from
Aug 9, 2024
Merged

[GLUTEN-6737]package delta into bundle jar when specify delta profile #6738

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1f71824
package delta into bundle jar when specify delta profile
dcoliversun Aug 7, 2024
a618cfd
update velox.md
dcoliversun Aug 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 1 addition & 6 deletions docs/get-started/Velox.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -314,10 +314,7 @@ First of all, compile gluten-delta module by a `delta` profile, as follows:
mvn clean package -Pbackends-velox -Pspark-3.3 -Pdelta -DskipTests
```

Then, put the additional `gluten-delta-XX-SNAPSHOT.jar` to the class path (usually it's `$SPARK_HOME/jars`).
The gluten-delta jar is in `gluten-delta/target` directory.

After the two steps, you can query delta table by gluten/velox without scan's fallback.
Once built successfully, delta features will be included in gluten-velox-bundle-X jar. Then you can query delta table by gluten/velox without scan's fallback.

Gluten with velox backends also support the column mapping of delta tables.
About column mapping, see more [here](https://docs.delta.io/latest/delta-column-mapping.html).
Expand All @@ -336,8 +333,6 @@ mvn clean package -Pbackends-velox -Pspark-3.3 -Piceberg -DskipTests

Once built successfully, iceberg features will be included in gluten-velox-bundle-X jar. Then you can query iceberg table by gluten/velox without scan's fallback.

After the two steps, you can query iceberg table by gluten/velox without scan's fallback.

# Coverage

Spark3.3 has 387 functions in total. ~240 are commonly used. To get the support status of all Spark built-in functions, please refer to [Velox Backend's Supported Operators & Functions](../velox-backend-support-progress.md).
Expand Down
4 changes: 0 additions & 4 deletions gluten-delta/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,10 +62,6 @@
<type>test-jar</type>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.binary.version}</artifactId>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.binary.version}</artifactId>
Expand Down
10 changes: 10 additions & 0 deletions package/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,16 @@
</dependency>
</dependencies>
</profile>
<profile>
<id>delta</id>
<dependencies>
<dependency>
<groupId>org.apache.gluten</groupId>
<artifactId>gluten-delta</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
</profile>
</profiles>

<build>
Expand Down
Loading