Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[VL][DOC] Branch-1.2: Port #6533 to add uniffle doc #6542

Merged
merged 1 commit into from
Jul 22, 2024
Merged

[VL][DOC] Branch-1.2: Port #6533 to add uniffle doc #6542

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions docs/get-started/Velox.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -264,6 +264,40 @@ spark.celeborn.storage.hdfs.dir hdfs://<namenode>/celeborn
spark.dynamicAllocation.enabled false
```

## Uniffle support

Uniffle with velox backend supports [Uniffle](https://github.com/apache/incubator-uniffle) as remote shuffle service. Currently, the supported Uniffle versions are `0.8.0`.

First refer to this URL(https://uniffle.apache.org/docs/intro) to get start with uniffle.

When compiling the Gluten Java module, it's required to enable `uniffle` profile, as follows:

```
mvn clean package -Pbackends-velox -Pspark-3.3 -Puniffle -DskipTests
```

Then add the Uniffle and Spark Celeborn Client packages to your Spark application's classpath(usually add them into `$SPARK_HOME/jars`).

- Uniffle: rss-client-spark3-shaded-[uniffleVersion].jar
- Gluten: gluten-uniffle-velox-xxx-SNAPSHOT-3.x.jar

Currently to use Gluten following configurations are required in `spark-defaults.conf`

```
spark.shuffle.manager org.apache.spark.shuffle.gluten.uniffle.UniffleShuffleManager
# uniffle coordinator address
spark.rss.coordinator.quorum ip:port
# Support for Spark AQE
spark.sql.adaptive.localShuffleReader.enabled false
spark.shuffle.service.enabled false
# Uniffle support mutilple storage types, you can choose one of them.
# Such as MEMORY,LOCALFILE,MEMORY_LOCALFILE,HDFS,MEMORY_HDFS,LOCALFILE_HDFS,MEMORY_LOCALFILE_HDFS
spark.rss.storage.type LOCALFILE_HDFS
# If you want to use dynamic resource allocation,
# please refer to this URL (https://github.com/apache/incubator-uniffle/tree/master/patch/spark) to apply the patch into your own Spark.
spark.dynamicAllocation.enabled false
```

## DeltaLake Support

Gluten with velox backend supports [DeltaLake](https://delta.io/) table.
Expand Down
Loading