Please set them via --, e.g. --build_type=Release.
| Parameters | Description | Default value |
|---|---|---|
| build_type | Gluten build type(for arrow/velox/cpp), CMAKE_BUILD_TYPE | Release |
| build_test | build test code in cpp folder and arrow | OFF |
| build_benchmarks | build benchmark code in cpp folder and arrow | OFF |
| build_jemalloc | build with jemalloc | ON |
| build_protobuf | build protobuf lib | ON |
| enable_qat | enable QAT for shuffle data de/compression | OFF |
| enable_hbm | enable HBM allocator | OFF |
| enable_s3 | build with s3 lib | OFF |
| enable_hdfs | build with hdfs lib | OFF |
| enable_ep_cache | enable caching for external project build (Arrow and Velox) | OFF |
Please set them via --, e.g., --arrow_home=/YOUR/PATH.
| Parameters | Description | Default value |
|---|---|---|
| arrow_home | Arrow build path | GLUTEN_DIR/ep/build-arrow/build |
| build_type | ARROW build type, CMAKE_BUILD_TYPE | Release |
Please set them via --, e.g., --arrow_home=/YOUR/PATH.
| Parameters | Description | Default value |
|---|---|---|
| arrow_home | Arrow build path | GLUTEN_DIR/ep/build-arrow/build |
| build_type | ARROW build type, CMAKE_BUILD_TYPE | Release |
| build_test | Build arrow with -DARROW_JSON=ON | OFF |
Please set them via --, e.g., --velox_home=/YOUR/PATH.
| Parameters | Description | Default value |
|---|---|---|
| velox_home | Velox build path | GLUTEN_DIR/ep/build-velox/build/velox_ep |
| build_type | Velox build type, CMAKE_BUILD_TYPE | Release |
| enable_s3 | Build Velox with -DENABLE_S3 | OFF |
| enable_hdfs | Build Velox with -DENABLE_HDFS | OFF |
| build_protobuf | build protobuf from source | ON |
To build different backends, there are 3 parameters can be set via -P for mvn.
| Parameters | Description | Activation state by default |
|---|---|---|
| backends-velox | Add -Pbackends-velox in maven command to compile the JVM part of Velox backend. | disabled |
| backends-gazelle | Add -Pbackends-gazelle in maven command to compile the JVM part of Gazelle backend. | disabled |
| backends-clickhouse | Add -Pbackends-clickhouse in maven command to compile the JVM part of ClickHouse backend. | disabled |
The gluten jar's name pattern is gluten-spark<sparkbundle.version>_<scala.binary.version>-<version>-SNAPSHOT-jar-with-dependencies.jar.
| Spark Version | sparkbundle.version | scala.binary.version |
|---|---|---|
| 3.2.2 | 3.2 | 2.12 |
| 3.3.1 | 3.3 | 2.12 |
The velox backend and the clickhouse backend support both spark-3.2.2 and spark-3.3.1.