Skip to content

Commit

Permalink
[GLUTEN-2274] Preparing Gluten release v1.0.0 (#2331)
Browse files Browse the repository at this point in the history
  • Loading branch information
xieqi authored Jul 14, 2023
1 parent 8f3b369 commit bfe394b
Show file tree
Hide file tree
Showing 22 changed files with 27 additions and 27 deletions.
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ spark-shell \
--conf spark.plugins=io.glutenproject.GlutenPlugin \
--conf spark.gluten.loadLibFromJar=true \
--conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager \
--jars https://github.com/oap-project/gluten/releases/download/0.5.0/gluten-velox-bundle-spark3.2_2.12-ubuntu_20.04-0.5.0-SNAPSHOT.jar,https://github.com/oap-project/gluten/releases/download/0.5.0/gluten-thirdparty-lib-ubuntu-20.04.jar
--jars https://github.com/oap-project/gluten/releases/download/1.0.0/gluten-velox-bundle-spark3.2_2.12-ubuntu_20.04-1.0.0.jar
```

# 3.2 Custom Build
Expand All @@ -78,7 +78,7 @@ Alternatively, you can build gluten from source, then do some configurations to

```
export gluten_jvm_jar = /PATH/TO/GLUTEN/backends-velox/target/<gluten-jar>
spark-shell
spark-shell
--master yarn --deploy-mode client \
--conf spark.plugins=io.glutenproject.GlutenPlugin \
--conf spark.driver.extraClassPath=${gluten_jvm_jar} \
Expand Down
2 changes: 1 addition & 1 deletion backends-clickhouse/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<artifactId>gluten-parent</artifactId>
<groupId>io.glutenproject</groupId>
<version>0.5.0-SNAPSHOT</version>
<version>1.0.0</version>
</parent>
<modelVersion>4.0.0</modelVersion>

Expand Down
2 changes: 1 addition & 1 deletion backends-velox/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<artifactId>gluten-parent</artifactId>
<groupId>io.glutenproject</groupId>
<version>0.5.0-SNAPSHOT</version>
<version>1.0.0</version>
</parent>
<modelVersion>4.0.0</modelVersion>

Expand Down
4 changes: 2 additions & 2 deletions docs/developers/NewToGluten.md
Original file line number Diff line number Diff line change
Expand Up @@ -360,11 +360,11 @@ spark-shell --name run_gluten \
--conf spark.memory.offHeap.enabled=true \
--conf spark.memory.offHeap.size=20g \
--conf spark.gluten.loadLibFromJar=true \
--jars https://github.com/oap-project/gluten/releases/download/0.5.0/gluten-velox-bundle-spark3.2_2.12-ubuntu_20.04-0.5.0-SNAPSHOT.jar,https://github.com/oap-project/gluten/releases/download/0.5.0/gluten-thirdparty-lib-ubuntu-20.04.jar
--jars https://github.com/oap-project/gluten/releases/download/1.0.0/gluten-velox-bundle-spark3.2_2.12-ubuntu_20.04-1.0.0.jar
```

# How to prioritize loading Gluten jars in Spark
# How to prioritize loading Gluten jars in Spark

To implement the Insert into directory function in gluten, it is necessary to overwrite the implementation of HiveFileFormat in vanilla spark. Therefore, when running a program that uses gluten, it is essential to ensure that the gluten jar is loaded prior to the vanilla spark jar. In this section, we will provide some configuration settings in `$SPARK_HOME/conf/spark-defaults.conf` for Yarn client, Yarn cluster, and Local&Standalone mode to guarantee that the gluten jar is prioritized.

Expand Down
2 changes: 1 addition & 1 deletion gluten-celeborn/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<artifactId>gluten-parent</artifactId>
<groupId>io.glutenproject</groupId>
<version>0.5.0-SNAPSHOT</version>
<version>1.0.0</version>
<relativePath>../pom.xml</relativePath>
</parent>
<modelVersion>4.0.0</modelVersion>
Expand Down
2 changes: 1 addition & 1 deletion gluten-core/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<artifactId>gluten-parent</artifactId>
<groupId>io.glutenproject</groupId>
<version>0.5.0-SNAPSHOT</version>
<version>1.0.0</version>
</parent>
<modelVersion>4.0.0</modelVersion>

Expand Down
2 changes: 1 addition & 1 deletion gluten-data/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<artifactId>gluten-parent</artifactId>
<groupId>io.glutenproject</groupId>
<version>0.5.0-SNAPSHOT</version>
<version>1.0.0</version>
<relativePath>../pom.xml</relativePath>
</parent>
<modelVersion>4.0.0</modelVersion>
Expand Down
2 changes: 1 addition & 1 deletion gluten-ut/common/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<artifactId>gluten-ut</artifactId>
<groupId>io.glutenproject</groupId>
<version>0.5.0-SNAPSHOT</version>
<version>1.0.0</version>
<relativePath>../pom.xml</relativePath>
</parent>
<modelVersion>4.0.0</modelVersion>
Expand Down
2 changes: 1 addition & 1 deletion gluten-ut/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
<parent>
<groupId>io.glutenproject</groupId>
<artifactId>gluten-parent</artifactId>
<version>0.5.0-SNAPSHOT</version>
<version>1.0.0</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion gluten-ut/spark32/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<artifactId>gluten-ut</artifactId>
<groupId>io.glutenproject</groupId>
<version>0.5.0-SNAPSHOT</version>
<version>1.0.0</version>
<relativePath>../pom.xml</relativePath>
</parent>
<modelVersion>4.0.0</modelVersion>
Expand Down
2 changes: 1 addition & 1 deletion gluten-ut/spark33/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<artifactId>gluten-ut</artifactId>
<groupId>io.glutenproject</groupId>
<version>0.5.0-SNAPSHOT</version>
<version>1.0.0</version>
<relativePath>../pom.xml</relativePath>
</parent>
<modelVersion>4.0.0</modelVersion>
Expand Down
2 changes: 1 addition & 1 deletion package/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
<parent>
<groupId>io.glutenproject</groupId>
<artifactId>gluten-parent</artifactId>
<version>0.5.0-SNAPSHOT</version>
<version>1.0.0</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
<modelVersion>4.0.0</modelVersion>
<groupId>io.glutenproject</groupId>
<artifactId>gluten-parent</artifactId>
<version>0.5.0-SNAPSHOT</version>
<version>1.0.0</version>
<packaging>pom</packaging>

<name>Gluten Parent Pom</name>
Expand Down
2 changes: 1 addition & 1 deletion shims/common/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
<parent>
<groupId>io.glutenproject</groupId>
<artifactId>spark-sql-columnar-shims</artifactId>
<version>0.5.0-SNAPSHOT</version>
<version>1.0.0</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion shims/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
<parent>
<groupId>io.glutenproject</groupId>
<artifactId>gluten-parent</artifactId>
<version>0.5.0-SNAPSHOT</version>
<version>1.0.0</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion shims/spark32/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
<parent>
<groupId>io.glutenproject</groupId>
<artifactId>spark-sql-columnar-shims</artifactId>
<version>0.5.0-SNAPSHOT</version>
<version>1.0.0</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion shims/spark33/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
<parent>
<groupId>io.glutenproject</groupId>
<artifactId>spark-sql-columnar-shims</artifactId>
<version>0.5.0-SNAPSHOT</version>
<version>1.0.0</version>
<relativePath>../pom.xml</relativePath>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion substrait/substrait-spark/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<artifactId>gluten-parent</artifactId>
<groupId>io.glutenproject</groupId>
<version>0.5.0-SNAPSHOT</version>
<version>1.0.0</version>
<relativePath>../../pom.xml</relativePath>
</parent>
<modelVersion>4.0.0</modelVersion>
Expand Down
6 changes: 3 additions & 3 deletions tools/gluten-it/common/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,11 @@
<parent>
<groupId>io.glutenproject</groupId>
<artifactId>gluten-it</artifactId>
<version>0.5.0-SNAPSHOT</version>
<version>1.0.0</version>
</parent>

<artifactId>gluten-it-common</artifactId>
<version>0.5.0-SNAPSHOT</version>
<version>1.0.0</version>
<packaging>jar</packaging>

<properties>
Expand All @@ -21,7 +21,7 @@
<spark32.version>3.2.2</spark32.version>
<spark33.version>3.3.1</spark33.version>
<scala.binary.version>2.12</scala.binary.version>
<gluten.version>0.5.0-SNAPSHOT</gluten.version>
<gluten.version>1.0.0</gluten.version>
</properties>

<profiles>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ public Map<String, String> extraSparkConf() {

private static class LocalClusterMode implements Mode {
// We should transfer the jars to be tested in the integration testing to executors
public static final String[] EXTRA_JARS = new String[]{"gluten-package-0.5.0-SNAPSHOT.jar"};
public static final String[] EXTRA_JARS = new String[]{"gluten-package-1.0.0.jar"};

@CommandLine.Option(names = {"--local-cluster"}, description = "Run in Spark local cluster mode", required = true)
private boolean enabled;
Expand Down
2 changes: 1 addition & 1 deletion tools/gluten-it/package/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
<parent>
<groupId>io.glutenproject</groupId>
<artifactId>gluten-it</artifactId>
<version>0.5.0-SNAPSHOT</version>
<version>1.0.0</version>
</parent>
<artifactId>gluten-it-package</artifactId>
<name>Archetype - assembly</name>
Expand Down
4 changes: 2 additions & 2 deletions tools/gluten-it/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

<groupId>io.glutenproject</groupId>
<artifactId>gluten-it</artifactId>
<version>0.5.0-SNAPSHOT</version>
<version>1.0.0</version>
<packaging>pom</packaging>
<modules>
<module>common</module>
Expand All @@ -20,6 +20,6 @@
<spark32.version>3.2.2</spark32.version>
<spark33.version>3.3.1</spark33.version>
<scala.binary.version>2.12</scala.binary.version>
<gluten.version>0.5.0-SNAPSHOT</gluten.version>
<gluten.version>1.0.0</gluten.version>
</properties>
</project>

0 comments on commit bfe394b

Please sign in to comment.