From b20b1f1fc83bf86e25acb8f5d01591649b2538ed Mon Sep 17 00:00:00 2001 From: Luis Ponce Date: Wed, 5 Jun 2019 12:54:37 -0500 Subject: [PATCH] Update build hibench readme * Update 2.4 version to Specify Spark Version. * Add Specify Hadoop version documentation. * Add Build using JDK 11 documentation. Signed-off-by: Luis Ponce --- docs/build-hibench.md | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/docs/build-hibench.md b/docs/build-hibench.md index 4709f90cf..a022df762 100644 --- a/docs/build-hibench.md +++ b/docs/build-hibench.md @@ -28,7 +28,7 @@ Because some Maven plugins cannot support Scala version perfectly, there are som ### Specify Spark Version ### -To specify the spark version, use -Dspark=xxx(1.6, 2.0, 2.1 or 2.2). By default, it builds for spark 2.0 +To specify the spark version, use -Dspark=xxx(1.6, 2.0, 2.1, 2.2 or 2.4). By default, it builds for spark 2.0 mvn -Psparkbench -Dspark=1.6 -Dscala=2.11 clean package tips: @@ -37,6 +37,11 @@ default . For example , if we want use spark2.0 and scala2.11 to build hibench. package` , but for spark2.0 and scala2.10 , we need use the command `mvn -Dspark=2.0 -Dscala=2.10 clean package` . Similarly , the spark1.6 is associated with the scala2.10 by default. +### Specify Hadoop Version ### +To specify the spark version, use -Dhadoop=xxx(3.2). By default, it builds for hadoop 2.4 + + mvn -Psparkbench -Dhadoop=3.2 -Dspark=2.4 -Dscala=2.12 clean package + ### Build a single module ### If you are only interested in a single workload in HiBench. You can build a single module. For example, the below command only builds the SQL workloads for Spark. @@ -48,3 +53,10 @@ Supported modules includes: micro, ml(machine learning), sql, websearch, graph, For Spark 2.0 and Spark 2.1, we add the benchmark support for Structured Streaming. This is a new module which cannot be compiled in Spark 1.6. And it won't get compiled by default even if you specify the spark version as 2.0 or 2.1. You must explicitly specify it like this: mvn -Psparkbench -Dmodules -PstructuredStreaming clean package + +### Build using JDK 1.11 +If you are interested in building using Java 11 specify scala, spark and hadoop version as below + + mvn -Psparkbench -Pflinkbench -Phadoopbench -Pstormbench -Dhadoop=3.2 -Dspark=2.4 -Dscala=2.12 clean package + +Supported frameworks only: hadoopbench, sparkbench, flinkbench, stormbench \ No newline at end of file