diff --git a/docs/spark-standalone.md b/docs/spark-standalone.md index 7a89c8124bdfe..25d2fba47ce1c 100644 --- a/docs/spark-standalone.md +++ b/docs/spark-standalone.md @@ -735,18 +735,38 @@ In order to enable this recovery mode, you can set SPARK_DAEMON_JAVA_OPTS in spa spark.deploy.recoveryMode NONE - The recovery mode setting to recover submitted Spark jobs with cluster mode when it failed and relaunches. - Set to FILESYSTEM to enable single-node recovery mode, ZOOKEEPER to use Zookeeper-based recovery mode, and + The recovery mode setting to recover submitted Spark jobs with cluster mode when it failed and relaunches. Set to + FILESYSTEM to enable file-system-based single-node recovery mode, + ROCKSDB to enable RocksDB-based single-node recovery mode, + ZOOKEEPER to use Zookeeper-based recovery mode, and CUSTOM to provide a customer provider class via additional `spark.deploy.recoveryMode.factory` configuration. + NONE is the default value which disables this recovery mode. 0.8.1 spark.deploy.recoveryDirectory "" - The directory in which Spark will store recovery state, accessible from the Master's perspective. + The directory in which Spark will store recovery state, accessible from the Master's perspective. + Note that the directory should be clearly manualy if spark.deploy.recoveryMode, + spark.deploy.recoverySerializer, or spark.deploy.recoveryCompressionCodec is changed. + 0.8.1 + + spark.deploy.recoverySerializer + JAVA + A serializer for writing/reading objects to/from persistence engines; JAVA (default) or KRYO. + Java serializer has been the default mode since Spark 0.8.1. + Kryo serializer is a new fast and compact mode from Spark 4.0.0. + 4.0.0 + + + spark.deploy.recoveryCompressionCodec + (none) + A compression codec for persistence engines. none (default), lz4, lzf, snappy, and zstd. Currently, only FILESYSTEM mode supports this configuration. + 4.0.0 + spark.deploy.recoveryMode.factory ""