Skip to content

Commit

Permalink
[SPARK-46299][DOCS] Make spark.deploy.recovery* docs up-to-date
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

This PR aims to update `Spark Standalone` cluster recovery configurations.

### Why are the changes needed?

We need to document
- apache#44173
- apache#44129
- apache#44113

![Screenshot 2023-12-06 at 7 15 24 PM](https://github.com/apache/spark/assets/9700541/04f0be6f-cdfb-4d87-b1b5-c4bf131f460a)

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Manual review.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#44227 from dongjoon-hyun/SPARK-46299.

Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
  • Loading branch information
dongjoon-hyun committed Dec 7, 2023
1 parent d026747 commit 43ca0b9
Showing 1 changed file with 23 additions and 3 deletions.
26 changes: 23 additions & 3 deletions docs/spark-standalone.md
Original file line number Diff line number Diff line change
Expand Up @@ -735,18 +735,38 @@ In order to enable this recovery mode, you can set SPARK_DAEMON_JAVA_OPTS in spa
<tr>
<td><code>spark.deploy.recoveryMode</code></td>
<td>NONE</td>
<td>The recovery mode setting to recover submitted Spark jobs with cluster mode when it failed and relaunches.
Set to FILESYSTEM to enable single-node recovery mode, ZOOKEEPER to use Zookeeper-based recovery mode, and
<td>The recovery mode setting to recover submitted Spark jobs with cluster mode when it failed and relaunches. Set to
FILESYSTEM to enable file-system-based single-node recovery mode,
ROCKSDB to enable RocksDB-based single-node recovery mode,
ZOOKEEPER to use Zookeeper-based recovery mode, and
CUSTOM to provide a customer provider class via additional `spark.deploy.recoveryMode.factory` configuration.
NONE is the default value which disables this recovery mode.
</td>
<td>0.8.1</td>
</tr>
<tr>
<td><code>spark.deploy.recoveryDirectory</code></td>
<td>""</td>
<td>The directory in which Spark will store recovery state, accessible from the Master's perspective.</td>
<td>The directory in which Spark will store recovery state, accessible from the Master's perspective.
Note that the directory should be clearly manualy if <code>spark.deploy.recoveryMode</code>,
<code>spark.deploy.recoverySerializer</code>, or <code>spark.deploy.recoveryCompressionCodec</code> is changed.
</td>
<td>0.8.1</td>
</tr>
<tr>
<td><code>spark.deploy.recoverySerializer</code></td>
<td>JAVA</td>
<td>A serializer for writing/reading objects to/from persistence engines; JAVA (default) or KRYO.
Java serializer has been the default mode since Spark 0.8.1.
Kryo serializer is a new fast and compact mode from Spark 4.0.0.</td>
<td>4.0.0</td>
</tr>
<tr>
<td><code>spark.deploy.recoveryCompressionCodec</code></td>
<td>(none)</td>
<td>A compression codec for persistence engines. none (default), lz4, lzf, snappy, and zstd. Currently, only FILESYSTEM mode supports this configuration.</td>
<td>4.0.0</td>
</tr>
<tr>
<td><code>spark.deploy.recoveryMode.factory</code></td>
<td>""</td>
Expand Down

0 comments on commit 43ca0b9

Please sign in to comment.