[CORE][SPARK-4.0] Only require one config spark.plugins to enable gluten

[ulysses-you]

Description

For now, there are at least three required configs to enable gluten:

• spark.plugins
• spark.memory.offHeap.enabled
• spark.shuffle.manager

Less required configs is always better. Fortunately, We can set spark.memory.offHeap.enabled and spark.shuffle.manager inside GlutenDriverPlugin since Spark 4.0.0, so only spark.plugins is required.

• spark.memory.offHeap.enabled: [SPARK-46947][CORE] Delay memory manager initialization until Driver plugin is loaded spark#45052
• spark.shuffle.manager: [SPARK-45762][CORE] Support shuffle managers defined in user jars by changing startup order spark#43627

[FelixYBW]

We may set shuffle manager default as local shuffle manager. We may overwrite offheap.enabled=true once offheap.size is set.

[acvictor]

@FelixYBW as long as ColumnarShuffleManager is in the classpath, do you see any issues with setting default shuffle manager to ColumnarShuffleManager?

[FelixYBW]

@FelixYBW as long as ColumnarShuffleManager is in the classpath, do you see any issues with setting default shuffle manager to ColumnarShuffleManager?

No. go ahead to submit a PR.

[acvictor]

@FelixYBW as long as ColumnarShuffleManager is in the classpath, do you see any issues with setting default shuffle manager to ColumnarShuffleManager?

No. go ahead to submit a PR.

Where would this be set? I think in Spark 3.x it must be set as part of session builder?