[GLUTEN-6876] Support Spark-352 #7138

zhouyuan · 2024-09-06T00:05:39Z

What changes were proposed in this pull request?

Support Spark 352

fixes #6876

How was this patch tested?

Pass GHA

github-actions · 2024-09-06T00:05:56Z

#6876

github-actions · 2024-09-06T00:06:11Z

Run Gluten Clickhouse CI

github-actions · 2024-09-06T00:50:08Z

Run Gluten Clickhouse CI

wForget · 2024-09-06T02:21:36Z

gluten-substrait/src/main/scala/org/apache/spark/shuffle/sort/ColumnarShuffleManager.scala

@@ -107,7 +107,7 @@ class ColumnarShuffleManager(conf: SparkConf) extends ShuffleManager with Loggin
          metrics,
          shuffleExecutorComponents)
      case other: BaseShuffleHandle[K @unchecked, V @unchecked, _] =>
-        new SortShuffleWriter(other, mapId, context, shuffleExecutorComponents)
+        new SortShuffleWriter(other, mapId, context, _, shuffleExecutorComponents)


Does this mean that gluten will not guarantee spark minor version compatibility?

Hi @wForget,
Not guaranteed for all minor versions indeed. Actually it's the same for Spark 32 & 3.3 & 3.4. It's only tested with the latest minor release.
The issue is mostly due to the shim layer design, and the CI/CD resources are also not enough for so many small versions.

I see, thank you.

github-actions · 2024-09-06T06:19:55Z

Run Gluten Clickhouse CI

xhumanoid · 2024-09-06T06:42:40Z

shims/spark35/src/main/scala/org/apache/spark/sql/execution/GlutenExplainUtils.scala

+      p: SparkPlan,
+      reason: String,
+      fallbackNodeToReason: mutable.HashMap[String, String]): Unit = {
+    p.getTagValue(QueryPlan.OP_ID_TAG).foreach {


https://issues.apache.org/jira/browse/SPARK-48610

QueryPlan.OP_ID_TAG doesn't exist in spark 3.5.2+ / 4.0.0

Thanks, there's also a minor change on sortshufflewriter, will move these changes to shim layer

github-actions · 2024-09-06T06:46:14Z

Run Gluten Clickhouse CI

github-actions · 2024-09-06T06:47:23Z

Run Gluten Clickhouse CI

github-actions · 2024-09-06T09:28:07Z

Run Gluten Clickhouse CI

github-actions · 2024-09-06T09:30:55Z

Run Gluten Clickhouse CI

github-actions · 2024-09-09T09:43:13Z

Run Gluten Clickhouse CI

github-actions · 2024-10-04T14:55:47Z

Run Gluten Clickhouse CI

github-actions · 2024-10-09T02:59:12Z

Run Gluten Clickhouse CI

github-actions · 2024-10-09T03:46:01Z

Run Gluten Clickhouse CI

github-actions · 2024-10-09T05:35:25Z

Run Gluten Clickhouse CI

leoluan2009 · 2024-10-09T06:22:16Z

Spark 3.5.3 is out https://spark.apache.org/news/spark-3-5-3-released.html, should we upgrade to 3.5.3?

zhouyuan · 2024-10-09T06:49:31Z

Spark 3.5.3 is out https://spark.apache.org/news/spark-3-5-3-released.html, should we upgrade to 3.5.3?

Yes, plan to submit one new patch for Spark-353 after this - actually there's not much change from 352 to 353
here's the discussion before the PRC holiday:
#7336

thanks, -yuan

github-actions · 2024-10-09T08:42:16Z

Run Gluten Clickhouse CI

zhouyuan · 2024-10-11T03:09:45Z

@baibaichen @zhztheplayer

baibaichen · 2024-10-11T07:43:58Z

let me try performance for ch backend,

github-actions · 2024-10-11T12:17:06Z

Run Gluten Clickhouse CI

github-actions · 2024-10-14T09:35:32Z

Run Gluten Clickhouse CI

Signed-off-by: Yuan Zhou <[email protected]>

github-actions · 2024-10-14T13:02:09Z

Run Gluten Clickhouse CI

baibaichen

LGTM

github-actions bot added CORE works for Gluten Core INFRA TOOLS labels Sep 6, 2024

zhouyuan force-pushed the wip_spark352 branch from e844f5d to 2497695 Compare September 6, 2024 00:49

wForget reviewed Sep 6, 2024

View reviewed changes

zhouyuan marked this pull request as draft September 6, 2024 06:11

xhumanoid reviewed Sep 6, 2024

View reviewed changes

zhouyuan force-pushed the wip_spark352 branch from 2f439e6 to 7ce863d Compare September 6, 2024 06:46

zhouyuan mentioned this pull request Sep 28, 2024

[VL] Bump Spark to 3.5.3 #7336

Closed

zhouyuan force-pushed the wip_spark352 branch from ecd2cae to 4775901 Compare October 9, 2024 05:34

github-actions bot added the DATA_LAKE label Oct 9, 2024

zhouyuan marked this pull request as ready for review October 9, 2024 05:35

zhztheplayer mentioned this pull request Oct 9, 2024

[VL] Prepare shim API for breaking change in SPARK-48610 #7445

Merged

zhztheplayer force-pushed the wip_spark352 branch from df34d44 to a988a6b Compare October 9, 2024 09:01

baibaichen force-pushed the wip_spark352 branch from 264ac93 to 6031293 Compare October 11, 2024 12:16

baibaichen force-pushed the wip_spark352 branch from 6031293 to 082196c Compare October 14, 2024 09:35

zhouyuan and others added 20 commits October 14, 2024 21:01

fix sort shuffle

9c265f2

Signed-off-by: Yuan Zhou <[email protected]>

fix sort and op tag

5232db3

Signed-off-by: Yuan Zhou <[email protected]>

fix explain utils

8108980

Signed-off-by: Yuan Zhou <[email protected]>

fix format

9450f40

Signed-off-by: Yuan Zhou <[email protected]>

fix version

aff0e79

Signed-off-by: Yuan Zhou <[email protected]>

revert big shim layer

c344779

Signed-off-by: Yuan Zhou <[email protected]>

fix Spark 352 build

ed564d0

fix shim layer

bc59b55

Signed-off-by: Yuan Zhou <[email protected]>

fix API change

f2bf361

Signed-off-by: Yuan Zhou <[email protected]>

fix package

6be621d

Signed-off-by: Yuan Zhou <[email protected]>

fix data lake connectors

98aeb3f

Signed-off-by: Yuan Zhou <[email protected]>

exclude csv varcha test

ff6f3a3

Signed-off-by: Yuan Zhou <[email protected]>

simplify shim 1

1bf13f4

simplify shim 2

88b257a

fix shim layer code style

b7d0fdd

Signed-off-by: Yuan Zhou <[email protected]>

fix spark-352 source

eec45b2

Signed-off-by: Yuan Zhou <[email protected]>

ignore csv varchar test

0fe5663

Signed-off-by: Yuan Zhou <[email protected]>

ignore base64 unit tests

a303a1b

Signed-off-by: Yuan Zhou <[email protected]>

disable csv varchar ut in v1 suite

ef3bc44

Signed-off-by: Yuan Zhou <[email protected]>

ignore more unit tests

6125e92

Signed-off-by: Yuan Zhou <[email protected]>

baibaichen force-pushed the wip_spark352 branch from 082196c to 6125e92 Compare October 14, 2024 13:01

baibaichen approved these changes Oct 15, 2024

View reviewed changes

zhouyuan merged commit 32cd1dc into apache:main Oct 15, 2024
45 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[GLUTEN-6876] Support Spark-352 #7138

[GLUTEN-6876] Support Spark-352 #7138

zhouyuan commented Sep 6, 2024

github-actions bot commented Sep 6, 2024

github-actions bot commented Sep 6, 2024

github-actions bot commented Sep 6, 2024

wForget Sep 6, 2024

zhouyuan Sep 6, 2024

wForget Sep 9, 2024

github-actions bot commented Sep 6, 2024

xhumanoid Sep 6, 2024

zhouyuan Sep 6, 2024

github-actions bot commented Sep 6, 2024

github-actions bot commented Sep 6, 2024

github-actions bot commented Sep 6, 2024

github-actions bot commented Sep 6, 2024

github-actions bot commented Sep 9, 2024

github-actions bot commented Oct 4, 2024

github-actions bot commented Oct 9, 2024

github-actions bot commented Oct 9, 2024

github-actions bot commented Oct 9, 2024

leoluan2009 commented Oct 9, 2024

zhouyuan commented Oct 9, 2024

github-actions bot commented Oct 9, 2024

zhouyuan commented Oct 11, 2024

baibaichen commented Oct 11, 2024

github-actions bot commented Oct 11, 2024

github-actions bot commented Oct 14, 2024

github-actions bot commented Oct 14, 2024

baibaichen left a comment

[GLUTEN-6876] Support Spark-352 #7138

[GLUTEN-6876] Support Spark-352 #7138

Conversation

zhouyuan commented Sep 6, 2024

What changes were proposed in this pull request?

How was this patch tested?

github-actions bot commented Sep 6, 2024

github-actions bot commented Sep 6, 2024

github-actions bot commented Sep 6, 2024

wForget Sep 6, 2024

Choose a reason for hiding this comment

zhouyuan Sep 6, 2024

Choose a reason for hiding this comment

wForget Sep 9, 2024

Choose a reason for hiding this comment

github-actions bot commented Sep 6, 2024

xhumanoid Sep 6, 2024

Choose a reason for hiding this comment

zhouyuan Sep 6, 2024

Choose a reason for hiding this comment

github-actions bot commented Sep 6, 2024

github-actions bot commented Sep 6, 2024

github-actions bot commented Sep 6, 2024

github-actions bot commented Sep 6, 2024

github-actions bot commented Sep 9, 2024

github-actions bot commented Oct 4, 2024

github-actions bot commented Oct 9, 2024

github-actions bot commented Oct 9, 2024

github-actions bot commented Oct 9, 2024

leoluan2009 commented Oct 9, 2024

zhouyuan commented Oct 9, 2024

github-actions bot commented Oct 9, 2024

zhouyuan commented Oct 11, 2024

baibaichen commented Oct 11, 2024

github-actions bot commented Oct 11, 2024

github-actions bot commented Oct 14, 2024

github-actions bot commented Oct 14, 2024

baibaichen left a comment

Choose a reason for hiding this comment