[CORE] Enable SortShuffleSuite with ColumnarShuffleManager

[acvictor]

What changes were proposed in this pull request?

Enable Spark's SortShuffleSuite with shuffle manager set to ColumnarShuffleManager to ensure that shuffle is always compatible with existing Spark cases.

(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
Added a new test suite.

[github-actions]

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

• Other pull requests

[github-actions]

Run Gluten Clickhouse CI

[github-actions]

Run Gluten Clickhouse CI

[acvictor]

@rui-mo @zhli1142015 can you please review?

[rui-mo]

Hi @zzcclp, do we need to exclude this suite for CH backend? Thanks.

[rui-mo]

@acvictor Could you rebase this PR to try the CH CI again? Thanks.

[github-actions]

Run Gluten Clickhouse CI

[acvictor]

@rui-mo I think the Clickhouse CI has failed but I'm not sure how to access it. Should I disable the suite for CH?

[rui-mo]

@acvictor We provide an account to log into CH CI system at https://github.com/apache/incubator-gluten/blob/main/docs/get-started/ClickHouse.md#new-ci-system.

name: gluten, password: hN2xX3uQ4m

But I find the failure looks irrelevant to this change. @zzcclp Could you help confirm?

[zzcclp]

please help to exclude the GlutenSortShuffleSuite for CH backend first, thanks.

[acvictor]

please help to exclude the GlutenSortShuffleSuite for CH backend first, thanks.

Done. Can you please review?

[github-actions]

Run Gluten Clickhouse CI

[github-actions]

Run Gluten Clickhouse CI

[acvictor]

@zzcclp @rui-mo can you please review? CH CI is passing.

[rui-mo]

Thanks. Will merge after CI passes.

[GlutenPerfBot]

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query	log/native_5816_time.csv	log/native_master_05_28_2024_d7374bd8f_time.csv	difference	percentage
q1	35.85	34.04	-1.812	94.95%
q2	23.74	23.62	-0.111	99.53%
q3	38.90	36.93	-1.965	94.95%
q4	32.32	32.10	-0.218	99.33%
q5	67.01	69.91	2.902	104.33%
q6	7.51	5.97	-1.545	79.43%
q7	79.84	80.25	0.415	100.52%
q8	84.52	84.79	0.266	100.31%
q9	120.83	121.55	0.725	100.60%
q10	45.32	43.64	-1.678	96.30%
q11	19.84	22.27	2.435	112.27%
q12	23.83	26.90	3.075	112.91%
q13	54.56	52.40	-2.152	96.05%
q14	17.68	21.59	3.913	122.14%
q15	31.52	31.46	-0.058	99.82%
q16	14.13	13.97	-0.159	98.87%
q17	103.18	102.83	-0.354	99.66%
q18	145.11	145.92	0.812	100.56%
q19	13.52	14.82	1.297	109.59%
q20	26.96	31.06	4.099	115.20%
q21	262.22	261.69	-0.534	99.80%
q22	12.14	12.27	0.128	101.06%
total	1260.49	1269.97	9.480	100.75%

[GlutenPerfBot]

===== Performance report for TPCDS SF2000 with Velox backend, for reference only ====

query	log/native_master_05_29_2024_time.csv	log/native_master_05_28_2024_d7374bd8f5_time.csv	difference	percentage
q1	16.97	14.54	-2.429	85.69%
q2	14.94	15.27	0.332	102.22%
q3	4.96	4.95	-0.011	99.79%
q4	64.00	63.51	-0.493	99.23%
q5	7.81	8.07	0.264	103.38%
q6	2.42	2.31	-0.111	95.41%
q7	5.64	5.15	-0.492	91.28%
q8	5.59	3.29	-2.303	58.79%
q9	18.71	16.20	-2.506	86.60%
q10	9.65	9.74	0.086	100.89%
q11	35.04	36.46	1.425	104.07%
q12	1.39	1.35	-0.040	97.12%
q13	6.52	6.67	0.155	102.38%
q14a	44.10	43.33	-0.768	98.26%
q14b	43.05	41.66	-1.385	96.78%
q15	2.47	3.69	1.215	149.12%
q16	39.09	39.54	0.453	101.16%
q17	5.64	5.39	-0.241	95.72%
q18	6.47	6.21	-0.263	95.93%
q19	1.91	3.56	1.645	186.03%
q20	1.21	1.59	0.379	131.33%
q21	2.41	1.09	-1.318	45.22%
q22	8.78	7.79	-0.989	88.74%
q23a	81.60	83.10	1.498	101.84%
q23b	102.04	102.20	0.164	100.16%
q24a	76.07	69.27	-6.810	91.05%
q24b	68.64	68.74	0.102	100.15%
q25	4.14	4.01	-0.133	96.80%
q26	3.07	4.40	1.337	143.60%
q27	3.01	2.93	-0.086	97.16%
q28	22.76	20.32	-2.436	89.29%
q29	6.60	7.82	1.225	118.57%
q30	4.28	9.08	4.794	211.95%
q31	5.97	6.79	0.821	113.75%
q32	0.96	1.23	0.275	128.76%
q33	4.84	4.80	-0.034	99.29%
q34	6.25	4.81	-1.432	77.08%
q35	6.40	6.41	0.013	100.21%
q36	2.98	3.08	0.108	103.64%
q37	3.86	3.70	-0.164	95.76%
q38	12.00	11.80	-0.200	98.34%
q39a	3.22	3.14	-0.078	97.59%
q39b	2.75	2.90	0.145	105.28%
q40	4.75	3.64	-1.103	76.76%
q41	0.58	0.58	-0.006	98.96%
q42	0.86	0.84	-0.026	97.02%
q43	3.47	3.54	0.072	102.06%
q44	6.87	10.58	3.707	153.97%
q45	3.47	3.39	-0.086	97.53%
q46	5.11	3.12	-1.992	61.03%
q47	15.72	14.59	-1.130	92.81%
q48	4.50	4.29	-0.210	95.33%
q49	7.64	7.27	-0.370	95.15%
q50	22.54	21.31	-1.229	94.55%
q51	8.32	10.20	1.876	122.55%
q52	1.01	0.97	-0.043	95.70%
q53	1.74	2.85	1.112	164.10%
q54	3.06	3.11	0.059	101.92%
q55	0.98	0.98	-0.002	99.84%
q56	4.27	4.34	0.069	101.62%
q57	8.43	8.63	0.198	102.34%
q58	2.52	2.47	-0.046	98.19%
q59	13.85	14.13	0.281	102.03%
q60	4.58	4.59	0.010	100.22%
q61	6.91	5.15	-1.758	74.55%
q62	5.46	3.70	-1.759	67.77%
q63	1.91	1.83	-0.083	95.65%
q64	48.49	50.79	2.298	104.74%
q65	15.70	15.15	-0.550	96.50%
q66	3.11	6.39	3.273	205.17%
q67	353.82	348.77	-5.051	98.57%
q68	3.62	3.54	-0.080	97.78%
q69	9.38	6.62	-2.765	70.53%
q70	8.84	8.43	-0.413	95.33%
q71	2.28	2.23	-0.053	97.68%
q72	187.72	187.81	0.096	100.05%
q73	2.29	2.21	-0.076	96.67%
q74	21.11	21.75	0.638	103.02%
q75	23.43	23.68	0.251	101.07%
q76	7.32	6.98	-0.344	95.30%
q77	1.85	1.73	-0.120	93.51%
q78	40.94	37.38	-3.562	91.30%
q79	3.38	3.49	0.113	103.34%
q80	10.89	10.51	-0.381	96.50%
q81	4.55	4.53	-0.026	99.42%
q82	6.34	6.09	-0.250	96.06%
q83	1.49	1.47	-0.017	98.84%
q84	4.82	2.47	-2.349	51.30%
q85	6.85	6.83	-0.020	99.71%
q86	3.24	3.11	-0.129	96.00%
q87	12.48	12.27	-0.210	98.31%
q88	16.41	17.80	1.385	108.44%
q89	2.61	2.60	-0.007	99.75%
q90	3.25	3.35	0.104	103.20%
q91	2.57	2.68	0.101	103.93%
q92	2.46	1.15	-1.310	46.66%
q93	30.56	28.51	-2.049	93.29%
q94	21.42	25.26	3.844	117.95%
q9	81.92	80.22	-1.698	97.93%
q5	5.38	2.56	-2.821	47.53%
q96	12.10	12.20	0.105	100.87%
q97	1.83	2.02	0.196	110.71%
q98	9.29	9.16	-0.131	98.59%
q99	9.29	9.16	-0.131	98.59%
total	1882.45	1859.70	-22.754	98.79%

[GlutenPerfBot]

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query	log/native_master_05_29_2024_time.csv	log/native_master_05_28_2024_d7374bd8f_time.csv	difference	percentage
q1	33.56	34.04	0.483	101.44%
q2	23.85	23.62	-0.227	99.05%
q3	37.37	36.93	-0.441	98.82%
q4	32.42	32.10	-0.317	99.02%
q5	69.83	69.91	0.078	100.11%
q6	7.54	5.97	-1.577	79.09%
q7	81.75	80.25	-1.499	98.17%
q8	86.57	84.79	-1.779	97.94%
q9	118.35	121.55	3.201	102.71%
q10	44.16	43.64	-0.516	98.83%
q11	22.23	22.27	0.041	100.19%
q12	26.72	26.90	0.181	100.68%
q13	53.60	52.40	-1.198	97.76%
q14	17.59	21.59	3.999	122.73%
q15	32.89	31.46	-1.435	95.64%
q16	13.41	13.97	0.565	104.22%
q17	103.62	102.83	-0.797	99.23%
q18	144.53	145.92	1.388	100.96%
q19	13.59	14.82	1.232	109.07%
q20	29.66	31.06	1.402	104.73%
q21	260.43	261.69	1.264	100.49%
q22	13.84	12.27	-1.574	88.63%
total	1267.50	1269.97	2.473	100.20%