Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](mtmv) Fix rewrite fail by materialized view when filter or join condition has alias #44779

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

seawinde
Copy link
Contributor

What problem does this PR solve?

query and mv def are as following,partsupp.public_col as public_col is alias, this would cause rewritting fail by materialized view with msg, the graph logic between query and view is different.

the pr fix this

      select 
      o_custkey, 
      o_orderdate, 
      o_shippriority, 
      o_comment, 
      o_orderkey, 
      orders.public_col as col1, 
      l_orderkey, 
      l_partkey, 
      l_suppkey, 
      lineitem.public_col as col2, 
      ps_partkey, 
      ps_suppkey, 
      partsupp.public_col as col3, 
      partsupp.public_col * 2 as col4, 
      o_orderkey + l_orderkey + ps_partkey * 2, 
      sum(
        o_orderkey + l_orderkey + ps_partkey * 2
      ), 
      count() as count_all 
    from 
      (
        select 
          o_custkey, 
          o_orderdate, 
          o_shippriority, 
          o_comment, 
          o_orderkey, 
          orders.public_col as public_col 
        from 
          orders
      ) orders 
      left join (
        select 
          l_orderkey, 
          l_partkey, 
          l_suppkey, 
          lineitem.public_col as public_col 
        from 
          lineitem 
        where 
          lineitem.public_col is null 
          or lineitem.public_col <> 1
      ) lineitem on l_orderkey = o_orderkey 
      inner join (
        select 
          ps_partkey, 
          ps_suppkey, 
          partsupp.public_col as public_col 
        from 
          partsupp
      ) partsupp on ps_partkey = o_orderkey
    where 
      lineitem.public_col is null 
      or lineitem.public_col <> 1 
      and o_orderkey = 2
    group by 
      1, 
      2, 
      3, 
      4, 
      5, 
      6, 
      7, 
      8, 
      9, 
      10, 
      11, 
      12, 
      13, 
      14;

Issue Number: close #xxx

Related PR: #27922

Problem Summary:

Rewrite fail by materialized view when filter or join condition has alias

Release note

Fix rewrite fail by materialized view when filter or join condition has alias

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@seawinde
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39951 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit eb9ca0c1fef7a2de20825fb75129b51488efdce2, data reload: false

------ Round 1 ----------------------------------
q1	17789	7669	7312	7312
q2	2053	177	175	175
q3	10665	1082	1171	1082
q4	10582	749	714	714
q5	7583	2726	2691	2691
q6	242	149	146	146
q7	992	649	596	596
q8	9251	1875	1878	1875
q9	6695	6504	6517	6504
q10	7025	2297	2343	2297
q11	470	257	256	256
q12	419	217	224	217
q13	17931	3090	3031	3031
q14	236	210	212	210
q15	580	531	521	521
q16	632	602	596	596
q17	969	537	491	491
q18	7323	6695	6599	6599
q19	1357	942	1007	942
q20	508	180	180	180
q21	3963	3199	3299	3199
q22	379	326	317	317
Total cold run time: 107644 ms
Total hot run time: 39951 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7307	7241	7516	7241
q2	332	231	235	231
q3	2882	2851	2885	2851
q4	2083	1781	1814	1781
q5	5671	5667	5683	5667
q6	231	146	152	146
q7	2279	1791	1823	1791
q8	3466	3516	3523	3516
q9	9086	9114	9090	9090
q10	3609	3557	3578	3557
q11	591	508	514	508
q12	822	582	585	582
q13	9716	3294	3248	3248
q14	319	265	269	265
q15	583	527	525	525
q16	707	673	635	635
q17	1856	1649	1655	1649
q18	8434	7851	7699	7699
q19	2732	1520	1489	1489
q20	2118	1873	1882	1873
q21	5650	5475	5458	5458
q22	641	563	555	555
Total cold run time: 71115 ms
Total hot run time: 60357 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 198899 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit eb9ca0c1fef7a2de20825fb75129b51488efdce2, data reload: false

query1	1282	957	929	929
query2	6236	2109	2076	2076
query3	11123	4677	4597	4597
query4	33056	23615	23635	23615
query5	3572	504	493	493
query6	297	187	173	173
query7	3975	309	314	309
query8	300	230	238	230
query9	9321	2677	2672	2672
query10	455	259	271	259
query11	17964	15375	15491	15375
query12	160	103	100	100
query13	1543	426	398	398
query14	8700	7222	7378	7222
query15	268	201	182	182
query16	7981	472	501	472
query17	1611	562	561	561
query18	2117	308	302	302
query19	223	168	154	154
query20	121	115	109	109
query21	202	107	104	104
query22	4856	4521	4546	4521
query23	35389	34696	34718	34696
query24	10404	2564	2664	2564
query25	619	415	448	415
query26	1104	157	160	157
query27	2239	294	292	292
query28	6896	2503	2471	2471
query29	857	443	440	440
query30	244	161	159	159
query31	1046	874	856	856
query32	96	58	58	58
query33	807	303	299	299
query34	1046	505	531	505
query35	933	784	798	784
query36	1129	981	968	968
query37	118	74	79	74
query38	4539	4513	4522	4513
query39	1545	1476	1496	1476
query40	212	99	99	99
query41	45	44	44	44
query42	113	103	105	103
query43	554	498	501	498
query44	1322	849	844	844
query45	192	176	168	168
query46	1196	765	736	736
query47	1989	1926	1913	1913
query48	443	325	323	323
query49	922	401	385	385
query50	852	447	410	410
query51	7502	7318	7230	7230
query52	106	88	86	86
query53	273	190	193	190
query54	1230	412	407	407
query55	89	76	77	76
query56	258	240	243	240
query57	1286	1155	1136	1136
query58	239	230	222	222
query59	3404	3191	3208	3191
query60	284	253	257	253
query61	162	101	133	101
query62	884	657	660	657
query63	225	191	179	179
query64	3854	672	635	635
query65	3342	3244	3216	3216
query66	853	295	313	295
query67	16138	15982	15566	15566
query68	4477	557	553	553
query69	417	252	248	248
query70	1217	1166	1148	1148
query71	395	255	255	255
query72	6365	3991	4080	3991
query73	798	361	364	361
query74	10253	9109	9083	9083
query75	3377	2679	2714	2679
query76	2692	1049	1148	1049
query77	407	271	270	270
query78	10319	9438	9366	9366
query79	2014	598	611	598
query80	766	524	441	441
query81	512	227	232	227
query82	690	127	116	116
query83	167	148	145	145
query84	250	71	75	71
query85	1351	306	294	294
query86	431	281	307	281
query87	4721	4613	4641	4613
query88	3714	2221	2182	2182
query89	466	294	287	287
query90	2108	193	194	193
query91	144	101	105	101
query92	58	48	50	48
query93	1954	552	553	552
query94	826	275	299	275
query95	359	250	249	249
query96	623	281	271	271
query97	2881	2685	2651	2651
query98	225	194	193	193
query99	1593	1341	1325	1325
Total cold run time: 298913 ms
Total hot run time: 198899 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.33 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit eb9ca0c1fef7a2de20825fb75129b51488efdce2, data reload: false

query1	0.03	0.02	0.03
query2	0.07	0.03	0.03
query3	0.23	0.08	0.07
query4	1.61	0.11	0.10
query5	0.43	0.42	0.41
query6	1.14	0.66	0.65
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.57	0.52	0.51
query10	0.56	0.56	0.57
query11	0.14	0.10	0.11
query12	0.14	0.11	0.12
query13	0.61	0.60	0.59
query14	2.74	2.83	2.73
query15	0.92	0.83	0.83
query16	0.38	0.38	0.37
query17	1.01	0.99	0.98
query18	0.23	0.21	0.21
query19	1.86	1.89	2.05
query20	0.01	0.01	0.01
query21	15.36	0.58	0.58
query22	2.67	2.49	2.32
query23	16.97	1.00	0.83
query24	3.02	1.02	0.99
query25	0.28	0.10	0.09
query26	0.57	0.14	0.15
query27	0.05	0.04	0.05
query28	10.84	1.10	1.08
query29	12.53	3.29	3.26
query30	0.25	0.06	0.06
query31	2.86	0.39	0.39
query32	3.27	0.47	0.47
query33	2.95	3.03	3.01
query34	16.95	4.50	4.53
query35	4.55	4.55	4.57
query36	0.68	0.50	0.48
query37	0.09	0.07	0.06
query38	0.04	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.13	0.13
query41	0.08	0.03	0.03
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 107.02 s
Total hot run time: 33.33 s

@seawinde
Copy link
Contributor Author

seawinde commented Dec 2, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40067 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dffce5e42b13344be5808859a6fbd7981fba8f5f, data reload: false

------ Round 1 ----------------------------------
q1	17594	7520	7257	7257
q2	2042	183	186	183
q3	10572	1075	1233	1075
q4	10541	804	743	743
q5	7608	2778	2734	2734
q6	241	147	150	147
q7	1001	643	597	597
q8	9231	1851	1942	1851
q9	6662	6498	6478	6478
q10	7000	2321	2318	2318
q11	477	286	262	262
q12	432	227	227	227
q13	17785	3071	2998	2998
q14	265	207	214	207
q15	574	536	531	531
q16	652	563	569	563
q17	975	522	572	522
q18	7398	6793	6655	6655
q19	1329	1084	1012	1012
q20	459	181	190	181
q21	4061	3208	3323	3208
q22	382	323	318	318
Total cold run time: 107281 ms
Total hot run time: 40067 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7277	7253	7258	7253
q2	323	227	231	227
q3	2876	2825	2905	2825
q4	2015	1818	1843	1818
q5	5656	5668	5704	5668
q6	225	143	146	143
q7	2182	1829	1826	1826
q8	3464	3549	3533	3533
q9	8952	9120	9047	9047
q10	3626	3562	3583	3562
q11	616	503	514	503
q12	829	648	608	608
q13	11620	3331	3237	3237
q14	321	283	288	283
q15	588	525	528	525
q16	681	649	649	649
q17	1836	1627	1615	1615
q18	8258	7871	7556	7556
q19	1747	1688	1653	1653
q20	2093	1868	1894	1868
q21	5870	5469	5502	5469
q22	668	599	568	568
Total cold run time: 71723 ms
Total hot run time: 60436 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197670 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dffce5e42b13344be5808859a6fbd7981fba8f5f, data reload: false

query1	1229	969	956	956
query2	6251	2070	2034	2034
query3	10985	4281	4505	4281
query4	67250	28747	23852	23852
query5	4950	462	454	454
query6	396	197	188	188
query7	5473	308	299	299
query8	317	246	241	241
query9	8577	2718	2730	2718
query10	412	251	256	251
query11	16997	15225	15906	15225
query12	152	102	101	101
query13	1411	436	418	418
query14	11106	6823	7687	6823
query15	227	184	208	184
query16	7223	495	502	495
query17	1246	599	572	572
query18	1842	311	322	311
query19	215	168	167	167
query20	118	113	114	113
query21	218	110	104	104
query22	4619	4685	4568	4568
query23	35558	34653	34458	34458
query24	5717	2483	2519	2483
query25	482	376	425	376
query26	677	157	153	153
query27	1890	279	297	279
query28	4808	2478	2490	2478
query29	661	444	425	425
query30	232	147	151	147
query31	1034	835	846	835
query32	68	55	56	55
query33	418	306	296	296
query34	949	505	527	505
query35	905	769	771	769
query36	1082	963	974	963
query37	129	75	78	75
query38	4438	4482	4345	4345
query39	1526	1476	1463	1463
query40	211	100	103	100
query41	45	44	43	43
query42	120	103	108	103
query43	534	490	495	490
query44	1207	861	841	841
query45	188	175	166	166
query46	1169	747	732	732
query47	2047	1970	1943	1943
query48	426	317	337	317
query49	754	417	398	398
query50	835	397	384	384
query51	7522	7171	7122	7122
query52	103	84	88	84
query53	249	176	179	176
query54	519	392	386	386
query55	80	77	78	77
query56	241	229	234	229
query57	1271	1133	1123	1123
query58	223	205	209	205
query59	3159	3010	3058	3010
query60	262	243	234	234
query61	111	109	109	109
query62	842	668	665	665
query63	209	189	196	189
query64	1759	665	624	624
query65	3371	3220	3234	3220
query66	722	312	315	312
query67	15987	15910	15745	15745
query68	3898	623	559	559
query69	441	250	256	250
query70	1226	1044	1141	1044
query71	358	259	253	253
query72	6094	4042	4064	4042
query73	756	359	360	359
query74	10308	8925	9039	8925
query75	3456	2667	2648	2648
query76	2053	1137	1125	1125
query77	489	267	272	267
query78	10720	9633	9582	9582
query79	1512	606	600	600
query80	884	424	419	419
query81	497	227	231	227
query82	1257	128	116	116
query83	251	144	139	139
query84	277	69	72	69
query85	899	298	304	298
query86	355	309	297	297
query87	4629	4554	4580	4554
query88	3730	2225	2211	2211
query89	425	314	308	308
query90	2038	186	190	186
query91	133	102	102	102
query92	69	53	53	53
query93	1897	541	537	537
query94	789	290	295	290
query95	344	245	253	245
query96	622	278	285	278
query97	2904	2721	2725	2721
query98	223	195	197	195
query99	1714	1334	1284	1284
Total cold run time: 321202 ms
Total hot run time: 197670 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.92 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dffce5e42b13344be5808859a6fbd7981fba8f5f, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.04
query3	0.23	0.08	0.07
query4	1.63	0.10	0.10
query5	0.40	0.43	0.41
query6	1.16	0.66	0.65
query7	0.01	0.02	0.01
query8	0.05	0.03	0.04
query9	0.59	0.50	0.52
query10	0.55	0.55	0.56
query11	0.14	0.11	0.10
query12	0.14	0.11	0.12
query13	0.60	0.60	0.59
query14	2.83	2.74	2.72
query15	0.92	0.81	0.84
query16	0.39	0.37	0.38
query17	1.03	1.05	1.02
query18	0.21	0.20	0.20
query19	1.97	1.84	2.03
query20	0.01	0.01	0.01
query21	15.38	0.60	0.58
query22	2.69	2.04	1.97
query23	17.08	0.95	0.87
query24	2.91	1.09	1.61
query25	0.28	0.10	0.14
query26	0.56	0.16	0.15
query27	0.05	0.05	0.04
query28	10.05	1.10	1.07
query29	12.54	3.22	3.20
query30	0.25	0.06	0.06
query31	2.87	0.39	0.38
query32	3.26	0.46	0.46
query33	2.94	2.99	3.04
query34	17.10	4.49	4.47
query35	4.52	4.55	4.55
query36	0.70	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.02
query40	0.18	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.04
Total cold run time: 106.65 s
Total hot run time: 32.92 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants