Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[VL] Eliminate pre local sort after offload date type range frame window #6667

Merged
merged 1 commit into from
Aug 2, 2024

Conversation

zml1206
Copy link
Contributor

@zml1206 zml1206 commented Aug 1, 2024

What changes were proposed in this pull request?

Follow up #6653 .
Offload date type range frame window rewrite orderSpec, it will produce new local sort. Pre local sort is unnecessary, so remove it.

 sql("select max(l_partkey) over" +
      " (partition by l_suppkey order by l_commitdate" +
      " RANGE BETWEEN 1 PRECEDING AND CURRENT ROW) from lineitem ").collect()

before PR

  +- == Final Plan ==
   VeloxColumnarToRowExec (15)
   +- ^ ProjectExecTransformer (13)
      +- ^ WindowExecTransformer (12)
         +- ^ SortExecTransformer (11)
            +- ^ ProjectExecTransformer (10)
               +- ^ SortExecTransformer (9)      <--- Useless one
                  +- ^ InputIteratorTransformer (8)
                     +- ShuffleQueryStage (6), Statistics(sizeInBytes=1175.3 KiB, rowCount=6.02E+4)
                        +- ColumnarExchange (5)
                           +- VeloxAppendBatches (4)
                              +- ^ ProjectExecTransformer (2)
                                 +- ^ Scan parquet  (1)

after pr

+- == Final Plan ==
   VeloxColumnarToRowExec (14)
   +- ^ ProjectExecTransformer (12)
      +- ^ WindowExecTransformer (11)
         +- ^ SortExecTransformer (10)
            +- ^ ProjectExecTransformer (9)
               +- ^ InputIteratorTransformer (8)
                  +- ShuffleQueryStage (6), Statistics(sizeInBytes=1175.3 KiB, rowCount=6.02E+4)
                     +- ColumnarExchange (5)
                        +- VeloxAppendBatches (4)
                           +- ^ ProjectExecTransformer (2)
                              +- ^ Scan parquet  (1)

How was this patch tested?

UT

Copy link

github-actions bot commented Aug 1, 2024

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

Copy link

github-actions bot commented Aug 1, 2024

Run Gluten Clickhouse CI

@zml1206
Copy link
Contributor Author

zml1206 commented Aug 2, 2024

cc @JkSelf @ulysses-you thanks.

Copy link
Contributor

@JkSelf JkSelf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for your fixing.

@JkSelf JkSelf merged commit b7dd9a8 into apache:main Aug 2, 2024
44 checks passed
@zml1206
Copy link
Contributor Author

zml1206 commented Aug 2, 2024

Thank you for review. @JkSelf

@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCDS SF2000 with Velox backend, for reference only ====

query log/native_master_08_02_2024_time.csv log/native_master_08_01_2024_e472e5dac3_time.csv difference percentage
q1 13.40 13.43 0.030 100.22%
q2 13.64 11.57 -2.074 84.80%
q3 4.41 3.93 -0.473 89.26%
q4 69.94 68.93 -1.009 98.56%
q5 7.81 8.84 1.025 113.12%
q6 2.26 3.50 1.238 154.67%
q7 4.93 6.56 1.631 133.10%
q8 5.38 3.29 -2.087 61.18%
q9 24.03 24.03 0.003 100.01%
q10 9.68 9.65 -0.038 99.61%
q11 37.31 36.74 -0.578 98.45%
q12 1.43 1.38 -0.051 96.41%
q13 6.54 6.05 -0.487 92.56%
q14a 46.61 46.13 -0.481 98.97%
q14b 41.84 42.47 0.630 101.51%
q15 2.40 2.69 0.286 111.90%
q16 45.79 45.73 -0.059 99.87%
q17 4.91 4.78 -0.133 97.29%
q18 6.78 6.90 0.123 101.81%
q19 2.10 2.15 0.043 102.06%
q20 2.33 1.40 -0.928 60.14%
q21 2.36 1.21 -1.150 51.28%
q22 7.84 8.72 0.881 111.25%
q23a 101.91 101.52 -0.394 99.61%
q23b 124.13 125.98 1.855 101.49%
q24a 94.01 91.79 -2.218 97.64%
q24b 98.07 98.41 0.337 100.34%
q25 4.17 4.14 -0.031 99.26%
q26 3.37 3.30 -0.069 97.94%
q27 4.02 4.00 -0.029 99.27%
q28 32.53 32.38 -0.147 99.55%
q29 10.36 11.53 1.172 111.31%
q30 4.89 4.81 -0.080 98.36%
q31 8.22 7.15 -1.069 87.00%
q32 1.47 1.16 -0.309 79.05%
q33 4.81 4.31 -0.498 89.63%
q34 3.76 3.87 0.104 102.76%
q35 7.32 8.18 0.863 111.80%
q36 4.43 4.63 0.196 104.43%
q37 4.52 4.47 -0.044 99.02%
q38 14.04 14.47 0.432 103.08%
q39a 3.45 3.05 -0.400 88.40%
q39b 3.06 2.72 -0.341 88.85%
q40 3.84 4.05 0.202 105.27%
q41 0.62 0.64 0.019 103.07%
q42 0.90 0.95 0.050 105.58%
q43 4.23 4.59 0.360 108.49%
q44 9.83 10.05 0.221 102.24%
q45 3.30 3.30 0.000 100.00%
q46 3.80 3.76 -0.044 98.85%
q47 17.78 17.92 0.145 100.82%
q48 5.13 5.21 0.083 101.61%
q49 8.57 8.60 0.030 100.35%
q50 21.54 21.19 -0.350 98.37%
q51 9.74 9.36 -0.375 96.15%
q52 1.08 1.06 -0.014 98.73%
q53 2.33 2.25 -0.074 96.82%
q54 3.95 3.79 -0.159 95.99%
q55 1.09 1.10 0.011 101.00%
q56 4.13 4.29 0.155 103.75%
q57 10.98 10.84 -0.139 98.73%
q58 2.50 2.37 -0.135 94.60%
q59 10.62 10.66 0.043 100.40%
q60 4.03 4.11 0.072 101.79%
q61 4.16 4.06 -0.104 97.49%
q62 4.86 5.03 0.177 103.65%
q63 2.26 2.32 0.058 102.59%
q64 58.92 59.43 0.513 100.87%
q65 18.93 20.11 1.183 106.25%
q66 3.88 5.38 1.505 138.82%
q67 403.58 386.18 -17.398 95.69%
q68 3.49 3.64 0.148 104.24%
q69 5.34 5.36 0.015 100.28%
q70 11.38 11.30 -0.079 99.31%
q71 2.38 2.35 -0.032 98.68%
q72 220.08 215.13 -4.953 97.75%
q73 2.40 4.03 1.621 167.42%
q74 23.51 22.14 -1.366 94.19%
q75 26.60 26.55 -0.048 99.82%
q76 11.08 11.00 -0.077 99.30%
q77 2.42 2.22 -0.203 91.61%
q78 53.41 59.88 6.462 112.10%
q79 3.66 3.84 0.184 105.02%
q80 12.72 13.41 0.699 105.50%
q81 4.66 5.41 0.754 116.18%
q82 7.58 7.16 -0.421 94.44%
q83 1.68 1.61 -0.066 96.03%
q84 2.90 2.74 -0.165 94.33%
q85 7.77 7.35 -0.423 94.55%
q86 3.86 3.86 0.002 100.05%
q87 13.24 13.25 0.006 100.05%
q88 21.42 21.42 0.000 100.00%
q89 3.68 3.70 0.023 100.63%
q90 3.40 3.15 -0.247 92.72%
q91 2.59 2.46 -0.135 94.79%
q92 1.26 1.21 -0.054 95.74%
q93 38.99 38.42 -0.568 98.54%
q94 25.08 23.94 -1.134 95.48%
q9 89.81 86.96 -2.852 96.82%
q5 2.64 2.74 0.108 104.11%
q96 17.40 17.38 -0.023 99.87%
q97 1.93 1.98 0.053 102.75%
q98 10.76 10.37 -0.388 96.39%
q99 10.76 10.37 -0.388 96.39%
total 2161.95 2140.50 -21.450 99.01%

@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_master_08_02_2024_time.csv log/native_master_08_01_2024_acb50c472a_time.csv difference percentage
q1 40.71 39.31 -1.395 96.57%
q2 29.99 29.53 -0.461 98.46%
q3 51.10 54.54 3.435 106.72%
q4 42.20 41.78 -0.428 98.99%
q5 104.03 105.12 1.086 101.04%
q6 11.29 12.79 1.492 113.21%
q7 116.66 116.82 0.160 100.14%
q8 115.83 113.59 -2.243 98.06%
q9 167.98 169.30 1.326 100.79%
q10 65.15 65.98 0.826 101.27%
q11 27.62 27.92 0.307 101.11%
q12 29.91 29.66 -0.250 99.16%
q13 51.44 51.37 -0.070 99.86%
q14 25.27 23.87 -1.400 94.46%
q15 55.54 54.56 -0.980 98.24%
q16 18.78 18.94 0.164 100.88%
q17 128.58 130.37 1.789 101.39%
q18 198.43 197.07 -1.358 99.32%
q19 25.19 27.52 2.334 109.27%
q20 39.44 41.72 2.281 105.78%
q21 380.24 377.96 -2.274 99.40%
q22 15.78 15.56 -0.221 98.60%
total 1741.18 1745.30 4.119 100.24%

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants