Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GLUTEN-5314][VL] Separate FileSink instantiation for different file systems #5881

Merged
merged 2 commits into from
May 28, 2024

Conversation

PHILO-HE
Copy link
Contributor

What changes were proposed in this pull request?

(Please fill in changes proposed in this fix)

(Fixes: #ISSUE-ID)

How was this patch tested?

(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)

(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

Copy link

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

Copy link
Contributor

@JkSelf JkSelf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for the fix.

@PHILO-HE
Copy link
Contributor Author

@RaoZhiRou-Z, could you confirm whether this patch can fix you issue?

@PHILO-HE PHILO-HE changed the title [VL] Separate FileSink instantiation for different file systems [GLUTEN-5314][VL] Separate FileSink instantiation for different file systems May 28, 2024
Copy link

#5314

@PHILO-HE PHILO-HE merged commit d7374bd into apache:main May 28, 2024
39 checks passed
@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_5881_time.csv log/native_master_05_27_2024_efd6f31fb_time.csv difference percentage
q1 33.32 34.01 0.694 102.08%
q2 26.33 25.69 -0.639 97.57%
q3 36.46 36.59 0.129 100.35%
q4 34.12 34.28 0.169 100.49%
q5 69.35 70.33 0.973 101.40%
q6 7.48 7.60 0.119 101.59%
q7 82.31 82.15 -0.166 99.80%
q8 84.03 88.29 4.260 105.07%
q9 122.10 122.17 0.069 100.06%
q10 43.95 46.70 2.749 106.26%
q11 20.56 20.07 -0.481 97.66%
q12 26.53 23.62 -2.904 89.05%
q13 52.33 53.35 1.018 101.95%
q14 18.86 17.13 -1.729 90.83%
q15 31.42 31.65 0.226 100.72%
q16 13.86 13.86 0.000 100.00%
q17 103.16 103.47 0.314 100.30%
q18 147.50 144.12 -3.384 97.71%
q19 13.74 13.72 -0.025 99.82%
q20 26.70 28.10 1.400 105.25%
q21 260.20 260.32 0.128 100.05%
q22 11.79 12.53 0.737 106.25%
total 1266.10 1269.76 3.657 100.29%

@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCDS SF2000 with Velox backend, for reference only ====

query log/native_master_05_28_2024_time.csv log/native_master_05_27_2024_efd6f31fb4_time.csv difference percentage
q1 14.54 15.37 0.827 105.69%
q2 15.27 15.78 0.511 103.34%
q3 4.95 4.39 -0.553 88.82%
q4 63.51 64.02 0.513 100.81%
q5 8.07 8.19 0.115 101.43%
q6 2.31 2.67 0.368 115.95%
q7 5.15 6.21 1.061 120.62%
q8 3.29 3.35 0.068 102.07%
q9 16.20 18.08 1.886 111.64%
q10 9.74 10.51 0.771 107.92%
q11 36.46 35.26 -1.206 96.69%
q12 1.35 1.51 0.163 112.15%
q13 6.67 5.60 -1.073 83.92%
q14a 43.33 44.21 0.877 102.02%
q14b 41.66 41.63 -0.032 99.92%
q15 3.69 3.81 0.121 103.29%
q16 39.54 39.72 0.171 100.43%
q17 5.39 5.72 0.324 106.01%
q18 6.21 9.08 2.872 146.28%
q19 3.56 3.49 -0.067 98.10%
q20 1.59 1.45 -0.144 90.98%
q21 1.09 1.20 0.110 110.15%
q22 7.79 7.98 0.184 102.36%
q23a 83.10 81.64 -1.456 98.25%
q23b 102.20 99.94 -2.263 97.79%
q24a 69.27 72.11 2.840 104.10%
q24b 68.74 79.45 10.706 115.57%
q25 4.01 4.56 0.552 113.78%
q26 4.40 10.31 5.907 234.12%
q27 2.93 2.66 -0.267 90.89%
q28 20.32 22.02 1.703 108.38%
q29 7.82 7.00 -0.823 89.49%
q30 9.08 4.12 -4.960 45.34%
q31 6.79 5.95 -0.834 87.71%
q32 1.23 1.24 0.008 100.63%
q33 4.80 4.75 -0.053 98.89%
q34 4.81 4.66 -0.151 96.86%
q35 6.41 7.93 1.518 123.69%
q36 3.08 3.18 0.094 103.06%
q37 3.70 3.93 0.232 106.27%
q38 11.80 11.84 0.041 100.35%
q39a 3.14 3.35 0.205 106.52%
q39b 2.90 2.79 -0.106 96.33%
q40 3.64 3.83 0.187 105.13%
q41 0.58 0.56 -0.017 96.98%
q42 0.84 0.94 0.106 112.61%
q43 3.54 3.55 0.006 100.17%
q44 10.58 7.08 -3.491 66.99%
q45 3.39 3.51 0.119 103.51%
q46 3.12 3.03 -0.094 96.99%
q47 14.59 14.84 0.246 101.68%
q48 4.29 4.40 0.107 102.50%
q49 7.27 7.41 0.143 101.97%
q50 21.31 24.44 3.131 114.69%
q51 10.20 8.72 -1.471 85.57%
q52 0.97 0.99 0.018 101.85%
q53 2.85 1.79 -1.060 62.79%
q54 3.11 3.30 0.188 106.03%
q55 0.98 0.94 -0.032 96.77%
q56 4.34 4.38 0.046 101.05%
q57 8.63 8.52 -0.117 98.65%
q58 2.47 2.71 0.243 109.83%
q59 14.13 16.41 2.275 116.10%
q60 4.59 5.10 0.511 111.12%
q61 5.15 5.28 0.131 102.54%
q62 3.70 4.25 0.555 115.00%
q63 1.83 1.85 0.025 101.36%
q64 50.79 49.92 -0.872 98.28%
q65 15.15 13.56 -1.585 89.54%
q66 6.39 3.01 -3.374 47.16%
q67 348.77 354.16 5.396 101.55%
q68 3.54 3.50 -0.040 98.88%
q69 6.62 6.78 0.163 102.47%
q70 8.43 8.76 0.327 103.88%
q71 2.23 2.27 0.036 101.60%
q72 187.81 191.86 4.047 102.15%
q73 2.21 2.29 0.079 103.59%
q74 21.75 21.60 -0.155 99.29%
q75 23.68 23.74 0.061 100.26%
q76 6.98 7.03 0.052 100.75%
q77 1.73 1.78 0.058 103.37%
q78 37.38 39.90 2.521 106.74%
q79 3.49 3.47 -0.018 99.49%
q80 10.51 10.58 0.070 100.67%
q81 4.53 4.53 0.007 100.17%
q82 6.09 8.65 2.554 141.92%
q83 1.47 1.46 -0.014 99.03%
q84 2.47 2.93 0.456 118.41%
q85 6.83 7.13 0.308 104.51%
q86 3.11 3.19 0.087 102.78%
q87 12.27 12.28 0.016 100.13%
q88 17.80 16.77 -1.023 94.25%
q89 2.60 2.65 0.042 101.61%
q90 3.35 3.40 0.048 101.43%
q91 2.68 2.57 -0.102 96.20%
q92 1.15 1.14 -0.005 99.57%
q93 28.51 31.14 2.637 109.25%
q94 25.26 21.35 -3.912 84.51%
q9 80.22 81.54 1.321 101.65%
q5 2.56 6.35 3.791 248.35%
q96 12.20 11.98 -0.224 98.17%
q97 2.02 1.82 -0.201 90.05%
q98 9.16 9.80 0.643 107.02%
q99 9.16 9.80 0.643 107.02%
total 1859.70 1895.41 35.711 101.92%

@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_master_05_28_2024_time.csv log/native_master_05_27_2024_efd6f31fb_time.csv difference percentage
q1 34.04 34.01 -0.027 99.92%
q2 23.62 25.69 2.070 108.76%
q3 36.93 36.59 -0.339 99.08%
q4 32.10 34.28 2.184 106.80%
q5 69.91 70.33 0.417 100.60%
q6 5.97 7.60 1.635 127.41%
q7 80.25 82.15 1.895 102.36%
q8 84.79 88.29 3.501 104.13%
q9 121.55 122.17 0.620 100.51%
q10 43.64 46.70 3.059 107.01%
q11 22.27 20.07 -2.196 90.14%
q12 26.90 23.62 -3.279 87.81%
q13 52.40 53.35 0.947 101.81%
q14 21.59 17.13 -4.457 79.36%
q15 31.46 31.65 0.190 100.60%
q16 13.97 13.86 -0.111 99.21%
q17 102.83 103.47 0.644 100.63%
q18 145.92 144.12 -1.801 98.77%
q19 14.82 13.72 -1.101 92.57%
q20 31.06 28.10 -2.962 90.46%
q21 261.69 260.32 -1.366 99.48%
q22 12.27 12.53 0.261 102.13%
total 1269.97 1269.76 -0.216 99.98%

@RaoZhiRou-Z
Copy link

report

thanks, I will try this patch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants