Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[VL] Prefer to use path.getFileSystem instead of FileSystem.get to create FileSystem #6123

Merged
merged 1 commit into from
Jun 19, 2024

Conversation

yikf
Copy link
Contributor

@yikf yikf commented Jun 18, 2024

What changes were proposed in this pull request?

Prefer to use path.getFileSystem instead of FileSystem.get to create FileSystem, this can always default to the path's schema to get fs which handle the path.

How was this patch tested?

exist tests and manual tests

Copy link

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

@FelixYBW FelixYBW requested a review from JkSelf June 18, 2024 19:00
Copy link
Contributor

@JkSelf JkSelf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks.

@Yohahaha Yohahaha merged commit 27c32f1 into apache:main Jun 19, 2024
36 checks passed
@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_6123_time.csv log/native_master_06_18_2024_800cadd0f_time.csv difference percentage
q1 33.47 34.68 1.206 103.60%
q2 23.75 27.07 3.324 114.00%
q3 38.71 38.23 -0.476 98.77%
q4 33.03 36.14 3.110 109.41%
q5 69.58 71.13 1.557 102.24%
q6 7.86 7.39 -0.473 93.98%
q7 83.82 84.79 0.971 101.16%
q8 84.48 84.78 0.301 100.36%
q9 125.59 117.48 -8.109 93.54%
q10 50.55 45.02 -5.535 89.05%
q11 20.28 22.82 2.547 112.56%
q12 26.06 27.15 1.092 104.19%
q13 39.16 39.59 0.427 101.09%
q14 19.93 18.08 -1.853 90.70%
q15 29.59 30.03 0.438 101.48%
q16 13.99 13.98 -0.016 99.89%
q17 103.99 103.10 -0.893 99.14%
q18 149.07 144.12 -4.949 96.68%
q19 17.37 13.72 -3.647 79.00%
q20 27.84 26.04 -1.800 93.53%
q21 260.91 263.69 2.781 101.07%
q22 13.33 12.17 -1.165 91.27%
total 1272.37 1261.21 -11.162 99.12%

@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCDS SF2000 with Velox backend, for reference only ====

query log/native_master_06_19_2024_time.csv log/native_master_06_16_2024_a08a57c61f_time.csv difference percentage
q1 14.77 14.69 -0.081 99.45%
q2 14.76 15.18 0.421 102.86%
q3 4.71 4.91 0.195 104.14%
q4 66.03 66.82 0.795 101.20%
q5 7.52 8.33 0.813 110.81%
q6 2.29 2.16 -0.127 94.44%
q7 7.96 6.30 -1.660 79.13%
q8 4.65 5.96 1.313 128.23%
q9 20.08 17.57 -2.515 87.48%
q10 12.10 11.03 -1.066 91.19%
q11 37.17 35.51 -1.663 95.53%
q12 1.46 1.44 -0.015 98.96%
q13 5.71 5.31 -0.396 93.06%
q14a 42.20 43.42 1.213 102.87%
q14b 42.07 38.81 -3.262 92.25%
q15 4.17 3.85 -0.319 92.36%
q16 38.26 42.00 3.743 109.78%
q17 5.88 4.73 -1.144 80.55%
q18 7.68 6.16 -1.516 80.26%
q19 2.47 3.59 1.122 145.39%
q20 1.56 2.71 1.149 173.47%
q21 1.30 1.35 0.052 104.00%
q22 8.44 8.13 -0.304 96.40%
q23a 79.83 82.95 3.119 103.91%
q23b 104.47 100.23 -4.239 95.94%
q24a 72.01 70.78 -1.222 98.30%
q24b 80.65 76.15 -4.500 94.42%
q25 4.72 6.22 1.495 131.65%
q26 6.24 4.83 -1.411 77.39%
q27 3.15 3.25 0.108 103.44%
q28 20.75 21.89 1.139 105.49%
q29 7.75 7.91 0.157 102.03%
q30 4.47 4.35 -0.124 97.24%
q31 6.45 6.50 0.052 100.80%
q32 1.22 1.11 -0.107 91.26%
q33 5.85 7.95 2.103 135.94%
q34 4.98 5.74 0.762 115.28%
q35 8.04 7.07 -0.972 87.92%
q36 5.19 3.56 -1.628 68.61%
q37 4.48 4.21 -0.267 94.04%
q38 12.17 12.83 0.654 105.38%
q39a 3.31 3.18 -0.130 96.07%
q39b 2.83 2.83 0.002 100.08%
q40 5.33 6.35 1.018 119.10%
q41 0.66 0.61 -0.045 93.07%
q42 1.07 1.10 0.033 103.06%
q43 5.33 3.90 -1.433 73.11%
q44 9.00 13.59 4.589 150.97%
q45 3.91 4.16 0.248 106.35%
q46 4.53 3.39 -1.145 74.74%
q47 15.33 14.38 -0.947 93.82%
q48 4.76 4.63 -0.137 97.13%
q49 9.50 9.65 0.154 101.62%
q50 27.73 24.71 -3.030 89.08%
q51 8.79 8.97 0.177 102.01%
q52 1.05 1.11 0.055 105.26%
q53 5.41 2.99 -2.420 55.28%
q54 3.47 3.36 -0.113 96.74%
q55 1.10 1.17 0.063 105.69%
q56 4.90 4.57 -0.333 93.21%
q57 9.25 9.20 -0.053 99.43%
q58 2.64 2.66 0.019 100.71%
q59 14.06 14.06 0.001 100.01%
q60 4.87 4.64 -0.226 95.36%
q61 5.29 5.22 -0.071 98.65%
q62 3.88 6.60 2.725 170.27%
q63 2.18 2.10 -0.080 96.32%
q64 56.68 53.36 -3.326 94.13%
q65 14.09 16.93 2.835 120.12%
q66 4.95 3.16 -1.785 63.91%
q67 377.50 363.54 -13.960 96.30%
q68 5.25 3.72 -1.529 70.87%
q69 6.80 9.96 3.156 146.40%
q70 9.45 8.93 -0.521 94.48%
q71 2.43 2.57 0.142 105.82%
q72 186.85 192.42 5.571 102.98%
q73 2.06 2.26 0.196 109.53%
q74 21.63 21.15 -0.477 97.79%
q75 24.15 24.94 0.784 103.25%
q76 12.94 12.49 -0.451 96.52%
q77 2.12 2.33 0.216 110.19%
q78 44.17 42.10 -2.069 95.31%
q79 3.58 3.64 0.055 101.53%
q80 16.00 15.73 -0.268 98.32%
q81 5.32 5.14 -0.179 96.64%
q82 7.89 7.76 -0.134 98.30%
q83 1.53 2.36 0.823 153.68%
q84 3.20 2.82 -0.382 88.08%
q85 8.62 8.11 -0.510 94.08%
q86 3.27 3.26 -0.010 99.70%
q87 12.32 12.39 0.072 100.58%
q88 28.67 28.73 0.060 100.21%
q89 3.18 3.43 0.256 108.04%
q90 4.35 4.35 -0.003 99.93%
q91 2.61 2.62 0.017 100.63%
q92 1.38 1.25 -0.123 91.08%
q93 32.47 30.88 -1.594 95.09%
q94 23.49 22.55 -0.944 95.98%
q9 86.18 87.33 1.150 101.33%
q5 3.85 3.78 -0.076 98.02%
q96 12.06 11.99 -0.061 99.50%
q97 2.15 2.02 -0.128 94.02%
q98 12.40 13.03 0.628 105.06%
q99 12.40 13.03 0.628 105.06%
total 1991.46 1969.68 -21.782 98.91%

@GlutenPerfBot
Copy link
Contributor

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_master_06_19_2024_time.csv log/native_master_06_18_2024_800cadd0f_time.csv difference percentage
q1 34.61 34.68 0.065 100.19%
q2 25.46 27.07 1.608 106.31%
q3 38.59 38.23 -0.360 99.07%
q4 33.98 36.14 2.164 106.37%
q5 70.97 71.13 0.161 100.23%
q6 6.54 7.39 0.843 112.88%
q7 85.08 84.79 -0.289 99.66%
q8 87.11 84.78 -2.328 97.33%
q9 123.67 117.48 -6.191 94.99%
q10 44.85 45.02 0.170 100.38%
q11 20.56 22.82 2.264 111.01%
q12 27.28 27.15 -0.133 99.51%
q13 39.58 39.59 0.011 100.03%
q14 18.81 18.08 -0.736 96.09%
q15 33.35 30.03 -3.323 90.04%
q16 14.28 13.98 -0.306 97.86%
q17 105.60 103.10 -2.498 97.63%
q18 144.60 144.12 -0.482 99.67%
q19 13.77 13.72 -0.044 99.68%
q20 29.04 26.04 -3.006 89.65%
q21 265.23 263.69 -1.535 99.42%
q22 12.88 12.17 -0.706 94.52%
total 1275.86 1261.21 -14.653 98.85%

@yikf yikf deleted the fs branch June 20, 2024 06:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants