Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[VL] Daily Update Velox Version (2024_06_30) #6284

Merged
merged 12 commits into from
Jul 1, 2024
Merged

[VL] Daily Update Velox Version (2024_06_30) #6284

merged 12 commits into from
Jul 1, 2024

Conversation

GlutenPerfBot
Copy link
Contributor

@GlutenPerfBot GlutenPerfBot commented Jun 30, 2024 •
edited by marin-ma
Loading

Upstream Velox's New Commits:

0ef0ac8e4 by Jia Ke, Enable right join in smj (10148)
c54e59dbb by wypb, Fix HashStringAllocator::clear() and cumulativeBytes_ (10260)
4963d7116 by duanmeng, Add recursive spill for RowNumber (8654)
e3de4ea9d by Sandino Flores, Add support for Protobuf v22+ (10294)
0d8022846 by PHILO-HE, Support finding installed arrow libraries from system (9992)
fd955bff4 by liangyongyuan, Add float/double types support for Spark mod function (9848)
0ced9e5f0 by NEUpanning, Fix typo in expression evaluation documentation (10304)
8803bfbd1 by lingbin, Fix typo in SIMD document (10319)
bcfc8f8c3 by PHILO-HE, Allow returning Status from callNullable and callNullFree methods (10274)
258db516d by PHILO-HE, Use legacySizeOfNull argument to determine the behavior of Spark size function (10100)

@github-actions GitHub Actions
Copy link

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

@PHILO-HE
Copy link
Contributor

PHILO-HE commented Jul 1, 2024

@marin-ma, we still need to apply some changes for arrow build (if arrow lib is not found, velox will resolve the dependency by
building from source as before). Its cmake file is moved here:
https://github.com/facebookincubator/velox/tree/main/CMake/resolve_dependency_modules/arrow

@PHILO-HE
Copy link
Contributor

PHILO-HE commented Jul 1, 2024

ARROW_HOME needs to be set to

"${VELOX_HOME}/_build/release/CMake/resolve_dependency_modules/arrow/arrow_ep/" (replace release with debug for debug mode).

https://github.com/apache/incubator-gluten/blob/main/cpp/CMakeLists.txt#L71-L73

@marin-ma marin-ma merged commit 5d6d214 into apache:main Jul 1, 2024
40 checks passed
@GlutenPerfBot
Copy link
Contributor Author

===== Performance report for TPCDS SF2000 with Velox backend, for reference only ====

query log/native_master_07_01_2024_time.csv log/native_master_06_30_2024_86449d0c8_time.csv difference percentage
q1 19.66 14.83 -4.822 75.47%
q2 13.98 14.85 0.872 106.24%
q3 4.21 4.21 0.007 100.16%
q4 63.74 63.51 -0.229 99.64%
q5 7.86 7.38 -0.481 93.88%
q6 2.39 2.32 -0.079 96.70%
q7 5.24 6.03 0.791 115.08%
q8 3.70 5.14 1.448 139.16%
q9 22.95 21.68 -1.265 94.49%
q10 11.68 11.58 -0.097 99.17%
q11 35.50 36.42 0.927 102.61%
q12 2.44 2.59 0.153 106.26%
q13 5.53 9.04 3.517 163.63%
q14a 44.64 41.82 -2.822 93.68%
q14b 40.59 38.81 -1.773 95.63%
q15 3.93 3.62 -0.314 92.01%
q16 38.53 39.65 1.118 102.90%
q17 7.25 5.07 -2.185 69.88%
q18 6.66 6.20 -0.459 93.11%
q19 2.08 2.34 0.258 112.40%
q20 1.49 1.40 -0.089 94.05%
q21 1.06 1.02 -0.041 96.15%
q22 7.67 7.79 0.119 101.56%
q23a 84.05 84.02 -0.026 99.97%
q23b 100.62 101.13 0.513 100.51%
q24a 69.49 69.39 -0.103 99.85%
q24b 68.46 72.20 3.747 105.47%
q25 8.18 4.35 -3.835 53.14%
q26 4.43 2.85 -1.573 64.46%
q27 2.86 3.15 0.286 109.98%
q28 20.85 28.74 7.892 137.86%
q29 9.59 7.11 -2.482 74.12%
q30 5.23 4.19 -1.039 80.13%
q31 6.16 6.70 0.549 108.91%
q32 1.14 2.13 0.989 186.97%
q33 4.80 4.74 -0.063 98.68%
q34 3.66 3.65 -0.010 99.73%
q35 6.73 6.57 -0.163 97.58%
q36 3.29 3.35 0.068 102.06%
q37 4.02 4.05 0.027 100.68%
q38 11.92 11.85 -0.068 99.43%
q39a 3.33 5.69 2.361 170.99%
q39b 2.86 3.27 0.418 114.63%
q40 8.52 3.73 -4.793 43.77%
q41 0.64 0.62 -0.021 96.71%
q42 0.96 0.94 -0.019 98.05%
q43 5.34 3.89 -1.447 72.89%
q44 8.81 8.62 -0.190 97.84%
q45 3.54 3.40 -0.137 96.12%
q46 3.40 3.22 -0.178 94.76%
q47 14.29 14.40 0.108 100.75%
q48 4.59 4.58 -0.006 99.87%
q49 10.03 9.80 -0.233 97.68%
q50 23.46 24.23 0.776 103.31%
q51 8.63 8.51 -0.116 98.66%
q52 1.03 1.04 0.008 100.74%
q53 1.99 2.01 0.029 101.45%
q54 3.37 3.23 -0.138 95.90%
q55 1.05 0.99 -0.062 94.08%
q56 5.48 4.53 -0.950 82.67%
q57 8.61 9.28 0.673 107.82%
q58 2.71 2.59 -0.118 95.63%
q59 17.09 16.43 -0.663 96.12%
q60 4.95 4.83 -0.122 97.53%
q61 7.46 5.67 -1.794 75.95%
q62 4.34 4.76 0.416 109.58%
q63 2.24 2.32 0.080 103.57%
q64 48.99 49.99 1.001 102.04%
q65 13.85 13.70 -0.141 98.98%
q66 3.43 4.25 0.817 123.81%
q67 355.58 355.08 -0.502 99.86%
q68 3.79 3.65 -0.140 96.32%
q69 6.55 6.25 -0.295 95.49%
q70 8.73 8.60 -0.127 98.54%
q71 4.20 2.45 -1.748 58.38%
q72 189.48 188.05 -1.430 99.25%
q73 2.37 2.26 -0.111 95.31%
q74 21.60 21.26 -0.336 98.44%
q75 23.29 23.20 -0.097 99.59%
q76 12.12 9.42 -2.697 77.75%
q77 2.10 2.21 0.109 105.20%
q78 39.33 41.97 2.638 106.71%
q79 3.45 3.51 0.056 101.64%
q80 11.00 11.16 0.154 101.40%
q81 5.29 5.19 -0.100 98.11%
q82 7.99 10.14 2.157 127.01%
q83 1.48 1.55 0.077 105.24%
q84 2.99 2.68 -0.309 89.65%
q85 6.89 6.54 -0.354 94.86%
q86 3.17 3.21 0.042 101.34%
q87 13.43 12.09 -1.340 90.03%
q88 24.68 24.76 0.076 100.31%
q89 7.29 3.25 -4.039 44.58%
q90 4.32 9.55 5.230 221.11%
q91 2.68 5.39 2.714 201.44%
q92 1.41 1.40 -0.011 99.25%
q93 28.70 28.51 -0.191 99.33%
q94 21.24 21.60 0.359 101.69%
q9 85.76 83.91 -1.853 97.84%
q5 3.47 3.78 0.318 109.18%
q96 12.28 12.34 0.060 100.49%
q97 2.08 1.96 -0.129 93.80%
q98 9.15 9.18 0.024 100.26%
q99 9.15 9.18 0.024 100.26%
total 1921.10 1914.13 -6.975 99.64%

@FelixYBW
Copy link
Contributor

FelixYBW commented Jul 1, 2024

@PHILO-HE just to confirm, with this PR, how many times Arrow will be built and where it's built?

FelixYBW
FelixYBW reviewed Jul 1, 2024
View reviewed changes
backends-velox/src/test/scala/org/apache/gluten/execution/TestOperator.scala
@@ -1017,7 +1017,7 @@ class TestOperator extends VeloxWholeStageTransformerSuite with AdaptiveSparkPla
}
}

test("test explode/posexplode function") {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@marin-ma why these operators test should be ignored?

Copy link
Contributor

@PHILO-HE PHILO-HE Jul 2, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@FelixYBW, let me help clarify. This is a known test failure since we need some code change in Gluten. I have created a Gluten pr to do the code change and re-enable these tests: #6014.

@GlutenPerfBot
Copy link
Contributor Author

===== Performance report for TPCH SF2000 with Velox backend, for reference only ====

query log/native_master_07_01_2024_time.csv log/native_master_06_30_2024_86449d0c8_time.csv difference percentage
q1 34.57 35.24 0.667 101.93%
q2 23.98 23.80 -0.185 99.23%
q3 39.13 40.97 1.838 104.70%
q4 32.81 32.74 -0.069 99.79%
q5 70.64 70.02 -0.618 99.12%
q6 8.16 7.93 -0.236 97.11%
q7 83.92 82.49 -1.427 98.30%
q8 85.80 83.59 -2.204 97.43%
q9 120.92 118.63 -2.295 98.10%
q10 46.13 49.82 3.684 107.99%
q11 21.88 20.68 -1.198 94.53%
q12 25.17 25.51 0.349 101.39%
q13 41.24 38.43 -2.802 93.20%
q14 20.05 17.87 -2.188 89.09%
q15 33.15 30.75 -2.394 92.78%
q16 13.35 14.20 0.845 106.33%
q17 104.45 104.33 -0.121 99.88%
q18 148.04 148.98 0.933 100.63%
q19 13.81 14.66 0.850 106.16%
q20 27.79 29.28 1.490 105.36%
q21 266.41 261.77 -4.637 98.26%
q22 13.75 12.54 -1.202 91.26%
total 1275.14 1264.22 -10.919 99.14%

@PHILO-HE
Copy link
Contributor

PHILO-HE commented Jul 2, 2024

@PHILO-HE just to confirm, with this PR, how many times Arrow will be built and where it's built?

@FelixYBW, currently, there are still two builds, since a build issue for Velox was found yesterday. We need wait for this velox pr: facebookincubator/velox#10355

@FelixYBW
Copy link
Contributor

FelixYBW commented Jul 2, 2024

Can we add this PR to OAP/Velox temporarily?

In future if a PR from us to Velox may impact Gluten, let's submit the PR to Gluten also and make sure once Velox's PR is merged Gluten can work as expected.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@FelixYBW FelixYBW FelixYBW left review comments

@PHILO-HE PHILO-HE PHILO-HE left review comments

@marin-ma marin-ma marin-ma approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@GlutenPerfBot @PHILO-HE @FelixYBW @marin-ma