[VL] Fix for centos9 build of Gluten #6183

deepashreeraghu · 2024-06-22T16:08:53Z

What changes were proposed in this pull request?

TO support building Gluten on Centos9

How was this patch tested?

I am trying to build Gluten on Centos9. Will update here once I am able to build successfully.

github-actions · 2024-06-22T16:09:10Z

Thanks for opening a pull request!

Could you open an issue for this pull request on Github Issues?

https://github.com/apache/incubator-gluten/issues

Then could you also rename commit message and pull request title in the following format?

[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}

See also:

Other pull requests

github-actions · 2024-06-26T15:49:29Z

Run Gluten Clickhouse CI

deepashreeraghu · 2024-06-27T04:38:34Z

@FelixYBW - I am able to build Gluten on Centos9 with these changes. Have got the jar generated, now have to consume the jar and test it.

[INFO] Reactor Summary for Gluten Parent Pom 1.2.0-SNAPSHOT:
[INFO] 
[INFO] Gluten Parent Pom .................................. SUCCESS [  0.216 s]
[INFO] Gluten UI .......................................... SUCCESS [  9.042 s]
[INFO] Gluten Shims ....................................... SUCCESS [  0.160 s]
[INFO] Gluten Shims Common ................................ SUCCESS [ 10.171 s]
[INFO] Gluten Shims for Spark 3.4 ......................... SUCCESS [  9.944 s]
[INFO] Gluten Ras ......................................... SUCCESS [  0.150 s]
[INFO] Gluten Ras Common .................................. SUCCESS [ 23.889 s]
[INFO] Gluten Core ........................................ SUCCESS [ 53.233 s]
[INFO] Gluten Data ........................................ SUCCESS [ 10.314 s]
[INFO] Gluten Backends Velox .............................. SUCCESS [ 47.250 s]
[INFO] Gluten Celeborn .................................... SUCCESS [  0.024 s]
[INFO] Gluten Celeborn Common ............................. SUCCESS [  4.708 s]
[INFO] Gluten Celeborn Velox .............................. SUCCESS [  4.012 s]
[INFO] Gluten Celeborn Package ............................ SUCCESS [  0.041 s]
[INFO] Gluten Uniffle ..................................... SUCCESS [  0.153 s]
[INFO] Gluten Uniffle Velox ............................... SUCCESS [  4.328 s]
[INFO] Gluten Uniffle Package ............................. SUCCESS [  0.212 s]
[INFO] Gluten Package ..................................... SUCCESS [01:10 min]
[INFO] Gluten Ras Planner ................................. SUCCESS [  0.158 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  04:09 min
[INFO] Finished at: 2024-06-26T19:58:05Z
[INFO] ------------------------------------------------------------------------
+ mkdir -p /incubator-gluten/dev/../package/target/thirdparty-lib
+ [[ centos == \u\b\u\n\t\u ]]
+ [[ centos == \p\o\p ]]
+ '[' centos == centos ']'
+ '[' 9 == 9 ']'
+ process_setup_centos_9
+ cp /lib64/libre2.so.9 /lib64/libdouble-conversion.so.3 /lib64/libevent-2.1.so.7 /lib64/libdwarf.so.0 /lib64/libgsasl.so.7 /lib64/libicudata.so.67 /lib64/libicui18n.so.67 /lib64/libicuuc.so.67 /lib64/libidn.so.12 /lib64/libntlm.so.0 /lib64/libsodium.so.23 /incubator-gluten/dev/../package/target/thirdparty-lib/
+ cp /usr/local/lib/libhdfs3.so.1 /usr/local/lib/libboost_context.so.1.84.0 /usr/local/lib/libboost_filesystem.so.1.84.0 /usr/local/lib/libboost_program_options.so.1.84.0 /usr/local/lib/libboost_regex.so.1.84.0 /usr/local/lib/libboost_system.so.1.84.0 /usr/local/lib/libboost_thread.so.1.84.0 /usr/local/lib/libboost_atomic.so.1.84.0 /usr/local/lib/libprotobuf.so.32 /incubator-gluten/dev/../package/target/thirdparty-lib/
+ cp /usr/local/lib64/libgflags.so.2.2 /usr/local/lib64/libglog.so.1 /incubator-gluten/dev/../package/target/thirdparty-lib/
+ cd /incubator-gluten/dev/../package/target/thirdparty-lib/
+ jar cvf gluten-thirdparty-lib-centos-9-x86_64.jar ./
added manifest
adding: libre2.so.9(in = 478136) (out= 209171)(deflated 56%)
adding: libdouble-conversion.so.3(in = 86680) (out= 38620)(deflated 55%)
adding: libevent-2.1.so.7(in = 366424) (out= 161644)(deflated 55%)
adding: libdwarf.so.0(in = 497200) (out= 197190)(deflated 60%)
adding: libgsasl.so.7(in = 139832) (out= 61127)(deflated 56%)
adding: libicudata.so.67(in = 28408488) (out= 11295428)(deflated 60%)
adding: libicui18n.so.67(in = 3201200) (out= 1278222)(deflated 60%)
adding: libicuuc.so.67(in = 2024088) (out= 847392)(deflated 58%)
adding: libidn.so.12(in = 213288) (out= 71597)(deflated 66%)
adding: libntlm.so.0(in = 28264) (out= 10484)(deflated 62%)
adding: libsodium.so.23(in = 362928) (out= 171315)(deflated 52%)
adding: libhdfs3.so.1(in = 40390056) (out= 13882117)(deflated 65%)
adding: libboost_context.so.1.84.0(in = 16848) (out= 2921)(deflated 82%)
adding: libboost_filesystem.so.1.84.0(in = 180784) (out= 64226)(deflated 64%)
adding: libboost_program_options.so.1.84.0(in = 534136) (out= 168539)(deflated 68%)
adding: libboost_regex.so.1.84.0(in = 397512) (out= 155365)(deflated 60%)
adding: libboost_system.so.1.84.0(in = 15648) (out= 1854)(deflated 88%)
adding: libboost_thread.so.1.84.0(in = 156392) (out= 48431)(deflated 69%)
adding: libboost_atomic.so.1.84.0(in = 18296) (out= 5742)(deflated 68%)
adding: libprotobuf.so.32(in = 10773312) (out= 2351675)(deflated 78%)
adding: libgflags.so.2.2(in = 424336) (out= 104170)(deflated 75%)
adding: libglog.so.1(in = 268064) (out= 80727)(deflated 69%)

deepashreeraghu · 2024-06-28T11:24:59Z

@zhztheplayer - Could you please check on this PR if there is some pending action from me ?

wanglinsong · 2024-07-02T06:10:07Z

centos 9 or centos stream 9?
https://www.redhat.com/en/blog/faq-centos-stream-updates#Q3

deepashreeraghu · 2024-07-02T06:37:16Z

@wanglinsong - Yes, I am using FROM quay.io/centos/centos:stream9 and building on top of it.

deepashreeraghu · 2024-07-11T17:28:55Z

@FelixYBW - Could you please help reviewing ?

majetideepak · 2024-07-15T13:19:18Z

@deepashreeraghu I am hitting the following warning which is being handled as an error. Are you not seeing this?

In file included from /root/incubator-gluten/ep/build-velox/build/velox_ep/./velox/vector/BaseVector.h:37,
                 from /root/incubator-gluten/ep/build-velox/build/velox_ep/./velox/vector/arrow/Bridge.h:20,
                 from /root/incubator-gluten/ep/build-velox/build/velox_ep/velox/vector/arrow/Bridge.cpp:17:
In member function ‘void facebook::velox::SelectivityVector::clearAll()’,
    inlined from ‘std::enable_if_t<is_same_v<U, facebook::velox::StringView>, void> facebook::velox::SimpleVector<T>::invalidateIsAscii() [with U = facebook::velox::StringView; T = facebook::velox::StringView]’ at /root/incubator-gluten/ep/build-velox/build/velox_ep/./velox/vector/SimpleVector.h:315:55,
    inlined from ‘void facebook::velox::SimpleVector<T>::resetDataDependentFlags(const facebook::velox::SelectivityVector*) [with T = facebook::velox::StringView]’ at /root/incubator-gluten/ep/build-velox/build/velox_ep/./velox/vector/SimpleVector.h:434:26:
/root/incubator-gluten/ep/build-velox/build/velox_ep/./velox/vector/SelectivityVector.h:163:18: error: writing 1 byte into a region of size 0 [-Werror=stringop-overflow=]
  163 |     allSelected_ = false;
      |     ~~~~~~~~~~~~~^~~~~~~

majetideepak · 2024-07-15T13:25:45Z

ep/build-velox/src/get_velox.sh

+  # make this function Reentrant
+  git checkout scripts/setup-centos9.sh
+  # need set BUILD_SHARED_LIBS flag for thrift
+  #sed -i "/cd fbthrift/{n;s/cmake_install -Denable_tests=OFF/cmake_install -Denable_tests=OFF -DBUILD_SHARED_LIBS=OFF/;}" scripts/setup-centos9.sh


remove commented command?

ep/build-velox/src/get_velox.sh

majetideepak · 2024-07-15T13:34:50Z

error: writing 1 byte into a region of size 0 [-Werror=stringop-overflow=]

@yma11 How was this fixed on the Gluten side?
facebookincubator/velox#9578 (comment)

majetideepak · 2024-07-15T13:37:44Z

Looks like we need this fix #5686 for Centos Stream 9 + GCC11 as well.

yma11 · 2024-07-16T01:27:40Z

error: writing 1 byte into a region of size 0 [-Werror=stringop-overflow=]

@yma11 How was this fixed on the Gluten side? facebookincubator/velox#9578 (comment)

It doesn't pop in most of our environments so we don't handle this formally at Gluten side yet. Haven't found a fix better than the workaround by adding flag -Wno-stringop-overflow.

majetideepak · 2024-07-16T01:45:42Z

@yma11 thanks. I am fixing this on the Velox side: https://github.com/facebookincubator/velox/pull/10469/files

majetideepak

I also verified that this PR works for Centos9 with one fix facebookincubator/velox#10469 on the Velox side.

majetideepak · 2024-07-16T18:29:29Z

I see that there are conflicts for this branch.

deepashreeraghu · 2024-07-17T17:26:23Z

@majetideepak - Thanks for reviewing. I have rebased. Now I am rebuilding with latest changes and will retest it to confirm

deepashreeraghu · 2024-07-18T16:38:17Z

@majetideepak - I am able to build with latest set of changes too. Now using the built Gluten jar for further testing.

deepashreeraghu · 2024-07-19T07:17:54Z

@majetideepak - I have tested this and basic tests are going through fine with latest build.

majetideepak · 2024-07-19T14:23:54Z

ep/build-velox/src/get_velox.sh

+    sed -i '/^  dnf_install ninja-build/a\  dnf_install yasm\' scripts/setup-centos9.sh
+  fi
+  if [[ $ENABLE_HDFS == "ON" ]]; then
+    sed -i '/cd protobuf/{n;s/\.\/configure --prefix=\/usr/\.\/configure CXXFLAGS="-fPIC" --prefix=\/usr\/local/;}' scripts/setup-centos9.sh


Based on process_setup_centos8, should this be above and not under if [[ $ENABLE_HDFS == "ON" ]];?

Updated it to be like centos8

majetideepak · 2024-07-19T15:05:30Z

@FelixYBW, @zhztheplayer Can you help review this PR? Thanks!

majetideepak · 2024-07-20T11:46:36Z

@yma11 can you help review this PR as well? Thanks!

yma11 · 2024-07-22T02:46:09Z

@deepashreeraghu can you add some code to ensure the sed happens or pops up errors? You can refer code. Thanks.

deepashreeraghu · 2024-07-22T11:57:53Z

@yma11 - Updated and verified it too. Please review.

majetideepak · 2024-07-23T14:10:57Z

@PHILO-HE are you able to review this and merge it so that we can refactor it along with #6559 ?

majetideepak · 2024-07-23T14:16:21Z

Thanks! @yma11

PHILO-HE · 2024-07-24T08:47:59Z

@deepashreeraghu, thanks so much for your work! We have a weekly build job to verify no build issue on some mainstream OS. Could you add centos-9 for tracking?
https://github.com/apache/incubator-gluten/blob/main/.github/workflows/velox_weekly.yml

deepashreeraghu · 2024-07-24T08:50:40Z

@PHILO-HE - Sure, will do . I will create a separate PR for that. Updating this branch will be a hassle. Hope it is ok ?

PHILO-HE · 2024-07-24T08:52:30Z

@PHILO-HE - Sure, will do . I will create a separate PR for that. Updating this branch will be a hassle. Hope it is ok ?

@deepashreeraghu, yes, a separate PR. Thanks!

deepashreeraghu marked this pull request as ready for review June 26, 2024 18:24

FelixYBW changed the title ~~Fix for centos9 build of Gluten~~ [VL] Fix for centos9 build of Gluten Jul 2, 2024

majetideepak reviewed Jul 15, 2024

View reviewed changes

ep/build-velox/src/get_velox.sh Outdated Show resolved Hide resolved

majetideepak approved these changes Jul 16, 2024

View reviewed changes

chetanbhatia force-pushed the centos9-build-support branch from c236d73 to dbc4b8c Compare July 17, 2024 17:00

deepashreeraghu force-pushed the centos9-build-support branch from dbc4b8c to 2d78033 Compare July 17, 2024 17:01

Merge conflicts resolution

e278d5d

deepashreeraghu force-pushed the centos9-build-support branch from 2d78033 to e278d5d Compare July 17, 2024 17:04

Deepa8 added 2 commits July 17, 2024 22:47

New build script updated

72dbf52

Handled review comments

4dbf0c4

removed BUILD_PROTOBUF from centos9

a3696e1

majetideepak reviewed Jul 19, 2024

View reviewed changes

Updated based on review comment

fd8139e

majetideepak approved these changes Jul 19, 2024

View reviewed changes

majetideepak mentioned this pull request Jul 20, 2024

[GLUTEN-6523][VL] fix: Remove fix for stringop-overflow warning in alinux3 #6522

Merged

deepashreeraghu and others added 2 commits July 22, 2024 15:46

Merge branch 'apache:main' into centos9-build-support

00cb1a8

Handling review comments

dfb146d

Typo during merge added back

5233786

yma11 approved these changes Jul 23, 2024

View reviewed changes

yma11 merged commit c4d9db2 into apache:main Jul 23, 2024
39 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[VL] Fix for centos9 build of Gluten #6183

[VL] Fix for centos9 build of Gluten #6183

deepashreeraghu commented Jun 22, 2024

github-actions bot commented Jun 22, 2024

github-actions bot commented Jun 26, 2024

deepashreeraghu commented Jun 27, 2024

deepashreeraghu commented Jun 28, 2024

wanglinsong commented Jul 2, 2024

deepashreeraghu commented Jul 2, 2024

deepashreeraghu commented Jul 11, 2024

majetideepak commented Jul 15, 2024

majetideepak Jul 15, 2024

deepashreeraghu Jul 17, 2024

majetideepak commented Jul 15, 2024

majetideepak commented Jul 15, 2024 •

edited

Loading

yma11 commented Jul 16, 2024

majetideepak commented Jul 16, 2024

majetideepak left a comment •

edited

Loading

majetideepak commented Jul 16, 2024

deepashreeraghu commented Jul 17, 2024 •

edited

Loading

deepashreeraghu commented Jul 18, 2024

deepashreeraghu commented Jul 19, 2024

majetideepak Jul 19, 2024

deepashreeraghu Jul 19, 2024

majetideepak commented Jul 19, 2024

majetideepak commented Jul 20, 2024

yma11 commented Jul 22, 2024

deepashreeraghu commented Jul 22, 2024

majetideepak commented Jul 23, 2024

majetideepak commented Jul 23, 2024

PHILO-HE commented Jul 24, 2024

deepashreeraghu commented Jul 24, 2024

PHILO-HE commented Jul 24, 2024

[VL] Fix for centos9 build of Gluten #6183

[VL] Fix for centos9 build of Gluten #6183

Conversation

deepashreeraghu commented Jun 22, 2024

What changes were proposed in this pull request?

How was this patch tested?

github-actions bot commented Jun 22, 2024

github-actions bot commented Jun 26, 2024

deepashreeraghu commented Jun 27, 2024

deepashreeraghu commented Jun 28, 2024

wanglinsong commented Jul 2, 2024

deepashreeraghu commented Jul 2, 2024

deepashreeraghu commented Jul 11, 2024

majetideepak commented Jul 15, 2024

majetideepak Jul 15, 2024

Choose a reason for hiding this comment

deepashreeraghu Jul 17, 2024

Choose a reason for hiding this comment

majetideepak commented Jul 15, 2024

majetideepak commented Jul 15, 2024 • edited Loading

yma11 commented Jul 16, 2024

majetideepak commented Jul 16, 2024

majetideepak left a comment • edited Loading

Choose a reason for hiding this comment

majetideepak commented Jul 16, 2024

deepashreeraghu commented Jul 17, 2024 • edited Loading

deepashreeraghu commented Jul 18, 2024

deepashreeraghu commented Jul 19, 2024

majetideepak Jul 19, 2024

Choose a reason for hiding this comment

deepashreeraghu Jul 19, 2024

Choose a reason for hiding this comment

majetideepak commented Jul 19, 2024

majetideepak commented Jul 20, 2024

yma11 commented Jul 22, 2024

deepashreeraghu commented Jul 22, 2024

majetideepak commented Jul 23, 2024

majetideepak commented Jul 23, 2024

PHILO-HE commented Jul 24, 2024

deepashreeraghu commented Jul 24, 2024

PHILO-HE commented Jul 24, 2024

majetideepak commented Jul 15, 2024 •

edited

Loading

majetideepak left a comment •

edited

Loading

deepashreeraghu commented Jul 17, 2024 •

edited

Loading