Cannot build Gluten in Ubuntu 20.04 container

[obedmr]

Problem description

I'm trying to compile gluten in a new Ubuntu 20.04 container. Below the steps:

1. Start the Ubuntu 20.04 container

docker run --rm --name gluten-build-20.04 -it -v $(pwd):/mnt ubuntu:20.04 bash

2. Install dependencies (running inside the container)

export DEBIAN_FRONTEND=noninteractive

apt update -y

apt install -y openjdk-8-jdk

apt install -y sudo locales wget tar tzdata git ccache cmake \
    ninja-build build-essential llvm-11-dev clang-11 vim pip sysstat gcc \
    git curl zip unzip tar autoconf bison flex autoconf-archive \
    maven pkg-config libtool libre2-dev

3. Compile (running inside the container)

cd /mnt
git clone https://github.com/apache/incubator-gluten.git gluten

pushd gluten
time ./dev/buildbundle-veloxbe.sh --enable_hdfs=ON --enable_gcs=ON --spark_version=3.3 --enable_vcpkg=ON
popd

There are dozens of errors that will appear, I pasted the last cmake log entries below.

System information

Velox System Info v0.0.2
Commit: 86449d0c87437002d6d2636e079420cacb258d82
CMake Version: 3.16.3
System: Linux-5.15.0-86-generic
Arch: x86_64
CPU Name: Model name:                         Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz
C++ Compiler: /usr/bin/c++
C++ Compiler Version: 9.4.0
C Compiler: /usr/bin/cc
C Compiler Version: 9.4.0
CMake Prefix Path: /usr/local;/usr;/;/usr;/usr/local;/usr/X11R6;/usr/pkg;/opt

CMake log

[131/980] Building CXX object _deps/absl-build/absl/strings/CMakeFiles/cord.dir/cord.cc.o
FAILED: _deps/absl-build/absl/strings/CMakeFiles/cord.dir/cord.cc.o
/usr/bin/ccache /usr/bin/c++ -DFOLLY_HAVE_INT128_T=1 -DVELOX_ENABLE_HDFS3 -DVELOX_ENABLE_PARQUET -I/mnt/gluten/dev/vcpkg/vcpkg_installed/x64-linux-avx/includ
e -isystem /mnt/gluten/ep/build-velox/build/velox_ep/_build/release/_deps/absl-src -Wno-missing-field-initializers -march=native -std=c++17 -mno-avx512f -mbm
i2 -D USE_VELOX_COMMON_BASE -D HAS_UNCAUGHT_EXCEPTIONS -Wall -Wextra -Wno-unused        -Wno-unused-parameter        -Wno-sign-compare        -Wno-ignored-qu
alifiers        -Wnon-virtual-dtor        -Wno-implicit-fallthrough          -Wno-class-memaccess          -Wno-comment          -Wno-int-in-bool-context
      -Wno-redundant-move          -Wno-array-bounds          -Wno-maybe-uninitialized          -Wno-unused-result          -Wno-format-overflow          -Wn
o-strict-aliasing -O3 -DNDEBUG -std=gnu++17 -fPIC -fdiagnostics-color=always -Wall -Wextra -Wcast-qual -Wconversion-null -Wformat-security -Wmissing-declarat
ions -Woverlength-strings -Wpointer-arith -Wundef -Wunused-local-typedefs -Wunused-result -Wvarargs -Wvla -Wwrite-strings -DNOMINMAX -MD -MT _deps/absl-build
/absl/strings/CMakeFiles/cord.dir/cord.cc.o -MF _deps/absl-build/absl/strings/CMakeFiles/cord.dir/cord.cc.o.d -o _deps/absl-build/absl/strings/CMakeFiles/cor
d.dir/cord.cc.o -c /mnt/gluten/ep/build-velox/build/velox_ep/_build/release/_deps/absl-src/absl/strings/cord.cc
/mnt/gluten/ep/build-velox/build/velox_ep/_build/release/_deps/absl-src/absl/strings/cord.cc:1219:26: error: no declaration matches 'absl::lts_20230125::Cord
::CharIterator absl::lts_20230125::Cord::FindImpl(absl::lts_20230125::Cord::CharIterator, absl::lts_20230125::string_view) const'
 1219 | absl::Cord::CharIterator absl::Cord::FindImpl(CharIterator it,
      |                          ^~~~
/mnt/gluten/ep/build-velox/build/velox_ep/_build/release/_deps/absl-src/absl/strings/cord.cc:1219:26: note: no functions named 'absl::lts_20230125::Cord::Cha
rIterator absl::lts_20230125::Cord::FindImpl(absl::lts_20230125::Cord::CharIterator, absl::lts_20230125::string_view) const'
In file included from /mnt/gluten/ep/build-velox/build/velox_ep/_build/release/_deps/absl-src/absl/strings/cord.cc:15:
/mnt/gluten/dev/vcpkg/vcpkg_installed/x64-linux-avx/include/absl/strings/cord.h:150:7: note: 'class absl::lts_20230125::Cord' defined here
  150 | class Cord {
      |       ^~~~
/mnt/gluten/ep/build-velox/build/velox_ep/_build/release/_deps/absl-src/absl/strings/cord.cc:1261:26: error: no declaration matches 'absl::lts_20230125::Cord
::CharIterator absl::lts_20230125::Cord::Find(absl::lts_20230125::string_view) const'
 1261 | absl::Cord::CharIterator absl::Cord::Find(absl::string_view needle) const {
      |                          ^~~~
/mnt/gluten/ep/build-velox/build/velox_ep/_build/release/_deps/absl-src/absl/strings/cord.cc:1261:26: note: no functions named 'absl::lts_20230125::Cord::Cha
rIterator absl::lts_20230125::Cord::Find(absl::lts_20230125::string_view) const'
In file included from /mnt/gluten/ep/build-velox/build/velox_ep/_build/release/_deps/absl-src/absl/strings/cord.cc:15:
/mnt/gluten/dev/vcpkg/vcpkg_installed/x64-linux-avx/include/absl/strings/cord.h:150:7: note: 'class absl::lts_20230125::Cord' defined here
  150 | class Cord {
      |       ^~~~
/mnt/gluten/ep/build-velox/build/velox_ep/_build/release/_deps/absl-src/absl/strings/cord.cc:1311:26: error: no declaration matches 'absl::lts_20230125::Cord
::CharIterator absl::lts_20230125::Cord::Find(const absl::lts_20230125::Cord&) const'
 1311 | absl::Cord::CharIterator absl::Cord::Find(const absl::Cord& needle) const {
      |                          ^~~~
/mnt/gluten/ep/build-velox/build/velox_ep/_build/release/_deps/absl-src/absl/strings/cord.cc:1311:26: note: no functions named 'absl::lts_20230125::Cord::Cha
rIterator absl::lts_20230125::Cord::Find(const absl::lts_20230125::Cord&) const'
In file included from /mnt/gluten/ep/build-velox/build/velox_ep/_build/release/_deps/absl-src/absl/strings/cord.cc:15:
/mnt/gluten/dev/vcpkg/vcpkg_installed/x64-linux-avx/include/absl/strings/cord.h:150:7: note: 'class absl::lts_20230125::Cord' defined here
  150 | class Cord {
      |       ^~~~
/mnt/gluten/ep/build-velox/build/velox_ep/_build/release/_deps/absl-src/absl/strings/cord.cc:1355:6: error: no declaration matches 'bool absl::lts_20230125::
Cord::Contains(absl::lts_20230125::string_view) const'
 1355 | bool Cord::Contains(absl::string_view rhs) const {
      |      ^~~~
/mnt/gluten/ep/build-velox/build/velox_ep/_build/release/_deps/absl-src/absl/strings/cord.cc:1355:6: note: no functions named 'bool absl::lts_20230125::Cord:
:Contains(absl::lts_20230125::string_view) const'
In file included from /mnt/gluten/ep/build-velox/build/velox_ep/_build/release/_deps/absl-src/absl/strings/cord.cc:15:
/mnt/gluten/dev/vcpkg/vcpkg_installed/x64-linux-avx/include/absl/strings/cord.h:150:7: note: 'class absl::lts_20230125::Cord' defined here
  150 | class Cord {
      |       ^~~~
/mnt/gluten/ep/build-velox/build/velox_ep/_build/release/_deps/absl-src/absl/strings/cord.cc:1359:6: error: no declaration matches 'bool absl::lts_20230125::
Cord::Contains(const absl::lts_20230125::Cord&) const'
 1359 | bool Cord::Contains(const absl::Cord& rhs) const {
      |      ^~~~
/mnt/gluten/ep/build-velox/build/velox_ep/_build/release/_deps/absl-src/absl/strings/cord.cc:1359:6: note: no functions named 'bool absl::lts_20230125::Cord:
:Contains(const absl::lts_20230125::Cord&) const'
In file included from /mnt/gluten/ep/build-velox/build/velox_ep/_build/release/_deps/absl-src/absl/strings/cord.cc:15:
/mnt/gluten/dev/vcpkg/vcpkg_installed/x64-linux-avx/include/absl/strings/cord.h:150:7: note: 'class absl::lts_20230125::Cord' defined here
  150 | class Cord {
      |       ^~~~
[136/980] Performing build step for 'libstemmer'
make[2]: Entering directory '/mnt/gluten/ep/build-velox/build/velox_ep/_build/release/_deps/libstemmer/src/libstemmer'
cc -O2 -Iinclude -fPIC  -c -o src_c/stem_UTF_8_arabic.o src_c/stem_UTF_8_arabic.c
cc -O2 -Iinclude -fPIC  -c -o src_c/stem_UTF_8_armenian.o src_c/stem_UTF_8_armenian.c
.
.
.
.
.
.
cc -O2 -Iinclude -fPIC  -c -o libstemmer/libstemmer.o libstemmer/libstemmer.c
ar -cru libstemmer.a src_c/stem_UTF_8_arabic.o src_c/stem_UTF_8_armenian.o src_c/stem_ISO_8859_1_basque.o src_c/stem_UTF_8_basque.o src_c/stem_ISO_8859_1_catalan.o src_c/stem_UTF_8_catalan.o src_c/stem_ISO_8859_1_danish.o src_c/stem_UTF_8_danish.o src_c/stem_ISO_8859_1_dutch.o src_c/stem_UTF_8_dutch.o src_c/stem_ISO_8859_1_english.o src_c/stem_UTF_8_english.o src_c/stem_ISO_8859_1_finnish.o src_c/stem_UTF_8_finnish.o src_c/stem_ISO_8859_1_french.o src_c/stem_UTF_8_french.o src_c/stem_ISO_8859_1_german.o src_c/stem_UTF_8_german.o src_c/stem_UTF_8_greek.o src_c/stem_UTF_8_hindi.o src_c/stem_ISO_8859_2_hungarian.o src_c/stem_UTF_8_hungarian.o src_c/stem_ISO_8859_1_indonesian.o src_c/stem_UTF_8_indonesian.o src_c/stem_ISO_8859_1_irish.o src_c/stem_UTF_8_irish.o src_c/stem_ISO_8859_1_italian.o src_c/stem_UTF_8_italian.o src_c/stem_UTF_8_lithuanian.o src_c/stem_UTF_8_nepali.o src_c/stem_ISO_8859_1_norwegian.o src_c/stem_UTF_8_norwegian.o src_c/stem_ISO_8859_1_porter.o src_c/stem_UTF_8_porter.o src_c/stem_ISO_8859_1_portuguese.o src_c/stem_UTF_8_portuguese.o src_c/stem_ISO_8859_2_romanian.o src_c/stem_UTF_8_romanian.o src_c/stem_KOI8_R_russian.o src_c/stem_UTF_8_russian.o src_c/stem_UTF_8_serbian.o src_c/stem_ISO_8859_1_spanish.o src_c/stem_UTF_8_spanish.o src_c/stem_ISO_8859_1_swedish.o src_c/stem_UTF_8_swedish.o src_c/stem_UTF_8_tamil.o src_c/stem_UTF_8_turkish.o src_c/stem_UTF_8_yiddish.o runtime/api.o runtime/utilities.o libstemmer/libstemmer.o
ar: `u' modifier ignored since `D' is the default (see `U')
cc -O2 -Iinclude -fPIC  -c -o examples/stemwords.o examples/stemwords.c
cc -O2 -o stemwords examples/stemwords.o libstemmer.a
make[2]: Leaving directory '/mnt/gluten/ep/build-velox/build/velox_ep/_build/release/_deps/libstemmer/src/libstemmer'
[140/980] Building CXX object velox/serializers/CMakeFiles/velox_presto_serializer.dir/PrestoSerializer.cpp.o
ninja: build stopped: subcommand failed.
make[1]: *** [Makefile:103: build] Error 1
make[1]: Leaving directory '/mnt/gluten/ep/build-velox/build/velox_ep'
make: *** [Makefile:110: release] Error 2

[PHILO-HE]

@obedmr, seems similar to this issue: #6003 when gcs build is enabled. I will fix it. Before that, you may try with --enable_vcpkg=OFF if you don't insist on building with vcpkg.

[PHILO-HE]

@obedmr, could you try the latest main branch?

[zhztheplayer]

Besides, as a possible temporary solution, would you please try with this method to create a build in docker container? It should be a more general way for building.

[obedmr]

@obedmr, could you try the latest main branch?

Hi @PHILO-HE, I've been a bit busy on last days, hope to get a chance this week, will let you know. Thanks for your help.

[FelixYBW]

Let's close this since docker script is changed a lots