Assistance on building 32-bit wheels for PyTables

[amotl]

Dear @joerick, @YannickJadoul, @Czaki, @henryiii and all other contributors,

thank you so much for conceiving and maintaining this excellent tool. It saves so many lives!

I am currently working on PyTables/PyTables#823 on behalf of PyTables/PyTables#872 and want to humbly ask for your assistance. While there might be more things to discuss in this regard, I currently want to focus on one specific detail.

Attaching to #336 and #338, I see that you added the possibility to run jobs within Docker through linux32, which is also referenced at [1]. If I am getting this right, linux32 might be used to build 32-bit wheels for Linux even when running on a 64-bit platform / container image?

Now, I am asking about the correct usage. At https://github.com/joerick/cibuildwheel/blob/acd93000777eed59c0a9b4e3bf8986a50aeb56e6/cibuildwheel/linux.py#L120, the simulate_32_bit option is enabled conditionally when the name of the image's platform tag ends with i686. So, I figure this would not trigger when using a x64 build image for both variants like

CIBW_MANYLINUX_X86_64_IMAGE: dockcross/manylinux2014-x64
CIBW_MANYLINUX_I686_IMAGE: dockcross/manylinux2014-x64

right?

Maybe you can help in this regard how I could make cibuildwheel use this simulate_32_bit option when being configured with the same 64-bit image for running both manylinux targets (32-bit and 64-bit) like outlined above. Maybe I am getting this completely wrong and it would in fact never work. If this is the case, I will also be happy about any enlightenment.

With kind regards,
Andreas.

[1] https://github.com/pypa/manylinux#example

[amotl]

Oh, I can see it works already [1]. The proof is that running linux32 yum ... like

docker run -it --rm dockcross/manylinux2014-x64 bash
linux32 yum install -y blosc-devel bzip2-devel hdf5-devel lz4-devel lzo-devel snappy-devel libzstd-devel zlib-devel

will unfortunately (but probably obviously) croak with

Cannot find a valid baseurl for repo: centos-sclo-rh/i386

Bummer!

[1] https://github.com/daq-tools/PyTables/runs/1798360495?check_suite_focus=true#step:7:565

[amotl]

So, I just tried to run a recipe I was imagining without using cibuildwheel yet, like

# Use a modern variant of `manylinux`
docker run -it --rm dockcross/manylinux2014-x64 bash

# Install prerequisites
yum install -y python3 python3-devel python3-numpy python3-numexpr
yum install -y blosc-devel bzip2-devel hdf5-devel lz4-devel lzo-devel snappy-devel libzstd-devel zlib-devel
pip3 install numpy numexpr cython wheel

# Get sources
git clone https://github.com/PyTables/PyTables
cd PyTables

# Configure build
export PYTABLES_NO_EMBEDDED_LIBS=true
export DISABLE_AVX2=true

# Build 64-bit wheel
python3 setup.py bdist_wheel

# Build 32-bit wheel
linux32 python3 setup.py bdist_wheel

and the outcome was

ls -l dist
tables-3.6.2.dev0-cp36-cp36m-linux_i686.whl
tables-3.6.2.dev0-cp36-cp36m-linux_x86_64.whl

Do you believe it is sane what I am attempting here in order to create both 32-bit and 64-bit wheels for Linux? If so, do you believe we could make this recipe work with cibuildwheel in any way?

[YannickJadoul]

the simulate_32_bit option is enabled conditionally when the name of the image's platform tag ends with i686.

@amotl, no, that's not true. platform_tag is the last part of the the build identifier. It's completely unrelated to the name of the docker image. All build identifier ending with manylinux_i686 will be built using CIBW_MANYLINUX_I686_IMAGE and with linux32 prepended.

[YannickJadoul]

Is there a reason you cannot use CIBW_MANYLINUX_I686_IMAGE: dockcross/manylinux2014-x86?

[amotl]

Hi Yannick,

thanks for looking into this. I am already past my initial bummer:
Oh, I can see it works already [1].

Is there a reason you cannot use CIBW_MANYLINUX_I686_IMAGE: dockcross/manylinux2014-x86?

Oh, I didn't try. Apologies! Somehow I did misinterpret something that there would be no images for x86 and overlooked that somehow. Thanks!

I will try that and report back to you how it goes along building wheels for PyTables.

With kind regards,
Andreas.

[YannickJadoul]

thanks for looking into this. I am already past my initial bummer:
Oh, I can see it works already [1].

Right, sorry! Just wanted to clarify :-)

If not, I can dive deeper. But normally, using 32-bit images for 32-bit builds is the most straightforward. Cross-compilation is not really supported (yet) by cibuildwheel.

If things don't work out, let me know, and I'll have a deeper dive than these two superficial messages!

[amotl]

Hi Yannick,

trying dockcross/manylinux2014-x86 croaks on another end, see dockcross/dockcross#467. The issue has been seen before on the 64-bit images, see dockcross/dockcross#363.

Just to keep you in the loop: Of course I tried the regular manylinux1, manylinux2010, and manylinux2014 images beforehand, but only manylinux2014 (64-bit) was able to install those prerequisites:

yum install -y blosc-devel bzip2-devel hdf5-devel lz4-devel lzo-devel snappy-devel libzstd-devel zlib-devel

If you can have a deeper look, I would appreciate that very much.

With kind regards,
Andreas.

[Czaki]

@amotl You may look to my repository for building imagecodces wheels https://github.com/Czaki/imagecodecs_build.

I just use build my own docker images https://github.com/Czaki/imagecodecs_build/blob/master/build_utils/Dockerfile_i686 and build part of libraries present for x86_64 and missed for i686.

I do not know which ones are missed. Maybe they are present in https://hub.docker.com/r/bokota/imagecodecs_i686, but this image is not updated regularly.

[YannickJadoul]

Hah, that's annoying. But you will need blosc-devel for i686 anyway, right, if you're building i686 wheels?

[henryiii]

So you want to do the yum install in 64 bit mode, but the build in 32 bit? I'm guessing blosc would then have to only be used for some stage of the build but not included in any way in the wheel itself? I'm sure things like the hdf5 libraries would need to match the build arch.

[amotl]

Hi again,

thanks for your answers. I understand that the idea of building 32-bit wheels on 64-bit Linux is utter Quatsch. Apologies!

Now, I am still struggling to find appropriate 32-bit images to get hold of the build prerequisites. Thank you for suggesting bokota/imagecodecs_i686, @Czaki. However, yum is also broken there:

docker run -it --rm bokota/imagecodecs_i686 bash

yum install -y blosc-devel bzip2-devel hdf5-devel lz4-devel lzo-devel snappy-devel libzstd-devel zlib-devel

Loaded plugins: fastestmirror, ovl
Setting up Install Process
Determining fastest mirrors
YumRepo Error: All mirror URLs are not using ftp, http[s] or file.
 Eg. Invalid release/repo/arch combination/
removing mirrorlist with no valid mirrors: /var/cache/yum/i386/6/base/mirrorlist.txt
Error: Cannot find a valid baseurl for repo: base

With kind regards,
Andreas.

[amotl]

Even after fixing dockcross/manylinux2014-x86 like

docker run -it --rm dockcross/manylinux2014-x86 bash
echo $'#!/bin/bash\nLD_PRELOAD=/usr/lib/libcurl.so.4 /usr/bin/yum "$@"' > /usr/local/bin/yum

and manually installing EPEL like

yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

there is no joy because EPEL apparently stopped being available for 32-bit starting with CentOS version 7, see [1,2].

yum install -y blosc-devel bzip2-devel hdf5-devel lz4-devel lzo-devel snappy-devel libzstd-devel zlib-devel

Cannot retrieve metalink for repository: epel/i386. Please verify its path and try again

[1] https://archives.fedoraproject.org/pub/archive/epel/6/i386/
[2] https://archives.fedoraproject.org/pub/archive/epel/7/

[Czaki]

This images contains compiled from source these libraries:
zlib 1.2.11
lz4 1.9.3
zstd 1.4.8
blosc 1.21.0
bzip2 1.0.8
liblzma 5.2.5
liblzf 3.6
libpng 1.6.37
libwebp 1.1.0
libtiff 4.2.0
libjpeg-turbo 2.0.6 (8 and 12-bit)
libjpeg 9d
charls 2.2.0
openjpeg 2.4.0
jxrlib 1.1
zfp 0.5.5
bitshuffle 0.3.5
libaec 1.0.4
snappy 1.1.8
zopfli-1.0.3
brotli 1.0.9
brunsli 0.1
giflib 5.2.1
lerc 2.2.1
libdeflate 1.7
libavif 0.8.4
dav1d 0.8.1
rav1e 0.3.5
aom 2.0.1
lcms 2.11

Maybe it covers these libraries which are missed from repositories. Otherwise, I suggest basing on my work and prepare own Dockerfile with libraries compiled from the source.

[amotl]

Dear Grzegorz,

this looks really nice, thanks. Apparently, only hdf5 and lzo are missing from this list. If yum would work within both bokota/imagecodecs_64 and bokota/imagecodecs_i686, it would be easy to add both libraries at runtime.

With kind regards,
Andreas.

[amotl]

Hi again,

I took a quick look and found that even more libraries are installed. Let me write down a short summary about what I was able to confirm.

• find / -iname '*hdf5*' shows that libhdf5 is installed within both image variants.
• find / -iname '*lzo*' shows that liblzo is installed within bokota/imagecodecs_64, but not within bokota/imagecodecs_i686.
• find / -iname '*blosc*' shows that libblosc is not installed into standard system locations.

Regarding yum croaking, I have been able to fix it ad hoc by adjusting /etc/yum.repos.d/CentOS-Base.repo like [1]

baseurl=https://vault.centos.org/6.10/os/$basearch/
baseurl=https://vault.centos.org/6.10/updates/$basearch/
baseurl=https://vault.centos.org/6.10/extras/$basearch/
baseurl=https://vault.centos.org/6.10/centosplus/$basearch/
baseurl=https://vault.centos.org/6.10/contrib/$basearch/

Would you accept respective patches to imagecodecs_build [2], @Czaki?

With kind regards,
Andreas.

[1] https://forums.cpanel.net/threads/yumrepo-error-and-cannot-find-valid-baseurl.682465/
[2] https://github.com/Czaki/imagecodecs_build

[Czaki]

Create PR. i will rebuild docker images these evening.

[YannickJadoul]

Regarding yum croaking, I have been able to fix it ad hoc by adjusting /etc/yum.repos.d/CentOS-Base.repo like [1]

Shouldn't you be able to add them to a manylinux image as well, and get things going this way?

[amotl]

Hi Yannick,

I would prefer using vanilla manylinux, but the 32-bit variants don't offer packages for blosc-devel and hdf5-devel, see [1,2]. I've summarized my findings so far at [3]. There might be well other methods to bring in those prerequisites like manually installing them. However, I still try to follow the path of "installation using official packages (or upstream images)".

That's why I appreciated the idea by @Czaki because, apparently, many of those prerequisites are already satisfied by the imagecodecs_build images he is providing on behalf of @cgohlke's imagecodecs.

On the other hand, I also added a fix to make yum on dockcross/manylinux2014-x86 work again, see dockcross/dockcross#468.

With kind regards,
Andreas.

[1] https://pkgs.org/search/?q=blosc-devel
[2] https://pkgs.org/search/?q=hdf5-devel
[3] https://github.com/daq-tools/PyTables/blob/88017e14dceb2443159db688b8253f2a79ff8617/.github/workflows/wheels.yml#L86-L100

[YannickJadoul]

@amotl, right, sorry. I definitely don't know too much about this either, but I saw the https://vault.centos.org/6.10/.../$basearch/ urls you posed before, so I though this would maybe help :-) Thanks for the clear summary!

[YannickJadoul]

[1] https://pkgs.org/search/?q=blosc-devel
[2] https://pkgs.org/search/?q=hdf5-devel

Hmmm, but there is libblosc-devel-1.18.1-alt1.i586.rpm and libhdf5-devel-1.8.13-alt1.qa4.i586.rpm. Aren't those also compatible with i686?

[amotl]

Dear Yannick,

I tried a few of those packages from non-CentOS distributions by attempting to install them directly (using the HTTP URL) without using any package repository flavor of CentOS at all. The outcome is usually that the process croaks on dependency hell aspects like those:

yum install -y http://ftp.altlinux.org/pub/distributions/ALTLinux/Sisyphus/i586/RPMS.classic/libblosc-devel-1.18.1-alt1.i586.rpm

Error: Package: libblosc-devel-1.18.1-alt1.i586 (/libblosc-devel-1.18.1-alt1.i586)
           Requires: libblosc = -alt1:sisyphus+250012.100.1.1

yum install -y http://ftp.altlinux.org/pub/distributions/ALTLinux/Sisyphus/i586/RPMS.classic/libblosc-1.18.1-alt1.i586.rpm

Error: Package: libblosc-1.18.1-alt1.i586 (/libblosc-1.18.1-alt1.i586)
           Requires: libzstd.so.1 >= liEXtCElKAXDD2

In fact, all of libblosc-devel, libhdf5-devel and libzstd-devel are not available through vanilla manylinux 32-bit images.

Don't even get me started on libhdf5-devel, which, for example when following [1], you will see that it needs six(!) other packages. Trying to install them discretely will in turn croak mostly on wrong libc6 dependency requirements. I don't claim that I iterated through all permutations, but I tried many.

With kind regards,
Andreas.

[1] Springdale Computational i386: https://centos.pkgs.org/6/springdale-computational-i386/hdf5-devel-1.8.5.patch1-7.sdl6.i686.rpm.html

[amotl]

In fact, all of libblosc-devel, libhdf5-devel and libzstd-devel are not available through vanilla manylinux 32-bit images.

I have to correct this statement. On manylinux2010_i686, both libhdf5-devel and libzstd-devel are available but not libblosc-devel.

docker run -it --rm quay.io/pypa/manylinux2010_i686 bash

cat /etc/centos-release
CentOS release 6.10 (Final)

yum install -y blosc-devel bzip2-devel hdf5-devel lz4-devel lzo-devel snappy-devel libzstd-devel zlib-devel

No package blosc-devel available.

However, trying to add those libblosc packages on top yields:

yum install -y http://ftp.altlinux.org/pub/distributions/ALTLinux/Sisyphus/i586/RPMS.classic/libblosc-devel-1.18.1-alt1.i586.rpm

Error: Package: libblosc-devel-1.18.1-alt1.i586 (/libblosc-devel-1.18.1-alt1.i586)
           Requires: libblosc = -alt1:sisyphus+250012.100.1.1

blocking on:

yum install -y http://ftp.altlinux.org/pub/distributions/ALTLinux/Sisyphus/i586/RPMS.classic/libblosc-1.18.1-alt1.i586.rpm

ERROR You need to update rpm to handle:
rpmlib(SetVersions) is needed by libblosc-1.18.1-alt1.i586

[amotl]

By following the suggestion to use rpm --nodeps in order to ignore rpmlib(SetVersions) [1], I have been able to distill a recipe based on the libblosc 32-bit packages from "ALT Linux Sisyphus" as you recommended at #568 (comment), @YannickJadoul. It actually seems to work.

# Use classic `manylinux2010_i686`
docker run -it --rm quay.io/pypa/manylinux2010_i686 bash

# Install baseline prerequisites
yum install -y bzip2-devel hdf5-devel lz4-devel lzo-devel snappy-devel libzstd-devel zlib-devel

# Install libblosc from "ALT Linux Sisyphus"
yum install -y wget
wget http://ftp.altlinux.org/pub/distributions/ALTLinux/Sisyphus/i586/RPMS.classic/libblosc-1.18.1-alt1.i586.rpm
wget http://ftp.altlinux.org/pub/distributions/ALTLinux/Sisyphus/i586/RPMS.classic/libblosc-devel-1.18.1-alt1.i586.rpm
rpm --install --nodeps libblosc-1.18.1-alt1.i586.rpm libblosc-devel-1.18.1-alt1.i586.rpm

# Build 32-bit wheel
export PATH=/opt/python/cp39-cp39/bin:$PATH
pip install numpy numexpr cython wheel
git clone https://github.com/PyTables/PyTables
cd PyTables
export PYTABLES_NO_EMBEDDED_LIBS=true
export DISABLE_AVX2=true
python setup.py bdist_wheel

# Outcome
ls dist/
tables-3.6.2.dev0-cp39-cp39-linux_i686.whl

After installing it, it shows

>>> tables.print_versions()
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
PyTables version:    3.6.2.dev0
HDF5 version:        1.8.5-patch1
NumPy version:       1.20.0
Numexpr version:     2.7.2 (not using Intel's VML/MKL)
Zlib version:        1.2.3 (in Python interpreter)
LZO version:         2.03 (Apr 30 2008)
BZIP2 version:       1.0.5 (10-Dec-2007)
Blosc version:       1.18.1 (2020-03-29)
Blosc compressors:   blosclz (2.0.0), lz4 (1.9.2), lz4hc (1.9.2), snappy (unknown), zlib (1.2.11), zstd (1.4.4)
Blosc filters:       shuffle, bitshuffle
Cython version:      0.29.21
Python version:      3.9.1 (default, Jan 24 2021, 15:58:04)
[GCC 8.3.1 20190311 (Red Hat 8.3.1-3)]
Platform:            Linux-4.19.76-linuxkit-i686-with-glibc2.12
Byte-ordering:       little
Detected cores:      4
Default encoding:    utf-8
Default FS encoding: utf-8
Default locale:      (en_US, UTF-8)
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

[1] https://serverfault.com/questions/381018/tell-yum-to-ignore-a-single-dependency

[YannickJadoul]

Nice work, @amotl, congrats!

[henryiii]

Nice piece of work!

[amotl]

Hi @Czaki,

as you can see, I have been able to base the recipe upon the baseline quay.io/pypa/manylinux2010_i686 and implement it like [1]. Nevertheless, thanks to all of you for your support and suggestions!

However, I would like to eventually come back to submitting a patch to imagecodecs_build in order to improve some details as outlined within #568 (comment).

Maybe I should carry those notes over to another issue on the imagecodecs_build repository in order to be able to close this one?

With kind regards,
Andreas.

[1] https://github.com/PyTables/PyTables/blob/1acef23a79f54c404549c147cfc9e403f3f8e195/.github/workflows/wheels.yml#L86-L113

[Czaki]

However, I would like to eventually come back to submitting a patch to imagecodecs_build in order to improve some details as outlined within #568 (comment).

Maybe I should carry those notes over to another issue on the imagecodecs_build repository in order to be able to close this one?

do it

[amotl]

Thanks, I've just added Czaki/imagecodecs_build#1. So, I am closing this now and again - thanks for all the fish!

[YannickJadoul]

Thanks for the patience and explanations, @amotl!