[WIP] dissociate python optional test wheels builds + python313 #6282

th0ma7 · 2024-10-14T20:01:25Z

Description

Intent is to:

Remove wheel build testing from python310/python311 base Makefiles to ease updates
Add python313 package (potentially also add python312)
Dissociate crossenv creation from python3*/Makefile -> Now uses spksrc.crossenv.mk

$ ls -la ~/spksrc/mk/crossenv/requirements-default.txt
$ make crossenvclean
$ make crossenc-<arch>-<version>

Allow wheel specific crossenv enablement using a mk/crossenv/ directory containing wheel crossenv definitions:

$ ls -la ~/spksrc/mk/crossenv/requirements-aiohttp-3.8.5.txt
$ make crossenvclean
$ WHEEL="aiohttp-3.8.5" make crossenv-x64-7.1

Checklist

Build rule all-supported completed successfully
New installation of package completed successfully
Package upgrade completed successfully (Manually install the package again)
Package functionality was tested
Any needed documentation is updated/created

Type of change

TODO

Migrate spksrc.wheel.mk to use $(which cross-pip)
Remove / fix / validate usage of pip cache through $HOME/.cache/pip

2024-11-14T02:37:23.1044954Z WARNING: The directory '/github/home/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.

Fix handling of requirements.txt entry with trailing comment test==1234 # This is a test wheel
Have crossenv wheels use #$(WORK_DIR) caching directory instead of default $(HOME)/.cache/pip
Review OPENSSL_*_DIR variables and logic usage throughout python related mk files
Use status cookie for crossenv creation
Use status cookie for wheel building
Have crossenv add status info to status-build.log file
Have spksrc.python-wheel.mk to use status cookie to avoid always rebuilding
Avoid zlib creation + remove of symlinks from spksrc.python.mk to eliminate rebuilding

cross/python313/Makefile

th0ma7 · 2024-10-14T23:34:58Z

@hgy59 I'll let this run but this patch in particular 9c764a7 needs to be in its own PR... interesting from not having noticed it earlier. I'll also mark as [WIP] as not even close to be merged, this was early work being push to initiate exchange with python github relatively to the build issue I had.

th0ma7 · 2024-10-15T11:18:43Z

@hgy59 bind compiling tries to install packages ?!?!? I'll try to find where that comes from but in case it ring any bell...

===>  Patching for bind
===>  Configuring for bind
===>  - Configure ARGS: --host=x86_64-pc-linux-gnu --build=i686-pc-linux --prefix=/usr/local/bind --localstatedir=/usr/local/bind/var --enable-full-report BUILD_CC=cc BUILD_CFLAGS=-I/home/spksrc/python-wheels/spksrc/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/x86_64-pc-linux-gnu/sys-root/usr/include -I/home/spksrc/python-wheels/spksrc/cross/bind/work-x64-7.1/install/usr/local/bind/include -O2 -O2 BUILD_CPPFLAGS=-I/home/spksrc/python-wheels/spksrc/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/x86_64-pc-linux-gnu/sys-root/usr/include -I/home/spksrc/python-wheels/spksrc/cross/bind/work-x64-7.1/install/usr/local/bind/include -O2 -O2 BUILD_LDFLAGS=-L/home/spksrc/python-wheels/spksrc/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/x86_64-pc-linux-gnu/sys-root/lib -L/home/spksrc/python-wheels/spksrc/cross/bind/work-x64-7.1/install/usr/local/bind/lib -Wl,--rpath-link,/home/spksrc/python-wheels/spksrc/cross/bind/work-x64-7.1/install/usr/local/bind/lib -Wl,--rpath,/usr/local/bind/lib  BUILD_LIBS=
===>  - Install prefix: /usr/local/bind
===>  Install libtool binary
sudo apt-get update
[sudo] password for spksrc:

hgy59 · 2024-10-15T16:11:27Z

@hgy59 bind compiling tries to install packages ?!?!? I'll try to find where that comes from but in case it ring any bell...

===>  Patching for bind
===>  Configuring for bind
===>  - Configure ARGS: --host=x86_64-pc-linux-gnu --build=i686-pc-linux --prefix=/usr/local/bind --localstatedir=/usr/local/bind/var --enable-full-report BUILD_CC=cc BUILD_CFLAGS=-I/home/spksrc/python-wheels/spksrc/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/x86_64-pc-linux-gnu/sys-root/usr/include -I/home/spksrc/python-wheels/spksrc/cross/bind/work-x64-7.1/install/usr/local/bind/include -O2 -O2 BUILD_CPPFLAGS=-I/home/spksrc/python-wheels/spksrc/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/x86_64-pc-linux-gnu/sys-root/usr/include -I/home/spksrc/python-wheels/spksrc/cross/bind/work-x64-7.1/install/usr/local/bind/include -O2 -O2 BUILD_LDFLAGS=-L/home/spksrc/python-wheels/spksrc/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/x86_64-pc-linux-gnu/sys-root/lib -L/home/spksrc/python-wheels/spksrc/cross/bind/work-x64-7.1/install/usr/local/bind/lib -Wl,--rpath-link,/home/spksrc/python-wheels/spksrc/cross/bind/work-x64-7.1/install/usr/local/bind/lib -Wl,--rpath,/usr/local/bind/lib  BUILD_LIBS=
===>  - Install prefix: /usr/local/bind
===>  Install libtool binary
sudo apt-get update
[sudo] password for spksrc:

@th0ma7 I will remove this in #6269. It is obsolete since spksrc image on debian 12.

th0ma7 · 2024-10-16T00:11:00Z

@hgy59, @mreid-tt and other @SynoCommunity/developers I'd appreciate having some thoughts on this...

I always kept within spk/python3* examples to build from and test cross-compiled wheels. While it served its purpose it also led to complexifying the overall maintenance of default python makefiles. In hope to simplify this I first thought it would make sense moving all of that under a python-wheels or similar sub-package, sort of placeholder to test wheel building when upgrading python / pip / setuptools and al (around where this PR is atm).

Doing so I thought why not building them all and include other remanants dispersed into other sub spk ...

While it may look neat on paper this becomes a burden to maintain as indivudual spk will evolve over time and this centralized copy won't serve it's purpose any longer...
On the other hand I do believe we need a central location providing clear and up-2-date examples and to test new versions of complex wheels while upgrading python versions.

Then I recall someone mentioning a while back ago, why not having our own wheel repository?

This could avoid maintaining cross-compiled wheels into individual packages + reducing package size as it would then download from our wheel repository at install time.
This centralized python-wheels could potentially meet that purpose.
While neat on paper I wonder what effect would lead upgrading dependent cross/* existing wheels ...

Thoughts on this would be much appreciated.

Also on my TODO (to which babysteps may be best):

make available a python 3.13 package (undergoing)
fix CMake based wheel building (requires sort of autodetection to allow passing proper parameters to at build-time)
fix meson based wheel building (similar issue to CMake type)
decouple crossenv from cross/python3* makefiles
allow either on-demand OR legacy+latest crossenv (for compatibility reasons)

mreid-tt · 2024-10-17T09:25:52Z

@th0ma7, I’m not very experienced with Python builds, so I had to familiarize myself with the basics (e.g., using Real Python's guide on wheels). Here are some initial thoughts:

Moving test code examples to "build from and test cross-compiled wheels" seems like a solid idea, aligning well with the purpose of the demoservice and demowebservice packages.
Setting up and maintaining our own wheel repository appears to be a significant long-term commitment.

I might be missing some key points due to my limited background, so I'd appreciate more details. For instance, you mentioned integrating other "remnants" scattered across sub SPKs. What exactly are these remnants, and what benefits would centralizing them bring? Are they also related to test code?

Regarding the internal wheel repository, is it a matter of PyPI's offerings being insufficient? Are there commonly missing platform-specific wheels for Synology hardware? Would it be feasible to advocate for the Python Package Index to include the wheels we need, perhaps by reaching out to the relevant projects and requesting support?

I might not fully grasp the situation, but I’m keen to understand the challenges better and contribute more meaningful suggestions.

th0ma7 · 2024-10-19T15:40:54Z

I’m not very experienced with Python builds, so I had to familiarize myself with the basics (e.g., using Real Python's guide on wheels). Here are some initial thoughts:

Moving test code examples to "build from and test cross-compiled wheels" seems like a solid idea, aligning well with the purpose of the demoservice and demowebservice packages.

Indeed. Package could even be renamed similarly.

Setting up and maintaining our own wheel repository appears to be a significant long-term commitment.

Not really. That would mostly be just a web page listing our wheels, that you can query through json to list our pre-compiled wheel packages. From there in our requirements.txt file provided with each package, we add a URL to fetch from. Then when wheels gets installed at package installation, it can download them from our source, similarly to using pypi.

I might be missing some key points due to my limited background, so I'd appreciate more details. For instance, you mentioned integrating other "remnants" scattered across sub SPKs. What exactly are these remnants, and what benefits would centralizing them bring? Are they also related to test code?

Regarding the internal wheel repository, is it a matter of PyPI's offerings being insufficient? Are there commonly missing platform-specific wheels for Synology hardware? Would it be feasible to advocate for the Python Package Index to include the wheels we need, perhaps by reaching out to the relevant projects and requesting support?

Issue is, pypi provides tons of pre-compiled wheels but ppc, armv5, (and often armv7 and even aarch64) are always missing. Therefore, on our NAS at installation time pip will only find the source of the requested wheels and try to compile them locally... and failing miserably as our Synology NAS does not provide a pre-installed build system toolchain and we're not providing the include files of the various dependencies.

Therefore when we build our spk package using our spksrc framework, we pull source packages from pypi usng pip, then cross-compile them and generate a resulting *.whl file. This wheel file gets packaged withing the spk so at package installation installation time it is being fetched by pip after failing to find a valid source online.

I might not fully grasp the situation, but I’m keen to understand the challenges better and contribute more meaningful suggestions.

The challenge is that the python wheel build system is undergoing a lot of changes currently and impacting the overall wheel building approaches. As pypi modules documentation and maintenance varies from one to the next, this ends-up breaking things all over the place.

So question is, would it be easier to have a single location to manage all wheel cross-compiling, providing numerous versions online to ease package management? I've been thinking of this a little more and still unsure, further as this would probably mean statically linking wheels so it is compatible with any installation OR using rpath so dependencies are available from that python-wheel package for instance.

Safihre · 2024-10-20T06:42:41Z

It makes sense to have our own repository. Although it will be quite a lot of work to implement and significantly add to maintenance burden..
I saw there's applications available that allow this, for example https://github.com/pypiserver/pypiserver

th0ma7 · 2024-11-14T02:37:06Z

@hgy59 I believe I may now have a feature-wise functional code ready for testing... I'll be away next week (SC24) so cycles until my return may be limited. Although I would much like your opinion on this invasive but in theory fully backward compatible code change.

TL;DR;

Added the ability to create on-demand multiple crossenv (see description above)
In brief, when faced with a legacy or wheels not so well supported, you can now create a mk/crossenv/requirements-<wheel>-<version>.txt OR it falls back to mk/crossenv/requirements-<wheel>.txt if exist a matching crossenv configuration.
Resulting wheel specific crossenv will only be of use for that specific wheel you're trying to compile (unless using fallback mechanism that may work for multiple versions).

Nice addition (from my perspective), ability to re-generate crossenv on demand (see description above for howto).

Besides the non-blocker TODO items, in theory it should be ready for testing and shaking out bugs and/or adjusting proposed strategy. I'm mostly thinking of weird corner-cases, such as the ones found in homeassistant that I hope can be addressed with this (to be tested + wheel specific crossenv configurations to be created).

Let me know if you have a moment to test-bed this, your pair of 👀 would be much appreciated.

Lastly: I did not forgot you with your requirement to join ffmpeg + wheel cross-compiling. With the previous addition of spksrc.ffmpeg.mk, along with this code refactor, it shouldn't be too complext in theory to connect the dots and tweak the code to make this a reality.

EDIT: Looking at github-action output it seems there are still a few rough edges to look for... on my todo list.

hgy59 · 2024-11-14T19:53:53Z

initial analysis:

when building python311 for the first time, the build-crossenv target fails, because some variables are not set, i.e. do not contain version info (I added those variables to the log)

make[3]: Leaving directory '/spksrc/cross/pip'
===>  crossenv wheel packages: pip==24.3.1, setuptools==75.4.0, wheel==0.45.0
===>  crossenv requirements file = /spksrc/mk/crossenv/requirements-default.txt
===>  PYTHON_PKG_VERS =
===>  PYTHON_PKG_NAME = python
===>  PYTHON_PKG_VERS_MAJOR_MINOR = .
mkdir -p /spksrc/spk/python311/work-x64-7.1/Python-/build/lib.linux-x86_64-.
cp -RL /spksrc/native/python/work-native/Python-/build/lib.linux-x86_64-. /spksrc/spk/python311/work-x64-7.1/Python-/build
cp: cannot stat '/spksrc/native/python/work-native/Python-/build/lib.linux-x86_64-.': No such file or directory
make[2]: *** [../../mk/spksrc.crossenv.mk:146: build-crossenv] Error 1
make[2]: Leaving directory '/spksrc/spk/python311'
make[1]: *** [../../mk/spksrc.supported.mk:74: build-arch-x64-7.1] Error 1
make[1]: Leaving directory '/spksrc/spk/python311'

when you call make again, the build-crossenv target is not built again (this looks like a missing .-response file)

but after call make crossenvclean

the the variables are set (and build finally succeeds):

make[3]: Leaving directory '/spksrc/cross/pip'
mkdir -p /spksrc/spk/python311/work-x64-7.1/crossenv-default/build
===>  crossenv wheel packages: pip==24.3.1, setuptools==75.4.0, wheel==0.45.0
===>  crossenv requirements file = /spksrc/mk/crossenv/requirements-default.txt
===>  PYTHON_PKG_VERS = 3.11.10
===>  PYTHON_PKG_NAME = python311
===>  PYTHON_PKG_VERS_MAJOR_MINOR = 3.11
mkdir -p /spksrc/spk/python311/work-x64-7.1/Python-3.11.10/build/lib.linux-x86_64-3.11
cp -RL /spksrc/native/python311/work-native/Python-3.11.10/build/lib.linux-x86_64-3.11 /spksrc/spk/python311/work-x64-7.1/Python-3.11.10/build
/spksrc/native/python311/work-native/install/usr/local/bin/python3 -m crossenv /spksrc/spk/python311/work-x64-7.1/install/var/packages/python311/target/bin/python3.11 --cc /spksrc/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/bin/x86_64-pc-linux-gnu-gcc --cxx /spksrc/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/bin/x86_64-pc-linux-gnu-c++ --ar /spksrc/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/bin/x86_64-pc-linux-gnu-ar --sysroot /spksrc/toolchain/syno-x64-7.1/work/x86_64-pc-linux-gnu/x86_64-pc-linux-gnu/sys-root --env LIBRARY_PATH= --manylinux manylinux2014 /spksrc/spk/python311/work-x64-7.1/crossenv-default
===>  Setting default crossenv /spksrc/spk/python311/work-x64-7.1/crossenv-default
===>  ln -sf crossenv-default /spksrc/spk/python311/work-x64-7.1/crossenv
...

hgy59 · 2024-11-14T20:13:01Z

some more details to the analysis above

all lines from build log starting with ===> (i.e. @$(MSG)) without dependent packages:

	Zeile  8633: ===>  Extracting for python311
	Zeile  8635: ===>  Patching for python311
	Zeile  8644: ===>  Configuring for python311
	Zeile  8645: ===>  - Configure ARGS: --host=x86_64-pc-linux-gnu --build=i686-pc-linux --prefix=/var/packages/python311/target --localstatedir=/var/packages/python311/var --enable-shared --without-static-libpython --enable-ipv6 --without-ensurepip --enable-loadable-sqlite-extensions --with-computed-gotos=yes --with-build-python --disable-test-modules --with-lto ac_cv_buggy_getaddrinfo=no ac_cv_file__dev_ptmx=no ac_cv_file__dev_ptc=no ac_cv_have_long_long_format=yes --with-ssl-default-suites=openssl --with-dbmliborder=gdbm:ndbm:bdb --with-system-expat --with-system-ffi
	Zeile  8646: ===>  - Install prefix: /var/packages/python311/target
	Zeile  9394: ===>  Compiling for python311
	Zeile  9943: ===>  Installing for python311
	Zeile 13671: ===>  Correcting pkg-config file lib/pkgconfig/python-3.11-embed.pc
	Zeile 13672: ===>  Correcting pkg-config file lib/pkgconfig/python-3.11.pc
	Zeile 13673: ===>  Correcting pkg-config file lib/pkgconfig/python3-embed.pc
	Zeile 13674: ===>  Correcting pkg-config file lib/pkgconfig/python3.pc
	Zeile 13677: ===>  Downloading files for pip
	Zeile 13678: ===>    File pip-23.2.1.tar.gz already downloaded
	Zeile 13679: ===>  Verifying files for pip
	Zeile 13680: ===>    Checking sha1sum of file pip-23.2.1.tar.gz
	Zeile 13681: ===>    Checking sha256sum of file pip-23.2.1.tar.gz
	Zeile 13682: ===>    Checking md5sum of file pip-23.2.1.tar.gz
	Zeile 13684: ===>  Processing dependencies of pip
	Zeile 13685: ===>  Extracting for pip
	Zeile 13687: ===>  Patching for pip
	Zeile 13688: ===>  Configuring for pip
	Zeile 13689: ===>  - Configure ARGS:
	Zeile 13690: ===>  - Install prefix: /var/packages/python311/target
	Zeile 13696: ===>  crossenv wheel packages: pip==24.3.1, setuptools==75.4.0, wheel==0.45.0
	Zeile 13697: ===>  crossenv requirements file = /spksrc/mk/crossenv/requirements-default.txt
	Zeile 13698: ===>  PYTHON_PKG_VERS = 3.11.10
	Zeile 13699: ===>  PYTHON_PKG_NAME = python311
	Zeile 13700: ===>  PYTHON_PKG_VERS_MAJOR_MINOR = 3.11
	Zeile 13704: ===>  Setting default crossenv /spksrc/spk/python311/work-x64-7.1/crossenv-default
	Zeile 13705: ===>  ln -sf crossenv-default /spksrc/spk/python311/work-x64-7.1/crossenv
	Zeile 13707: ===>  /spksrc/spk/python311/work-x64-7.1/crossenv-default/bin/build-python /spksrc/spk/python311/work-x64-7.1/crossenv-default/build/get-pip.py pip==24.3.1 --no-setuptools --no-wheel --disable-pip-version-check
	Zeile 13717: ===>  /spksrc/spk/python311/work-x64-7.1/crossenv-default/bin/cross-python /spksrc/spk/python311/work-x64-7.1/crossenv-default/build/get-pip.py pip==24.3.1 --no-setuptools --no-wheel --disable-pip-version-check
	Zeile 13727: ===>  /spksrc/spk/python311/work-x64-7.1/crossenv-default/bin/build-pip --disable-pip-version-check install setuptools==75.4.0 wheel==0.45.0
	Zeile 13740: ===>  /spksrc/spk/python311/work-x64-7.1/crossenv-default/bin/cross-pip --disable-pip-version-check install setuptools==75.4.0 wheel==0.45.0
	Zeile 13753: ===>  [/spksrc/spk/python311/work-x64-7.1/crossenv-default] Processing /spksrc/mk/crossenv/requirements-default.txt
	Zeile 13754: ===>  /spksrc/spk/python311/work-x64-7.1/crossenv-default/bin/build-pip --disable-pip-version-check install -r /spksrc/mk/crossenv/requirements-default.txt
	Zeile 13922: ===>  /spksrc/spk/python311/work-x64-7.1/crossenv-default/bin/cross-pip --disable-pip-version-check install -r /spksrc/mk/crossenv/requirements-default.txt
	Zeile 14084: ===>  Package list for /spksrc/spk/python311/work-x64-7.1/crossenv-default:
	Zeile 14085: ===>  /spksrc/spk/python311/work-x64-7.1/crossenv-default/bin/cross-pip list
	Zeile 14147: ===>  Compiling for pip
	Zeile 14148: ===>  CROSSENV: /spksrc/spk/python311/work-x64-7.1/crossenv-default/bin/activate
	Zeile 14150: ===>  Installing for pip
	Zeile 15841: ===>  crossenv wheel packages: pip==24.3.1, setuptools==75.4.0, wheel==0.45.0
	Zeile 15842: ===>  crossenv requirements file = /spksrc/mk/crossenv/requirements-default.txt
	Zeile 15843: ===>  PYTHON_PKG_VERS =
	Zeile 15844: ===>  PYTHON_PKG_NAME = python
	Zeile 15845: ===>  PYTHON_PKG_VERS_MAJOR_MINOR = .

This shows that crossenv-build target is used twice

first for Configuring for pip
second for Installing for pip

the first has correct variables, but the second is missing the python version

This might trigger you...

hgy59 · 2024-11-14T20:26:09Z

@th0ma7 another idea I want to share

When wheels couldn't be built without additional wheels in crossenv (like expandvars to build frozenlist), I was looking in the pyproject.toml file of the source to find the build dependencies.
A mutch more elegant solution would be an automatic population of the crossenv with all build requirements taken from pyproject.toml when the package source has such a file.

th0ma7 · 2024-11-14T20:31:44Z

@th0ma7 another idea I want to share

When wheels couldn't be built without additional wheels in crossenv (like expandvars to build frozenlist), I was looking in the pyproject.toml file of the source to find the build dependencies.
A mutch more elegant solution would be an automatic population of the crossenv with all build requirements taken from pyproject.toml when the package source has such a file.

That would be elegant indeed, and this pr could set the stage as a start to move towards gainin more flexibility with wheel building.

th0ma7 · 2024-11-15T16:50:15Z

@hgy59 now fixed and ready for testing. I also migrated the code to make use of status cookie handling like other pieces of the framework which simplify things a lot.

If by any chances you have a moment to look at py313 cross-compiling... they moved away from setup.py and I believe there may be some work to somehow port the 004-xcompile-paths.patch patch. Help would be welcomed. Note that from testing this switch seems to have started with py312...

hgy59 · 2024-11-15T18:11:08Z

not yet testing...
current build fails for DSM 6.2.4

it fails to install requirements-default.txt as it does not only install the listed modules but additional dependencies.
list of all aditional dependencies that are installed into crossenv see below

For DSM6.2.4 it fails to install msgpack==1.1.0 because it would need additional cflags (use option -std=c99, -std=gnu99, -std=c11 or -std=gnu11 to compile your code)

We must either

use --no-deps to install requirements-default.txt
add msgpack==1.0.5 to requirements-default.txt
install each module in requirements-default.txt with a dedicated call of build-pip/cross-pip with the handling of additional defined flags (as we do for package wheels)

List of additional modules installed into crossenv (those are not listed in requirements-default.txt)

build==1.2.2.post1
CacheControl==0.14.1
certifi==2024.8.30
charset-normalizer==3.4.0
cleo==2.1.0
click==8.1.7
crashtest==0.4.1
distlib==0.3.9
distro==1.9.0
docutils==0.21.2
dulwich==0.21.7
fastjsonschema==2.20.0
filelock==3.16.1
flit_core==3.10.1
idna==3.10
installer==0.7.0
jaraco.classes==3.4.0
jeepney==0.8.0
keyring==24.3.1
more-itertools==10.5.0
msgpack==1.1.0
packaging==24.2
pexpect==4.9.0
pkginfo==1.11.2
platformdirs==4.3.6
poetry-core==1.9.1
poetry-plugin-export==1.8.0
ptyprocess==0.7.0
pycparser==2.22
pyproject_hooks==1.2.0
RapidFuzz==3.10.1
requests==2.32.3
requests-toolbelt==1.0.0
SecretStorage==3.3.3
semantic-version==2.10.0
shellingham==1.5.4
tomli_w==1.1.0
tomlkit==0.13.2
trove-classifiers==2024.10.21.16
urllib3==2.2.3
virtualenv==20.27.1

th0ma7 · 2024-11-15T21:53:52Z

@hgy59 question of managing expectations... This PR really still is a [WIP] and my main focus until now has been on keeping a working python 3.10-3.11 while enhancing the crossenv functionality to have a per-wheel crossenv as needed. Python 3.13 isn't yet fully getting my attraction until I can confirm new crossenv functionality allows re-producing all python based packages.

I was hoping that at this stage it could be tested to confirm wetter this suffice to resolve the immediate build failures we have and allow getting reproducibility back for some of our packages (homeassistant is a good candidate).

not yet testing... current build fails for DSM 6.2.4

it fails to install requirements-default.txt as it does not only install the listed modules but additional dependencies. list of all aditional dependencies that are installed into crossenv see below

Interestingly it's only python 3.13 that fails, and I'm glad it only fails on this as I had other issues previously which now looks solved, at least thru github-action.

For DSM6.2.4 it fails to install msgpack==1.1.0 because it would need additional cflags (use option -std=c99, -std=gnu99, -std=c11 or -std=gnu11 to compile your code)

We must either:

use --no-deps to install requirements-default.txt

The crossenv requires all dependencies to be installed as otherwise it may fail when cross-compile wheels later-on as the build environment would not allow functionalities of said tools to work without its needed dependencies. Also note, this is the exact same behavior as current.

add msgpack==1.0.5 to requirements-default.txt

Currently testing that to see if this helps... Also, on my local branch cross/python313 currenly fails with the following but builds fine using spk/python313 up the the crossenv.

checking for --with-build-python... configure: error: invalid or missing build python binary "python3.13"

I'll have another look at it upon my return (feel free to push fixes if you hapen to have cycles).

install each module in requirements-default.txt with a dedicated call of build-pip/cross-pip with the handling of additional defined flags (as we do for package wheels)

I tried this... and it's really tricky as ordering of install is important. On the other hand that may ease passing -std=c99, -std=gnu99, -std=c11 or -std=gnu11 but even then, we'd need a place to hold such compiler or platform specific crossenv build process configurations beyond current mk/crossenv. An idea could be of using a similar directory functionality as the patch/ directory, something worth investigating later-on.

But you're right, while downgrading msgpack==1.0.5 may solve the immediate issue, we will need extra build options if we intent to maintain backward compatibility with our older platforms (until gcc4 no longer works).

List of additional modules installed into crossenv (those are not listed in requirements-default.txt)

yup, due to the dependency chain as we need to provide all dependencies for cross-compiling to actually work. I find it handy to print the list of crossenv installed wheels to track exactly what is the build environment in use when cross-compiling.

Lastly, my next step is to review the wheel building code to use status cookies and be closer to the remaining of the framework code. I want to divide it such as the following for a start:

spksrc.wheel-download.mk
spksrc.wheel-compile.mk

So then it will become much easier to maintain and add extra functionalities as needed such as meson and cmake toolchain file support, and potentially automating crossenv based on wheel internal information like you suggested.

th0ma7 · 2024-11-16T01:47:42Z

Qoriq failure may be related to python/cpython#125269

hgy59 · 2024-11-25T11:19:59Z

@th0ma7 finding issues on gh
AFAICR there were local issues that were gone when calling make again (the one with the python path?)

probably you can reproduce issues locally by use of make clean (this is not so slow, since we have prebuilt python).

th0ma7 · 2024-11-25T11:55:02Z

@hgy59 one of the key thing I noticed is pycparser ... when investigating I noticed that both cffi and pycparser are dependencies of cross/cryptography to which they are being installed into the crossenv for cryptography to actually build. This is totally useless as the crossenv already has that.

Further, I noticed a few legacy python things in there needing to be cleaned-up... something for a different PR.

As for debuging, the best would be to find out how (and if it is possible) to get into the gh container directly. I haven't search howto yet, I'll leave this one complete to confirm if a few things got fixed already.

th0ma7 · 2024-11-26T12:35:47Z

@hgy59 🚀 now time for some code cleanups...

- add evaluation of python313 dependent packages - evaluate packages to build and changed dependency folders in build.yml to avoid overflow of bash variables (limited to 1024 chars) (fixes SynoCommunity#6284)

hgy59 · 2024-11-26T22:02:14Z

@th0ma7 sorry for adding a fix for #6284 to this PR, but it would be too much effort to create a dedicated PR to reproduce this error to fix it.

It is still possible that not all packages are found by dependencies because the DEPENDENCY_LIST variable still might be truncated (at 1024 chars). This will be (hopefully) fixed in #6255.
But packages with changes under spk folder are found now.

th0ma7 · 2024-11-26T22:26:30Z

@hgy59 i sincerely have no clue. And I had not looked into why python-wheel wasn't being trigered yet, I just thought this was due to a naming issue conflicting with python3* on our gh hooks. I was actually thinking of renaming it to py311-wheels so we can then prepare migration with a py31* later on.

- python dependent packages with name sorted after python31* where removed from packages to build (in this context: python-wheels and rdiff-backup)

hgy59 · 2024-11-26T22:53:16Z

@hgy59 i sincerely have no clue. And I had not looked into why python-wheel wasn't being trigered yet, I just thought this was due to a naming issue conflicting with python3* on our gh hooks. I was actually thinking of renaming it to py311-wheels so we can then prepare migration with a py31* later on.

it was the alphabetic order (python-wheels and rdiff-backup were skipped) - now fixed.

BTW currently it fails to download python sources (403 forbidden) so all is red...

hgy59 · 2024-11-26T22:58:13Z

@th0ma7 my local build of python-wheels succeeded only for the following archs

x64-7.1
aarch64-7.1
evansport-7.1
x64-6.2.4
aarch64-6.2.4

all other archs fail to build cross/mariadb-connector-c

- add space to separate packages

hgy59 · 2024-11-26T23:15:00Z

BTW currently it fails to download python sources (403 forbidden) so all is red...

known issue python/cpython#127307

th0ma7 · 2024-11-26T23:54:59Z

all other archs fail to build cross/mariadb-connector-c

I thought I had it fixed, but it's been a while. Let me double check, then hopefully this can get moved into a distinct pr.

th0ma7 · 2024-11-27T01:10:04Z

@hgy59 I'm getting this weird build failure with openssl3 only with qoriq, rings any bell?

/powerpc-e500v2-linux-gnuspe-gcc} -pthread -I/home/spksrc/python-wheels3/spksrc/toolchain/syno-qoriq-6.2.4/work/powerpc-e500v2-linux-gnuspe/powerpc-e500v2-linux-gnuspe/sysroot/usr/include -mcpu=8548 -mhard-float -mfloat-gprs=double -I/home/spksrc/python-wheels3/spksrc/cross/openssl3/work-qoriq-6.2.4/install/usr/local/openssl3/include  -L.  -L/home/spksrc/python-wheels3/spksrc/toolchain/syno-qoriq-6.2.4/work/powerpc-e500v2-linux-gnuspe/powerpc-e500v2-linux-gnuspe/sysroot/lib -mcpu=8548 -mhard-float -mfloat-gprs=double -L/home/spksrc/python-wheels3/spksrc/cross/openssl3/work-qoriq-6.2.4/install/usr/local/openssl3/lib -Wl,--rpath-link,/home/spksrc/python-wheels3/spksrc/cross/openssl3/work-qoriq-6.2.4/install/usr/local/openssl3/lib -Wl,--rpath,/usr/local/openssl3/lib  \
	-o apps/openssl \
	apps/lib/openssl-bin-cmp_mock_srv.o \
	apps/openssl-bin-asn1parse.o apps/openssl-bin-ca.o \
	apps/openssl-bin-ciphers.o apps/openssl-bin-cmp.o \
	apps/openssl-bin-cms.o apps/openssl-bin-crl.o \
	apps/openssl-bin-crl2pkcs7.o apps/openssl-bin-dgst.o \
	apps/openssl-bin-dhparam.o apps/openssl-bin-dsa.o \
	apps/openssl-bin-dsaparam.o apps/openssl-bin-ec.o \
	apps/openssl-bin-ecparam.o apps/openssl-bin-enc.o \
	apps/openssl-bin-engine.o apps/openssl-bin-errstr.o \
	apps/openssl-bin-fipsinstall.o apps/openssl-bin-gendsa.o \
	apps/openssl-bin-genpkey.o apps/openssl-bin-genrsa.o \
	apps/openssl-bin-info.o apps/openssl-bin-kdf.o \
	apps/openssl-bin-list.o apps/openssl-bin-mac.o \
	apps/openssl-bin-nseq.o apps/openssl-bin-ocsp.o \
	apps/openssl-bin-openssl.o apps/openssl-bin-passwd.o \
	apps/openssl-bin-pkcs12.o apps/openssl-bin-pkcs7.o \
	apps/openssl-bin-pkcs8.o apps/openssl-bin-pkey.o \
	apps/openssl-bin-pkeyparam.o apps/openssl-bin-pkeyutl.o \
	apps/openssl-bin-prime.o apps/openssl-bin-progs.o \
	apps/openssl-bin-rand.o apps/openssl-bin-rehash.o \
	apps/openssl-bin-req.o apps/openssl-bin-rsa.o \
	apps/openssl-bin-rsautl.o apps/openssl-bin-s_client.o \
	apps/openssl-bin-s_server.o apps/openssl-bin-s_time.o \
	apps/openssl-bin-sess_id.o apps/openssl-bin-smime.o \
	apps/openssl-bin-speed.o apps/openssl-bin-spkac.o \
	apps/openssl-bin-srp.o apps/openssl-bin-storeutl.o \
	apps/openssl-bin-ts.o apps/openssl-bin-verify.o \
	apps/openssl-bin-version.o apps/openssl-bin-x509.o \
	apps/libapps.a -lssl -lcrypto -ldl -pthread -latomic 
./libcrypto.so: undefined reference to `__dlsym'
./libcrypto.so: undefined reference to `__dlopen'
./libcrypto.so: undefined reference to `__dlclose'
./libcrypto.so: undefined reference to `__dladdr'
./libcrypto.so: undefined reference to `__dlerror'
collect2: error: ld returned 1 exit status
make[4]: *** [Makefile:20901: apps/openssl] Error 1
make[4]: Leaving directory '/home/spksrc/python-wheels3/spksrc/cross/openssl3/work-qoriq-6.2.4/openssl-3.1.7'
make[3]: *** [Makefile:2858: build_sw] Error 2
make[3]: Leaving directory '/home/spksrc/python-wheels3/spksrc/cross/openssl3/work-qoriq-6.2.4/openssl-3.1.7'

th0ma7 · 2024-11-27T01:46:06Z

@hgy59 getting mariadb-connector-c to compile properly is a real pain... anyhow, now sorted although any version > 3.3.11 will make aarch64-6.2.4 to fail on assembly with Error: CFI instruction used without previous .cfi_startproc. Feel free to chime in if you have an idea on how to fix that so we can update to a newer version. Note that I tried the classics of chaging -O0 (which is tricky as it enforces -O3 by other means) and tons of other cflags and failed.

Otherwise, this qoriq failure on openssl is new to me... ?

hgy59 · 2024-11-27T06:17:40Z

unfortunaltely build of python-wheels is still missing here...

th0ma7 mentioned this pull request Oct 14, 2024

cross-compiling python 3.13 readline error - unable to find ncurses.pc while ncursesw.pc is available python/cpython#125452

Closed

hgy59 reviewed Oct 14, 2024

View reviewed changes

cross/python313/Makefile Show resolved Hide resolved

SynoCommunity deleted a comment from hacscred Oct 14, 2024

th0ma7 changed the title ~~python311:Dissociate python optional test wheels builds + python313 test drive~~ [WIP] dissociate python optional test wheels builds + python313 Oct 14, 2024

th0ma7 self-assigned this Oct 14, 2024

th0ma7 added the status/work-in-progress label Oct 14, 2024

SynoCommunity deleted a comment from hacscred Oct 15, 2024

th0ma7 mentioned this pull request Oct 15, 2024

readline: Fix readline.pc to depend on ncursesw #6283

Merged

10 tasks

th0ma7 force-pushed the python-wheels branch from d66464e to b66819d Compare October 15, 2024 21:07

th0ma7 mentioned this pull request Oct 16, 2024

Creation of a new spk fails to trigger build #6284

Open

1 task

th0ma7 force-pushed the python-wheels branch from 5a6a413 to ba8d9bd Compare November 3, 2024 21:31

hgy59 mentioned this pull request Nov 10, 2024

rdiff-backup: update and use alternate tmp dir (#4427) #6049

Merged

7 tasks

th0ma7 force-pushed the python-wheels branch from bb787f5 to a7a0997 Compare November 11, 2024 15:05

hgy59 added the framework label Nov 14, 2024

th0ma7 added 4 commits November 25, 2024 01:23

dtlssocket: Add -std=c99 for gcc < 5

dd306d8

python.mk: Erase only $(ARCH)-$(TCVERSION) related files

1825e05

deluge: Update wheels and fix build

f3f4dc3

boost_1.82: Allow builds from python.mk

89dcd4c

th0ma7 force-pushed the python-wheels branch from 6c89b34 to 89dcd4c Compare November 25, 2024 01:24

th0ma7 added 2 commits November 25, 2024 11:10

cross/cryptography: crossenv already has cffi and pycparser

d3e67a7

crossenv.mk: Add debug traces to find issue on gh

77f44c0

th0ma7 added 4 commits November 25, 2024 22:02

crossenv.mk: Testing alternative method to get python version dir

6f45c95

crossenv.mk: Send msg to status file

94c6249

wheel.mk: Remove trailing # comment using $${wheel%% *}

d7edc67

crossenv.mk: Use $(WORK_DIR)/pip caching dir for crossenv creation

708218a

hgy59 added 2 commits November 26, 2024 21:22

python-wheels: include wheels in package

3b29da6

update github build action

2b552da

- add evaluation of python313 dependent packages - evaluate packages to build and changed dependency folders in build.yml to avoid overflow of bash variables (limited to 1024 chars) (fixes SynoCommunity#6284)

adjust variable GH_DEPENDENCY_FOLDERS

c261b65

real fix for SynoCommunity#6284

2f537de

- python dependent packages with name sorted after python31* where removed from packages to build (in this context: python-wheels and rdiff-backup)

fix github prepare script

194e2c1

- add space to separate packages

crossenv.mk: Miscelanious cleanups

1431ce3

mariadb-connector-c: Fix compilation for some archs

c56d379

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] dissociate python optional test wheels builds + python313 #6282

[WIP] dissociate python optional test wheels builds + python313 #6282

th0ma7 commented Oct 14, 2024 •

edited

Loading

th0ma7 commented Oct 14, 2024

th0ma7 commented Oct 15, 2024

hgy59 commented Oct 15, 2024

th0ma7 commented Oct 16, 2024

mreid-tt commented Oct 17, 2024

th0ma7 commented Oct 19, 2024

Safihre commented Oct 20, 2024

th0ma7 commented Nov 14, 2024 •

edited

Loading

hgy59 commented Nov 14, 2024

hgy59 commented Nov 14, 2024

hgy59 commented Nov 14, 2024

th0ma7 commented Nov 14, 2024

th0ma7 commented Nov 15, 2024

hgy59 commented Nov 15, 2024

th0ma7 commented Nov 15, 2024

th0ma7 commented Nov 16, 2024

hgy59 commented Nov 25, 2024

th0ma7 commented Nov 25, 2024

th0ma7 commented Nov 26, 2024

hgy59 commented Nov 26, 2024

th0ma7 commented Nov 26, 2024

hgy59 commented Nov 26, 2024

hgy59 commented Nov 26, 2024

hgy59 commented Nov 26, 2024

th0ma7 commented Nov 26, 2024

th0ma7 commented Nov 27, 2024

th0ma7 commented Nov 27, 2024

hgy59 commented Nov 27, 2024

[WIP] dissociate python optional test wheels builds + python313 #6282

Are you sure you want to change the base?

[WIP] dissociate python optional test wheels builds + python313 #6282

Conversation

th0ma7 commented Oct 14, 2024 • edited Loading

Description

Checklist

Type of change

TODO

th0ma7 commented Oct 14, 2024

th0ma7 commented Oct 15, 2024

hgy59 commented Oct 15, 2024

th0ma7 commented Oct 16, 2024

mreid-tt commented Oct 17, 2024

th0ma7 commented Oct 19, 2024

Safihre commented Oct 20, 2024

th0ma7 commented Nov 14, 2024 • edited Loading

TL;DR;

hgy59 commented Nov 14, 2024

hgy59 commented Nov 14, 2024

hgy59 commented Nov 14, 2024

th0ma7 commented Nov 14, 2024

th0ma7 commented Nov 15, 2024

hgy59 commented Nov 15, 2024

th0ma7 commented Nov 15, 2024

th0ma7 commented Nov 16, 2024

hgy59 commented Nov 25, 2024

th0ma7 commented Nov 25, 2024

th0ma7 commented Nov 26, 2024

hgy59 commented Nov 26, 2024

th0ma7 commented Nov 26, 2024

hgy59 commented Nov 26, 2024

hgy59 commented Nov 26, 2024

hgy59 commented Nov 26, 2024

th0ma7 commented Nov 26, 2024

th0ma7 commented Nov 27, 2024

th0ma7 commented Nov 27, 2024

hgy59 commented Nov 27, 2024

th0ma7 commented Oct 14, 2024 •

edited

Loading

th0ma7 commented Nov 14, 2024 •

edited

Loading