Wheels (which means buildsystems, documentation, CI etc)

[llimeht]

The build system for SasView (and friends) carries a lot of historical baggage and now gets in the way of working on SasView like a 'normal' Python package. In looking at build systems, documentation and CI, I've prototyped some ideas that I think address many of the issues.

Since this is a set of changes across sasview, sasmodels, sasdata, and bumps, I've created this as a design discussion rather jumping to four (or more!?) separate PRs.

Current state

• SasView doesn't currently produce usable wheels - setup.py could probably be convinced to make a .whl, but with documentation being an integral part of SasView, the wheels are incomplete.
• The SasView source has a very unusual property - it requires the source of other packages to be available to be able to successfully build it. In particular, building the SasView docs currently requires the source to be checked out in a particular way. Source dependencies don't match the normal software distribution paradigm so aren't supported by pip/Python build systems, linux distributions, flatpak, snap etc. The use of source dependencies complicates and slows down CI.
• Finding resources that are part of the modules currently uses lots of file path manipulation code which is a great source of bugs, and causes issues with pyinstaller; using importlib and putting resources inside wheels is more robust. (This is the failure cause for the doc-regen code under linux from what I can see too, but more on that in a separate discussion)

Desired future state

• pip install sasview works.
• running SasView only needs the sasview wheel to be installed, not the module to be unpacked in a certain way and have separate files like documentation shipped with it
• building and CI is simpler and less entangled
• building a SasView wheel requires only toolchain packages and wheels from dependencies (e.g. some system libraries and then pip install ...)

Approach

There are a few things to do:

• Move to declarative packaging as much as possible - i.e. a simple pyproject.toml config file and not Python code to maintain.
• Ensure that everything needed at runtime is in the wheel (i.e. make it so that listing the package in a dependencies field is enough)
• Ensure that everything needed by users of the module is available in the wheel (i.e. make it so that listing the package as a build-dependency is enough; the wheel satisfies build dependencies properly)
• While making the packaging simpler and more declarative, simplifying the sphinx build calls is also appropriate.

As a build system, hatchling is being developed by the pypa folks (i.e. the people who maintain the tool chain packages build, installer, wheel, pip, setuptools). While it is possible to use hatch as a driver for managing builds, build environments, test environments etc, that is not necessary. As a standards-compliant backend builder, hatchling is invisible to the person who invokes the build.

For routine work of assembling python files and data files into a wheel, hatchling is fine. There are then plugins to deal with versions from git or from magic strings in files. There is a growing selection of plugins to deal with metadata or building; I've written one that can build documentation with sphinx.

To ensure that the wheels contain everything that is needed for all uses of the module, the wheel then needs to contain the sphinx source for the documentation. By embedding the sphinx source for sasdata into the sasdata wheel as a set of data files that are part of the module, those files are available for the next build step; likewise for the other projects whose documentation is embedded into sasview's documentation.

When building sasview, the sphinx source for each module can be located and extracted (a couple of lines with importlib) and then the assembled documentation source can be given to sphinx.

The sasview wheel needs to contain both the assembled sphinx source ready to build, and also the pre-built documentation.

• A user who just wants to view the documentation can do so straight out of the installation (locating the documentation via importlib once again).
• A user who is interested in editing the model documentation can have the source for the docs extracted from the wheel (using importlib), copied into the user's profile directory (perhaps using platformdirs to do this more portably?) and then the docs can be rebuilt as needed.

If the wheels are more self-contained, then pyinstaller usage can also be improved. It's possible to let each module be a bit more self-describing of its resources for pyinstaller via entry-points.

Prototypes

The above plan is not just hypothesis... there are prototypes available for discussion

https://github.com/llimeht/sasmodels/tree/tmp/pyproject
https://github.com/llimeht/sasdata/tree/tmp/pyproject
https://github.com/llimeht/bumps/tree/tmp/docs-source
https://github.com/llimeht/sasview/tree/pyproject-hatchling

These are obviously not perfect but any problems can be treated as bugs to fix rather than showstoppers that invalidate the approach.

Known issues:

• the tinkering with pyinstaller hooks is incomplete and so some files are missing from the installer package at the moment
• the doc-editing-regen code is not tweaked to find the shipped source... but that's not actually regression under linux; I think that actually gets easier to fix once there's a clear idea of where the docs are stored and how to think of them.
• the assembly of sphinx sources from the sub-projects can be simplified a lot further in time
• with more tweaking of the pyinstaller hooks, the sasview.spec can be further simplified.

For discussion

• Can we reach high level agreement that it is useful to simplify the build-dependencies and runtime-dependencies?
• Are we happy to move away from the on-going pain of setuptools to standardised interfaces with hatchling?
• Are we ready to adopt standard interfaces/methods for building, packaging, and distributing the modules?
• Are we happy to include sphinx sources as module data file resources - since that is basically what they have become?
• (what else do we need to discuss before getting into the implementation details?)

[butlerpd]

Sounds like something that needs some serious discussion, preferably soon! Guessing that will have to wait till the new year at this point?

Certainly build processes and the CI have long been a pain point though part of that has been the need to pin versions of some dependencies. I assume this will not help with that?

Other long standing issues with the build process such as the need to pull in the documentation from various dependencies before building, the various path issues and the perennial issues with Unix deployments sound like they might be addressed by this proposal? Given most facilities seem to deploy SasView locally for users on Unix machines that would probably be helpful.
.
I do have a naive, but important question: how would these changes impact the windows installer (and mac OS as well) build as that is currently the largest user base?

[jamescrake-merani]

I do have a naive, but important question: how would these changes impact the windows installer (and mac OS as well) build as that is currently the largest user base?

I don't see any problems that this work would raise in this regard. There is of course going to be some work to make sure everything is tested, and working as expected on both platforms but I think that would be significantly simpler, and easier than the current system we have at the moment.

[llimeht]

All good questions, @butlerpd

Sounds like something that needs some serious discussion, preferably soon! Guessing that will have to wait till the new year at this point?

Most likely. Much of this work is pretty easy to rebase onto future additions to git without us stomping on each-other's toes. The exception is bumps where there's a lot of divergence between the 0.9 and 1.0 branches and this work is based on the 0.9.3 release since that is what sasview continues to need; the work will need to be ported over to 1.0 and merged in with the other build system changes that have been worked on there.

Certainly build processes and the CI have long been a pain point though part of that has been the need to pin versions of some dependencies. I assume this will not help with that?

That's more a philosophical problem with a "move fast and break things" mentality of Python module development where API stability is not prioritised. API instability and conflicting version requirements for different dependencies are things that can't be engineered around.

Other long standing issues with the build process such as the need to pull in the documentation from various dependencies before building, the various path issues and the perennial issues with Unix deployments sound like they might be addressed by this proposal?Given most facilities seem to deploy SasView locally for users on Unix machines that would probably be helpful. .

There are a few things to unpack there:

• build process needing to pull in docs: by making wheels of sasmodels, sasdata and bumps provide the relevant sphinx source as module data, the build requirement is now only "install the wheels" for these packages - i.e. they are no different to any other dependency, can just sit in requirements.txt etc. The doc sources are still merged for compilation into a unified output, but lots of code trying to find the doc sources can go away
• the path issues come from (incorrect) assumptions about how sasview is installed and that building from source like any other python application doesn't work here. To a first approximation, every use of functions like os.path.join in the code that is looking for module resources is a bug and should be replaced by importlib.resources. This work helps a bit because it puts all the data files into the wheels, meaning there is only one location for the resources and they can be located.
• the issues with linux deployments are mostly about making incorrect path assumptions; the doc regen bug, for example, appears to come from assuming that sasview is in the same directory as other things, when it will be in /usr/bin/sasview or ~/.bin/sasview or ..../some/venv/bin/sasview and the modules are in a completely different location.
• By making everything go via wheels, we reduce the number of ways that sasview gets built and installed. That's a substantial simplification that will help reduce the places for bugs to hide. The wheel can still get unpacked into some sort of bundle (pyinstaller, conda-pack, whatever), but by removing all the manual steps of copying around files and removing the code trying to guess where resources are installed, this all gets a lot simpler.
• I would expect that the facility installation would end up just looking like pipx install sasview at that stage. That only has to download wheels from pypi and unzip them - no building required on the target machines.

I do have a naive, but important question: how would these changes impact the windows installer (and mac OS as well) build as that is currently the largest user base?

It lets us get to those installers more easily; we don't make changes to those installers with this work.

We'd actually have just a single build of the sasview wheel — it is platform-agnostic so that build can be done on the fastest builders in the CI farm (the linux ones). The wheels can be unpacked into the platform-specific installer bundles. That should be a faster time-to-complete for CI as well as a substantial reduction in the total CPU time on CI.

In saying there are "no changes" to the installer bundles... there is one deliberate change: the docs and the sphinx sources for the docs would move into the sas python module rather than just kicking around in the top-level directory of the installer bundles. There's a code change required for that, obviously, but the docs would then be in the same place in all installations via all methods - fewer code paths, fewer assumptions.

[jamescrake-merani]

I agree with pretty much all of this. Just to add my own contributions:

perhaps using platformdirs to do this more portably?

I think this would be a really good idea, and would also fix #3142. When I got the Flatpak reviewed, the reviewers really didn't like how we are putting a lot of configuration, and other data into the .sasview directory of the home folder. The workaround we had to use was to get Flatpak to automatically store them in .var/app/org.sasview.sasview, but this location will probably be less obvious to the user. Using a library that already has the right directories preconfigured for us for each operating system seems ideal to me.

I think also (as alluded to), the documentation system is something we are going to need to take a close look at. If my understanding is correct (which it may not be), currently, documentation is copied to the home folder on installation, and is built from there? This won't work under the Flatpak as it is built under a separate container, and won't have access to the user's home folder at all at this stage (not least because it will not even be built on the user's machine).

My opinion in general is that we need to move away from custom installation, and build logic to using pre-existing systems as much as possible. Your work in my eyes marks significant progress in this area.

[pkienzle]

Consider using conda-pack as the distribution rather than pyinstaller. Rather than a zip-file bundle the resources can be located within the install tree as a usual python package. This simplifies bundling the package and finding the resources from within the bundle.

[llimeht]

I don't really have an opinion on conda-pack vs pyinstaller - looking at conda-pack, I think this work is still needed because getting sasview actually installable as a python module is still needed. If we can get sasmodels, sasdata and bumps to be 'normal' build-dependencies and get a sasview wheel out the end, then there are lots of possibilities.

[pkienzle]

Agreed this is orthogonal. I brought it up because the complexity with finding resource files comes from our original wx+py2exe bundle. In retrospect we may have been able to work it out even then (there is a blog post from 2008 that suggests a solution) so I should not let this bias me against pyinstaller.

[pkienzle]

Since bumps has a life outside of SasView I posted this discussion as a bumps issue: bumps/bumps#208

[pkienzle]

Moving to pyproject.toml seems reasonable (we already did so for bumps and periodictable) but I don't find hatchling to be particularly compelling compared to setuptools for the build system. I would personally wait until its usage passes poetry-core (see download trends) before switching from known pain to unknown pain.

Moving to importlib.resources would be good if it means we can remove code. That depends on whether the bundling of data works with the sasview installer.

SasView doc build is going to be weird no matter how it is handled. It might help if we can tell sphinx to use files in another tree when building so at least we don't have the copy step. Carrying the .rst files around in the package just to save a little work in the SasView CI seems a little ugly.

If SasView is using doc files from bumps [ I don't know why you would in their current state :-( ] then put a ticket into bumps. We can pre-compile the tutorials (i.e., run gentut.py) as part of the build so SasView doesn't need to know about that.

pyopencl is already using platformdirs for its cache. It makes debugging harder since I never know where it hides things 8-)