Goals of this repository

[fingolfin]

Let's collect some goals for this repository, to see if we can agree on a common vision; then next we can see how to implement them and who will do it. CC @ZachNewbery @alex-konovalov @james-d-mitchell @wilfwilson please feel free to edit this and/or to comment on it all.

Overarching goal is to merge all existing GAP docker repositories into this one. The resulting images shall be used for...

• CI of individual packages
• CI of the GAP package distribution
• possibly also GAP CI
• end users who want to try GAP??
• other things (e.g. that Deepnote work by @ZachNewbery and @alex-konovalov ?)

We'll want a "matrix" of image variants along several axes:

1. GAP variants

   • master branch (updated nightly)
   • stable-4.x branches (updated nightly, but ideally only if there really were changes to the branch?)
   • for each GAP release , i.e., based on v4.x.y tags in the GAP repo, and accessible here under the same tags
   • ideally also something to say "give me the latest 4.11.x release" -- say via tag v4.11 or v4.11-latest or so
   • perhaps also a latest tag which maps to whatever is the latest GAP release

2. dependencies / packages

   • a "base" image which all other images are based on; adds that "gap" user and whatever common config we might want?
   • an "all-deps" image which has "all" (or at least as many as sensibly possible) dependencies for package installed (see also PR WIP: Attempt to unify building some of the containers #3), but nothing else (in particular no GAP)
     • most other images will be layered atop this one
     • could be useful for package distro CI
   • a "full" image for each GAP variant, which has all (or as many as sensibly possible) GAP packages compiled which need to be compiled
   • a "minimal" image for each GAP variant, which has GAP and its required packages but basically nothing else (in particular, not the full package distro!!!)
     • for use in CI of GAP packages
     • actually maybe add in PackageManager, for obvious reasons
     • this is mainly interesting if it provides measurable size savings, resp. is faster to use in real world CI runs
   • a "doc" image (possible for each GAP variant?) which is based on "minimal" and adds AutoDoc, but also latex
     • for use in CI jobs of GAP packages which build manuals

Some further ideas (should perhaps be moved to separate issues):

• size of images is a concern if they are to be used heavily in CI; e.g. in one case, I found it was faster to get the GAP source code and build it each time plus a few select packages, than to download and start a relevant container.
  • perhaps investigate whether basing our images on Ubuntu is ideal; there are lighter alternatives; e.g. Alpine Linux.
  • OTOH Ubuntu really provides most external dependencies we want, so that has to be kept in mind
  • some people say that using Alpine is risky, see https://pythonspeed.com/articles/base-image-python-docker-images/
• ...

[wilfwilson]

Thanks for posting this issue. I will come back to it soon.

[fingolfin]

Just did a quick experiment and adapted the dockerfile from PR #3 to use ubuntu:21.10 as base and then followed my suggested edits on that PR. I've also dropped the 32bit support. A quick size comparison:

• Ubuntu 20.04 base image: 65.6MB
• Ubuntu 21.10 base image: 69.9MB
• result of my experiment: 2.68GB (!)
• gap-docker-master from 6 days ago: 5.63GB (!!!)

So the base image size probably isn't the problem. But the packages we install are. Yet it still saves substantially compared to gap-docker-master ...

One interesting task would be to figure out how much each individual package we install adds. And then perhaps some could be omitted.

As a first quick test, I checked what happens if I reduce the packages to what is necessary to build just GAP:

RUN    apt-get update -qq \
    && apt-get -qq install -y \
            autoconf \
            automake \
            build-essential \
            curl \
            g++ \
            gcc \
            git \
            libgmp-dev \
            libreadline6-dev \
            libtool \
            m4 \
            sudo \
            unzip \
            wget

The result already weighs 798MB.

As it turns out, installing the macaulay2 packages adds 1.3 GB (!). Ironically, we install this for the sake of homalg tests, but actually currently the homalg packages don't even specify a TestFile, so this is for nothing 😂

The pari-gp and polymake packages each seem to install about ~450 MB. Singular ~150 MB.

I wonder: is there a way to "extend" the standard GitHub images in a way that leverage layering? What I mean is this: they already use Ubuntu images for their runners, and clearly those are preloaded on the runner machines, so are fast. If we could just layer our images atop those, and if GH's infrastructure is able to make use of this (by reusing the already present base image) then we could potentially safe quite a bit, as at least (most of) those 798 MB I just listed are already in there.

One final thought: some of these packages are needed to build packages. Some of them are only needed to run packages. The latter include some big ones: macaulay2, pari-gp, polymake, singular. Perhaps we should just omit those in the "all-deps" image (resp. in a subimage of that, called "all-build-deps" or so), and then install them only during CI, and only in those CI jobs that really need them...

[wucas]

In PR #5 I've tried to make the images as small as I could. But even so given the many combinations we would like this adds up to around 130GB without(!) the doc variant. Where are we going to host that? As far as I can tell ghcr.io offers at most 50GB without additional monthly fees.
I ended up with image size of around 1.3GB for the minimal variants and around 4.3GB for the full variant. I expect the doc variant will be somewhere between 2 and 3 GB.

[wucas]

I stand corrected, I simply didn't read carefully enough and oversaw the complete left side of my monitor where it says "everything free for open source" 🤦🏼‍♂️ my bad.

[zickgraf]

I just stumbled across this issue and have some things to add:

• I have opened two PRs which reduce the image size of the existing docker container quite a bit: Reduce image size gap-docker-base#32 and Do not store packages-master.tar.gz on disk gap-docker-master#14
• When I added Macaulay2 to the image, the increase in size was only 195MB (Install Macaulay2 in the image gap-docker-base#23 (comment)). I guess the 1.3 GB @fingolfin has measured include lots of dependencies which also get pulled in by other packages like polymake, or something drastically has changed since then.
• In any case, homalg does not work with current versions of Macaulay2 anyway and we have dropped it from our tests -> feel free to drop it from the container.