Alpine static builds using github action built musl-cross-make toolchains

[userdocs]

Hello, first, thank your for the project.

I was recently made aware of this project and I noticed that you deploy Linux static binaries using Alpine and docker images from muslcc/x86_64 that currently use gcc version 10.2.1 20210227 (GCC)

docker run -it muslcc/x86_64:aarch64-linux-musl ash -c 'gcc -v'

As this is similar to some build processes that I use I thought I might try to apply my my custom musl-cross-make toolchains that are configured to use the latest available options from the parent project

CONFIG_SUB_REV = 888c8e3d5f7b
GCC_VER = 11-20211113
BINUTILS_VER = 2.37
MUSL_VER = git-master
GMP_VER = 6.2.1
MPC_VER = 1.2.1
MPFR_VER = 4.1.0

LINUX_VER = 5.15.2

These builds are created from a 100% github action based repo that currently uses triples of specific interest to me but as it's a matrix build I am sure it could be extended to use all supported triples and provide comparable toolchains options.

So i build my own toolchains using github and they are provided as releases here - https://github.com/userdocs/musl-cross-make/releases/latest

I then used them to build btop - https://github.com/userdocs/btop-crossbuilds/releases/latest

One successful, full toolchain action, using all supported triples could provide the option to deploy a musl static binary using gcc 11 and provide you more control over the build and release than those docker images.

My builds in the examples are working, based on existing and demonstrable actions so i think the idea is a achievable with out too much drama. It was not too hard for to adapt my workflow to build your app.

[userdocs]

I thought i'd try the full 51 triples available and i got 50/51

https://github.com/userdocs/musl-cross-make/runs/4256989016?check_suite_focus=true

The riscv32-linux-musl failure appears muslc related so i'll need to pick that up with the dev over at gitlab

2021-11-18T22:21:38.3505460Z checking target system type... riscv32-linux-musl
2021-11-18T22:21:38.3507026Z ../src_musl/configure: unknown or unsupported target "riscv32-linux-musl"
2021-11-18T22:21:38.3519862Z make[1]: *** [Makefile:238: obj_musl/.lc_configured] Error 1
2021-11-18T22:21:38.3521213Z make[1]: Leaving directory '/root/musl-cross-make/build/local/riscv32-linux-musl'
2021-11-18T22:21:38.3548883Z make: *** [Makefile:194: all] Error 2
2021-11-18T22:21:38.3660523Z ##[error]Process completed with exit code 2.

But 50 latest toolchains from a github action is ok I think.

https://github.com/userdocs/musl-cross-make/releases/tag/11-2021

With 46/51 crossbuilds

https://github.com/userdocs/btop-crossbuilds/actions/runs/1478838385

https://github.com/userdocs/btop-crossbuilds/releases/tag/v1.1.1

[userdocs]

As far as i understand this:

tl;dr: You got lucky in not including the riscv32 patches in your build, and should build that target last after temporarily renaming the directory containing those patches while you build the rest of the toolchains. The latest patch set (in testing still, incomplete) is for git-b76f37fd5625d038141b52184956fb4b7838e9a5.

When i set MUSL_VER = git-b76f37fd5625d038141b52184956fb4b7838e9a5 it will use all the patches in the folder

https://git.zv.io/toolchains/musl-cross-make/-/tree/master/patches/musl-git-b76f37fd5625d038141b52184956fb4b7838e9a5

So what i am confused about is, if i am to set this in my config.mak

MUSL_VER = git-b76f37fd5625d038141b52184956fb4b7838e9a5

But then rename the dir as to avoid using the patches

patches/musl-git-b76f37fd5625d038141b52184956fb4b7838e9a5

I am not using any of those patches?

Am i missing something here about what you said?

I want to use the patches but i don't understand how if i rename the dir?

[zv-io]

The patches are applied only if the names match exactly. For example, MUSL_VER = 1.2.2 looks for a directory called patches/musl-1.2.2/. For git commits, patches/musl-git-FOO/. If the directory does not exist, then it silently does not apply.

I usually just append .bak to the directory before building, then rename it back, and push that to the repo.

[userdocs]

I think i understand that part but the bit i am struggling with is understanding how to implement this

These riscv32 patches, if enabled, must not be included when building other targets. Imagine you started with MUSL_VER = git-b76f37fd5625d038141b52184956fb4b7838e9a5 and built only the riscv32-linux-musl target. Now you want to build the others, so you must rename patches/musl-git-b76f37fd5625d038141b52184956fb4b7838e9a5 temporarily and start with a pristine tree.

So if i do those for the other 50 targets, there are no musl patches applied, and there are 14 patches here:

https://git.zv.io/toolchains/musl-cross-make/-/tree/master/patches/musl-git-b76f37fd5625d038141b52184956fb4b7838e9a5

7 are have riscv in the name so i assume these are specific to this target, but what about the other 7? do i need them? are all 14 riscv specifc?

I don't understand how using this

MUSL_VER = git-b76f37fd5625d038141b52184956fb4b7838e9a5

Provides access to any patches if I rename to folder for 50/51 targets.

Instead of rename the folder should I delete the patches 08-14

This is what I am trying to underhand. Which patches I need for musl, for all 51 targets,

[zv-io]

The current musl matches (patches/musl-git-b76f37fd5625d038141b52184956fb4b7838e9a5/) are currently:

• Only for riscv32 and highly experimental
• Unique in that they will break other targets in subtle ways if they're applied
  Normally the patches are non-disruptive in that they can safely be applied without undesirable side-effects. For now this is not true of this patch set.

The names (first 7 vs second 7) are not reliable. This entire patch set is for riscv32 only. There are three additional patches that are not included yet because I haven't finished testing them.

Because these 14 patches are all riscv32-specific, renaming is necessary for the other 50 targets.

Only apply them for riscv32-linux-musl.

[userdocs]

Ok, I understand how it works now. So I did that and I got all 51/51 https://github.com/userdocs/musl-cross-make/runs/4274867415?check_suite_focus=true

Since that particular commit is also the latest in the master branch the outcome is effectively the same for these build as I still used = git-master for the other 50. In future ill look to see if the patches or non breaking but if not match the commit.

I imported the repo over here and i'll try to mimic the workflow and see if i can set up a like for like CI.

edit: fixed link

[aristocratos]

@userdocs
Hi, the current release builds are scripted and built locally on my dev machine (for more control and quicker debugging) and I will swap over to use your toolchain for the these builds.

Changing the "Continuous Build Linux" workflow to use a musl gcc 11 toolchain seems like a great idea, I not sure if it really matters that much if 32-bit RISC can't be built, I'm gonna assume that anyone running that platform are likely gonna be comfortable compiling btop themselves if need be :)

I would like to ask @jan-guenter who setup the current workflow if he has any thoughts on the subject?

[userdocs]

The workflow here has a larger matrix so I tried to build against that https://github.com/aristocratos/btop/blob/main/.github/workflows/continuous-build-linux.yml

I was mostly successful in this but if you are using more standard targets to actually release (14 linux targets?) https://github.com/aristocratos/btop/releases/tag/v1.1.1

You also do release a btop-1.1.1-riscv32-linux-musl.tbz build so i was not sure how important it was to achieve this

[aristocratos]

I think he vast majority of people is probably after the x86 and arm based builds, the other targets included in the release builds I mostly just threw in there, so no worries about removing any that are troublesome.

[userdocs]

If that is the case and since you are a small binary you have the option of full emulation builds as i mentioned to @jan-guenter

https://github.com/userdocs/iperf3-static/blob/master/.github/workflows/multiarch_qemu_iperf3.yml

arch: [i386, amd64, arm32v6, arm32v7, arm64v8, ppc64le, s390x]

From this method (which is part of my matrix build) https://github.com/multiarch/qemu-user-static

Using these dockers:

https://hub.docker.com/r/s390x/alpine/
https://hub.docker.com/r/ppc64le/alpine/
https://hub.docker.com/r/arm64v8/alpine/
https://hub.docker.com/r/arm32v7/alpine/
https://hub.docker.com/r/arm32v6/alpine/
https://hub.docker.com/r/amd64/alpine/
https://hub.docker.com/r/i386/alpine/

The caveat with this method is the build time can be greatly increased but since you are a small binary taking about 2 minutes i don't think it really matters if it takes 5-10 minutes instead.

So this is another valid option but you lose many of the targets available from musl-cross

[userdocs]

on no, ignore me. we don't get access to gcc 11 this way and that is why we are using musl-cross. I forgot that part.

[aristocratos]

The build method for the release builds doesn't really matter that much since I will be running them locally, so will just have to change my current build script a bit to use your toolchain instead.

However the continuous build workflow I intend for to include as many architectures as possible.

[jan-guenter]

In general I think it's a good idea to upgrade gcc, however I have to wonder why neither Alpine nor muslcc have yet released a gcc 11 version. For the past releases there have always been some patches necessary to ensure a fully working toolchain. (see the patches directory of musl-cross-make)

@zv-io, is there a special reason why muslcc has not been built with gcc 11 yet? Are you aware of any known complications using gcc 11 and musl?

@userdocs, were you able to complete the gcc test suit with your toolchain builds?
On the first glance I see nothing wrong the builds. I'll take closer look tonight and try to run the gcc test suits myself.

@userdocs, is there a reason why your github action uses docker manually instead of container.image and then running the steps directly inside the container? Also what's the qemu installation used for?

@aristocratos

Also, I think if we copy over README.md and the Makefile to the btop-completed directory, we've got a working installer from the workflow.

Told you some time ago that you could automate the whole release process ;-)
Perhaps another make target like package or similar producing the installer archive would be beneficial for a simple automated package build.

[zv-io]

This is very interesting. I wonder if it is related to another GCC bug I found; I will spend some time today bisecting GCC first and then binutils once I get more information.

[zv-io]

or1k-linux-musl was 100% unmodified; the sh* variants have a nasty patch that I'll need to review more closely.

[zv-io]

The file patches/gcc-11-20211120/0004-static-pie.diff triggers this issue.

[zv-io]

This is definitely a GCC regression and I'll triage it after work today. The good news is this isn't an unrelated issue :)

[userdocs]

I rebuilt the github toolchain using current settings and i did not get these failures as my results were consistent with the previous toolchain version:

https://github.com/userdocs/btop-crossbuilds/actions/runs/1495346456

47/51 but if we minus microblaze as it's not built here then I am at 47/49 with no seg faults using an identical configuration to your release.

I'm not sure what it means but i think the consistent outcome has some value?

[zv-io]

@aristocratos

Honestly, I see no issue in using @userdocs's toolchain, or is there some reasoning I'm missing for waiting for the future musl.cc gcc11 release?

There should be little or no issue in using these toolchains for your project here. But please be warned, that if you promote them to other developers for use in their projects, that's potentially a lot of untested and experimental code making its way into the wild and there may be subtle breakage. So my recommendation is use it here if you need bleeding-edge, but don't promote it elsewhere.

The toolchains on musl.cc are built with very specific versions of core system utilities and libraries.

@userdocs

... but you might need a more modern option 6 months later and musl.cc might not release until 12 months later. This is a perfectly normal scenario to consider.

Prior to this past spring, releases were done every 1-2 months. I want to get back into that cadence.

@jan-guenter

Alternatively you could take up Zachs offer of providing the RC versions of his toolchain builds, which could provide him with better feedback in case of errors than a 3rd party build.

This won't be necessary.

[zv-io]

@userdocs it's been on my TODO list /forever/ to set up automatic build/test infrastructure on my end, which will make it easy to plumb in packaging (e.g. Docker, .deb, .rpm, etc.). I would be doing this on GitLab, though. Would you have any interest in helping with that?

[userdocs]

The only issue is I have never used Gitlab CI and one reason is that it's quite different from the github workflow syntax. Also there are many actions I use on github for creating debs and stuff. So I have no idea how this is done on GitLab and would need to learn.

I am open minded to trying but I guess I'll have to start with a basic goal and go figure it out over there. Perhaps like exporting the current workflow to GitLab and getting it to work and provide an identical outcome

[zv-io]

OK. For now, I'll work on this next release and poke at automation later. Are you in IRC?

[userdocs]

I am loitering in a few libera.chat channels or i use discord (which i prefer) or even matrix (via element)

[jan-guenter]

@zv-io @userdocs I work in DevOps now (former c++ dev) and have been working with an on-premise GitLab in our company since about 6 years now. If you like my help in setting up the automation feel free to contact me. @zv-io I also did the setup of our CI network and can lend a hand with the gitlab-runner setup for your instance.
My usual communication channels are email or Discord, but I can also hop on an IRC if you like.