Show how to move mkosi WorkspaceDirectory to /tmp(fs)

[marc-hb]

I've been selfishly enjoying this major optimization for a while; time to share it. Very small hence very easy to comment out or even revert if needed.

mkosi-workspace is located on /var/tmp/ by default, which is usually persistent and slow storage which wears out. On success, mkosi-workspace is deleted at the end of the build so it really does not need to be located on persistent storage. So, move it to /tmp which is usually tmpfs.

On one of my systems, moving to tmpfs lowers the build time from 4 min to 1 min! If you ever wondered how I could git bisect mkosi faster than reading its release notes: that's how.

[marc-hb]

cc:

• bwrap --tmpfs option disrupts --workspace-dir=/tmp/foo  systemd/mkosi#1894
• Reuse host /tmp in bwrap() systemd/mkosi#1905

[marc-hb]

According to https://repology.org/project/mkosi/versions, only Fedora 39 shipped version 16. Fedora 39 is EOL now.

[marc-hb]

Debian 13 (2025?) will finally move /tmp to tmpfs
https://lists.debian.org/debian-devel/2024/05/msg00456.html?ref=news.itsfoss.com

Maybe Ubuntu will eventually switch too?
https://ubuntu.com/blog/data-driven-analysis-tmp-on-tmpfs
https://askubuntu.com/questions/62928/why-doesnt-tmp-use-tmpfs

[marc-hb]

This configuration option was first added in mkosi v10 commit https://github.com/systemd/mkosi/commits/59d8d913a5f67d9

At the time, mkosi.md documented it like this: mkosi.workspace/ else $TMPDIR else /var/tmp. It relied on tmp_dir().

[marc-hb]

This is a simplifcation. Since v19, mkosi is trying to default to somewhere in /home in order to use reflinks systemd/mkosi@94c1e78a496713e0

In my experience across multiple versions, tmpfs is just faster - especially when qbuild is symlinked to /tmp/qbuild.real too.

[marc-hb]

@davejiang found a failure with v24.3, looking into it.

[stellarhopper]

In case there are instances where mkosi leaves behind a /tmp/mkosi-workspace*, it might be worth adding a sudo rm -rf to remove it in the -r wipe case.

[marc-hb]

@davejiang found a failure with v24.3, looking into it.

So this setting is incompatible with v24 commit systemd/mkosi@f4eba02dd3ca08

:: Synchronizing package databases...
 core is up to date
 extra        
chmod: changing permissions of '/tmp': Operation not permitted

Bubblewrap does a lot of weird things with bind mounts and what not.

v25 replaced Bubblewrap and works again.

I think I found a workaround. Will that workaround be compatible with other, random mkosi versions? Maybe not. Maybe the best is to make this opt-in. Easy enough to just copy this file to a different place.

[marc-hb]

So this setting is incompatible with v24 commit systemd/mkosi@f4eba02dd3ca08
...
v25 dropped and replaced Bubblewrap and works again.

So this is what seems to happen. I looked into this just to make sure the workaround does not work by chance/accident.

• mkosi/__init__.py#setup_workspace() starts with rmtree($WorkspaceDir/mkosi-workspace-12345)

• tree.py#rmtree(p) runs rm -rf $p in a bubblewrap sandbox where mounts=[(p.parent, p.parent),...]. $WorkspaceDir is the workspace parent p! This means bubblewrap is invoked with --bind /tmp /tmp.

• From v24 commit systemd/mkosi@f4eba02dd3ca08, every bubblewrap invocation starts with (among others) chmod 1777 /tmp where /tmp is harcoded; totally unrelated to $WorkspaceDir.

• That chmod 1777 seems incompatible with a bubblewrap bind mount: Operation not permitted.

So the workaround is very simple; simply add one directory level: WorkspaceDir=/tmp/rq_mkosi_wspaces/. As a bonus this keeps /tmp/ tidier.

Maybe the best is to make this opt-in.

I don't want to debug issues like this again so the next force push will make this opt-in. Everyone can use it or not without polluting their git status.