Server-less build system

[mob-sakai]

The build workflows will be triggered from the versioning-backend server.
However, most of the functionality can be replaced by GitHub Actions.

I propose a Server-less build system.

Summary

'Build All' workflow
'Build Editor' workflow

• Add workflows to build images without the versioning-backend server.
• Check out latest release and build base and hub.
  • The existed image tag building will be skipped.
  • Use skopeo to check for the existence of tags.
• After base and hub images are built, the workflow to build the editor image will be dispatched (on a per-module basis for readability).
  • Build All (build base and hub)
    • (dispatch) -> Build Editor (base, [2018.3.1f1, 2018.3.2f1, ...])
    • (dispatch) -> Build Editor (linux-il2cpp, [2018.3.1f1, 2018.3.2f1, ...])
      ...
    • (dispatch) -> Build Editor (webgl, [2018.3.1f1, 2018.3.2f1, ...])
• This workflow lists the available Unity versions without the versioning-backend server.
  • Use unity-changeset to get the available Unity versions.
  • As workflow settings, we can specify the minimum Unity version and the excluded Unity versions/module and alpha/beta support.
• The build workflows for a new release takes about a day.
  • If no new build is needed, it will take about 5 minutes.
• This workflow will run daily to build for new released Unity versions or to rebuild.
  • If other build workflows ('Build All' or 'Build Editor') are in progress or queued, this workflow will be skipped.
    • It prevents workflow congestion.
  • 'Build Editor' workflow may fail with a timeout, but in most cases the rebuild will succeed.

For example

Trigger	base/hub image	base (=linux)	linux-il2cpp	windows-mono	mac-mono	android	ios	webgl
Release v0.9	✅ (147 Unity versions found)	✅ 147	✅ 147	✅ 147	✅ 145 ❌ 2	✅ 142 ❌ 5	✅ 145 ❌ 2	✅ 139 ❌ 8
Scheduled (rebuild)	✅ (150 Unity versions found)	✅ 3	✅ 3	✅ 3	✅ 5	✅ 8	✅ 5	✅ 11
Scheduled (rebuild)	✅ (150 Unity versions found)	-	-	-	-	-	-	-

• In the 1st run after the release, several builds failed due to timeouts and bandwidth limitations.
• In the 2nd run, not only the failed versions but also the newly released versions (2019.4.20f1, 2020.2.4f1, 2021.1.0b6) were built and published.
• In the 3rd run, all versions had already been released. So there were no additional builds.
• Published images: https://hub.docker.com/repository/docker/mobsakai/unity_editor
  • Supported versions: 2018.3 or later (2021.1 beta included)

[webbertakken]

Note that the backend is already based on cloud functions (serverless), but I'm assuming that's not what you meant.

There is a lot more to the versioning backend than scheduling builds.

Firstly there are several triggers:

• Tagging docker repo (as you mentioned)
• Unity creates a new release
• Unity changes an older release
• Retry after fix without having to rebuild all images (with is about 1TB).

Only these would be hard to cover in actions only.

Then there are several functionalities unique to a programmable backend:

• Easily accessible information about all the builds and when they were modified
• Reporting new builds to discord
• Reporting failures or other useful information in the alert channel (for maintainers)
• Retrying failed builds (currently we retry 14 times for a total of 15 tries)

And what we want to add in the near future:

• Display a list of all available images and using search engine in some frontend
• Inspecting specific failures and having a user friendly (admin) interface for retrying builds
• Adding notes for specific builds, to keep track of per-version caveats

Therefor I do not think it is a good idea to create a very complex actions workflow.

[mob-sakai]

I understood.
To be frank, I just wanted a "customizable" build workflow for myself (for #50). 😄

[webbertakken]

I could give you admin access to the project if you like, so you could reproduce code changes?