Intermittent build failures due to inconsistent redirects on apt.armbian.com

[matthijskooijman]

TL;DR: apt.armbian.com redirects to different mirrors for different requests, leading to apt update and build failures.

Not sure if this is the right place for this report, but the repo title seemed appropriate.

Over the last weeks, I've been seeing some intermittent build failures, related to failed apt updates inside the rootfs chroot. Usually it helped to just restart the build, but yesterday they were quite persistent (something like 6 out of 10 attempts failed), so I decided to investigate.

The error I get in install.log is:

E: Failed to fetch https://imola.armbian.com/apt/dists/focal/main/binary-armhf/Packages.bz2  File has unexpected size (199533 != 123546). Mirror sync in progress? [IP: 93.103.15.56 443]
   Hashes of expected file:
    - Filesize:123546 [weak]
    - SHA512:cfa454cb1bbcb61b47ca00d2b6970feaf9ab7085740b864d7e021b5a6d2f830097703c7fe86a4e591508bab6bc3fce7a8edc1484df62767cdce169b076712226
    - SHA256:65c20abb7cdf3a958056f6e6c8420b6c64d2972db8ff06a3c4ae1f839665a831
    - SHA1:12fb81cbdd36bc04236551dcee503ed42ff25d09 [weak]
    - MD5Sum:06a04d0d482b380776dc4fb5f292d5a1 [weak]
   Release file created at: Tue, 01 Feb 2022 17:25:15 +0000

I've also seen 404 errors occasionally, which I suspect are the result of the same underlying issue.

Apt seems to be configured (by default) to use apt.armbian.com, which redirects to some (local?) mirror. However, this redirect is not consistent, in that it seems to decide which mirror to use for each request separately, rather than consistently redirecting the same client (IP) to the same mirror all the time time.

This approach seems to cause a problem when the different mirrors are not completely in sync. What I saw happening, is that the InRelease file is fetched from one mirror, and then the Packages file is fetched from another mirror. If these mirrors are not exactly in sync, the checksums from the InRelease file will not match the Packages file, failing the update and thus the build.

It seems like this should be fixed by adding some stability to the redirection mechanism (i.e. when there are multiple options based on the geolocation, use a hash of the client IP or so to decide on the mirror to use?).

Running apt -oDebug::Acquire::http=1 update inside the chroot manually shows the problem clearly. Here's an exerpt, full log at https://gist.github.com/3683adddbd1dbf8dd3e2412e9cd6c034

Answer for: http://apt.armbian.com/dists/focal/InRelease
HTTP/1.1 302 FOUND^M
Server: nginx/1.18.0 (Ubuntu)^M
Date: Mon, 07 Mar 2022 21:49:00 GMT^M
Content-Type: text/html; charset=utf-8^M
Content-Length: 328^M
Connection: keep-alive^M
Location: http://xogium.performanceservers.nl/apt/dists/focal/InRelease^M

...

Answer for: http://apt.armbian.com/dists/focal/main/binary-armhf/Packages
HTTP/1.1 302 FOUND^M
Server: nginx/1.18.0 (Ubuntu)^M
Date: Mon, 07 Mar 2022 21:49:20 GMT^M
Content-Type: text/html; charset=utf-8^M
Content-Length: 340^M
Connection: keep-alive^M
Location: http://imola.armbian.com/apt/dists/focal/main/binary-armhf/Packages^M

[matthijskooijman]

Oh, just found https://github.com/armbian/dl-router, which seems like a better place for this issue. Resubmitted as armbian/dl-router#29.