Intermittent build failures due to inconsistent redirects on apt.armbian.com

[matthijskooijman]

TL;DR: apt.armbian.com redirects to different mirrors for different requests, leading to apt update and build failures.

Not sure if this is the right place for this report, but the repo title seemed appropriate.

Over the last weeks, I've been seeing some intermittent build failures, related to failed apt updates inside the rootfs chroot. Usually it helped to just restart the build, but yesterday they were quite persistent (something like 6 out of 10 attempts failed), so I decided to investigate.

The error I get in install.log is:

E: Failed to fetch https://imola.armbian.com/apt/dists/focal/main/binary-armhf/Packages.bz2  File has unexpected size (199533 != 123546). Mirror sync in progress? [IP: 93.103.15.56 443]
   Hashes of expected file:
    - Filesize:123546 [weak]
    - SHA512:cfa454cb1bbcb61b47ca00d2b6970feaf9ab7085740b864d7e021b5a6d2f830097703c7fe86a4e591508bab6bc3fce7a8edc1484df62767cdce169b076712226
    - SHA256:65c20abb7cdf3a958056f6e6c8420b6c64d2972db8ff06a3c4ae1f839665a831
    - SHA1:12fb81cbdd36bc04236551dcee503ed42ff25d09 [weak]
    - MD5Sum:06a04d0d482b380776dc4fb5f292d5a1 [weak]
   Release file created at: Tue, 01 Feb 2022 17:25:15 +0000

I've also seen 404 errors occasionally, which I suspect are the result of the same underlying issue.

Apt seems to be configured (by default) to use apt.armbian.com, which redirects to some (local?) mirror. However, this redirect is not consistent, in that it seems to decide which mirror to use for each request separately, rather than consistently redirecting the same client (IP) to the same mirror all the time time.

This approach seems to cause a problem when the different mirrors are not completely in sync. What I saw happening, is that the InRelease file is fetched from one mirror, and then the Packages file is fetched from another mirror. If these mirrors are not exactly in sync, the checksums from the InRelease file will not match the Packages file, failing the update and thus the build.

It seems like this should be fixed by adding some stability to the redirection mechanism (i.e. when there are multiple options based on the geolocation, use a hash of the client IP or so to decide on the mirror to use?).

Running apt -oDebug::Acquire::http=1 update inside the chroot manually shows the problem clearly. Here's an exerpt, full log at https://gist.github.com/3683adddbd1dbf8dd3e2412e9cd6c034

Answer for: http://apt.armbian.com/dists/focal/InRelease
HTTP/1.1 302 FOUND^M
Server: nginx/1.18.0 (Ubuntu)^M
Date: Mon, 07 Mar 2022 21:49:00 GMT^M
Content-Type: text/html; charset=utf-8^M
Content-Length: 328^M
Connection: keep-alive^M
Location: http://xogium.performanceservers.nl/apt/dists/focal/InRelease^M

...

Answer for: http://apt.armbian.com/dists/focal/main/binary-armhf/Packages
HTTP/1.1 302 FOUND^M
Server: nginx/1.18.0 (Ubuntu)^M
Date: Mon, 07 Mar 2022 21:49:20 GMT^M
Content-Type: text/html; charset=utf-8^M
Content-Length: 340^M
Connection: keep-alive^M
Location: http://imola.armbian.com/apt/dists/focal/main/binary-armhf/Packages^M

[umiddelb]

Have you tried
sudo apt clean
so far?

[matthijskooijman]

Have you tried
sudo apt clean
so far?

I had not yet, and just did, without success.

I did investigate the state of the various mirrors a bit more, and found that it's just not that some mirrors are less up-to-date than others, but also that some mirrors are actually internally inconsistent (InRelease file checksums not matching the actual files). However, I believe that this internal inconsistency of mirrors is not actually required for the problem described in this issue (inconsistent redirects by apt.armbian.com) to cause issues (just unequal sync between mirrors is needed for this issue to occur). The current (long-standing, already since Feb 1) internally inconsistent mirrors do make this issue easy to reproduce currently.

Since the mirror state is a separate issue, I filed that in (what I think is) a more appropriate place: armbian/mirror#7

As I said, I still believe that redirecting different requests to different mirrors (as shown above) by apt.armbian.com is problematic and should be fixed, regardless of the mirror sync issues.

[umiddelb]

Hm,

I've experienced the same issue and `sudo apt clean´ fixed the checksum mismatch.

[matthijskooijman]

I've experienced the same issue and `sudo apt clean´ fixed the checksum mismatch.

I suspect that the retry just just got you redirected to a different combination of mirrors that did not show the problem (from the looks of it, if you get an up-to-date InRelease file from e.g. imola.armbian.com, then downloading the other files from any other mirror works ok). IOW, it would also have worked if you had just retried and omitted the clean. I think.