Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Getting 403 error randomly when installing temurin packages #923

Open
angaaruriakhil opened this issue Oct 20, 2023 · 57 comments
Open

Getting 403 error randomly when installing temurin packages #923

angaaruriakhil opened this issue Oct 20, 2023 · 57 comments
Labels
bug Something isn't working Waiting on OP

Comments

@angaaruriakhil
Copy link

angaaruriakhil commented Oct 20, 2023

We are conducting ubuntu 20.04 image builds using code in the runner-images repository.

When installing temurin JDK packages, we now get 403 errors randomly on differing packages.

We conducted a build yesterday which failed on installing temurin 21 with the following error:

azure-arm: E: Failed to fetch https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-21/temurin-21-jdk_21.0.0.0.0+35_amd64.deb 403 Forbidden [IP: 146.75.107.42 443]

We conducted a build today which failed on temurin 11 with the following error:

azure-arm.build_vhd: E: Failed to fetch https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-11/temurin-11-jdk_11.0.20.1.0%2b1_amd64.deb Error reading from server. Remote end closed connection [IP: 146.75.107.42 443]

We have also had some successful builds where all the packages install.

To Reproduce

Run code in java-tools.sh to install temurin packages.

Expected behavior
Every package is able to be installed reliably

@angaaruriakhil angaaruriakhil added the bug Something isn't working label Oct 20, 2023
@gdams gdams transferred this issue from adoptium/api.adoptium.net Oct 20, 2023
@karianna
Copy link
Contributor

@angaaruriakhil Can you try updating you ca-certificates package and try again?

@KrustyDaClown
Copy link

KrustyDaClown commented Oct 23, 2023

Got the same issue here - even after
sudo apt-get install ca-certificates; sudo update-ca-certificates

It worked after an sudo apt update --fix-missing

@karianna
Copy link
Contributor

@gdams over to you - I can only imagine a CDN issue?

@jlgoolsbee
Copy link

We are intermittently seeing this issue as well, specifically via yum but I was able to reproduce (albeit with somewhat different error messaging) with curl as well (output below).

Example yum install failure:

$ sudo yum install -y temurin-11-jdk
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd

# [ dependency resolution omitted for brevity's sake ]

Total size: 160 M
Total download size: 159 M
Installed size: 314 M
Downloading packages:
No Presto metadata available for Adoptium
temurin-11-jdk-11.0.20.1.0.1-1 FAILED                                          
https://packages.adoptium.net/artifactory/rpm/amazonlinux/2/x86_64/Packages/temurin-11-jdk-11.0.20.1.0.1-1.x86_64.rpm: [Errno 14] HTTPS Error 403 - Forbidden                                                                                                 ]  0.0 B/s |    0 B  --:--:-- ETA 
Trying other mirror.


Error downloading packages:
  temurin-11-jdk-11.0.20.1.0.1-1.x86_64: [Errno 256] No more mirrors to try.

I encountered this failure several times over starting at 14:55 UTC today, but in my case only for the temurin-11-jdk package; I was able to successfully download and install temurin-11-jre as well as temurin-17-jdk, but temurin-11-jdk continued to fail. As had been suggested on this issue and issue #554, I tried both updating the ca-certificates package and running update-ca-trust – neither of which updated anything – but installation of temurin-11-jdk continued to fail. At this point I tried to curl the failing URL, and I did not receive a 403, but I also wasn't able to download the RPM:

$ curl -vLO https://packages.adoptium.net/artifactory/rpm/amazonlinux/2/x86_64/Packages/temurin-11-jdk-11.0.20.1.0.1-1.x86_64.rpm
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 146.75.39.42:443...
* Connected to packages.adoptium.net (146.75.39.42) port 443
* ALPN: curl offers h2,http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
{ [5 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [100 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [3978 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=packages.adoptium.net
*  start date: Oct  2 11:22:18 2023 GMT
*  expire date: Dec 31 11:22:17 2023 GMT
*  subjectAltName: host "packages.adoptium.net" matched cert's "packages.adoptium.net"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
} [5 bytes data]
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://packages.adoptium.net/artifactory/rpm/amazonlinux/2/x86_64/Packages/temurin-11-jdk-11.0.20.1.0.1-1.x86_64.rpm
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: packages.adoptium.net]
* [HTTP/2] [1] [:path: /artifactory/rpm/amazonlinux/2/x86_64/Packages/temurin-11-jdk-11.0.20.1.0.1-1.x86_64.rpm]
* [HTTP/2] [1] [user-agent: curl/8.3.0]
* [HTTP/2] [1] [accept: */*]
} [5 bytes data]
> GET /artifactory/rpm/amazonlinux/2/x86_64/Packages/temurin-11-jdk-11.0.20.1.0.1-1.x86_64.rpm HTTP/2
> Host: packages.adoptium.net
> User-Agent: curl/8.3.0
> Accept: */*
> 
{ [5 bytes data]
< HTTP/2 200 
< x-amz-id-2: H5ntxCxOdyFhTEZa+JrnjbA2V9VdbpXoFdCeekzjgSGgWSBi/YUbTChNj90JUOuqUxyWHrANR+A=
< x-amz-request-id: QJZ3FQGCMJW8Z84N
< x-amz-replication-status: FAILED
< last-modified: Thu, 31 Aug 2023 12:51:43 GMT
< etag: "71b38a94dbeff6f692cf13a3b7038f35"
< x-amz-server-side-encryption: AES256
< x-amz-version-id: umzWdTNJmMDdvnR0unSw_fRTy4j1NfZd
< content-disposition: attachment;filename="temurin-11-jdk-11.0.20.1.0.1-1.x86_64.rpm"
< content-type: application/x-rpm
< server: AmazonS3
< cache-control: public, max-age=15780096.000
< fastly-restarts: 1
< accept-ranges: bytes
< date: Tue, 24 Oct 2023 15:13:19 GMT
< via: 1.1 varnish
< age: 3022
< x-served-by: cache-iad-kcgs7200148-IAD
< x-cache: HIT
< x-cache-hits: 1
< x-timer: S1698160399.397782,VS0,VE2
< strict-transport-security: max-age=31557600
< content-length: 166352397
< 
{ [5 bytes data]
* HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)
 60  158M   60 96.0M    0     0   166M      0 --:--:-- --:--:-- --:--:--  166M
* Connection #0 to host packages.adoptium.net left intact
curl: (92) HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)

This failed command actually did download 96M – other attempts downloaded different amounts – but it always failed. The error message implying that the stream wasn't closed cleanly leads me to wonder if the cached asset is corrupt, or perhaps isn't fully-distributed/cached by the CDN? I'm not sure if/how yum is interpreting this as a 403, or if this error is somehow unique/different... meanwhile, downloading the RPM for temurin-17-jdk via curl worked just fine:

$ curl -vLO https://packages.adoptium.net/artifactory/rpm/amazonlinux/2/x86_64/Packages/temurin-17-jdk-17.0.8.1.0.1-1.x86_64.rpm
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 146.75.35.42:443...
* Connected to packages.adoptium.net (146.75.35.42) port 443
* ALPN: curl offers h2,http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
} [5 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
{ [5 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [100 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [3978 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=packages.adoptium.net
*  start date: Oct  2 11:22:18 2023 GMT
*  expire date: Dec 31 11:22:17 2023 GMT
*  subjectAltName: host "packages.adoptium.net" matched cert's "packages.adoptium.net"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
} [5 bytes data]
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://packages.adoptium.net/artifactory/rpm/amazonlinux/2/x86_64/Packages/temurin-17-jdk-17.0.8.1.0.1-1.x86_64.rpm
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: packages.adoptium.net]
* [HTTP/2] [1] [:path: /artifactory/rpm/amazonlinux/2/x86_64/Packages/temurin-17-jdk-17.0.8.1.0.1-1.x86_64.rpm]
* [HTTP/2] [1] [user-agent: curl/8.3.0]
* [HTTP/2] [1] [accept: */*]
} [5 bytes data]
> GET /artifactory/rpm/amazonlinux/2/x86_64/Packages/temurin-17-jdk-17.0.8.1.0.1-1.x86_64.rpm HTTP/2
> Host: packages.adoptium.net
> User-Agent: curl/8.3.0
> Accept: */*
> 
{ [5 bytes data]
< HTTP/2 200 
< x-amz-id-2: xUHOk3FxG3CALkYV/8zV6BjfkJl7o9PVSMO6jAXu1BBEKzEi8qRBbhPN+6SrCDSncmHJE9JYhqc=
< x-amz-request-id: 9FDX85EN3B8YFEC4
< x-amz-replication-status: FAILED
< last-modified: Thu, 31 Aug 2023 14:32:03 GMT
< etag: "a8ed74787d123dc1a9664b59a4c14ca2"
< x-amz-server-side-encryption: AES256
< x-amz-version-id: Z1zAzlEUB_ubAmeOULSp6EX2L.7XXn9A
< content-disposition: attachment;filename="temurin-17-jdk-17.0.8.1.0.1-1.x86_64.rpm"
< content-type: application/x-rpm
< server: AmazonS3
< cache-control: public, max-age=15780096.000
< fastly-restarts: 1
< accept-ranges: bytes
< date: Tue, 24 Oct 2023 16:12:30 GMT
< via: 1.1 varnish
< age: 0
< x-served-by: cache-iad-kjyo7100121-IAD
< x-cache: MISS
< x-cache-hits: 0
< x-timer: S1698163950.013750,VS0,VE913
< strict-transport-security: max-age=31557600
< content-length: 162689473
< 
{ [5 bytes data]
100  155M  100  155M    0     0  9496k      0  0:00:16  0:00:16 --:--:-- 10.2M
* Connection #0 to host packages.adoptium.net left intact

Diffing the two downloads, I noted this first download of temurin-17-jdk-17.0.8.1.0.1-1.x86_64.rpm showed a cache MISS; subsequent requests for the same URL were also successful and showed a cache HIT (as did all my requests for temurin-11-jdk-11.0.20.1.0.1-1.x86_64.rpm), but otherwise they appear functionally similar (no appreciable difference that might point to a reason for the "stream 1 was not closed cleanly" errors that I can see, anyways).

I lost track of the exact time while I was debugging all this, but at some point between 20-30 minutes later, a stray sudo yum install -y temurin-11-jdk completed successfully, and as I can't point to anything I did which resolved this issue on my end, I'm left to assume this issue must be present somewhere on the Artifactory/S3/Fastly-side of things.

@angaaruriakhil
Copy link
Author

angaaruriakhil commented Oct 25, 2023

Thanks for the responses folks,

I have not seen this issue this week on our builds, we've had 4 successful builds with temurin 8, 11, 17, and 21 installing on each build without issue. Haven't changed anything in our code since my last post. I'm in agreement with the above that it's possibly artifactory/CDN related...

@jlgoolsbee
Copy link

I don't agree that this issue should be closed – I experienced this issue just yesterday, as documented above – and even if we accept the premise that this is a transient issue, it is still a persistent issue that deserves further investigation.

@gdams gdams reopened this Oct 27, 2023
@KrustyDaClown
Copy link

KrustyDaClown commented Oct 27, 2023

Got the issue again. This time a variable ${VERSION} in the download URL target seems not to be resolved:

olger@phobos:~$ sudo apt upgrade
Paketlisten werden gelesen… Fertig
Abhängigkeitsbaum wird aufgebaut… Fertig
Statusinformationen werden eingelesen… Fertig
Paketaktualisierung (Upgrade) wird berechnet… Fertig
Die folgenden Pakete werden aktualisiert (Upgrade):
  temurin-17-jdk temurin-21-jdk
2 aktualisiert, 0 neu installiert, 0 zu entfernen und 0 nicht aktualisiert.
Es müssen 343 MB an Archiven heruntergeladen werden.
Nach dieser Operation werden 197 kB Plattenplatz zusätzlich benutzt.
Möchten Sie fortfahren? [J/n] 
Fehl:1 https://packages.adoptium.net/artifactory/deb bookworm/main amd64 temurin-17-jdk amd64 17.0.9.0+9
  400  Bad Request [IP: 146.75.123.42 443]
Fehl:2 https://packages.adoptium.net/artifactory/deb bookworm/main amd64 temurin-21-jdk amd64 21.0.1.0.0+12
  400  Bad Request [IP: 146.75.123.42 443]
E: Fehlschlag beim Holen von https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-${VERSION}/temurin-17-jdk_17.0.9.0%2b9_amd64.deb 400  Bad Request [IP: 146.75.123.42 443]
E: Fehlschlag beim Holen von https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-${VERSION}/temurin-21-jdk_21.0.1.0.0%2b12_amd64.deb 400  Bad Request [IP: 146.75.123.42 443]
E: Einige Archive konnten nicht heruntergeladen werden; vielleicht »apt-get update« ausführen oder mit »--fix-missing« probieren?

Is says "failed to get"
https//packages.adoptium.net/artifactory/deb/pool/main/t/temurin-${VERSION}/temurin-17-jdk_17.0.9.0%2b9_amd64.deb
https//packages.adoptium.net/artifactory/deb/pool/main/t/temurin-${VERSION}/temurin-21-jdk_21.0.1.0.0%2b12_amd64.deb
and I guess the string "${VERSION}" is not intended to be part of the URL...

@geenweb
Copy link

geenweb commented Oct 27, 2023

I'm having the same issue on Linux Mint 20 (Ubuntu 20.04)

The following packages will be upgraded:
  temurin-17-jdk temurin-17-jre
2 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 195 MB of archives.
After this operation, 2,048 B disk space will be freed.
Err:1 https://packages.adoptium.net/artifactory/deb focal/main amd64 temurin-17-jdk amd64 17.0.9.0+9
  400  Bad Request [IP: 151.101.131.42 443]
Err:2 https://packages.adoptium.net/artifactory/deb focal/main amd64 temurin-17-jre amd64 17.0.9.0+9
  400  Bad Request [IP: 151.101.131.42 443]
E: Failed to fetch https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-${VERSION}/temurin-17-jdk_17.0.9.0+9_amd64.deb  400  Bad Request [IP: 151.101.131.42 443]
E: Failed to fetch https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-${VERSION}/temurin-17-jre_17.0.9.0+9_amd64.deb  400  Bad Request [IP: 151.101.131.42 443]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

I updated certificates and tried --fix-missing. No joy.

@dustingrady
Copy link

Same issue here on Ubuntu 22.04 after updating ca-certs and trying --fix-missing

Err:1 https://packages.adoptium.net/artifactory/deb jammy/main amd64 temurin-17-jdk amd64 17.0.9.0+9
  400  Bad Request [IP: 151.101.3.42 443]
E: Failed to fetch https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-${VERSION}/temurin-17-jdk_17.0.9.0%2b9_amd64.deb  400  Bad Request [IP: 151.101.3.42 443]

@karianna
Copy link
Contributor

Hi folks, we're aware of the issue and are investigating with our CDN and package repo providers.

@gdams
Copy link
Member

gdams commented Oct 28, 2023

To anyone making an update to a URL that contains -${VERSION} this was an upload error which we've subsequently removed. It appears that artifactory didn't recalculate the index so I've triggered this manually which should fix most user issues. Please can people that were hitting this URL try updating their packages again?

The second bug is around the different behaviour that artifactory serves users if they are in a browser or on a command line. If you make a request in the browser it creates a 301 redirect to a /ui route so that you can explore the repo in artifactory whereas in the command line, it should always serve the binary. I’ve managed to create some code that synthetically handles these 301’s for browser users which should mean that Fastly stops caching the wrong response.

@geenweb
Copy link

geenweb commented Oct 28, 2023

Works now. Thank you.

@repudi8or
Copy link

this is not solved...

wget https://packages.adoptium.net/artifactory/rpm/amazonlinux/2/x86_64/Packages/temurin-17-jdk-17.0.9.0.0.9-1.x86_64.rpm
--2023-10-31 12:38:49-- https://packages.adoptium.net/artifactory/rpm/amazonlinux/2/x86_64/Packages/temurin-17-jdk-17.0.9.0.0.9-1.x86_64.rpm
Resolving packages.adoptium.net (packages.adoptium.net)... 151.101.23.42
Connecting to packages.adoptium.net (packages.adoptium.net)|151.101.23.42|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 162838233 (155M) [application/x-rpm]
Saving to: ‘temurin-17-jdk-17.0.9.0.0.9-1.x86_64.rpm.2’

61% [====================================================================> ] 100,663,296 312MB/s in 0.3s

2023-10-31 12:38:50 (312 MB/s) - Connection closed at byte 100663296. Retrying.

--2023-10-31 12:38:51-- (try: 2) https://packages.adoptium.net/artifactory/rpm/amazonlinux/2/x86_64/Packages/temurin-17-jdk-17.0.9.0.0.9-1.x86_64.rpm
Connecting to packages.adoptium.net (packages.adoptium.net)|151.101.23.42|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2023-10-31 12:38:51 ERROR 403: Forbidden.

wget https://packages.adoptium.net/artifactory/rpm/amazonlinux/2/x86_64/Packages/temurin-17-jdk-17.0.9.0.0.9-1.x86_64.rpm
--2023-10-31 12:41:40-- https://packages.adoptium.net/artifactory/rpm/amazonlinux/2/x86_64/Packages/temurin-17-jdk-17.0.9.0.0.9-1.x86_64.rpm
Resolving packages.adoptium.net (packages.adoptium.net)... 151.101.23.42
Connecting to packages.adoptium.net (packages.adoptium.net)|151.101.23.42|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 162838233 (155M) [application/x-rpm]
Saving to: ‘temurin-17-jdk-17.0.9.0.0.9-1.x86_64.rpm.3’

61% [====================================================================> ] 100,663,296 294MB/s in 0.3s

2023-10-31 12:41:40 (294 MB/s) - Connection closed at byte 100663296. Retrying.

--2023-10-31 12:41:41-- (try: 2) https://packages.adoptium.net/artifactory/rpm/amazonlinux/2/x86_64/Packages/temurin-17-jdk-17.0.9.0.0.9-1.x86_64.rpm
Connecting to packages.adoptium.net (packages.adoptium.net)|151.101.23.42|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2023-10-31 12:41:41 ERROR 403: Forbidden.
wget --no-cache https://packages.adoptium.net/artifactory/rpm/amazonlinux/2/x86_64/Packages/temurin-17-jdk-17.0.9.0.0.9-1.x86_64.rpm
--2023-10-31 12:43:18-- https://packages.adoptium.net/artifactory/rpm/amazonlinux/2/x86_64/Packages/temurin-17-jdk-17.0.9.0.0.9-1.x86_64.rpm
Resolving packages.adoptium.net (packages.adoptium.net)... 146.75.43.42
Connecting to packages.adoptium.net (packages.adoptium.net)|146.75.43.42|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 162838233 (155M) [application/x-rpm]
Saving to: ‘temurin-17-jdk-17.0.9.0.0.9-1.x86_64.rpm.4’

@angaaruriakhil
Copy link
Author

Hi, we have been experiencing this issue again today.

@gdams gdams reopened this Nov 2, 2023
@gdams
Copy link
Member

gdams commented Nov 2, 2023

I'm talking with Fastly again today, I think that Fastly is caching incomplete requests and then continuing to serve them rather than routing users directly to Artificatory. I'll provide an update here once I have one.

@xsnrg
Copy link

xsnrg commented Nov 2, 2023

Still having this issue. Failed to fetch temurin-21-jdk_21.0.1.0.0%2b12_amd64.deb.

Error reading from server. Remote end closed connection

@gdams
Copy link
Member

gdams commented Nov 3, 2023

Still having this issue. Failed to fetch temurin-21-jdk_21.0.1.0.0%2b12_amd64.deb.

Error reading from server. Remote end closed connection

@xsnrg can you share the full URL that you are trying to request?

@xsnrg
Copy link

xsnrg commented Nov 3, 2023

sure thing. It seems to be working again this morning.

E: Failed to fetch https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-21/temurin-21-jdk_21.0.1.0.0%2b12_amd64.deb  Error reading from server. Remote end closed connection [IP: 146.75.39.42 443]

E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

@ryantse
Copy link

ryantse commented Nov 3, 2023

@gdams: Seems like this issue is happening somewhat frequently for me as well:

22:15:08  �[1;31m==> x86_64-ami-builder: https://packages.adoptium.net/artifactory/rpm/amazonlinux/2/x86_64/Packages/temurin-11-jdk-11.0.20.1.0.1-1.x86_64.rpm: [Errno 14] curl#92 - "HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)"�[0m
22:15:08  �[1;31m==> x86_64-ami-builder: Trying other mirror.�[0m
22:15:08  �[1;31m==> x86_64-ami-builder:�[0m
22:15:08  �[1;31m==> x86_64-ami-builder:�[0m
22:15:08  �[1;31m==> x86_64-ami-builder: Error downloading packages:�[0m
22:15:08  �[1;31m==> x86_64-ami-builder:   temurin-11-jdk-11.0.20.1.0.1-1.x86_64: [Errno 256] No more mirrors to try.�[0m
22:15:08  �[1;31m==> x86_64-ami-builder:�[0m

@inodes
Copy link

inodes commented Nov 15, 2023

This is inconsistent and still requires attention.

Most of the time HTTPS Error 403 - Forbidden:

temurin-11-jdk-11.0.21.0.0.9-1 FAILED
https://packages.adoptium.net/artifactory/rpm/amazonlinux/2/x86_64/Packages/temurin-11-jdk-11.0.21.0.0.9-1.x86_64.rpm: [Errno 14] HTTPS Error 403 - Forbidden                      ]  0.0 B/s |    0 B  --:--:-- ETA
Trying other mirror.

In the very rare case the binary is able to download it often fails with HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)

temurin-11-jdk-11.0.21.0.0.9-1 FAILED                                                       30% [=========================                                                         ]  0.0 B/s |  50 MB  --:--:-- ETA
https://packages.adoptium.net/artifactory/rpm/amazonlinux/2/x86_64/Packages/temurin-11-jdk-11.0.21.0.0.9-1.x86_64.rpm: [Errno 14] curl#92 - "HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)"-:-- ETA
Trying other mirror.

When persistent, there will a rare occasion when it works (which some people posting above when it "appears to be working again".... are likely getting the lucky one off.

Total download size: 161 M
Installed size: 315 M
Downloading packages:
Public key for temurin-11-jdk-11.0.21.0.0.9-1.x86_64.rpm is not installed
--------------------------------------------------------------------------------
Total                                              187 MB/s | 161 MB  00:00
Retrieving key from https://packages.adoptium.net/artifactory/api/gpg/key/public
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : libXau-1.0.8-2.1.amzn2.0.2.x86_64                           1/10
  Installing : libxcb-1.12-1.amzn2.0.2.x86_64                              2/10
  Installing : alsa-lib-1.1.4.1-2.amzn2.x86_64                             3/10
  Installing : libX11-common-1.6.7-3.amzn2.0.4.noarch                      4/10
  Installing : libX11-1.6.7-3.amzn2.0.4.x86_64                             5/10
  Installing : libXext-1.3.3-3.amzn2.0.2.x86_64                            6/10
  Installing : libXi-1.7.9-1.amzn2.0.2.x86_64                              7/10
  Installing : libXtst-1.2.3-1.amzn2.0.2.x86_64                            8/10
  Installing : libXrender-0.9.10-1.amzn2.0.2.x86_64                        9/10
  Installing : temurin-11-jdk-11.0.21.0.0.9-1.x86_64                      10/10
  Verifying  : libXext-1.3.3-3.amzn2.0.2.x86_64                            1/10
  Verifying  : libXrender-0.9.10-1.amzn2.0.2.x86_64                        2/10
  Verifying  : libX11-common-1.6.7-3.amzn2.0.4.noarch                      3/10
  Verifying  : libX11-1.6.7-3.amzn2.0.4.x86_64                             4/10
  Verifying  : libXtst-1.2.3-1.amzn2.0.2.x86_64                            5/10
  Verifying  : alsa-lib-1.1.4.1-2.amzn2.x86_64                             6/10
  Verifying  : libXau-1.0.8-2.1.amzn2.0.2.x86_64                           7/10
  Verifying  : temurin-11-jdk-11.0.21.0.0.9-1.x86_64                       8/10
  Verifying  : libXi-1.7.9-1.amzn2.0.2.x86_64                              9/10
  Verifying  : libxcb-1.12-1.amzn2.0.2.x86_64                             10/10

Installed:
  temurin-11-jdk.x86_64 0:11.0.21.0.0.9-1

@nchammas
Copy link

I am seeing this as well today when installing Adoptium JDK 11 via Flintrock, which installs JDK on instances launched on EC2:

https://packages.adoptium.net/artifactory/rpm/amazonlinux/2/x86_64/Packages/temurin-11-jdk-11.0.21.0.0.9-1.x86_64.rpm: [Errno 14] HTTPS Error 403 - Forbidden
Trying other mirror.


Error downloading packages:
  temurin-11-jdk-11.0.21.0.0.9-1.x86_64: [Errno 256] No more mirrors to try.

Is there some more information I can share to help identify the root cause here?

@mdaloia
Copy link

mdaloia commented Nov 23, 2023

I was getting 403 Forbidden yesterday, some minutes later it worked ok without doing anything different.
This is a very unreliable and flaky networking issue.

I got the chance to get some information in case it is useful to find the root cause by using Ubuntu APT debugging options (-oDebug::Acquire::https=1, for example apt-get install -oDebug::Acquire::https=1 temurin-17-jdk=17.0.9.0.0+9)

Also, I observed that the 403 is not thrown at the very beginning but after some content is already downloaded (~15 %). I guess that APT opens multiple connections to download packages and a one of those workers got the 403 failing the rest of the download:

root@3d636ca4e8cf:/home/user# ls -la /var/cache/apt/archives/partial/
total 16396
drwx------ 1 _apt root     4096 Nov 22 21:36 .
drwxr-xr-x 1 root root     4096 Nov 22 21:29 ..
-rw-r--r-- 1 root root 16777216 Oct 30 14:12 temurin-17-jdk_17.0.9.0.0+9_amd64.deb
root@3d636ca4e8cf:/home/user# rm /var/cache/apt/archives/partial/temurin-17-jdk_17.0.9.0.0+9_amd64.deb

Here are the outputs of APT with debug options when it failed and when it worked:

Failed with 403 downloading latest version
root@3d636ca4e8cf:/home/user# DEBIAN_FRONTEND=noninteractive apt-get install -oDebug::Acquire::https=1 -y --no-install-recommends temurin-17-jdk=17.0.9.0.0+9
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  adoptium-ca-certificates java-common libp11-kit0 p11-kit p11-kit-modules
Suggested packages:
  default-jre
Recommended packages:
  fonts-dejavu-extra
The following NEW packages will be installed:
  adoptium-ca-certificates java-common p11-kit p11-kit-modules temurin-17-jdk
The following packages will be upgraded:
  libp11-kit0
1 upgraded, 5 newly installed, 0 to remove and 68 not upgraded.
Need to get 166 MB/166 MB of archives.
After this operation, 333 MB of additional disk space will be used.
17% [Working]GET /artifactory/deb/pool/main/t/temurin-17/temurin-17-jdk_17.0.9.0.0%2b9_amd64.deb HTTP/1.1
Host: packages.adoptium.net
User-Agent: Debian APT-HTTP/1.3 (1.6.12ubuntu0.1)


Answer for: https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-17/temurin-17-jdk_17.0.9.0.0+9_amd64.deb
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 165543916
x-amz-id-2: 6YwnzVTdwz8yhZkAUYs6NO9vF5PnAQQjCm+5XMRsuNOWikFKSXLJrgAvaI32rRIfzNOALiLxyF8=
x-amz-request-id: YKK0FBYJVA67M8PS
x-amz-replication-status: FAILED
Last-Modified: Mon, 30 Oct 2023 14:12:22 GMT
ETag: "cc468a88d7235d8911184b421b036031"
x-amz-server-side-encryption: AES256
x-amz-version-id: mOcoiHLQ7jeVUun7cgkPTP36Z3b_Hzu8
Content-Type: application/x-debian-package
Server: AmazonS3
Cache-Control: public, max-age=15780096.000
Fastly-Restarts: 1
Accept-Ranges: bytes
Date: Wed, 22 Nov 2023 21:41:36 GMT
Via: 1.1 varnish
Age: 3119
X-Served-By: cache-eze2230026-EZE
X-Cache: HIT
X-Cache-Hits: 1
X-Timer: S1700689297.581823,VS0,VE12

Get:1 https://packages.adoptium.net/artifactory/deb bionic/main amd64 temurin-17-jdk amd64 17.0.9.0.0+9 [166 MB]
24% [1 temurin-17-jdk 13.8 MB/166 MB 8%]GET /artifactory/deb/pool/main/t/temurin-17/temurin-17-jdk_17.0.9.0.0%2b9_amd64.deb HTTP/1.1
Host: packages.adoptium.net
Range: bytes=16777216-
If-Range: Mon, 30 Oct 2023 14:12:22 GMT
User-Agent: Debian APT-HTTP/1.3 (1.6.12ubuntu0.1)


25% [1 temurin-17-jdk 16.8 MB/166 MB 10%]Answer for: https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-17/temurin-17-jdk_17.0.9.0.0+9_amd64.deb
HTTP/1.1 403 Forbidden
Connection: keep-alive
x-amz-request-id: F5S03N8J3B9NJN7T
x-amz-id-2: 9R/nDMoJQWyrBQY14fvoph09ODiZOXaWEtVzKtvsngVZrg0FQIIy06hLuGnHn7NXoxnp88rwOyc=
Content-Type: application/xml
Server: AmazonS3
Fastly-Restarts: 1
Accept-Ranges: bytes
Date: Wed, 22 Nov 2023 21:41:38 GMT
Via: 1.1 varnish
X-Served-By: cache-eze2230058-EZE
X-Cache: MISS
X-Cache-Hits: 0
X-Timer: S1700689298.102621,VS0,VE749
transfer-encoding: chunked

Err:1 https://packages.adoptium.net/artifactory/deb bionic/main amd64 temurin-17-jdk amd64 17.0.9.0.0+9
  403  Forbidden [IP: 151.101.219.42 443]
E: Failed to fetch https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-17/temurin-17-jdk_17.0.9.0.0+9_amd64.deb  403  Forbidden [IP: 151.101.219.42 443]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
root@3d636ca4e8cf:/home/user#
Downloaded latest version some minutes later
root@3d636ca4e8cf:/home/user# DEBIAN_FRONTEND=noninteractive apt-get install -oDebug::Acquire::https=1 -y --no-install-recommends temurin-17-jdk=17.0.9.0.0+9
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  adoptium-ca-certificates java-common p11-kit p11-kit-modules
Suggested packages:
  default-jre
Recommended packages:
  fonts-dejavu-extra
The following NEW packages will be installed:
  adoptium-ca-certificates java-common p11-kit p11-kit-modules temurin-17-jdk
0 upgraded, 5 newly installed, 0 to remove and 68 not upgraded.
Need to get 166 MB of archives.
After this operation, 332 MB of additional disk space will be used.
0% [Working]GET /artifactory/deb/pool/main/a/adoptium-ca-certificates/adoptium-ca-certificates_1.0.2-1_all.deb HTTP/1.1
Host: packages.adoptium.net
User-Agent: Debian APT-HTTP/1.3 (1.6.12ubuntu0.1)


Answer for: https://packages.adoptium.net/artifactory/deb/pool/main/a/adoptium-ca-certificates/adoptium-ca-certificates_1.0.2-1_all.deb
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 2304
Content-Type: application/x-debian-package
X-JFrog-Version: Artifactory/7.71.1 77101900
X-Artifactory-Id: 5da408aece77f41502230d8ecf66e33487a6aab2
X-Artifactory-Node-Id: adoptium-artifactory-primary-0
Last-Modified: Tue, 20 Jun 2023 16:04:14 GMT
ETag: 0347ba993198898a773c8ba4084e8022d3dd1bfb
X-Checksum-Sha1: 0347ba993198898a773c8ba4084e8022d3dd1bfb
X-Checksum-Sha256: f944bcc23005177751212e578437065d6eedbfa57e26b1821b7f12456dcb5413
X-Checksum-Md5: 5db8bf5481fe4ad127e500d9ffc4576f
X-Artifactory-Filename: adoptium-ca-certificates_1.0.2-1_all.deb
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Request-ID: 3554394ae9c346a390b48a1de9713501
Cache-Control: public, max-age=15780096.000
Accept-Ranges: bytes
Date: Wed, 22 Nov 2023 21:50:36 GMT
Via: 1.1 varnish
Age: 1226858
X-Served-By: cache-eze2230064-EZE
X-Cache: HIT
X-Cache-Hits: 2
X-Timer: S1700689837.703204,VS0,VE0

Get:1 https://packages.adoptium.net/artifactory/deb bionic/main amd64 adoptium-ca-certificates all 1.0.2-1 [2304 B]
GET /artifactory/deb/pool/main/t/temurin-17/temurin-17-jdk_17.0.9.0.0%2b9_amd64.deb HTTP/1.1
Host: packages.adoptium.net
User-Agent: Debian APT-HTTP/1.3 (1.6.12ubuntu0.1)


Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 p11-kit-modules amd64 0.23.9-2ubuntu0.1 [234 kB]
Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 p11-kit amd64 0.23.9-2ubuntu0.1 [93.2 kB]
Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 java-common all 0.68ubuntu1~18.04.1 [14.5 kB]
16% [Waiting for headers]Answer for: https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-17/temurin-17-jdk_17.0.9.0.0+9_amd64.deb
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 165543916
x-amz-id-2: I31eBDk1re2GAh0BEOv+Nd0qnZVHNhMCAtTxW0InaJczvUInPLeqy1r81ucR0TlgaX2exAZDsEc=
x-amz-request-id: CX5K6GGZFPCHNFPD
x-amz-replication-status: FAILED
Last-Modified: Mon, 30 Oct 2023 14:12:22 GMT
ETag: "cc468a88d7235d8911184b421b036031"
x-amz-server-side-encryption: AES256
x-amz-version-id: mOcoiHLQ7jeVUun7cgkPTP36Z3b_Hzu8
Content-Type: application/x-debian-package
Server: AmazonS3
Cache-Control: public, max-age=15780096.000
Fastly-Restarts: 1
Accept-Ranges: bytes
Date: Wed, 22 Nov 2023 21:50:40 GMT
Via: 1.1 varnish
Age: 2
X-Served-By: cache-eze2230064-EZE
X-Cache: MISS
X-Cache-Hits: 0
X-Timer: S1700689837.713358,VS0,VE3556

Get:5 https://packages.adoptium.net/artifactory/deb bionic/main amd64 temurin-17-jdk amd64 17.0.9.0.0+9 [166 MB]
Fetched 166 MB in 49s (3365 kB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package p11-kit-modules:amd64.
(Reading database ... 25417 files and directories currently installed.)
Preparing to unpack .../p11-kit-modules_0.23.9-2ubuntu0.1_amd64.deb ...
Unpacking p11-kit-modules:amd64 (0.23.9-2ubuntu0.1) ...
Selecting previously unselected package p11-kit.
Preparing to unpack .../p11-kit_0.23.9-2ubuntu0.1_amd64.deb ...
Unpacking p11-kit (0.23.9-2ubuntu0.1) ...
Selecting previously unselected package adoptium-ca-certificates.
Preparing to unpack .../adoptium-ca-certificates_1.0.2-1_all.deb ...
Unpacking adoptium-ca-certificates (1.0.2-1) ...
Selecting previously unselected package java-common.
Preparing to unpack .../java-common_0.68ubuntu1~18.04.1_all.deb ...
Unpacking java-common (0.68ubuntu1~18.04.1) ...
Selecting previously unselected package temurin-17-jdk.
Preparing to unpack .../temurin-17-jdk_17.0.9.0.0+9_amd64.deb ...
Unpacking temurin-17-jdk (17.0.9.0.0+9) ...
Setting up java-common (0.68ubuntu1~18.04.1) ...
Setting up p11-kit-modules:amd64 (0.23.9-2ubuntu0.1) ...
Setting up p11-kit (0.23.9-2ubuntu0.1) ...
Setting up adoptium-ca-certificates (1.0.2-1) ...
/etc/ssl/certs/adoptium/cacerts successfully populated.
Setting up temurin-17-jdk (17.0.9.0.0+9) ...
update-alternatives: using /usr/lib/jvm/temurin-17-jdk-amd64/bin/jar to provide /usr/bin/jar (jar) in auto mode
update-alternatives: using /usr/lib/jvm/temurin-17-jdk-amd64/bin/jarsigner to provide /usr/bin/jarsigner (jarsigner) in auto mode
update-alternatives: using /usr/lib/jvm/temurin-17-jdk-amd64/bin/java to provide /usr/bin/java (java) in auto mode
update-alternatives: using /usr/lib/jvm/temurin-17-jdk-amd64/bin/javac to provide /usr/bin/javac (javac) in auto mode
update-alternatives: using /usr/lib/jvm/temurin-17-jdk-amd64/bin/javadoc to provide /usr/bin/javadoc (javadoc) in auto mode
update-alternatives: using /usr/lib/jvm/temurin-17-jdk-amd64/bin/javap to provide /usr/bin/javap (javap) in auto mode
update-alternatives: using /usr/lib/jvm/temurin-17-jdk-amd64/bin/jcmd to provide /usr/bin/jcmd (jcmd) in auto mode
update-alternatives: using /usr/lib/jvm/temurin-17-jdk-amd64/bin/jconsole to provide /usr/bin/jconsole (jconsole) in auto mode
update-alternatives: using /usr/lib/jvm/temurin-17-jdk-amd64/bin/jdb to provide /usr/bin/jdb (jdb) in auto mode
update-alternatives: using /usr/lib/jvm/temurin-17-jdk-amd64/bin/jdeprscan to provide /usr/bin/jdeprscan (jdeprscan) in auto mode
update-alternatives: using /usr/lib/jvm/temurin-17-jdk-amd64/bin/jdeps to provide /usr/bin/jdeps (jdeps) in auto mode
update-alternatives: using /usr/lib/jvm/temurin-17-jdk-amd64/bin/jfr to provide /usr/bin/jfr (jfr) in auto mode
update-alternatives: using /usr/lib/jvm/temurin-17-jdk-amd64/bin/jhsdb to provide /usr/bin/jhsdb (jhsdb) in auto mode
update-alternatives: using /usr/lib/jvm/temurin-17-jdk-amd64/bin/jimage to provide /usr/bin/jimage (jimage) in auto mode
update-alternatives: using /usr/lib/jvm/temurin-17-jdk-amd64/bin/jinfo to provide /usr/bin/jinfo (jinfo) in auto mode
update-alternatives: using /usr/lib/jvm/temurin-17-jdk-amd64/bin/jlink to provide /usr/bin/jlink (jlink) in auto mode
update-alternatives: using /usr/lib/jvm/temurin-17-jdk-amd64/bin/jmap to provide /usr/bin/jmap (jmap) in auto mode
update-alternatives: using /usr/lib/jvm/temurin-17-jdk-amd64/bin/jmod to provide /usr/bin/jmod (jmod) in auto mode
update-alternatives: using /usr/lib/jvm/temurin-17-jdk-amd64/bin/jpackage to provide /usr/bin/jpackage (jpackage) in auto mode
update-alternatives: using /usr/lib/jvm/temurin-17-jdk-amd64/bin/jps to provide /usr/bin/jps (jps) in auto mode
update-alternatives: using /usr/lib/jvm/temurin-17-jdk-amd64/bin/jrunscript to provide /usr/bin/jrunscript (jrunscript) in auto mode
update-alternatives: using /usr/lib/jvm/temurin-17-jdk-amd64/bin/jshell to provide /usr/bin/jshell (jshell) in auto mode
update-alternatives: using /usr/lib/jvm/temurin-17-jdk-amd64/bin/jstack to provide /usr/bin/jstack (jstack) in auto mode
update-alternatives: using /usr/lib/jvm/temurin-17-jdk-amd64/bin/jstat to provide /usr/bin/jstat (jstat) in auto mode
update-alternatives: using /usr/lib/jvm/temurin-17-jdk-amd64/bin/jstatd to provide /usr/bin/jstatd (jstatd) in auto mode
update-alternatives: using /usr/lib/jvm/temurin-17-jdk-amd64/bin/keytool to provide /usr/bin/keytool (keytool) in auto mode
update-alternatives: using /usr/lib/jvm/temurin-17-jdk-amd64/bin/rmiregistry to provide /usr/bin/rmiregistry (rmiregistry) in auto mode
update-alternatives: using /usr/lib/jvm/temurin-17-jdk-amd64/bin/serialver to provide /usr/bin/serialver (serialver) in auto mode
update-alternatives: using /usr/lib/jvm/temurin-17-jdk-amd64/lib/jexec to provide /usr/bin/jexec (jexec) in auto mode
update-alternatives: using /usr/lib/jvm/temurin-17-jdk-amd64/lib/jspawnhelper to provide /usr/bin/jspawnhelper (jspawnhelper) in auto mode
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
root@3d636ca4e8cf:/home/user#

I suggest looking to x-amz-replication-status (AWS doc, and how it can be monitored), the X-Cache and Age headers.

Just for reference, these are other reported issues that I come across while researching the problem.... it seems that the issue is not new:

@saamorim
Copy link

We too are facing this issue everyday, and it is not clear why.... Only during some moments we can actually install the package... is there any alternative for https://packages.adoptium.net/artifactory/deb that it is more reliable? or any known mitigation for this?

Thank you

@jappenzesr
Copy link

jappenzesr commented Jan 29, 2024

Also affected, when trying to build runner-images, blocking new releases of the same image. Thanks for looking into it.

@ijaouani
Copy link

I was reproducing this issue on last Thursday but not anymore today. Seems it has been fixed.

@panther7
Copy link

Still issue...

Do you want to continue? [Y/n]
Err:1 https://packages.adoptium.net/artifactory/deb bookworm/main amd64 temurin-17-jdk amd64 17.0.10.0.0+7
  403  Forbidden [IP: 199.232.19.42 443]
E: Failed to fetch https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-17/temurin-17-jdk_17.0.10.0.0%2b7_amd64.deb  403  Forbidden [IP: 199.232.19.42 443]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

@classicrocker883
Copy link

classicrocker883 commented Jan 29, 2024

one workaround is to simply download the .deb from the website, you may need to change the url to get it to work, but just use dpkg -i

@dustingrady
Copy link

dustingrady commented Jan 29, 2024 via email

@skalderqm
Copy link

received 403 and 503 errors as well as Hash Sum mismatch yesterday:

E: Failed to fetch https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-17/temurin-17-jdk_17.0.10.0.0%2b7_amd64.deb 503 Response object too large [IP: 151.101.3.42 443]

E: Failed to fetch https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-17/temurin-17-jdk_17.0.10.0.0%2b7_amd64.deb 403 Forbidden [IP: 151.101.67.42 443]

E: Failed to fetch https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-17/temurin-17-jdk_17.0.10.0.0%2b7_amd64.deb 403 Forbidden [IP: 151.101.131.42 443]

E: Failed to fetch https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-17/temurin-17-jdk_17.0.10.0.0%2b7_amd64.deb 503 Response object too large [IP: 151.101.195.42 443]

E: Failed to fetch https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-17/temurin-17-jdk_17.0.10.0.0%2b7_amd64.deb Hash Sum mismatch

@mhever
Copy link

mhever commented Feb 7, 2024

Still facing this issue

==> azure-arm.build_image: <- https:102%20Status%0aMessage:%20Waiting%20for%20headers%0aURI:%20https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-11/temurin-11-jdk_11.0.22.0.0%252b7_amd64.deb
==> azure-arm.build_image: Answer for: https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-11/temurin-11-jdk_11.0.22.0.0%2b7_amd64.deb
==> azure-arm.build_image: HTTP/1.1 403 Forbidden
==> azure-arm.build_image: Connection: keep-alive
==> azure-arm.build_image: x-amz-request-id: K1339498VN86JVVG
==> azure-arm.build_image: x-amz-id-2: U6hqQj+0Z2Fj/WHkuYEcqXzfuP70mOAB041ddaqxQp8nXxk4lY8UdsMAa/Euf1h+g0gPpHF0Q7Y=
==> azure-arm.build_image: Content-Type: application/xml
==> azure-arm.build_image: Server: AmazonS3
==> azure-arm.build_image: Fastly-Restarts: 1
==> azure-arm.build_image: Accept-Ranges: bytes
==> azure-arm.build_image: Date: Wed, 07 Feb 2024 14:14:41 GMT
==> azure-arm.build_image: Via: 1.1 varnish
==> azure-arm.build_image: X-Served-By: cache-ams21021-AMS
==> azure-arm.build_image: X-Cache: MISS
==> azure-arm.build_image: X-Cache-Hits: 0
==> azure-arm.build_image: X-Timer: S1707315282.777892,VS0,VE158
==> azure-arm.build_image: transfer-encoding: chunked
==> azure-arm.build_image:
==> azure-arm.build_image: <- https:400%20URI%20Failure%0aFailReason:%20HttpError403%0aMessage:%20403%20%20Forbidden%20[IP:%20151.101.39.42%20443]%0aURI:%20https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-11/temurin-11-jdk_11.0.22.0.0%252b7_amd64.deb
==> azure-arm.build_image: E: Failed to fetch https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-11/temurin-11-jdk_11.0.22.0.0%2b7_amd64.deb Error reading from server. Remote end closed connection [IP: 151.101.39.42 443]
==> azure-arm.build_image: E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
==> azure-arm.build_image: cat: /usr/lib/jvm/temurin-11-jdk-amd64/release: No such file or directory

@ericlayton
Copy link

This error is continuing to occur for at least Temurin 11 and 21 packages.

10:22:54 Reading package lists...
10:22:54 Building dependency tree...
10:22:54 Reading state information...
10:22:54 The following NEW packages will be installed:
10:22:54 temurin-21-jdk
10:22:54 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
10:22:54 Need to get 178 MB of archives.
10:22:54 After this operation, 362 MB of additional disk space will be used.
10:22:54 Get:1 https://packages.adoptium.net/artifactory/deb focal/main amd64 temurin-21-jdk amd64 21.0.2.0.0+13-2 [178 MB]
10:22:55 Err:1 https://packages.adoptium.net/artifactory/deb focal/main amd64 temurin-21-jdk amd64 21.0.2.0.0+13-2
10:22:55 403 Forbidden [IP: 146.75.35.42 443]
10:22:55 E: Failed to fetch https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-21/temurin-21-jdk_21.0.2.0.0+13-2_amd64.deb 403 Forbidden [IP: 146.75.35.42 443]
10:22:55 E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

11:22:06 Reading package lists...
11:22:06 Building dependency tree...
11:22:07 Reading state information...
11:22:07 The following NEW packages will be installed:
11:22:07 temurin-11-jdk
11:22:07 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
11:22:07 Need to get 168 MB of archives.
11:22:07 After this operation, 325 MB of additional disk space will be used.
11:22:07 Err:1 https://packages.adoptium.net/artifactory/deb focal/main amd64 temurin-11-jdk amd64 11.0.22.0.0+7
11:22:07 403 Forbidden [IP: 146.75.35.42 443]
11:22:07 E: Failed to fetch https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-11/temurin-11-jdk_11.0.22.0.0+7_amd64.deb 403 Forbidden [IP: 146.75.35.42 443]
11:22:07 E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

@jbonofre
Copy link

jbonofre commented Mar 2, 2024

We are facing this issue (a bunch of 403 errors trying to bootstrap, same as @ericlayton shared). @gdams any update in this one ? Maybe @mbarbero can help ?

@fabio-blanco
Copy link

fabio-blanco commented Mar 5, 2024

Forbidden error happening with an Ubuntu 20.04 while running sudo apt upgrade:

Lendo listas de pacotes... Pronto
Construindo árvore de dependências       
Lendo informação de estado... Pronto
Calculando atualização... Pronto
Obtenha mais atualizações de segurança através do Ubuntu Pro com 'esm-apps' habilitado:
  libavformat58 libmagick++-6.q16-8 libavfilter7 libimage-magick-perl
  libmagickcore-6.q16-6-extra libimage-magick-q16-perl imagemagick
  libswresample3 libgegl-0.4-0 libzmq5 libmagickwand-6.q16-6 libgegl-common
  libcgraph6 libpostproc55 liblab-gamut1 imagemagick-6.q16 libavcodec58
  libcdt5 libavutil56 libpathplan4 libswscale5 libgvpr2 libopenexr24
  libsdl2-2.0-0 libmysofa1 libmagickcore-6.q16-6 libgvc6 graphviz
  imagemagick-6-common
Saiba mais sobre o Ubuntu Pro em https://ubuntu.com/pro
Os pacotes a seguir serão atualizados:
  temurin-21-jdk
1 pacotes atualizados, 0 pacotes novos instalados, 0 a serem removidos e 0 não atualizados.
É preciso baixar 178 MB de arquivos.
Depois desta operação, 738 kB adicionais de espaço em disco serão usados.
Você quer continuar? [S/n] 
Err:1 https://packages.adoptium.net/artifactory/deb focal/main amd64 temurin-21-jdk amd64 21.0.2.0.0+13-2
  403  Forbidden [IP: 151.101.3.42 443]
E: Falhou ao buscar https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-21/temurin-21-jdk_21.0.2.0.0+13-2_amd64.deb  403  Forbidden [IP: 151.101.3.42 443]
E: Impossível buscar alguns arquivos, talvez executar apt-get update ou tentar com --fix-missing?

@jbonofre
Copy link

We are still facing this issue (randomly but pretty often). It's basically the same as @mhever. I'm trying a bunch of curl (with traceroute) to have some clues.

It would be great to have an update from Adoptium committers and/or Eclipse infra.

@jerboaa
Copy link

jerboaa commented Mar 27, 2024

It would be great to have an update from Adoptium committers and/or Eclipse infra.

I'm one of the Adoptium PMC members. We are aware of the issue and keep monitoring the situation. For example, we discussed this issue again on the PMC meeting of March 20'th, 2024. Unfortunately, we are unaware of a proper reproducer and given the volume of hits we get to the repository the failure cases are fairly infrequent (relatively speaking, considering the scale). In the absence of a reproducer, there is unfortunately only a limited amount that we can do on our end. We keep monitoring the situation with an eye on the failure rate, however.

Having said that, if somebody ends up figuring out what exactly is needed to reliably reproduce the issue we can work on a more permanent solution. As of yet, there is little we can do. Thanks for your understanding!

@jbonofre
Copy link

@jerboaa Thanks for your feedback ! I agree, I'm also struggling to reproduce the behavior on my machine, it seems way more frequent on our cloud infra.
Let me try to find a reproducer.

Thanks again !

@theadzik
Copy link

theadzik commented May 24, 2024

Got the error today:

wget -qO - https://packages.adoptium.net/artifactory/api/gpg/key/public | gpg --dearmor > /usr/share/keyrings/adoptium.gpg
echo "deb [signed-by=/usr/share/keyrings/adoptium.gpg] https://packages.adoptium.net/artifactory/deb/ $(lsb_release -cs) main" > /etc/apt/sources.list.d/adoptium.list
apt-get -q update
apt-get -q install temurin-17-jdk=\*
#13 194.0 Reading package lists...
#13 194.7 Building dependency tree...
#13 194.8 Reading state information...
#13 195.0 The following NEW packages will be installed:
#13 195.0   temurin-17-jdk
#13 195.4 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
#13 195.4 Need to get 166 MB of archives.
#13 195.4 After this operation, 331 MB of additional disk space will be used.
#13 195.4 Get:1 https://packages.adoptium.net/artifactory/deb jammy/main amd64 temurin-17-jdk amd64 17.0.11.0.0+9 [166 MB]
#13 203.2 Ign:1 https://packages.adoptium.net/artifactory/deb jammy/main amd64 temurin-17-jdk amd64 17.0.11.0.0+9
#13 205.1 Err:1 https://packages.adoptium.net/artifactory/deb jammy/main amd64 temurin-17-jdk amd64 17.0.11.0.0+9
#13 205.1   Error reading from server. Remote end closed connection [IP: 146.75.123.42 443]
#13 205.1 E: Failed to fetch https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-17/temurin-17-jdk_17.0.11.0.0%2b9_amd64.deb  Error reading from server. Remote end closed connection [IP: 146.75.123.42 443]
#13 205.1 E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

It worked 2 hours ago, now I can't get through after multiple tries

EDIT: It started working after an hour

@poiema72
Copy link

poiema72 commented May 25, 2024

I'm also experiencing similar issue today. (Experienced similar issue yesterday, As commented above, sometimes it worked yesterday.) However, again, happending today.

32.76 Err:6 https://packages.adoptium.net/artifactory/deb bullseye/main amd64 temurin-8-jdk amd64 8.0.412.0.0+8
32.76   Connection timed out [IP: 146.75.51.42 443]
32.77 E: Failed to fetch https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-8/temurin-8-jdk_8.0.412.0.0%2b8_amd64.deb  Connection timed out [IP: 146.75.51.42 443]
32.77 E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
32.77 Fetched 7195 kB in 31s (233 kB/s)

EDIT: Same as above. It started working now.

@ainoelina
Copy link

Getting the same issue when trying to use packer image for Ubuntu 20.04

azure-arm.build_image: Get:1 https://packages.adoptium.net/artifactory/deb focal/main amd64 temurin-11-jdk amd64 11.0.23.0.0+9 [168 MB]
azure-arm.build_image: Err:1 https://packages.adoptium.net/artifactory/deb focal/main amd64 temurin-11-jdk amd64 11.0.23.0.0+9
azure-arm.build_image:   403  Forbidden [IP: 151.101.39.42 443]
==> azure-arm.build_image: E: Failed to fetch https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-11/temurin-11-jdk_11.0.23.0.0+9_amd64.deb  403  Forbidden [IP: 151.101.39.42 443]
==> azure-arm.build_image: E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

@dracorp
Copy link

dracorp commented Aug 27, 2024

The same situation on Centos7, wget, curl, dnf, and yum.

@jessehersch
Copy link

Hit this today for temurin-8:

Err:5 https://packages.adoptium.net/artifactory/deb buster/main amd64 temurin-8-jdk amd64 8.0.422.0.0+5
  403  Forbidden [IP: 151.101.23.42 443]
Fetched 3053 kB in 0s (9175 kB/s)
E: Failed to fetch https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-8/temurin-8-jdk_8.0.422.0.0+5_amd64.deb  403  Forbidden [IP: 151.101.23.42 443]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

@jd-mateusz-prais
Copy link

If Fastly isn't able to solve this issue, it may be worth asking the good folks at Cloudflare if they'd be able to help out with CDN?

@rtrad89
Copy link

rtrad89 commented Oct 11, 2024

Facing the issue today too in a CD script

Err:16 https://packages.adoptium.net/artifactory/deb noble/main amd64 temurin-17-jdk amd64 17.0.12.0.0+7
  Error reading from server. Remote end closed connection [IP: 104.18.20.66 443]
Fetched 4075 kB in 4s (1112 kB/s)
E: Failed to fetch https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-17/temurin-17-jdk_17.0.12.0.0%2b7_amd64.deb  Error reading from server. Remote end closed connection [IP: 104.18.20.66 443]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

@Akkowicz
Copy link

This issue is blocking some playbooks from running when setting up new hosts.

@kolstae
Copy link

kolstae commented Oct 25, 2024

And today as well

21.19 E: Failed to fetch https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-11/temurin-11-jdk_11.0.24.0.0+8_amd64.deb  403  Forbidden [IP: 104.18.21.66 443]

@Langleu
Copy link

Langleu commented Oct 29, 2024

Same happening for me occasionally on AWS EC2 with Debian 12.

apt and wget resulting in a 403.

wget https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-21/temurin-21-jdk_21.0.4.0.0%2b7-1_amd64.deb
--2024-10-29 08:33:13--  https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-21/temurin-21-jdk_21.0.4.0.0%2b7-1_amd64.deb
Resolving packages.adoptium.net (packages.adoptium.net)... 104.18.20.66, 104.18.21.66, 2606:4700::6812:1442, ...
Connecting to packages.adoptium.net (packages.adoptium.net)|104.18.20.66|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 177364876 (169M) [application/x-debian-package]
Saving to: ‘temurin-21-jdk_21.0.4.0.0+7-1_amd64.deb’

temurin-21-jdk_21.0.4.0.0+7-1_amd64.deb                        9%[============>                                                                                                                               ]  16.00M  --.-KB/s    in 0.1s

2024-10-29 08:33:14 (151 MB/s) - Connection closed at byte 16777216. Retrying.

--2024-10-29 08:33:15--  (try: 2)  https://packages.adoptium.net/artifactory/deb/pool/main/t/temurin-21/temurin-21-jdk_21.0.4.0.0%2b7-1_amd64.deb
Connecting to packages.adoptium.net (packages.adoptium.net)|104.18.20.66|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2024-10-29 08:33:15 ERROR 403: Forbidden.

from my local machine it's working fine.

The only way for me to fix it was to change the Elastic IP / NAT Gateway as the AWS IP really seemed to be blacklisted.

@JBodkin
Copy link

JBodkin commented Nov 17, 2024

I've been encountering this issue, when I copy the link to my browser and try the same request, I can see that the request has already expired. Note: Expires vs ServerTime

<Error>
  <Code>AccessDenied</Code>
  <Message>Request has expired</Message>
  <X-Amz-Expires>900</X-Amz-Expires>
  <Expires>2024-11-17T12:53:20Z</Expires>
  <ServerTime>2024-11-17T13:33:23Z</ServerTime>
  <RequestId>SVJ548Z74ZRMA0NJ</RequestId>
  <HostId>d/c1uG/KnSKfBtEmEXOHXj5hz/2da7v4Lke30LZ88SC4rAZ6GMteAbnJ1FVJVrYQ1zmEaTZ5xPY=</HostId>
</Error>

Copy link

We are marking this issue as stale because it has not been updated for a while. This is just a way to keep the support issues queue manageable.
It will be closed soon unless the stale label is removed by a committer, or a new comment is made.

@github-actions github-actions bot added the stale label Feb 16, 2025
@karianna karianna removed the stale label Feb 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working Waiting on OP
Projects
None yet
Development

No branches or pull requests