Base container jenkins servers - Plugins 404 errors

[Harguer]

Service(s)

infra.ci.jenkins.io, jenkins.io, plugins.jenkins.io, Update center

Summary

After a recent restart of one of our Jenkins containers, it is unable to start up due to multiple 404 errors while attempting to download plugins from the Jenkins update servers. This issue appears to be specific to the US-East region, as testing the same setup in the US-West region works without issues.

 k logs -f jenkins-jenkins
Defaulted container "jenkins-master" out of: jenkins-master, backup
+ '[' '' == true ']'
+ echo 'To print debug messages set environment variable '\''DEBUG_JENKINS_OPERATOR'\'' to '\''true'\'''
+ mkdir -p /var/lib/jenkins/init.groovy.d
To print debug messages set environment variable 'DEBUG_JENKINS_OPERATOR' to 'true'
+ cp -n /var/jenkins/init-configuration/createOperatorUser.groovy /var/lib/jenkins/init.groovy.d
+ mkdir -p /var/lib/jenkins/scripts
+ cp /var/jenkins/scripts/init.sh /var/jenkins/scripts/install-plugins.sh /var/lib/jenkins/scripts
+ chmod +x /var/lib/jenkins/scripts/init.sh /var/lib/jenkins/scripts/install-plugins.sh
+ echo 'Installing plugins required by Operator - begin'
+ cat
Installing plugins required by Operator - begin
+ [[ -z '' ]]
+ /var/jenkins/scripts/install-plugins.sh
WARN: install-plugins.sh is deprecated, please switch to jenkins-plugin-cli
Cleaning up locks
Creating initial locks...
Analyzing war /usr/share/jenkins/jenkins.war...
Registering preinstalled plugins...
Using version-specific update center: https://eastamerica.cloudflare.jenkins.io/dynamic-2.427/...
Downloading plugins...
Downloading plugin: job-dsl from https://updates.jenkins.io/download/plugins/job-dsl/1.78.1/job-dsl.hpi
Downloading plugin: workflow-job from https://updates.jenkins.io/download/plugins/workflow-job/1189.va_d37a_e9e4eda_/workflow-job.hpi
Downloading plugin: workflow-aggregator from https://updates.jenkins.io/download/plugins/workflow-aggregator/2.6/workflow-aggregator.hpi
Downloading plugin: kubernetes from https://updates.jenkins.io/download/plugins/kubernetes/1.31.2/kubernetes.hpi
Downloading plugin: kubernetes-credentials-provider from https://updates.jenkins.io/download/plugins/kubernetes-credentials-provider/0.20/kubernetes-credentials-provider.hpi
Downloading plugin: configuration-as-code from https://updates.jenkins.io/download/plugins/configuration-as-code/1512.vb_79d418d5fc8/configuration-as-code.hpi
Downloading plugin: git from https://updates.jenkins.io/download/plugins/git/4.10.0/git.hpi
 > workflow-aggregator depends on lockable-resources:2.3,pipeline-stage-view:2.10,workflow-api:2.29,workflow-basic-steps:2.11,workflow-cps-global-lib:2.11,workflow-cps:2.56,workflow-durable-task-step:2.22,workflow-job:2.25,workflow-multibranch:2.20,workflow-scm-step:2.6,workflow-step-api:2.16,workflow-support:2.20,cloudbees-folder:6.6,credentials:2.1.18,git-client:2.7.3,jackson2-api:2.8.11.3,pipeline-build-step:2.7,pipeline-input-step:2.8,pipeline-milestone-step:1.3.1,pipeline-stage-step:2.3,scm-api:2.2.8,structs:1.15,pipeline-model-definition:1.3.2
Downloading plugin: lockable-resources from https://eastamerica.cloudflare.jenkins.io/dynamic-2.427//latest/lockable-resources.hpi
Downloading plugin: pipeline-stage-view from https://eastamerica.cloudflare.jenkins.io/dynamic-2.427//latest/pipeline-stage-view.hpi
Downloading plugin: workflow-api from https://eastamerica.cloudflare.jenkins.io/dynamic-2.427//latest/workflow-api.hpi
Downloading plugin: workflow-basic-steps from https://eastamerica.cloudflare.jenkins.io/dynamic-2.427//latest/workflow-basic-steps.hpi
Downloading plugin: workflow-cps-global-lib from https://eastamerica.cloudflare.jenkins.io/dynamic-2.427//latest/workflow-cps-global-lib.hpi
Downloading plugin: workflow-cps from https://eastamerica.cloudflare.jenkins.io/dynamic-2.427//latest/workflow-cps.hpi
Downloading plugin: workflow-durable-task-step from https://eastamerica.cloudflare.jenkins.io/dynamic-2.427//latest/workflow-durable-task-step.hpi
Downloading plugin: workflow-multibranch from https://eastamerica.cloudflare.jenkins.io/dynamic-2.427//latest/workflow-multibranch.hpi
Downloading plugin: workflow-scm-step from https://eastamerica.cloudflare.jenkins.io/dynamic-2.427//latest/workflow-scm-step.hpi
 > workflow-job depends on workflow-api:1143.v2d42f1e9dea_5,workflow-step-api:622.vb_8e7c15b_c95a_,workflow-support:818.v4eb_969241b_c7
 > kubernetes depends on caffeine-api:2.9.2-29.v717aac953ff3,workflow-api:1108.v57edf648f5d4,workflow-cps:2648.va9433432b33c;resolution:=optional,workflow-step-api:615.vb09dac339255,authentication-tokens:1.4,cloudbees-folder:6.17,credentials-binding:1.27,credentials:2.6.1,durable-task:493.v195aefbb0ff2,jackson2-api:2.13.1-242.v1a45bad25ceb,kubernetes-client-api:5.11.1-179.v12037658df90,metrics:4.0.2.8,plain-credentials:1.7,structs:308.v852b473a2b8c,variant:1.4,kubernetes-credentials:0.9.0,pipeline-model-extensions:1.9.3;resolution:=optional
Downloading plugin: workflow-support from https://eastamerica.cloudflare.jenkins.io/dynamic-2.427//latest/workflow-support.hpi
Downloading plugin: workflow-step-api from https://eastamerica.cloudflare.jenkins.io/dynamic-2.427//latest/workflow-step-api.hpi
Downloading plugin: caffeine-api from https://eastamerica.cloudflare.jenkins.io/dynamic-2.427//latest/caffeine-api.hpi
Downloading plugin: cloudbees-folder from https://eastamerica.cloudflare.jenkins.io/dynamic-2.427//latest/cloudbees-folder.hpi
 > git depends on configuration-as-code:1.54;resolution:=optional,workflow-scm-step:2.13,workflow-step-api:2.24,credentials-binding:1.27,credentials:2.6.1,git-client:3.10.0,mailer:1.34,matrix-project:1.19;resolution:=optional,parameterized-trigger:2.39;resolution:=optional,promoted-builds:3.9.1;resolution:=optional,scm-api:2.6.5,script-security:1.78,ssh-credentials:1.19,structs:1.23,token-macro:2.15;resolution:=optional
Skipping optional dependency workflow-cps
Downloading plugin: credentials from https://eastamerica.cloudflare.jenkins.io/dynamic-2.427//latest/credentials.hpi
Downloading plugin: git-client from https://eastamerica.cloudflare.jenkins.io/dynamic-2.427//latest/git-client.hpi
Skipping optional dependency configuration-as-code
Downloading plugin: jackson2-api from https://eastamerica.cloudflare.jenkins.io/dynamic-2.427//latest/jackson2-api.hpi
 > kubernetes-credentials-provider depends on aws-credentials:1.26;resolution:=optional,credentials:2.2.0,github-branch-source:2.7.1;resolution:=optional,kubernetes-client-api:5.4.1,openstack-cloud:2.47;resolution:=optional,plain-credentials:1.4;resolution:=optional,ssh-credentials:1.18.1;resolution:=optional,variant:1.1
Downloading plugin: pipeline-build-step from https://eastamerica.cloudflare.jenkins.io/dynamic-2.427//latest/pipeline-build-step.hpi
Skipping optional dependency aws-credentials
Downloading plugin: authentication-tokens from https://eastamerica.cloudflare.jenkins.io/dynamic-2.427//latest/authentication-tokens.hpi
Downloading plugin: pipeline-input-step from https://eastamerica.cloudflare.jenkins.io/dynamic-2.427//latest/pipeline-input-step.hpi
Downloading plugin: credentials-binding from https://eastamerica.cloudflare.jenkins.io/dynamic-2.427//latest/credentials-binding.hpi
Skipping optional dependency github-branch-source
curl: (22) The currequested URL returned error: 404
Downloading plugin: pipeline-milestone-step from https://eastamerica.cloudflare.jenkins.io/dynamic-2.427//latest/pipeline-milestone-step.hpi
l: (22) The requested URL returned error: 404
Skipping optional dependency openstack-cloud
20:38:38 Failure (22) Retrying in 1 seconds...
20:38:38 Failure (22) Retrying in 1 seconds...
curl: (22) The requested URL returned error: 404
Downloading plugin: durable-task from https://eastamerica.cloudflare.jenkins.io/dynamic-2.427//latest/durable-task.hpi
20:38:38 Failure (22) Retrying in 1 seconds...
Skipping optional dependency plain-credentials
Downloading plugin: kubernetes-client-api from https://eastamerica.cloudflare.jenkins.io/dynamic-2.427//latest/kubernetes-client-api.hpi
Skipping optional dependency ssh-credentials
Skipping optional dependency matrix-project
Downloading plugin: pipeline-stage-step from https://eastamerica.cloudflare.jenkins.io/dynamic-2.427//latest/pipeline-stage-step.hpi
Downloading plugin: mailer from https://eastamerica.cloudflare.jenkins.io/dynamic-2.427//latest/mailer.hpi
Skipping optional dependency parameterized-trigger
Downloading plugin: scm-api from https://eastamerica.cloudflare.jenkins.io/dynamic-2.427//latest/scm-api.hpi
ccurl: (22) The requested URL returned error: 404
curl: (22) The requested URL returned error: 404
Skipping optional dependency promoted-builds
url: (22) The requested URL returned error: 404
20:38:38 Failure (22) Retrying in 1 seconds...
curl: (22) The requested URL returned error: 404
c20:38:38 Failure (22) Retrying in 1 seconds...
url: (22) The requested Ucurl: (22) The requested URL returned error: 404
RL20:38:38 Failure (22) Retrying in 1 seconds...
 returned error: 404
20:38:38 Failure (22) Retrying in 1 seconds...
20:38:38 Failure (22) Retrying in 1 seconds...
Downloading plugin: structs from https://eastamerica.cloudflare.jenkins.io/dynamic-2.427//latest/structs.hpi
20:38:38 Failure (22) Retrying in 1 seconds...
Downloading plugin: metrics from https://eastamerica.cloudflare.jenkins.io/dynamic-2.427//latest/metrics.hpi
curl: (22) The requested URL returned error: 404
Downloading plugin: pipeline-model-definition from https://eastamerica.cloudflare.jenkins.io/dynamic-2.427//latest/pipeline-model-definition.hpi
Downloading plugin: variant from https://eastamerica.cloudflare.jenkins.io/dynamic-2.427//latest/variant.hpi
Downloading plugin: script-security from https://eastamerica.cloudflare.jenkins.io/dynamic-2.427//latest/script-security.hpi
20:38:38 Failure (22) Retrying in 1 seconds...
curl: (22) The requested URL returned error: 404
Downloading plugin: plain-credentials from https://eastamerica.cloudflare.jenkins.io/dynamic-2.427//latest/plain-credentials.hpi
20:38:38 Failure (22) Retrying in 1 seconds...
curl: (22) The requested URL returned error: 404
20:38:38 Failure (22) Retrying in 1 seconds...
curl: (22) The requested URL returned error: 404
curl: (22) The requested URL returned error: 404
Downloading plugin: ssh-credentials from https://eastamerica.cloudflare.jenkins.io/dynamic-2.427//latest/ssh-credentials.hpi
20:38:38 Failure (22) Retrying in 1 seconds...
Skipping optional dependency token-macro
20:38:38 Failure (22) Retrying in 1 seconds...
 > configuration-as-code depends on caffeine-api:2.9.3-65.v6a_47d0f4d1fe,snakeyaml-api:1.30.2-76.vc104f7ce9870
Skipping optional dependency pipeline-model-extensions
Downloading plugin: kubernetes-credentials from https://eastamerica.cloudflare.jenkins.io/dynamic-2.427//latest/kubernetes-credentials.hpi
curl: (22) The requested URL returned error: 404
20:38:38 Failure (22) Retrying in 1 seconds...
Downloading plugin: snakeyaml-api from https://eastamerica.cloudflare.jenkins.io/dynamic-2.427//latest/snakeyaml-api.hpi
curl: (22) The requested URL returned error: 404
ccuurrll::  ((2222))  ThTeh er erqeuqeusetsetde dU RUL RrLe truerned error: 4t0u4r n
ed error: 404
curl: (22) The requested URL returned error: 404
20:38:38 Failure (22) Retrying in 1 seconds...
20:38:38 Failure (22) Retrying in 1 seconds...
20:38:38 Failure (22) Retrying in 1 seconds...
20:38:38 Failure (22) Retrying in 1 seconds...
curl: (22) The requested URL returned error: 404
20:38:38 Failure (22) Retrying in 1 seconds...
 > job-dsl depends on config-file-provider:2.15.4;resolution:=optional,configuration-as-code:1.15;resolution:=optional,managed-scripts:1.3;resolution:=optional,cloudbees-folder:5.14,structs:1.19,script-security:1.54,vsphere-cloud:1.1.11;resolution:=optional
Skipping optional dependency config-file-provider
curl: (22) The requested URL returned error: 404
Skipping optional dependency configuration-as-code
Skipping optional dependency managed-scripts
20:38:38 Failure (22) Retrying in 1 seconds...
Skipping optional dependency vsphere-cloud
curl: (22) The requested URL returned error: 404
20:38:38 Failure (22) Retrying in 1 seconds...
...
...
...
Failed to download plugin: structs or structs-plugin
20:38:45 Failed in the last attempt (curl -sSfL --connect-timeout 20 --retry 3 --retry-delay 0 --retry-max-time 60 https://eastamerica.cloudflare.jenkins.io/dynamic-2.427//latest/snakeyaml-api-plugin.hpi -o /usr/share/jenkins/ref/plugins/snakeyaml-api-plugin.jpi)
Failed to download plugin: snakeyaml-api or snakeyaml-api-plugin
20:38:45 Failed in the last attempt (curl -sSfL --connect-timeout 20 --retry 3 --retry-delay 0 --retry-max-time 60 https://eastamerica.cloudflare.jenkins.io/dynamic-2.427//latest/metrics-plugin.hpi -o /usr/share/jenkins/ref/plugins/metrics-plugin.jpi)
Failed to download plugin: metrics or metrics-plugin
curl: (22) The requested URL returned error: 404
20:39:08 Failure (22) Retrying in 1 seconds...
20:39:09 Failed in the last attempt (curl -sSfL --connect-timeout 20 --retry 3 --retry-delay 0 --retry-max-time 60 https://eastamerica.cloudflare.jenkins.io/dynamic-2.427//latest/jackson2-api-plugin.hpi -o /usr/share/jenkins/ref/plugins/jackson2-api-plugin.jpi)
Failed to download plugin: jackson2-api or jackson2-api-plugin

WAR bundled plugins:


Installed plugins:
configuration-as-code:1512.vb_79d418d5fc8
git:4.10.0
job-dsl:1.78.1
kubernetes-credentials-provider:0.20
kubernetes:1.31.2
workflow-aggregator:2.6
workflow-job:1189.va_d37a_e9e4eda_
Some plugins failed to download! Not downloaded: workflow-scm-step
Not downloaded: lockable-resources
Not downloaded: git-client
Not downloaded: workflow-api
Not downloaded: workflow-cps-global-lib
Not downloaded: workflow-basic-steps
Not downloaded: caffeine-api
Not downloaded: pipeline-stage-view
Not downloaded: pipeline-input-step
Not downloaded: workflow-support
Not downloaded: credentials
Not downloaded: credentials-binding
Not downloaded: workflow-cps
Not downloaded: cloudbees-folder
Not downloaded: plain-credentials
Not downloaded: durable-task
Not downloaded: workflow-multibranch
Not downloaded: workflow-durable-task-step
Not downloaded: pipeline-milestone-step
Not downloaded: script-security
Not downloaded: kubernetes-credentials
Not downloaded: workflow-step-api
Not downloaded: authentication-tokens
Not downloaded: pipeline-stage-step
Not downloaded: mailer
Not downloaded: pipeline-model-definition
Not downloaded: kubernetes-client-api
Not downloaded: pipeline-build-step
Not downloaded: variant
Not downloaded: scm-api
Not downloaded: ssh-credentials
Not downloaded: structs
Not downloaded: snakeyaml-api
Not downloaded: metrics
Not downloaded: jackson2-api

Reproduction steps

restart any existing pod/container or run a new one and you will see the issue.

[grahamja]

I am having the same issue. It started this morning and appears to be intermittent.

[siddhubp]

I tried multiple jenkins versions but cannot make it work. Its intermittent with different amount of plugins that gets downloaded and makes jenkins operator to crash..

[MarkEWaite]

I believe that the warning message states the problem:

Installing plugins required by Operator - begin
+ [[ -z '' ]]
+ /var/jenkins/scripts/install-plugins.sh
WARN: install-plugins.sh is deprecated, please switch to jenkins-plugin-cli

Based on that message, it looks like you might be using the Kubernetes operator to install and configure Jenkins. Check your configuration and confirm that it is installing plugins from a container image (first preference) or is using the plugin installation manager tool to install plugins (second choice). You should also check that it is running a modern version of Jenkins, since Jenkins 2.427 has multiple known critical security vulnerabilities.

The reference to /dynamic-2.427/ is incorrect in that request.

Refer to https://www.jenkins.io/blog/2024/11/16/new-update-center/ for the announcement of the use of the new Jenkins update center that started today.

[dduportal]

For those using the Kubernetes Operator, this issue has been fixed in jenkinsci/kubernetes-operator#784 in 2022: it means you have to upgrade your operator to have this change available (e.g. to have the operator using the jenkins-plugin-cli.

If your operator is up to date, or if you are not using the operator, then the presence of the install-plugins.sh means you are using a very old version of the Jenkins Docker image subject to security issues and you must upgrade.

---

For other people here with HTTP/404 errors from the Update Center which are not pinpointed to the unsupported install-plugins.sh script, please share details of your setup so we can check what the problem could be.

[Harguer]

Thank you @dduportal @MarkEWaite , my jenkins has been running for +13hrs now (updated jenkins operator, jenkins, and plugins), indeed the issue was the jenkins operator, it was an old one, it was just a month old before the fix to the install-plugins.sh was made.

Thanks again for your quick responses :)

[dduportal]

Ping @grahamja @siddhubp any feedbacks?

[dduportal]

Closing as the initial requester answered (thanks!) and their problem is solved. Thanks for reporting!