Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[meta] mitigate scriptworker rollout bustage #399

Open
escapewindow opened this issue Sep 8, 2021 · 1 comment
Open

[meta] mitigate scriptworker rollout bustage #399

escapewindow opened this issue Sep 8, 2021 · 1 comment

Comments

@escapewindow
Copy link
Contributor

Recently, we rolled out an entire set of new k8s scriptworker pools via a push to the production branch, and a) the signingscript pool was busted because it was claiming tasks but had an incompatible version of osslsigncode due to a moved python:3.8 docker hub tag, and b) the rest of the pools were busted because they were using a version of scriptworker that didn't successfully claimWork.

On top of this, we didn't have an established, documented way of rolling back k8s pools; each person had their own way of doing so, or none at all. So:

  1. Let's run scriptworker integration tests in CI,
  2. pin the FROM image digest in Dockerfiles,
  3. add signingscript integration tests, and
  4. document scriptworker k8s pool rollback.
@MihaiTabara
Copy link
Contributor

MihaiTabara commented Sep 21, 2021

+1 to this. This is one thing that if done right pro-actively, can save us a lot of time reactively. Wondering if parts of this could be good first bugs in learning scriptworker. If we built the initial skeleton, we might also advertise this to external contributors.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants