Disconnected 4.12 install fails pivot due to missing podman credentials #1500

jensortenholm · 2023-02-08T12:21:42Z

jensortenholm
Feb 8, 2023

Describe the bug

When installing in an UPI baremetal disconnected scenario, the release-image-pivot.service fails.

Digging into it, the first rpm-ostree call in the pre-pivot.sh script fails to find /run/containers//auth.json to authenticate to my private registry. The error given is "permission denied".

I worked around it by copying the /root/.docker/config.json to /run/containers//auth.json and changing permissions on both it and /run/containers directory, then running the script manually.

Version

4.12.0-0.okd-2023-02-04-212953, installing UPI baremetal and disconnected with a private registry.

How reproducible

Tried a number of reinstalls, happened every time.

Log bundle

Sorry, I'm not allowed to share logs or must-gather from this environment due to company policy.

vrutkovs · 2023-02-08T13:13:56Z

vrutkovs
Feb 8, 2023
Maintainer

Not much we can help with when no logs are available. Replace hostnames / IP with stubs maybe?

1 reply

jensortenholm Feb 8, 2023
Author

Alright, I was thinking that the description of the problem and workaround might be enough.

I'll reproduce this in an environment from which I can share logs and I'll get back to you. :-)

jensortenholm · 2023-02-08T18:35:23Z

jensortenholm
Feb 8, 2023
Author

I have now reproduced this problem in my home lab. Not sure exactly which logs you need, but I tried to follow the pointers in the documentation for collecting information on the bootstrap node (no API up, so no oc adm must-gather).

If you need anything else, please just let me know and I will collect it.

Attached is the output of:

journalctl -b -xeu bootkube.service
journalctl -xeu release-image-pivot.service
contents of /var/log/openshift

line 51 in release-image-pivot.service.log is the error that i described in the original post. I can complete installation if I do the following on the bootstrap node:

mkdir /run/containers/62011
cp /root/.docker/config.json /run/containers/62011/auth.json
chmod a+r /run/containers/62011/auth.json
chmod a+rx /run/containers
systemctl start release-image-pivot.service

journalctl_b_xeu_bootkube_service.log.tgz
release-image-pivot.service.log.tgz
var_log_openshift.tgz

12 replies

jensortenholm Feb 11, 2023
Author

Reading through the rpm-ostree issue mentioned by cgwalters, the fix in that case seems to have been to force anonymous pulls.

Note that I'm pulling from a private registry in this scenario (an instance of https://github.com/quay/mirror-registry actually), and that authentication is required to access the release images. I do not expect forcing anonymous pulls to succeed.

GregoireJulien Aug 22, 2023

The problem still occur in all FCOS 38 release. Private registry (I deactivated the auth). There is just one file to copy to make everything work.

vrutkovs Sep 8, 2023
Maintainer

Do you have the logbundle?

GregoireJulien Sep 15, 2023

I willl collect the log, but there is already logs bundle in this discussion, it's exaclty the same error, the same workaround to make boostrap complete.

vrutkovs Sep 15, 2023
Maintainer

it's exaclty the same error

We can tell that for sure only when we have two log bundles to compare

jensortenholm · 2023-03-19T14:34:56Z

jensortenholm
Mar 19, 2023
Author

Tried this again today, with the fresh 4.12.0-0.okd-2023-03-18-084815 release, and also did a second test using the fedora-coreos-37.20230218.3.0-qemu.x86_64.qcow2 image, but the issue persists.

Looking at the issue mentioned by @cgwalters above, I don't think the issue is the same here. If I understand things correctly, that fix patched a similar issue when trying to pull images from a public repo.

In this case, it is a private repo but rpm-ostree is not able to find the necessary credentials to access it.

Not sure if rpm-ostree should be looking elsewhere for the credentials file, or if okd should be placing credentials somewhere else where rpm-ostree would find them.

1 reply

tkops Apr 26, 2023

I have the same issue. Using a private password protected registry.
After disabling authentication on specific registry rpm-ostree could pull the image and cluster comes up.
I agree with jensortenholm. rpm-ostree needs the secrets that podman uses to be able to pull the image.

[root@api bootstrap]# rpm-ostree rebase --experimental "ostree-unverified-registry:quay.io/openshift/okd-content@sha256:c858d7d2086ca4a7b3b340c2ea32b26b68967fb66b918f8ac027020ca18d5193"

Pulling manifest: ostree-unverified-image:docker://quay.io/openshift/okd-content@sha256:c858d7d2086ca4a7b3b340c2ea32b26b68967fb66b918f8ac027020ca18d5193
error: remote error: (Mirrors also failed: internal.registry.net/openshift/okd-content@sha256:c858d7d2086ca4a7b3b340c2ea32b26b68967fb66b918f8ac027020ca18d5193: reading manifest sha256:c858d7d2086ca4a7b3b340c2ea32b26b68967fb66b918f8ac027020ca18d5193 in internal.registry.net/openshift/okd-content: unauthorized: The client does not have permission for manifest: Download request for repo:path 'internal-repo:openshift/okd-content/sha256__c858d7d2086ca4a7b3b340c2ea32b26b68967fb66b918f8ac027020ca18d5193/manifest.json' is forbidden for user: 'anonymous'.]): quay.io/openshift/okd-content@sha256:c858d7d2086ca4a7b3b340c2ea32b26b68967fb66b918f8ac027020ca18d5193: pinging container registry quay.io: invalid status code from registry 503 (Service Unavailable)

eb4x · 2023-05-16T17:37:23Z

eb4x
May 16, 2023

I can confirm same or similar issue. I'm performing a disconnected baremetal ipi installation. I've replicated 4.12.0-0.okd-2023-04-16-041331 to a mirror-registry (https://github.com/quay/mirror-registry) running on the provisioner. I've even made the repositories public for good measure.

May 16 17:23:29 localhost.localdomain bootstrap-pivot.sh[1803]: ++ rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/openshift/okd-content@sha256:53abd41811f53c940ac41dc8b9d5475813d9c96e20ce893c8664ba951044d6a3
May 16 17:23:33 localhost.localdomain bootstrap-pivot.sh[3596]: Pulling manifest: ostree-unverified-image:docker://quay.io/openshift/okd-content@sha256:53abd41811f53c940ac41dc8b9d5475813d9c96e20ce893c8664ba951044d6a3
May 16 17:24:33 localhost.localdomain bootstrap-pivot.sh[3596]: error: remote error: (Mirrors also failed: [mirror-registry.openshift-utv.uio.no:8443/openshift/okd@sha256:53abd41811f53c940ac41dc8b9d5475813d9c96e20ce893c8664ba951044d6a3: getting username and password: 1 error occurred:
May 16 17:24:33 localhost.localdomain bootstrap-pivot.sh[3596]:         * reading JSON file "/run/containers/62011/auth.json": open /run/containers/62011/auth.json: permission denied
May 16 17:24:33 localhost.localdomain bootstrap-pivot.sh[3596]: ]): quay.io/openshift/okd-content@sha256:53abd41811f53c940ac41dc8b9d5475813d9c96e20ce893c8664ba951044d6a3: pinging container registry quay.io: Get "https://quay.io/v2/": dial tcp 50.17.122.58:443: i/o timeout

running skopeo inspect docker://mirror-registry.openshift-utv.uio.no:8443/openshift/okd@sha256:53abd41811f53c940ac41dc8b9d5475813d9c96e20ce893c8664ba951044d6a3 on the bootstrap-vm does output information on the image.

2 replies

eb4x May 17, 2023

I got around this by running cp /root/.docker/config.json /etc/ostree/auth.json on the bootstrap node where the contents were { "auths": { "fake": { "auth": "aWQ6cGFzcwo=" } } }, before the release-image-pivot.service gets to the rpm-ostree rebase point, though some might want their actual pull-secret in that file.

jele84 Jun 3, 2024

@vrutkovs hi there! have the same issue with okd 4.15 installation
cp /root/.docker/config.json /etc/ostree/auth.json helps, but is this workaround ok?

eb4x · 2023-06-14T07:15:43Z

eb4x
Jun 14, 2023

Still an issue with 4.13.0-0.okd-2023-06-04-080300

Jun 14 07:09:30 localhost.localdomain bootstrap-pivot.sh[1807]: ++ rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/openshift/okd-content@sha256:a70ed62bcd842bd75806638a64afeac3e691922566e7f0c312c60c80845a4953
Jun 14 07:09:33 localhost.localdomain bootstrap-pivot.sh[3606]: Pulling manifest: ostree-unverified-image:docker://quay.io/openshift/okd-content@sha256:a70ed62bcd842bd75806638a64afeac3e691922566e7f0c312c60c80845a4953
Jun 14 07:10:36 localhost.localdomain bootstrap-pivot.sh[3606]: error: remote error: (Mirrors also failed: [mirror-registry.openshift-utv.uio.no:8443/openshift/okd@sha256:a70ed62bcd842bd75806638a64afeac3e691922566e7f0c312c60c80845a4953: getting username and password: 1 error occurred:
Jun 14 07:10:36 localhost.localdomain bootstrap-pivot.sh[3606]:         * reading JSON file "/run/containers/62011/auth.json": open /run/containers/62011/auth.json: permission denied
Jun 14 07:10:36 localhost.localdomain bootstrap-pivot.sh[3606]: ]): quay.io/openshift/okd-content@sha256:a70ed62bcd842bd75806638a64afeac3e691922566e7f0c312c60c80845a4953: pinging container registry quay.io: Get "https://quay.io/v2/": dial tcp 52.44.162.2:443: i/o timeout

0 replies

davidecelano · 2023-06-20T16:05:05Z

davidecelano
Jun 20, 2023

I can confirm the issue in IPI installation on vSphere in a disconnected environment with a private registry (quay instance as a mirror of quay.io) as you can see from the attached log
release-image-pivot.log

4.12.0-0.okd-2023-04-16-041331
fedora-coreos-37.20221127.3.0-vmware.x86_64.ova

2 replies

davidecelano Jun 23, 2023

Also with Fedora CoreOS 37.20230322.3.0 I have the same issue

rebase --experimental ostree-unverified-registry:quay.io/openshift/okd-content@sha256:53abd41811f53c940ac41dc8b9d5475813d9c96e20ce893c8664ba951044d6a3 st: ostree-unverified-image:docker://quay.io/openshift/okd-content@sha256:53abd41811f53c940ac41dc8b9d5475813d9c96e20ce893c8664ba951044d6a3 g importer: Failed to invoke skopeo proxy method OpenImage: remote error: (Mirrors also failed: [okdmirrorhost:8443/okd4/openshift4@sha256:53abd41811f53c940> ing JSON file "/run/containers/62011/auth.json": open /run/containers/62011/auth.json: permission denied

GregoireJulien Aug 22, 2023

Having the Issue too. I have to log in to the bootstrap machine, copy the file, and then restart the pivot service.

I passed my registry with no auth, but rpm-ostree still looking for the auth.json

Tested on all FCOS 38 version available.

There is a workarround in ignition file for the bootstap machine ?

Best regards

dan1el-k · 2023-07-20T09:54:51Z

dan1el-k
Jul 20, 2023

Ah just found this Discussion.
Looks like I have the same issue but with registry proxy instead: #1636.

So does ostree needs another configuration additionally to "/etc/containers/registries.conf" ?

0 replies

GregoireJulien · 2023-08-22T15:17:43Z

GregoireJulien
Aug 22, 2023

Looks like coreos/rpm-ostree#4107 which should be fixed since 2022.15

Nope, Still occur

0 replies

engineater · 2024-02-15T04:20:28Z

engineater
Feb 15, 2024

The problem is still occur in 4.14.0-0.okd-2024-01-26-175629 when you try deploy AirGaped OKD cluster with only proxy "imageDigestSources on quay.io".

The solution is still working: run commands on boot node

sudo su
mkdir /run/containers/62011
cp /root/.docker/config.json /run/containers/62011/auth.json
chmod a+r /run/containers/62011/auth.json
chmod a+rx /run/containers
systemctl start release-image-pivot.service

0 replies

klaphi · 2024-03-15T11:46:24Z

klaphi
Mar 15, 2024

Version "4.15.0-0.okd-2024-03-10-010116" is also affected. Without using the workaround, a disconnected cluster installation fails.

0 replies

jensortenholm · 2024-03-20T22:33:45Z

jensortenholm
Mar 20, 2024
Author

I did offer to spend time on this, but alas got usurped by real life. Having changed employers a few times since, I now have more time to spend on stuff that matters (with an employer that cares :)). @vrutkovs you still up for helping me along the way to bring a fix for this?

0 replies

fortinj66 · 2024-11-25T21:08:39Z

fortinj66
Nov 25, 2024

This is still an issue with SCOS OKS 4.16

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Disconnected 4.12 install fails pivot due to missing podman credentials #1500

{{title}}

Replies: 12 comments 18 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

Disconnected 4.12 install fails pivot due to missing podman credentials #1500

Replies: 12 comments · 18 replies

vrutkovs Feb 8, 2023 Maintainer

jensortenholm Feb 8, 2023 Author

jensortenholm Feb 8, 2023 Author

jensortenholm Feb 11, 2023 Author

vrutkovs Sep 8, 2023 Maintainer

vrutkovs Sep 15, 2023 Maintainer

jensortenholm Mar 19, 2023 Author

jensortenholm Mar 20, 2024 Author

Replies: 12 comments 18 replies

vrutkovs
Feb 8, 2023
Maintainer

jensortenholm Feb 8, 2023
Author

jensortenholm
Feb 8, 2023
Author

jensortenholm Feb 11, 2023
Author

vrutkovs Sep 8, 2023
Maintainer

vrutkovs Sep 15, 2023
Maintainer

jensortenholm
Mar 19, 2023
Author

jensortenholm
Mar 20, 2024
Author