CAA: add support to look up imagePullSecrets for pods

[squarti]

This PR initializes auth.json with the imagePullSecrets listed on the pod and service account.

Fixes: #2231

[mkulke]

thx, looks good! the override semantics are interesting here, I'd intuitively expect that a cluster-wide auth.json is overridden by a pod-specific auth.json, but we're doing the opposite here. some questions:

does it make sense to retire the existing caa-wide "auth.json" mechanism in favor of passing this from kubernetes? the caa-wide param is roughly comparable to a node-wide auth.json in the kubelet config, which is discouraged (and which we don't consider it here, either).

I would suggest to at least deprecate the parameter and mark it for later removal.

If we don't want to remove the caa-wide auth.json, would we be able to "merge" both sources?

if we don't want to remove the caa-wide auth.json, would it make sense for pod-specific auth.json to take precedence?

What are the implications for auth.json that come from KBS?

[mkulke]

I guess the log message needs to be adjusted, to indicate that we fall back on image-pull-secrets from the pod

[stevenhorsman]

@bpradipt @snir911 - I think you added the CAA wide auth.json, so wanted to check if you had any concerned with this. I'm inclined to agree with Magnus, that it would be good to replace the CAA-wide approach with a pod specific one, but am unsure if this would cause challenges with the pause image pull on OpenShift?

[bpradipt]

@bpradipt @snir911 - I think you added the CAA wide auth.json, so wanted to check if you had any concerned with this. I'm inclined to agree with Magnus, that it would be good to replace the CAA-wide approach with a pod specific one, but am unsure if this would cause challenges with the pause image pull on OpenShift?

Afaicr, the pause image is embedded now.
Overall I like this approach, but I was thinking that imagePullSecrets, if present, would replace the default cluster wide auth.json as Magnus had mentioned.
In the current of using serviceAccount and imagePullSecrets may be we can have a flexibility to specify which service account to retrieve from. That could be a CAA config with default being the default SA. This will avoid any need for cluster wide auth.json to be used.

[squarti]

thx, looks good! the override semantics are interesting here, I'd intuitively expect that a cluster-wide auth.json is overridden by a pod-specific auth.json, but we're doing the opposite here. some questions:

I left the cluster wide auth.json overwriting the pod specific secrets because this way current code that uses cluster wide auth.json would not be broken. I think merging the secrets would also be ok as long as the pod secrets do not conflict with the cluster wide secret.

Let me know which approach to take and I will adjust the changes.

[mkulke]

thx, looks good! the override semantics are interesting here, I'd intuitively expect that a cluster-wide auth.json is overridden by a pod-specific auth.json, but we're doing the opposite here. some questions:

I left the cluster wide auth.json overwriting the pod specific secrets because this way current code that uses cluster wide auth.json would not be broken. I think merging the secrets would also be ok as long as the pod secrets do not conflict with the cluster wide secret.

Let me know which approach to take and I will adjust the changes.

I would opt for removing the option to have a cluster-wide auth.json in this case. If I understood it correctly there is no use case for that anymore. Propagating it from the cluster is the right call, IMO. since the manifest will still be pulled on the node different maintaining different sets of registry credentials in k8s and caa config can be pretty confusing.