Add ability to install Gatekeeper into separate namespace from operator's namespace

[font]

Currently in Kubernetes we support installing Gatekeeper into any namespace as long as that namespace is also where the operator is running.

This issue tracks the work necessary to have the operator install Gatekeeper into any namespace separate from where the operator is running.

On vanilla Kubernetes, the operator may be running in one namespace, but the user may want Gatekeeper to be installed to the gatekeeper-system canonical namespace, or another namespace separate from the operator's namespace.

On OpenShift, the operator may be installed into the openshift-operators namespace, but the operator should install Gatekeeper into a canonical namespace e.g. openshift-gatekeeper-system.

We should consider adding a namespace field to the Gatekeeper operator custom resource so the user could select which namespace is desired, and the operator should use a sane default if the user leaves it empty. This would require the operator to have permissions to create namespaces. Is this desirable?

[font]

We should consider adding a namespace field to the Gatekeeper operator custom resource so the user could select which namespace is desired, and the operator should use a sane default if the user leaves it empty. This would require the operator to have permissions to create namespaces. Is this desirable?

We should consider making such a namespace field immutable to aid in unintended consequences for the user i.e. operator would delete Gatekeeper installation and re-create it in another namespace. Or plan to document the behavior very well. Making this field immutable forces the user to perform the delete and create operations explicitly. However, currently K8s CRDs does not have built-in support for immutability using the readOnly OpenAPI feature, but it is being worked on. See the KEP for more details.

Instead, we'd need to use the traditional validating admission webhook approach to enforce immutability, which is a bit heavy. Considering the CRD support is being worked on, it would be preferable to use that approach if it will be resolved in the foreseeable future.

[JustinKuli]

@font, what's the current status of this? I was recently exploring installing the gatekeeper operator into different namespaces because in ROSA clusters, there is an admission webhook which prevents making new namespaces that start with openshift-. It seemed like the operator could be installed in an arbitrary namespace, but it was always installing gatekeeper into openshift-gatekeeper-system?

[font]

@font, what's the current status of this? I was recently exploring installing the gatekeeper operator into different namespaces because in ROSA clusters, there is an admission webhook which prevents making new namespaces that start with openshift-. It seemed like the operator could be installed in an arbitrary namespace, but it was always installing gatekeeper into openshift-gatekeeper-system?

There has been no progress on this as we are waiting for immutability support in CRDs before adding a namespace field to the CR to allow the user to pick the namespace to install Gatekeeper. The reason the operator installs GK to openshift-gatekeeper-system is because without giving the user the option (e.g. with a field in the CR) due to the problems described in this issue, we had to pick a canonical namespace to install Gatekeeper outside the openshift-operators namespace used to install the operator. We had to choose something with the openshift- prefix because we cannot guarantee the user doesn't already have the <pick-non-openshift-prefixed-namespace-name> name used up.

What admission webhook is preventing making new namespaces prefixed with openshift-? How can we make the webhook allow this? There is precedence here as there are other operators that do this as well. For example, the OpenShift Pipelines operator installs to openshift-operators and then installs OpenShift Pipelines to the newly created openshift-pipelines namespace.

[JustinKuli]

What admission webhook is preventing making new namespaces prefixed with openshift-? How can we make the webhook allow this?

Great questions, digging into this helped me understand why the "upstream" gatekeeper operator (installed via this ocm policy) wasn't working on ROSA but the "downstream" one (installed via this ocm policy) was working on ROSA. I believe the webhook is this one: https://github.com/openshift/managed-cluster-validating-webhooks/blob/master/pkg/webhooks/namespace/namespace.go

Service accounts in an openshift namespace are able to create new openshift namespaces. Via the ocm-policy, the "downstream" operator is installed into openshift-operators, so the operator can create the openshift-gatekeeper-system namespace.

I had recently changed the "upstream" operator policy to create and install into the gatekeeper-system namespace, since the configuration policy controller could not create the openshift-gatekeeper-operator namespace (since the policy controller is not in an openshift namespace). That policy still doesn't work on ROSA - it now installs the gatekeeper operator, but the operator can't create the Gatekeeper operand because it can't create the openshift-gatekeeper-system namespace. So that policy might need to be updated again.

TL;DR - installing the operator into openshift-operators will get around the admission webhook.

[JustinKuli]

I think this line in the original issue is confusing:

Currently we support installing Gatekeeper into any namespace as long as that namespace is also where the operator is running.

At this point, Gatekeeper is installed into either openshift-gatekeeper-system or gatekeeper-system, depending on whether the operator detects that it's in an OpenShift cluster. It does not install Gatekeeper into the same namespace where the operator is running.

[font]

I think this line in the original issue is confusing:

Currently we support installing Gatekeeper into any namespace as long as that namespace is also where the operator is running.

Ah yes, I've now updated the sentence to specify Kubernetes as this issue (and the operator in general) is primarily targeting Kubernetes.

At this point, Gatekeeper is installed into either openshift-gatekeeper-system or gatekeeper-system, depending on whether the operator detects that it's in an OpenShift cluster. It does not install Gatekeeper into the same namespace where the operator is running.

Where are you seeing that behavior? On Kubernetes, the operator installs Gatekeeper into the same namespace as the operator and this issue is primarily referring to that limitation. It's only on OpenShift that the operator uses a different canonical namespace openshift-gatekeeper-system.