Security context for shutdown-manager container is always hardcoded to default even if container spec has non-default values set

[saj235]

Description:
Deploying a GatewayClass and Gateway fails in Openshift cluster having strict security constraints in place.

Warning FailedCreate 4m5s (x79 over 6h33m) replicaset-controller Error creating: pods "envoy-test-apps-eg-5a4ee66b-7d96589c85-" is forbidden: unable to validate against any security context constraint

provider restricted-v2: .containers[0].runAsUser: Invalid value: 65532: must be in the ranges: [1000690000, 1000699999],

After adding customized configuration using EnvoyProxy as below, where the UID to be used for containers and pod are explictly set to be in acceptable range, the envoy container starts using the provided UID where as the shutdown-manager container still sticks to the default UID and the deployment still fails.

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: envoy-gateway-system
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyDeployment:
        container:
          securityContext:
            runAsGroup: 1000690011
            runAsUser: 1000690011
        pod:
          securityContext:
            runAsGroup: 1000690012
            runAsUser: 1000690012

Repro steps:
Deploy a GatewayClass and Gateway in an environment with strict security policy in place. eg: Openshift cluster where UIDs to be used by containers are restricted to a certain range that does not include the default UID of 65532.

Environment:
Envoy Gateway version: 1.2.3
Envoy version: 1.32.1
OCP version: 4.16.10
Kubernetes Version: v1.29.7+4510e9c

Logs:
Below errors seen while describing replica set for the failed deployment of GatewayClass and Gateway.

Warning FailedCreate 2m (x20 over 18m) replicaset-controller Error creating: pods "envoy-envoy-gateway-system-eg3-8902bb88-5f5fd5b4c4-" is forbidden: unable to validate against any security context constraint: [provider "anyuid": Forbidden: not usable by user or serviceaccount, provider restricted-v2: .containers[0].runAsUser: Invalid value: 65532: must be in the ranges: [1000690000, 1000699999],

Analysis:
After checking the source code, this seems to be a bug where the security context from container spec is taken into consideration only for the envoy container. Whereas for the shutdown manager, always default security context is set.

gateway/internal/infrastructure/kubernetes/proxy/resource.go

Line 387 in 337e5ac

func expectedShutdownManagerSecurityContext() *corev1.SecurityContext {

func expectedEnvoySecurityContext(containerSpec *egv1a1.KubernetesContainerSpec) *corev1.SecurityContext {
	if containerSpec != nil && containerSpec.SecurityContext != nil {
		return containerSpec.SecurityContext
	}

	sc := resource.DefaultSecurityContext()

	// run as non-root user
	sc.RunAsGroup = ptr.To(int64(65532))
	sc.RunAsUser = ptr.To(int64(65532))

	// Envoy container needs to write to the log file/UDS socket.
	sc.ReadOnlyRootFilesystem = nil
	return sc
}

func expectedShutdownManagerSecurityContext() *corev1.SecurityContext {
	sc := resource.DefaultSecurityContext()

	// run as non-root user
	sc.RunAsGroup = ptr.To(int64(65532))
	sc.RunAsUser = ptr.To(int64(65532))

	// ShutdownManger creates a file to indicate the connection drain process is completed,
	// so it needs file write permission.
	sc.ReadOnlyRootFilesystem = nil
	return sc
}

[arkodg]

One fix here could be to unset default for securityContext for shutdown manager and rely on pod securityContext for its settings

[saj235]

I guess we could reuse the logic in expectedEnvoySecurityContext method to use default ones only when there is nothing set in containerSpec.SecurityContext.

BTW, any possible workaround for temporary unblocking?

[arkodg]

@saj235 to temporarily unblock yourself you could set the securityContext using the patch field https://gateway.envoyproxy.io/docs/tasks/operations/customize-envoyproxy/#patching-deployment-for-envoyproxy