Cross Namespace ReferenceGrant is not working for SecurityPolicy [ExAuth]

[sekar-saravanan]

Description:
The ExAuth service could not grant the SecurityPolicy in a Cross Namespace setup. https://envoyproxy.slack.com/archives/C03E6NHLESV/p1722915047921119?thread_ts=1714637933.500289&cid=C03E6NHLESV

Repro steps:
Step 1: Create the following namespaces for different resources:

• Namespace1 (e.g., envoy-gateway) - For gateway components
• Namespace2 (e.g., default) - For service components
• Namespace3 (e.g., auth) - For auth components

Step 2: Deploy envoy-gateway in Namespace1 (envoy-gateway).
Step 3: Deploy service resources with httproute in Namespace2 (default).

apiVersion: v1
kind: ServiceAccount
metadata:
  name: backend
  namespace: default
---
apiVersion: v1
kind: Service
metadata:
  name: backend
  namespace: default
  labels:
    app: backend
    service: backend
spec:
  ports:
    - name: http
      port: 3000
      targetPort: 3000
  selector:
    app: backend
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: backend
      version: v1
  template:
    metadata:
      labels:
        app: backend
        version: v1
    spec:
      serviceAccountName: backend
      containers:
        - image: gcr.io/k8s-staging-gateway-api/echo-basic:v20231214-v1.0.0-140-gf544a46e
          imagePullPolicy: IfNotPresent
          name: backend
          ports:
            - containerPort: 3000
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: backend
  namespace: default
spec:
  parentRefs:
    - name: envoy-gateway
      namespace: envoy-gateway
  hostnames:
    - "www.example.com"
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: backend
          port: 3000
          weight: 1
      matches:
        - path:
            type: PathPrefix
            value: /

Step 4: Verify the app using the following curl command

curl --header "Host: www.example.com" https://${GATEWAY_EXTENAL_IP}/

Step 5: Deploy the Auth Service in Namespace3 (auth) using the command below.

kubectl apply -f https://raw.githubusercontent.com/envoyproxy/gateway/latest/examples/kubernetes/ext-auth-http-service.yaml -n auth

Step 6: Add a SecurityPolicy for the HTTPRoute in Namespace2 (default) to enable authentication

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: backend
  namespace: default
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: backend
  extAuth:
    http:
      backendRef:
        name: http-ext-auth
        namespace: auth
        port: 9002
      headersToBackend: ["x-current-user"]

Step 7: Add a ReferenceGrant in Namespace3 (auth) to allow the SecurityPolicy (in namespace: default) to use the Auth Service (in namespace: auth) across namespaces.

apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: auth
  namespace: auth
spec:
  from:
  - group: gateway.networking.k8s.io
    kind: SecurityPolicy
    namespace: default
  to:
  - group: ""
    kind: Service
    name: http-ext-auth

Environment:
Envoy Gateway Version: v1.1.0
Kubernetes Version: 1.28

Error:
Error noted in SecurityPolicy resource.

Status:
  Ancestors:
    Ancestor Ref:
      Group:      gateway.networking.k8s.io
      Kind:       Gateway
      Name:       envoy-gateway
      Namespace:  envoy-gateway
    Conditions:
      Last Transition Time:  2024-08-07T05:27:07Z
      Message:               ExtAuth: backend ref to Service auth/http-ext-auth not permitted by any ReferenceGrant.
      Observed Generation:   1
      Reason:                Invalid
      Status:                False
      Type:                  Accepted
    Controller Name:         gateway.envoyproxy.io/gatewayclass-controller

Expected Behaviour
SecurityPolicy backend ref to Service auth/http-ext-auth should be granted by ReferenceGrant.

[ardikabs]

@sekar-saravanan
Are you running multiple ReferenceGrant resources in your service namespace, in that case, the auth namespace?

If so, you might be encountering issue #2149

[sekar-saravanan]

@ardikabs No, this is the first ReferenceGrant resource which I deployed in auth namespace.

[ardikabs]

hi @sekar-saravanan , after taking a look once again, it seems you are mistakenly using the wrong group for SecurityPolicy, instead of gateway.networking.k8s.io, it should be gateway.envoyproxy.io .

[ardikabs]

I've tested with the manifest below, and no errors were encountered. So, this doesn't appear to be a bug, @arkodg.

apiVersion: gateway.networking.k8s.io/v1beta1
kind: ReferenceGrant
metadata:
  name: auth
  namespace: auth
spec:
  from:
  - group: gateway.envoyproxy.io
    kind: SecurityPolicy
    namespace: default
  to:
  - group: ""
    kind: Service
    name: http-ext-auth

[arkodg]

Thanks for triaging the issue @ardikabs, closing this one !

[raghukumarc]

hi @sekar-saravanan , after taking a look once again, it seems you are mistakenly using the wrong group for SecurityPolicy, instead of gateway.networking.k8s.io, it should be gateway.envoyproxy.io .

Than you, @ardikabs .
Couldn't find any example which showed the Referent to be SecurityPolicy. The assumption was the ReferenceGrant needs to be given to the targetRefs (HTTProute) within the SecurityPolicy whose request is being redirected to auth service. So, I was making the mistake of using:

spec:
from:
- group: gateway.networking.k8s.io
kind: HTTPRoute
namespace: frontend-ui

 instead of configuring it:

spec:
from:
- group: gateway.envoyproxy.io
kind: SecurityPolicy
namespace: frontend-ui