Envoy Gateway OIDC: Error with Self-Signed Certificate #4838

williamdlm · 2024-12-03T02:49:55Z

I have a Kubernetes cluster using Envoy Gateway and I need it to always authenticate with a Keycloak instance running outside the cluster. Since it is in a development environment, it uses a self-signed certificate.

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: oidc-example
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: eg
  oidc:
    provider:
      issuer: "https://my.keycloak.org/realms/myrealm"
    clientID: "envoy"
    clientSecret:
      name: "my-app-client-secret"
    redirectURL: "http://myapp.local:10080/app1"
    logoutPath: "/protocol/openid-connect/logout"

However, when I check this resource after it's created, I get the following error message:

message: 'OIDC: error fetching endpoints from issuer: Get "https://my.keycloak.org/realms/myrealm/.well-known/openid-configuration":
        tls: failed to verify certificate: x509: certificate signed by unknown authority.'

I found a similar case reported here: #3622

Based on the suggestion in the comments here: #3622 (comment)

I tried the following:

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: oidc-example
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: eg
  oidc:
    provider:
      issuer: "https://my.keycloak.org/realms/myrealm"
      backendRefs:
      - name: backend-policy-tls
        port: 443
        kind: BackendTLSPolicy
    clientID: "envoy"
    clientSecret:
      name: "my-app-client-secret"
    redirectURL: "http://myapp.local:10080/app1"
    logoutPath: "/protocol/openid-connect/logout"

apiVersion: gateway.networking.k8s.io/v1alpha3
kind: BackendTLSPolicy
metadata:
  name: backend-policy-tls
spec:
  targetRefs:
  - group: gateway.envoyproxy.io
    kind: SecurityPolicy
    name: oidc-example
  validation:
    caCertificateRefs:
    - group: ""
      kind: Secret
      name: keycloak-cert
    hostname: my.keycloak.org

However, the same error persists: "tls: failed to verify certificate: x509: certificate signed by unknown authority."

I need help with this, preferably with an example of the correct way to make this work.

The text was updated successfully, but these errors were encountered:

arkodg · 2024-12-05T03:05:31Z

cc @zhaohuabing

shivanidwivedi10 · 2024-12-05T14:26:03Z

Hi @arkodg I am also facing the same issue. Kindly find the details below

We are using Envoy Gateway (installed via Helm chart) as the Gateway API in our MicroK8s cluster running on a Red Hat VM in an air-gapped environment. Our setup includes:

Backend application running as a pod inside the MicroK8s cluster.
Keycloak server also deployed as a pod in the same cluster.
Domain: https://xxx.yyy.cloud, which resolves via /etc/hosts entries and is secured with a self-signed certificate.

Current Behavior:
Accessing https://xxx.yyy.cloud works as expected in Firefox (after bypassing the warning for the self-signed certificate).
Without Keycloak as the authentication server, everything functions correctly.
We have also made our host machine trust the self-signed certificates.

Problem:
Even after making host trust the certificate and after creating a security policy that uses Keycloak for authentication, we still receive the following error:
OIDC: error fetching endpoints from issuer: get "https://xxx.yyy.cloud/realms/myrealm/.well-known/openid-configuration": tls: failed to verify certificate: x509: certificate signed by unknown authority.

Goal:
Make the security policy trust the self-signed certificate we’ve created. We need to either:

Configure the security policy to trust the self-signed certificate, or
Find a way to bypass certificate validation in the security policy.

Please find the yaml files we have used:

 -backend.yaml
 apiVersion: gateway.envoyproxy.io/v1alpha1 
  kind: Backend
  metadata:
    name: backend-keycloak
    namespace: some-ns
  spec:
    endpoints:
    - fqdn:
        hostname: 'xxx.yyy.cloud'
        port: 443

-backend-tls-policy.yaml
apiVersion: gateway.networking.k8s.io/v1alpha3 
kind: BackendTLSPolicy
metadata:
  name: keycloak-backend-policy
  namespace: some-ns
spec:
  targetRef:
  -  group: gateway.envoyproxy.io
     kind: Backend
     name: backend-keycloak
  validation:
    caCertificateRefs:  
    -   group: ""
        name: tls-ca
        kind: Secret     #usedConfigMap as well
    hostname: xxx.yyy.cloud


-security-policy.yaml
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: xxx-security-policy
  namespace: some-ns
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: gateway-routes-oidc
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: gateway-api-routes-oidc
  oidc:
    provider:
      issuer: "https://xxx.yyy.cloud/realms/myrealm"
      backendRefs:
        - group: gateway.envoyproxy.io
          kind: Backend
          name: backend-keycloak
          port: 443
        backendSettings:
          retry:
            numRetries: 3
            perRetry:
              backOff:
                baseInterval: 1s
                maxInterval: 5s
            retryOn:
              triggers: ["5xx", "gateway-error", "reset"]
    clientID: "some.envoy-oidc.client"
    clientSecret:
      name: "my.oidc.client-secret"
    redirectURL: "https://xxx.yyy.cloud/xxxyyy/main/oauth2/callback"
    logoutPath: "/xxxyyy/main/logout"
    forwardAccessToken: true
    refreshToken: true
    cookieNames:
      accessToken: mmmmm
      idToken: nnnnnnn

Even after using backendrefs and BackendTLSPolicy, we are still facing the same OIDC error.
Kindly let us know if we are missing something in the yaml files I provided above or if you could share some example files we can refer to. This is something we are facing and trying to solve from past 2-3 days now.

zhaohuabing · 2024-12-06T09:35:25Z

The specified ca is only used by the envoy. Envoy Gateway doesn't use it to fetch the endpoints from issue's well-known openid configuration url.

@williamdlm @shivanidwivedi10 As a workaround, you can explicitly set the authorizationEndpoint adn tokenEndpoint.

shivanidwivedi10 · 2024-12-06T10:03:11Z

Thank you for checking this! @zhaohuabing I have tried with authorizationEndpoint and tokenEndpoint as well, with this we are able to get the Keycloak page after entering Username and password, it gives a blank page stating OAuth Flow Failed.
NOTE: I am doing this deployment in an air-gapped machine (RHEL 8).
Any suggestions!

zhaohuabing · 2024-12-06T14:16:57Z

d with authorizationEndpoint and tokenEndpoint as well, with this we are able to get the Keycloak page after entering Username and password, it gives a blank page stating OAuth Flow Failed.
NOTE: I am doing this deployment in an air-gapped machine (RHEL 8)

@shivanidwivedi10 Keycloak should work. Envoy Gateway runs an e2e test with Keycloak for every commit. You can comapre your configuration with the e2e to see if there are any noticable differences.

https://github.com/envoyproxy/gateway/blob/main/test/e2e/testdata/oidc-securitypolicy.yaml
https://github.com/envoyproxy/gateway/blob/main/test/e2e/testdata/oidc-keycloak.yaml
https://github.com/envoyproxy/gateway/blob/main/test/e2e/testdata/oidc-securitypolicy-backendcluster.yaml

Are there any errors in the envoy log when the OAuth flow failed?

williamdlm · 2024-12-06T18:50:24Z

The specified ca is only used by the envoy. Envoy Gateway doesn't use it to fetch the endpoints from issue's well-known openid configuration url.

@williamdlm @shivanidwivedi10 As a workaround, you can explicitly set the authorizationEndpoint adn tokenEndpoint.

Thank you for your response! Using authorizationEndpoint and tokenEndpoint, the SecurityPolicy works without the previous error.
However, like @shivanidwivedi10, I encountered the following error: "OAuth Flow Failed", but I couldn't find any logs related to this issue in Keycloak.

@shivanidwivedi10 Keycloak should work. Envoy Gateway runs an e2e test with Keycloak for every commit. You can comapre your configuration with the e2e to see if there are any noticable differences.

https://github.com/envoyproxy/gateway/blob/main/test/e2e/testdata/oidc-securitypolicy.yaml https://github.com/envoyproxy/gateway/blob/main/test/e2e/testdata/oidc-keycloak.yaml https://github.com/envoyproxy/gateway/blob/main/test/e2e/testdata/oidc-securitypolicy-backendcluster.yaml

Are there any errors in the envoy log when the OAuth flow failed?

Here are my configurations for comparison:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: app1-route
  namespace: default
spec:
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: app1-service
      port: 80
    matches:
    - path:
        type: PathPrefix
        value: /app1
        
---

apiVersion: v1
data:
  client-secret: MnkyWnpFMThsbWFDYXE0U0VFcVhZU3hmTnBQT2VKb2g=
kind: Secret
metadata:
  name: my-app-client-secret
  namespace: default
type: Opaque

---

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: oidc-example
  namespace: default
spec:
  oidc:
    clientID: envoy
    clientSecret:
      group: ""
      kind: Secret
      name: my-app-client-secret
    logoutPath: /protocol/openid-connect/logout
    provider:
      authorizationEndpoint: https://my.keycloak.org/realms/master/protocol/openid-connect/auth
      issuer: https://my.keycloak.org/realms/master
      tokenEndpoint: https://my.keycloak.org/realms/master/protocol/openid-connect/token
    redirectURL: http://myapp.local:10080/app1
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: app1-route

I reviewed them alongside your example, but I didn't notice anything unusual that could explain the issue.

williamdlm · 2024-12-06T22:23:42Z

The specified ca is only used by the envoy. Envoy Gateway doesn't use it to fetch the endpoints from issue's well-known openid configuration url.
@williamdlm @shivanidwivedi10 As a workaround, you can explicitly set the authorizationEndpoint adn tokenEndpoint.

Thank you for your response! Using authorizationEndpoint and tokenEndpoint, the SecurityPolicy works without the previous error. However, like @shivanidwivedi10, I encountered the following error: "OAuth Flow Failed", but I couldn't find any logs related to this issue in Keycloak.

@shivanidwivedi10 Keycloak should work. Envoy Gateway runs an e2e test with Keycloak for every commit. You can comapre your configuration with the e2e to see if there are any noticable differences.
https://github.com/envoyproxy/gateway/blob/main/test/e2e/testdata/oidc-securitypolicy.yaml https://github.com/envoyproxy/gateway/blob/main/test/e2e/testdata/oidc-keycloak.yaml https://github.com/envoyproxy/gateway/blob/main/test/e2e/testdata/oidc-securitypolicy-backendcluster.yaml
Are there any errors in the envoy log when the OAuth flow failed?

Here are my configurations for comparison:
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: app1-route
  namespace: default
spec:
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: app1-service
      port: 80
    matches:
    - path:
        type: PathPrefix
        value: /app1
        
---

apiVersion: v1
data:
  client-secret: MnkyWnpFMThsbWFDYXE0U0VFcVhZU3hmTnBQT2VKb2g=
kind: Secret
metadata:
  name: my-app-client-secret
  namespace: default
type: Opaque

---

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: oidc-example
  namespace: default
spec:
  oidc:
    clientID: envoy
    clientSecret:
      group: ""
      kind: Secret
      name: my-app-client-secret
    logoutPath: /protocol/openid-connect/logout
    provider:
      authorizationEndpoint: https://my.keycloak.org/realms/master/protocol/openid-connect/auth
      issuer: https://my.keycloak.org/realms/master
      tokenEndpoint: https://my.keycloak.org/realms/master/protocol/openid-connect/token
    redirectURL: http://myapp.local:10080/app1
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: app1-route
I reviewed them alongside your example, but I didn't notice anything unusual that could explain the issue.

I adjusted the redirectURL value in my SecurityPolicy to http://myapp.local:10080/oauth2/callback because I have an HTTPRoute that redirects to my service. After making this change, I noticed a change in behavior: when I send a request, I am redirected to the Keycloak login screen instead of receiving an unauthorized response, even though I did not provide a valid token. Furthermore, when I attempt to send a valid token, it still redirects me to the Keycloak login screen.

curl -i myapp.local:10080/app1
HTTP/1.1 302 Found
set-cookie: OauthNonce-76fd001b=1733523382007600;path=/;Max-Age=600;secure;HttpOnly
location: https://my.keycloak.org/realms/master/protocol/openid-connect/auth?client_id=envoy&redirect_uri=http%3A%2F%2Fmyapp.local%3A10080%2Foauth2%2Fcallback&response_type=code&scope=openid&state=url%3Dhttp%253A%252F%252Fmyapp.local%253A10080%252Fapp1%26nonce%3D1733523382007600
date: Fri, 06 Dec 2024 22:16:21 GMT
content-length: 0

I expect that when I make a request to the /app1 path with a valid token, the SecurityPolicy will authenticate with my Keycloak and redirect to the service specified in the redirectURL.

Any tips or advice on how to achieve the expected behavior?

zhaohuabing · 2024-12-07T02:26:07Z

@williamdlm The callback url should not be the same as your httproute path matching.

Change the redirect url to : http://myapp.local:10080/app1/callback in both the SecurityPlolicy and the Keyloak configuration, then it should work.

sachinshaji · 2024-12-07T05:55:25Z

I work with @shivanidwivedi10
We made it working with authorizationEndpoint and tokenEndpoint as discussed in #4838 (comment)

We were having an nginx(ingress) as a reverse proxy and it was blocking the request/authentication token from header, due the higher buffer size limit in ingress. We check the logs of ingress controller and found the error logs. Just put the logs in chatgpt and it suggested to add more annotations to ingress resource to increase the buffer size.

Wooho!! things starts to work after that. Thanks @zhaohuabing and for you support.

@williamdlm if needed more details or information @shivanidwivedi10 can share the annotations we use in ingress.

williamdlm · 2024-12-09T12:35:34Z

@zhaohuabing, thank you for your response.
I tried, but unfortunately, I am still facing the problem. I believe it might be related to some configuration on my side.
@sachinshaji, if @shivanidwivedi10 could share it with me, I would greatly appreciate it.

williamdlm · 2024-12-09T19:21:22Z

Hi @zhaohuabing,

I noticed that the log from envoy-default-eg contains the following entry:

2024-12-09T15:34:25.723298373Z envoy {"start_time":"2024-12-09T15:34:20.610Z","method":"POST","x-envoy-origin-path":"/app1","protocol":"HTTP/1.1","response_code":"302","response_flags":"-","response_code_details":"oauth.missing_credentials","connection_termination_details":"-","upstream_transport_failure_reason":"-","bytes_received":"0","bytes_sent":"0","duration":"2","x-envoy-upstream-service-time":"-","x-forwarded-for":"10.244.0.25","user-agent":"insomnia/10.1.1","x-request-id":"2e5d7a69-27ce-4d3d-821b-57602765a2b0",":authority":"myapp.local:10080","upstream_host":"-","upstream_cluster":"httproute/default/app1-route/rule/0","upstream_local_address":"-","downstream_local_address":"127.0.0.1:10080","downstream_remote_address":"127.0.0.1:55240","requested_server_name":"-","route_name":"httproute/default/app1-route/rule/0/match/0/*"}

The detail "response_code_details":"oauth.missing_credentials" caught my attention. It seems similar to issue #4718.

Could you provide guidance or insights on how to resolve this?

arkodg · 2024-12-10T04:26:25Z

@williamdlm which version are you on ? can you bump to v1.2.2

sachinshaji · 2024-12-10T05:09:34Z

Hello @williamdlm.
As discussed in #4718 we are also facing the issues sporadically. After the configuration which worked couple of days back, we cleaned up everything and installed again. Again 'OAuth flow failed' error starts popping up. Then we have deleted the ingress and security policy couple of times and tested, then is starts to work again.
Now we are also confused about this as it happens sporadically and without any change in configuration. We try with envoy version v1.2.3.
NOTE: We could also see the error message you have posted in the previous chat.

williamdlm · 2024-12-10T12:05:09Z

@arkodg,

@williamdlm which version are you on ? can you bump to v1.2.2

I'm actually using version 1.2.3; however, as per your recommendation, I tried v1.2.2, but the problem persists.

2024-12-10T11:58:39.455121669Z envoy {"start_time":"2024-12-10T11:58:32.755Z","method":"POST","x-envoy-origin-path":"/app1","protocol":"HTTP/1.1","response_code":"302","response_flags":"-","response_code_details":"oauth.missing_credentials","connection_termination_details":"-","upstream_transport_failure_reason":"-","bytes_received":"0","bytes_sent":"0","duration":"0","x-envoy-upstream-service-time":"-","x-forwarded-for":"10.244.0.10","user-agent":"insomnia/10.1.1","x-request-id":"a395385d-0d90-4fa1-9e8d-2fb35a706cc6",":authority":"myapp.local:10080","upstream_host":"-","upstream_cluster":"httproute/default/app1-route/rule/0","upstream_local_address":"-","downstream_local_address":"127.0.0.1:10080","downstream_remote_address":"127.0.0.1:50008","requested_server_name":"-","route_name":"httproute/default/app1-route/rule/0/match/0/*"}

zhaohuabing · 2024-12-10T13:02:02Z

Hi @zhaohuabing,

I noticed that the log from envoy-default-eg contains the following entry:

2024-12-09T15:34:25.723298373Z envoy {"start_time":"2024-12-09T15:34:20.610Z","method":"POST","x-envoy-origin-path":"/app1","protocol":"HTTP/1.1","response_code":"302","response_flags":"-","response_code_details":"oauth.missing_credentials","connection_termination_details":"-","upstream_transport_failure_reason":"-","bytes_received":"0","bytes_sent":"0","duration":"2","x-envoy-upstream-service-time":"-","x-forwarded-for":"10.244.0.25","user-agent":"insomnia/10.1.1","x-request-id":"2e5d7a69-27ce-4d3d-821b-57602765a2b0",":authority":"myapp.local:10080","upstream_host":"-","upstream_cluster":"httproute/default/app1-route/rule/0","upstream_local_address":"-","downstream_local_address":"127.0.0.1:10080","downstream_remote_address":"127.0.0.1:55240","requested_server_name":"-","route_name":"httproute/default/app1-route/rule/0/match/0/*"}

The detail "response_code_details":"oauth.missing_credentials" caught my attention. It seems similar to issue #4718.

Could you provide guidance or insights on how to resolve this?

@williamdlm @sachinshaji
This is not an error, Envoy redirected the user request to the provider's auth endpoint because the user hasn't logged in.

williamdlm · 2024-12-11T00:33:59Z

Hi @zhaohuabing,

Thanks again for your response. I understand that this is the default behavior of Keycloak, but I would like it to use the Client Credentials Flow.

When I send a request, Envoy's default behavior of redirecting to the Keycloak login screen triggers the Authorization Code Flow with User Authentication in Keycloak.

Is there a configuration in Envoy to force Keycloak to use the Client Credentials Flow?

What I need is for it to validate the token already sent in the request header, and if valid, redirect to the specified URL.

NOTE: I believe the necessary configurations for the Client Credentials Flow are already enabled in Keycloak.

zhaohuabing · 2024-12-11T03:25:06Z

@williamdlm the Envoy OAuth2 filter currently only supports the auth code flow. If you alreay have a token in the request header, you can use the JWT authn in the SecurityPolicy to validate it.

williamdlm · 2024-12-11T18:43:37Z

@zhaohuabing , Thank you for the clarification! I followed your suggestion and used the JWT authentication in the SecurityPolicy to validate the token, and it worked perfectly. Do you know if there are any plans to update the Envoy OAuth2 filter to support the Client Credentials flow in the future?

zhaohuabing · 2024-12-12T02:01:47Z

Do you know if there are any plans to update the Envoy OAuth2 filter to support the Client Credentials flow in the future?

@williamdlm
The Envoy OAuth2 filter is mainly designed for user login instead of machine to machine authentication. AFAIK, currently there is no plan to support Client Credential flow. You can raise an issue in Envoy to track this if you want it.

williamdlm added the triage label Dec 3, 2024

zhaohuabing self-assigned this Dec 6, 2024

zhaohuabing added kind/bug Something isn't working and removed triage labels Dec 6, 2024

zhaohuabing mentioned this issue Dec 6, 2024

fix: use tls config from BTP when connecting to the OIDC provider's well-known endpoint. #4857

Open

This comment was marked as off-topic.

Sign in to view

williamdlm closed this as completed Dec 11, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Envoy Gateway OIDC: Error with Self-Signed Certificate #4838

Envoy Gateway OIDC: Error with Self-Signed Certificate #4838

williamdlm commented Dec 3, 2024

arkodg commented Dec 5, 2024

shivanidwivedi10 commented Dec 5, 2024

zhaohuabing commented Dec 6, 2024 •

edited

Loading

shivanidwivedi10 commented Dec 6, 2024 •

edited

Loading

zhaohuabing commented Dec 6, 2024 •

edited

Loading

williamdlm commented Dec 6, 2024

williamdlm commented Dec 6, 2024

zhaohuabing commented Dec 7, 2024 •

edited

Loading

sachinshaji commented Dec 7, 2024 •

edited

Loading

williamdlm commented Dec 9, 2024

williamdlm commented Dec 9, 2024

arkodg commented Dec 10, 2024

sachinshaji commented Dec 10, 2024

williamdlm commented Dec 10, 2024 •

edited

Loading

This comment was marked as off-topic.

zhaohuabing commented Dec 10, 2024 •

edited

Loading

williamdlm commented Dec 11, 2024

zhaohuabing commented Dec 11, 2024 •

edited

Loading

williamdlm commented Dec 11, 2024

zhaohuabing commented Dec 12, 2024 •

edited

Loading

Envoy Gateway OIDC: Error with Self-Signed Certificate #4838

Envoy Gateway OIDC: Error with Self-Signed Certificate #4838

Comments

williamdlm commented Dec 3, 2024

arkodg commented Dec 5, 2024

shivanidwivedi10 commented Dec 5, 2024

zhaohuabing commented Dec 6, 2024 • edited Loading

shivanidwivedi10 commented Dec 6, 2024 • edited Loading

zhaohuabing commented Dec 6, 2024 • edited Loading

williamdlm commented Dec 6, 2024

williamdlm commented Dec 6, 2024

zhaohuabing commented Dec 7, 2024 • edited Loading

sachinshaji commented Dec 7, 2024 • edited Loading

williamdlm commented Dec 9, 2024

williamdlm commented Dec 9, 2024

arkodg commented Dec 10, 2024

sachinshaji commented Dec 10, 2024

williamdlm commented Dec 10, 2024 • edited Loading

This comment was marked as off-topic.

zhaohuabing commented Dec 10, 2024 • edited Loading

williamdlm commented Dec 11, 2024

zhaohuabing commented Dec 11, 2024 • edited Loading

williamdlm commented Dec 11, 2024

zhaohuabing commented Dec 12, 2024 • edited Loading

zhaohuabing commented Dec 6, 2024 •

edited

Loading

shivanidwivedi10 commented Dec 6, 2024 •

edited

Loading

zhaohuabing commented Dec 6, 2024 •

edited

Loading

zhaohuabing commented Dec 7, 2024 •

edited

Loading

sachinshaji commented Dec 7, 2024 •

edited

Loading

williamdlm commented Dec 10, 2024 •

edited

Loading

zhaohuabing commented Dec 10, 2024 •

edited

Loading

zhaohuabing commented Dec 11, 2024 •

edited

Loading

zhaohuabing commented Dec 12, 2024 •

edited

Loading