Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Jwks async fetching failed due to multiple requests per second. #4791

Open
sando38 opened this issue Nov 27, 2024 · 6 comments · Fixed by #4802
Open

Jwks async fetching failed due to multiple requests per second. #4791

sando38 opened this issue Nov 27, 2024 · 6 comments · Fixed by #4802
Labels
help wanted Extra attention is needed
Milestone
v1.3.0-rc.1

Comments

@sando38
Copy link

sando38 commented Nov 27, 2024 •
edited
Loading

Hello, I am using an Envoy Gateway SecurityPolicy with JWT inspection. I have an OIDC provider in the cluster. Periodically, envoy-proxy fires a lot of requests to the remoteJWKS.uri endpoint, i.e. 30 req/s.

<html>
<body>
<!--StartFragment-->
[2024-11-27 12:08:54.875][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
-- | --
  |   | [2024-11-27 12:08:54.502][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.502][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.502][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.502][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.502][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.502][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.502][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.502][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.502][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.502][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.502][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.502][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.502][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.501][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.501][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.501][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.501][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.501][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.501][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.501][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.501][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.501][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.501][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.501][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.501][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.501][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.501][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.501][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.498][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.498][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.498][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.497][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.497][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.497][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.497][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.497][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.497][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.497][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.497][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.497][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.497][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.497][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.497][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.497][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.496][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.496][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.496][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.496][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:54.496][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-11-27 12:08:46.517][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed

<!--EndFragment-->
</body>
</html>

I am running envoy gateway version 1.2.1 w/ 2x proxies, 8x SecurityPolicies in various namespaces . This overwhelms the OIDC provider time to time, therefore, would it make sense to configure the JWT with a retry_policy? I think 1 req/s would be totally sufficient.

Let me know what you think.

Thanks and have a great day
@sando38 sando38 added the triage label Nov 27, 2024
@arkodg
Copy link
Contributor

arkodg commented Nov 27, 2024

@zhaohuabing can we disable retry policy ?
https://github.com/envoyproxy/gateway/blame/a383dd97e4c7fe726f064c03f48a06e8b049001f/internal/xds/translator/jwt.go#L123

we already enable https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/http/jwt_authn/v3/config.proto#extensions-filters-http-jwt-authn-v3-jwksasyncfetch which has a failed_refetch_duration of 1s

@zhaohuabing
Copy link
Member

we already enable https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/http/jwt_authn/v3/config.proto#extensions-filters-http-jwt-authn-v3-jwksasyncfetch which has a failed_refetch_duration of 1s

Yes, I think retry can be removed as jwksasyncfetch has been enabled.

@arkodg arkodg added help wanted Extra attention is needed and removed triage labels Nov 28, 2024
@arkodg arkodg added this to the v1.3.0-rc.1 milestone Nov 28, 2024
@zhaohuabing zhaohuabing mentioned this issue Nov 28, 2024
Merged
@sando38
Copy link
Author

sando38 commented Dec 2, 2024 •
edited
Loading

Hello, I installed gateway 1.2.3 but still seeing the pattern as described above with appr. 200 rqs going to the OIDC server.

<html>
<body>
<!--StartFragment-->
[2024-12-02 08:23:15.748][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
-- | --
  |   | [2024-12-02 08:23:15.748][1][error][filter] [source/extensions/filters/http/common/jwks_fetcher.cc:51] fetch: fetch pubkey [uri = http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/] failed: [cluster = authentik-server_authentik_svc_cluster_local_80] is not configured |  
  |   | [2024-12-02 08:23:15.748][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-12-02 08:23:15.748][1][error][filter] [source/extensions/filters/http/common/jwks_fetcher.cc:51] fetch: fetch pubkey [uri = http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/] failed: [cluster = authentik-server_authentik_svc_cluster_local_80] is not configured |  
  |   | [2024-12-02 08:23:15.748][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-12-02 08:23:15.748][1][error][filter] [source/extensions/filters/http/common/jwks_fetcher.cc:51] fetch: fetch pubkey [uri = http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/] failed: [cluster = authentik-server_authentik_svc_cluster_local_80] is not configured |  
  |   | [2024-12-02 08:23:15.748][1][warning][jwt] [source/extensions/filters/http/jwt_authn/jwks_async_fetcher.cc:115] Jwks async fetching url=http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/: failed |  
  |   | [2024-12-02 08:23:15.748][1][error][filter] [source/extensions/filters/http/common/jwks_fetcher.cc:51] fetch: fetch pubkey [uri = http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/] failed: [cluster = authentik-server_authentik_svc_cluster_local_80] is not configured

<!--EndFragment-->
</body>
</html>

The policy enforcement works, until the OIDC becomes sometimes overloaded.

I am using SecurityPolicy objects like here: #4562 (comment)

I have 8x policies in 6 namespaces running two proxies.

Thank you and let me know if I can provide more information 👍

@arkodg arkodg reopened this Dec 2, 2024
@zhaohuabing
Copy link
Member

  |   | [2024-12-02 08:23:15.748][1][error][filter] [source/extensions/filters/http/common/jwks_fetcher.cc:51] fetch: fetch pubkey [uri = http://authentik-server.authentik.svc.cluster.local:80/application/o/kube-apiserver/jwks/] failed: [cluster = authentik-server_authentik_svc_cluster_local_80] is not configured

This seems like a different issue: the cluster doesn't exist. Can you get the lds and cds form envoy using gctl config envoy-proxy listener and gctl config envoy-proxy cluster when this happens?

@sando38
Copy link
Author

sando38 commented Dec 3, 2024

Sure, will do that!

@sando38
Copy link
Author

sando38 commented Dec 3, 2024 •
edited
Loading

@zhaohuabing I created "three" snapshots:

  • cluster.yaml is when I have freshly applied the JWT inspection, but only the original error code
  • cluster_2.yaml is when the "new" error occurs
  • cluster_without.yaml is the status without JWT in the SecurityPolicy

Same applies to listener*.yaml.

egctl_output.zip

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
help wanted Extra attention is needed
Projects
None yet
Milestone
v1.3.0-rc.1
Development

Successfully merging a pull request may close this issue.

fix: remove the default retry policy for jwks fetch zhaohuabing/gateway
3 participants
@zhaohuabing @arkodg @sando38