Reflector not watching secrets after period of time (still happening)

[gwenael-lebarzic]

Hello.

As issue #341 is closed, I open a new one.

As described in #341 , we encountered the same problem the 7th of October 2024, Reflector stopped replicating secrets.
Reflector did not log anything anymore (neither namespace watcher, configmap watcher or secret watcher).

Here is the end of the log :

2024-10-06 21:36:26.009 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Auto-reflected my-ns1/my-secret where permitted. Created 0 - Updated 19 - Deleted 0 - Validated 0.
2024-10-06 21:36:50.648 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretWatcher) Session closed. Duration: 00:35:36.4365983. Faulted: False.
2024-10-06 21:36:50.649 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretWatcher) Requesting V1Secret resources
2024-10-06 21:36:50.755 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Auto-reflected my-ns1/my-secret2 where permitted. Created 0 - Updated 0 - Deleted 0 - Validated 18.
2024-10-06 21:36:50.803 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Auto-reflected my-ns1/my-secret3 where permitted. Created 0 - Updated 0 - Deleted 0 - Validated 19.
2024-10-06 21:36:50.868 +00:00 [INF] (ES.Kubernetes.Reflector.Core.SecretMirror) Auto-reflected my-ns1/my-secret where permitted. Created 0 - Updated 0 - Deleted 0 - Validated 19.
2024-10-06 22:04:18.618 +00:00 [INF] (ES.Kubernetes.Reflector.Core.NamespaceWatcher) Session closed. Duration: 00:56:30.0338703. Faulted: False.
2024-10-06 22:04:18.618 +00:00 [INF] (ES.Kubernetes.Reflector.Core.NamespaceWatcher) Requesting V1Namespace resources
2024-10-06 22:08:14.073 +00:00 [INF] (ES.Kubernetes.Reflector.Core.ConfigMapWatcher) Session closed. Duration: 00:44:21.4212938. Faulted: False.
2024-10-06 22:08:14.073 +00:00 [INF] (ES.Kubernetes.Reflector.Core.ConfigMapWatcher) Requesting V1ConfigMap resources

After this time, there was no log at all anymore. Concerning the metrics, the pod reflector cpu was almost zero (seems normal because it wasn't doing anything anymore. Nothing specific about the memory usage just before the incident.

Here are informations about version :

• Reflector: emberstack/kubernetes-reflector:7.1.256
• cluster: GKE - 1.29.8-gke.1096000
• Cloud Provider: GCP

Is it possible to solve this problem please ? It makes reflector solution unstable unfortunately :(.

[gwenael-lebarzic]

Hello. I don't know if someone could check the problem ?

[gwenael-lebarzic]

Up

[enterdv]

Same for me

• Reflector: emberstack/kubernetes-reflector:7.1.288
• cluster: GKE - v1.30.5-gke.1014001
• Cloud Provider: GCP

[gwenael-lebarzic]

Hello.

Is it possible to get a status on this behaviour please ?

Best regards.

[iusergii]

Started to observe the same issue once our cluster reached 5k secrets in all namespaces. My guess is that Kubernetes API paginates responses ListSecretForAllNamespacesAsync and code is not handling pagination.

[gwenael-lebarzic]

Started to observe the same issue once our cluster reached 5k secrets in all namespaces. My guess is that Kubernetes API paginates responses ListSecretForAllNamespacesAsync and code is not handling pagination.

In my kubernetes cluster, where we have this problem, we have a total of 62 secrets.

[enterdv]

Started to observe the same issue once our cluster reached 5k secrets in all namespaces. My guess is that Kubernetes API paginates responses ListSecretForAllNamespacesAsync and code is not handling pagination.

I have 4 secrets in my cluster and the issue exists.

[iusergii]

Did you try to set watcher timeout ?