Reminder doesn't start after Kubernetes pod eviction

[aviviadisap]

Hi, I’m using .NET 6 and Orleans 3.7.2 with KubernetesHosting:
silo.ConfigureServices(services => services.UseKubernetesHosting());

We need to run a periodic single-silo task every 2 seconds. Since Orleans reminders have a minimum interval of 1 minute, we're simulating this with a long-running 1-year reminder and a while (shutdownCtk.IsCancelled == false) { doWork; sleep 2 seconds; } loop.

This approach works well after the initial deployment of the cluster. However, if Kubernetes evicts the pod running this reminder, the task doesn't resume on another pod until we manually restart one of the silos.

What might be causing this issue?

[ReubenBond]

It's hard to say what's causing this issue without code or logs - are you able to provide more details on your implementation? I assume "sleep 2 seconds" is a Task.Delay call?

Perhaps you want something similar to this: https://gist.github.com/ReubenBond/8dc47f68634faafe334fadc6f5444ad0