How to reduce impact to reconciliation time on operator start up?

[toshitabata]

Not really a bug report, but more of a discussion question

What did you do?

When an operator is first deployed, it will need to reconcile every custom resource (CR) it watches, regardless of whether there are changes.
This can significantly increase the reconciliation time when making an actual change to a CR.

What did you expect to see?

When I deploy an operator and make a change to a CR it watches, it should prioritise that change over automatic reconciliations, or there should be some way to reduce the impact to reconciliation times.

What did you see instead? Under which circumstances?

Reconciliation of the CR that is created is delayed until all other resources have been reconciled.

Possible Solution

I'm wondering if there's a way to use leader election to warm up the operator before switching over to the new deployment.
https://javaoperatorsdk.io/docs/features/#leader-election
We could also write some kind of priority queue but I'm wondering what's the least invasive way to resolve this kind of issue.

Additional context

Related issue, but for timed reconciliation #2293

[csviri]

Hi @toshitabata!
thx for raising this, basically the leader election works in a way that if a controller is not a leader it populates the caches of informers / event sources what might have already benefits since it might take long to do so; and the reconciliations happen after that (if becomes a leader).

Unfortunatelly when a controller starts, if I remember correctly the informers see all new events as "Add" events, so we cannot event distinguish based on that - if we would like to have some priority executor or something.

We discussed before the idea to have some priority queue and events with a flag that will indicate that reconciliation should be done on a priority queue. Not sure if this would help though, especially on startup; on the other hand will bring complexity for sure.