[1.17] Retry on leader lease renewal failure

[davidjumani]

Description

Backport of solo-io#9563

After a container has become a leader, any Kube API server unavailability results in the container crashing. This happens as the leader is unable to renew the lease. This is by design as outlined here

if we allowed gloo to continue to function as a leader during kube apiserver outage, we risk having two leaders in other failure modes. we should remove the panic and allow gloo to continue to serve last-known xds as a follower (effectively having two followers until kube apiserver recovers). this idea is similar to the role xds relay could play for gloo edge

However, crashing the gloo pods can also lead to an outage during scaling as the gateway-proxy pods that come up cannot fetch any configs and all routes result in 404s.

Now instead of crashing, the gloo pod falls back to a follower. This prevents an outage and any other pod can take over as leader if possible
This is fine as a leader only writes reports / statuses over here and here. On any failure, the pod becomes a follower and if elected back as a leader will continue to write reports.

Code changes

• Introduce a new Reset method on the identity implementation that allows an identity to fall back to a follower

CI changes

• Kind clusters that run in CI now do not come up with their own CNI as kindnet does not support custom network policies.
  Instead, cilium is installed as CNI as we need to test Kube API server unavailability

Context

Kube API unavailability results in a gloo container crash
When leader election fails, gloo crashes
Design Doc

Interesting decisions

Testing steps

Kube2e tests to verify the following :

• It can recover from a Kube APIServer unavailability
• It falls back to a follower
• Another leader can get elected

Notes for reviewers

Be sure to verify intended behavior by ...

Please proofread comments on ...

This is a complex PR and may require a huddle to discuss ...

Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added tests that prove my fix is effective or that my feature works

BOT NOTES:
resolves solo-io#8107

[solo-changelog-bot]

Issues linked to changelog:
solo-io#8107

[nfuden]

Im surprised that new_feature in changelog passed bot but looks good to me