Retry flanneld if it fails to start

[manuelbuil]

Proposed Changes

This PR does two things:
1 - Slightly changes one log message because it was raising alarms in our customers/support even though it was harmless: Flanneld can't start because it can't find node ==> Checking if node %s is already registered before starting flanneld
2 - Retrying executing flanneld if it failed. We try 5 times and give up if it always fails. We should retry at least 1 or 2 times, in the case of a windows node with only one interface as explained in the issue. 5 seems like a good number to give up

Types of Changes

Bugfix

Verification

The new log is easy to verify. Just check the rke2 logs

For the second part, make sure everything is running. Then execute: Stop-Process -Name flanneld. Then verify that the rke2 logs include:

4/11/2024 1:05:10 PM  Flanneld has an error: exit status 0xffffffff. Check c:\var\lib\rancher\rke2\agent\logs\flanneld.log for extra information
4/11/2024 1:05:10 PM  Running flanneld with envs: [NODE_NAME=mbuil-windows PATH=C:\var\lib\rancher\rke2\data\v1.29.3-rke2r1-windows-amd64-dfa86afd0b9e\bin:C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\Window
                      sPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program
                      Files\dotnet\;C:\ProgramData\chocolatey\bin;;c:\var\lib\rancher\rke2\bin;c:\usr\local\bin;C:\Windows\system32\config\systemprofile\AppData\Local\Microsoft\WindowsApps] and args:
                      [--kubeconfig-file=c:\var\lib\rancher\rke2\agent\flannel.kubeconfig --ip-masq --kube-subnet-mgr --iptables-forward-rules=false --iface=10.1.1.14
                      --net-config-path=c:\var\lib\rancher\rke2\agent\flanneld-net-conf.json]
4/11/2024 1:05:10 PM  Flanneld exited. Retrying.

Testing

Linked Issues

#5702

User-Facing Change

Retry flanneld.exe if it fails

Further Comments

[thomasferrandiz]

In flannel we use https://github.com/avast/retry-go/ for retries.
I don't know if it would be worth it here?

[manuelbuil]

Thanks for the suggestion Thomas! For this use case, I think it is fine with a simple loop. No need to carry another dependency

[brandond]

What specifically are we trying to fix by retrying here? All other prerequisites are polled for readiness before attempting to start the component. Other components that are started in a retry loop don't have a limit on them and are just retried indefinitely while RKE2 is up.

I feel like we should figure out what is causing flannel to fail initially, and poll for that to become come available before starting flannel, rather than artificially limiting the number of restarts.

If we do for some reason need to artificially limit the restarts, shouldn't we logrus.Fatalf after the for loop, rather than just leaving rke2 running without any further attempt to start the CNI?

[manuelbuil]

The first time flanneld runs, it creates the HNS network, which is an overlay flannel network bound with a physical interface. Because of how windows works, that interface then becomes unavailable for some time (according to my measurements from 10s - 25s). If it is closer to the 25s, flanneld dies because it failed to contact kube-api for too long. That's the case that we are trying to fix here. The 2nd time it runs, if the interface is back, as the network is already there, it does not create it, so there is no downtime for the connections

[manuelbuil]

One thing we could is: run flannel once and then, if it fails, check for the interface to come back. Once we are sure the interface is back, run flannel again. If it fails a second time, then do the fatal failure. What do you think?

[brandond]

Does flannel have an option to do a one-shot startup that will just create the HNS network if necessary, and then exit? We could do that, wait for the network to stabilize, and then start the "permanent" flannel that we expect to not crash?

That's pretty similar to what you were suggesting I guess, except that we would expect flannel to always exit the first time.

[manuelbuil]

Unfortunately that option does not exist. I have an idea I could try tomorrow in Flannel, hopefully before code freeze

[brandond]

What specifically are we trying to fix by retrying here? All other prerequisites are polled for readiness before attempting to start the component. Other components that are started in a retry loop don't have a limit on them and are just retried indefinitely while RKE2 is up.

I feel like we should figure out what is causing flannel to fail initially, and poll for that to become come available before starting flannel, rather than artificially limiting the number of restarts.

If we do for some reason need to artificially limit the restarts, shouldn't we logrus.Fatalf after the for loop, rather than just leaving rke2 running without any further attempt to start the CNI?

[manuelbuil]

Trying this instead #5747, hopefully solving the problem. Closing for the moment