K8S: cluster might be down and cannot schedule the job

[nuclearcat]

Some of jobs are failing, because kubernetes while listed in az cli tool as OK, doesnt really show any node and kubectl failed.

+ az aks list -o table
Name               Location        ResourceGroup    KubernetesVersion    CurrentKubernetesVersion    ProvisioningState    Fqdn
-----------------  --------------  ---------------  -------------------  --------------------------  -------------------  -----------------------------------------------------------
<<skipped>>
aks-kci-s-cent-us  southcentralus  kernelci         1.20.13              1.20.13                     Succeeded            aks-kci-s-cent-us-dns-fef8d055.hcp.southcentralus.azmk8s.io
<<skipped>>
[Pipeline] echo
K8S context: aks-kci-s-cent-us.  Current nodes:
[Pipeline] sh
+ kubectl --context aks-kci-s-cent-us get nodes
The connection to the server aks-kci-s-cent-us-dns-fef8d055.hcp.southcentralus.azmk8s.io:443 was refused - did you specify the right host or port?
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
$ docker stop --time=1 b4913a295f5b6bf109c50b58b701b0bf0780c8365208229c6f39c7ee17679311
$ docker rm -f b4913a295f5b6bf109c50b58b701b0bf0780c8365208229c6f39c7ee17679311
[Pipeline] // withDockerContainer
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 1
Finished: FAILURE

We need to find way, if it is useful to retry and how to get more diagnostic logs.

[padovan]

I am sure this is solved already :)