[Question] How to recover from provision state failed at Node Pool

[jkroepke]

Describe scenario
We are running multiple AKS and we have weekly automatic patching enabled.

Some of our critical pod have a PDB configured.

In some rate condition, the automatic update failed. The reason is that there is a timeout. A configured PDB, deny to drain a node.

Question

To recover from that situation, i have to manually restart the pod. Thats fine.

However. The node pool remains in Failed state, including the extra nodes remains as well.

How I can recover from that state? How I can re-trigger the automatic update?

One solution is manually delete the old VM from the VMSS. But thats kinda tricky on Node Pools which large amount of nodes.

[JoeyC-Dev]

az aks update -n $aks -g $rG

No other arguments/parameters.

[jkroepke]

So is no Portal Experience, right? I can't trigger update at portal, if its on the latest version.

[JoeyC-Dev]

So is no Portal Experience, right? I can't trigger update at portal, if its on the latest version.

From document, yes. Maybe someone else knows it is hidden somewhere.

https://learn.microsoft.com/troubleshoot/azure/azure-kubernetes/availability-performance/cluster-node-virtual-machine-failed-state

[jkroepke]

Thanks, I will try that on next incident!