Make k3s to shutdown faster

[hlacikd]

Hello,

we are deploying k3s to thousands of edge devices, that are getting turned off several times per day. Our goal is to make shutdown faster than default.
Having disabled all k3s components (coredns, servicelb, traefik,local-storage, metrics-server) shutdown is fast.
Having enabled them it takes around 2 minutes to shutdown, with no other deployments in place.

Is my understanding correct that only option is to make those deployments to shutdown faster by changing
terminationGracePeriodSeconds to less than 30s (which is default)?
Or am i missing something else here?
Thanks in advance.

[ChristianCiach]

I would not recommend this and I've never tried it, but maybe you can configure the shutdown timeouts using the systemd unit file at /etc/systemd/system/k3s*.service: https://unix.stackexchange.com/a/503064

Of course, this only applies if you actually run k3s using systemd.

[brandond]

What do you mean by shutting them down? Are you draining/cordoning the node, deleting the pods, or just trying to halt the node without stopping anything first?

[hlacikd]

halting node with k3s running. my deployments have terminationGracePeriodSeconds: 10 , however it takes 2-3minutes for k3s to gracefully shutdown and node is turned off

[brandond]

In order to allow for nondisruptive upgrades, K3s doesn't stop pods when shutting down. What you're seeing is systemd timing out while unmounting mounts for pod processes. If you want to stop the pods, you should do so before shutting down or restarting the OS. You might look into Kubelet graceful node shutdown.

[hlacikd]

Thank you @brandond , so i was able to pass to /etc/rancher/k3s/config.yaml

kubelet-arg: feature-gates=GracefulNodeShutdown=true

pardon me, but how to pass ShutdownGracePeriod=30s to kubelet config in k3s?

[brandond]

#4319 (reply in thread)