InhibitDelayMaxSec period is not honoured.

[ravisekhar-akkineni]

Hi

I have enabled GracefulNodeShutdownBasedOnPodPriority and also created different priority classes as listed below,

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
featureGates:
  GracefulNodeShutdownBasedOnPodPriority: true
shutdownGracePeriodByPodPriority:
   - priority: 5
     shutdownGracePeriodSeconds: 100
   - priority: 4
     shutdownGracePeriodSeconds: 100
   - priority: 3
     shutdownGracePeriodSeconds: 100
   - priority: 2
     shutdownGracePeriodSeconds: 100
   - priority: 1
     shutdownGracePeriodSeconds: 100
   - priority: 0
     shutdownGracePeriodSeconds: 100

terminationGracePeriodSeconds is not configured for any of the pods belong to above priority groups and also intentionally added code to ignore SIGTERM for all of these 6 PODs. As expected InhibitDelayMaxSec set to 600 in /etc/systemd/logind.conf.d/99-kubelet.conf file.

I expected that if "systemctl reboot" is executed, systemd waits for atleast 600 seconds as kubelet has to wait for 100 seconds for each of the pod (since SIGTERM is ignored) but device is getting rebooted always after 30 seconds.

What might be the reason for rebooting system after 30 seconds?

[brandond]

I suspect that graceful node shutdown's systemd inhibitor locks may expect that the kubelet is running in its own systemd unit. Can you confirm if rke2's shutdown is being delayed, or is it terminated immediately and the lock ignored?

[brandond]

I suspect you're going to have to tinker with this a bit to figure out what's going on. The kubelet is supposed to create an inhibitor lock that prevents systemd from actually starting the shutdown process. This is all managed directly between the kubelet and systemd; RKE2 is not involved. I suspect that the kubelet needs to be running directly under systemd in order to communicate with it via the notification socket to create the lock, but I don't know that for sure.

[ravisekhar-akkineni]

kubelet is taking inhibit lock as part of handling shutdown event. I have also dumped the systemd inhibit locks below.

Below code snippet taken from kubernetes/pkg/kubelet/nodeshutdown/systemd/inhibit_linux.go

84 // InhibitShutdown creates an systemd inhibitor by calling logind's Inhibt() and returns the inhibitor lock
85 // see https://www.freedesktop.org/wiki/Software/systemd/inhibit/ for more details.
86 func (bus *DBusCon) InhibitShutdown() (InhibitLock, error) {
87     obj := bus.SystemBus.Object(logindService, logindObject)
88     what := "shutdown"
89     who := "kubelet"
90     why := "Kubelet needs time to handle node shutdown"
91     mode := "delay"
92
93     call := obj.Call("org.freedesktop.login1.Manager.Inhibit", 0, what, who, why, mode)
94     if call.Err != nil {
95         return InhibitLock(0), fmt.Errorf("failed creating systemd inhibitor: %w", call.Err)
96     }
97
98     var fd uint32
99     err := call.Store(&fd)
100     if err != nil {
101         return InhibitLock(0), fmt.Errorf("failed storing inhibit lock file descriptor: %w", err)
102     }
103
104     return InhibitLock(fd), nil
105 }

systemd-inhibit --list
===============
WHO     UID USER PID  COMM    WHAT     WHY                                        MODE
kubelet 0   root 1581 kubelet shutdown Kubelet needs time to handle node shutdown delay.

[brandond]

Well it looks to me like the kubelet is successfully holding a lock, so I'm not sure why systemd is shutting down or rebooting the system after 30 seconds. I suspect you'll need to look more into the systemd logs and configuration to determine that.

[ravisekhar-akkineni]

Thanks a lot. Will look into systemd log

[piersharding]

I know this is late, but perhaps you have a systemd file like /etc/systemd/logind.conf.d/99-kubelet.conf with something like a InhibitDelayMaxSec=30 setting?