k3s version v1.27 installed using https://get.k3s.io making running pod's /tmp as read only after ndoe restart

[nsimha73]

Environmental Info:
K3s Version: v1.127
k3s version v1.27.15+k3s1 (3ced503)
go version go1.21.11

Node(s) CPU architecture, OS, and Version:

Linux stream9-node-1 5.14.0-496.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Aug 12 20:37:54 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:
2 nodes cluster with one master

Describe the bug:
On k3s 2 nodes cluster on CentOS stream9 OS, when we deploy helm charts all pods starts running without any issue ( most of our pods are running springboot services ) which are using /tmp of the container file system, once the kubernetes node restarts we are seeing the pod /tmp folder permissions are changed and pod fails to come up as the springboot service try to access /tmp to create files on start.

We would like to know if there are any issues reported like these.

[brandond]

I have no idea what /tmp is in your pod. Have you mounted an emptyDir volume there, or are you mounting /tmp from the host, or is it just part of your container image? There is no special behavior for /tmp in Kubernetes or containerd, by default it would just have whatever permissions that path has in your image. Do you perhaps have a process running within the pod that is changing its permissions or ownership?

[nsimha73]

Thank you for the details @brandond /tmp is what is part of the container file system that has access for the user who is running the pod, and /tmp is accessible. Pod run's without any issue until the node restarts where we see that /tmp inside the container changes it's permission and the process starts failing to come up as it can't write into /tmp any more. WE don't have any process inside the pod which is changing the /tmp permissions

[brandond]

There is not anything in k3s or containerd that would modify the image's root filesystem. Are you sure you don't have an init container or something else in the pod spec that's initially fixing the permissions on /tmp, and doesn't get re-run again when the pod is restarted?

Alternately, is it possible that you have something that is modifying the extracted image filesystem itself in the containerd image store?

I'd probably compare the /tmp ownership in the image, vs what you see when your pod is working, and what you see if you create a new test pod using that same image.