kubelet crashes

[R-Nabil]

Hi,

I’ve started to experience a weird situation one my fairly stable k3s cluster started to have one of its node suffering from rebooting.

It happens when the node in question runs a specific workload (nextcloud) and only when I try to sync my local laptop with that nextcloud (so basically trying to pull around 20Gb of data). Among these data, most are small files.

Thing to note : it’s a k3s cluster with Ubuntu 22.04 and kernel 6.5 backend. Network plugin is using cilium

What I’ve investigated so far :

• ressources are fine I have plenty of memory and cpu available
• it happens on any of my agent nodes it’s not specific to one of them
• syslog/kernlog aren’t showing anything specific
• kube events only says that “kubelet was rebooted”
• pods logs are not showing anything either. It could be because they crash before the log showing anything ?

I can 100% reproduce it any time so at least any suggestion can be tested easily.
I'm 100% sure this is related to nextcloud (either directly or indirectly tbh).

[jessebot]

This is an interesting one. I haven't experienced this exactly, but I've encountered some crashes depending on tuning. Some quick follow up questions:

1. What version of k3s are you running?

2. What helm chart version and nextcloud image version are you using? (Your values.yaml may also be helpful so others in the community can compare with their setups with yours. Ensure you anonymize any sensitive data before posting though.)

3. Do you have any resource requests/limits set for the nextcloud pod?

4. Have you tried looking at k9s or lens to see the resource usage while it is happening? Is it spiking above the required resources?

5. is the machine you're running on running out of open files? You can check your limits by checking ulimit/ulimit -n as the user that k3s runs as (likely root).

[jessebot]

1. Your k3s version looks fairly up to date. I'm running 1.29.3 right now, so not too far off. I think this is fine.

2. This repo is for support of the helm chart specifically, so it may be a little harder to help, but everyone is still absolutely welcome to give any thoughts or suggestions. I'll do my best, but I have limited resources to troubleshoot custom setups right now.

3. From your deployment, should limits be under a section in the yaml called resources? I'm not currently doing that much data ingress/egress, so I'm not taxing my nextcloud pod that hard, but those limits should normally be ok.

4. Let me know if you see anything specific happening for the nextcloud pod's cpu/mem when you reproduce it. I listed lens and k9s, but if you have another monitoring tool such as prometheus/grafana or datadog or newrelic or something, that would also work. I was just looking for an easy metric table or graph to check.

5. The limits for number of files is just one of the things that sometimes bites me on k3s on metal. Unlimited is good, but it may not be biting you in the first place. I use a program called teeldeer for checking for quick tldr man pages. When I run tldr ulimit it gives me (so you may also want to check the hard limit with ulimit -H, sorry for missing that before):

  Get and set user limits.
  More information: <https://manned.org/ulimit>.

  Get the properties of all the user limits:

      ulimit -a

  Get hard limit for the number of simultaneously opened files:

      ulimit -H -n

  Get soft limit for the number of simultaneously opened files:

      ulimit -S -n

  Set max per-user process limit:

      ulimit -u 30

The reason this matters is not because of the file limit on the nextcloud pod itself, but if your nextcloud pod is using a local-path or host-path for persistence, and you have other workloads on that cluster that also do so, if the soft or hard limits are too low, it can cause the whole cluster to fail in unpredictable ways. I have my soft limit set to 5120 and a hard limit of 6144. You may find this superuser thread useful if you'd like to adjust those limits yourself. Here's also the limits.conf man page.

6. I haven't used cilium a lot, but does it have any sort of body size limit on ingress/egress that you can adjust?
   For ingress-nginx you would normally annotate the ingress resource with something like nginx.ingress.kubernetes.io/proxy-body-size: 10G to allow more throughput, if you were using a nextcloud:fpm image with the nginx container integrated to the pod, but I'm afraid I've only recently started playing with cilium, so I can't offer too much more insights on this yet.

[R-Nabil]

1. It's actually one thing I could update easily. May I ask what's the underlying OS you use ?

2. Fair. My deployment was largely inspired by this helm chart and I intend to switch to that eventually. This could be the opportunity to.

3. Need to double check that.

4. I'm using rancher and haven't seen anything there. Will try to see if I can see anything in Lens.

5. Ok some more homework to do. Thanks a lot.

6. For my ingress/egress I use traefik cilium is just the pod networking backend. My files are all fairly small (45000 files but total size is 23G which is not much)

Thanks a lot for all the help. Will try come back quick on the above mentioned point.

[jessebot]

No problem, friend!

I'm running Debian 12 (bookworm) right now and then I run k3s directly on top of that, without rancher.

If you decide to the use the helm chart, and there's an issue with the chart itself, please feel free to open a Github Issue. Note: this is a community maintained chart, so volunteers come and go when they have the time, so if there's a delay in help, rest assured it will eventually come.

Best of luck out there when you get a chance to look further into this :)

[R-Nabil]

Apologies. My previous copy paste was wrong.

So for 3, it is indeed under the resoruces section. I've edited the deployment.

My soft limit is 1024, the hard one seems to be 1048576 ? I've raised the soft limit (4096) for now. But one question : why would anyone set a limit to that ? It may sound simple and stupid, but couldn't I just set it to... 1Mio ? Or in my case, 64000 (to cover for ALL nc file opened at once, a worst case scenario) ? Where would the drawback be ?

6. I've checked, traefik doesn't set a file size limit.

[jessebot]

why would anyone set a limit to that ? It may sound simple and stupid, but couldn't I just set it to... 1Mio ? Or in my case, 64000 (to cover for ALL nc file opened at once, a worst case scenario) ? Where would the drawback be ?

There are no stupid questions :) The limits are in place for mostly for security reasons, but sometimes also for restricting resource usage to accommodate hardware limitations. Increasing them to a known number of files that would generally be open is fine, but putting everything at unlimited may cause you to miss an intruder doing nefarious activities that require more than the average resource limit. The limits.conf file and ulimit command are designed as a bit of security sanity check, but in my opinion, the defaults tend to be a bit low for a kubernetes cluster with more than one major app and a prometheus stack haha (also sorry for the delay 🙏 )

[cloudymax]

Just a wild guess, but perhaps that 20gig sync job could be OOM-ing the kubelet?

If you aren't already - you could try setting some reserved resources and see if that helps 🤷

Here's how I have it in my K3s config:

kubelet-arg:
  kube-reserved=cpu=1,memory=2Gi,ephemeral-storage=1Gi
  system-reserved=cpu=1,memory=2Gi,ephemeral-storage=1Gi

[R-Nabil]

No the whole kubelet crashes not the pod only. Plus the pod has limits set at around half the total available resources. So in that case it wouldn't be possible right ?

[cloudymax]

with that context, I think you're correct that you'd protected from an OOM by the nextcloud pods resource limits.

[R-Nabil]

Ok its not much of an update but, for the sake of trying, I added a new node based on Debian 12 (instead of Ubuntu 22.04 + 6.5 Kernel).

It seems that for now, this has solved the issue (no crashes so far, but will wait further to be able to confirm)

[R-Nabil]

Ok that makes sense from that point of view. Thank you👍👌

…

On 17 May 2024 at 14:56 +0800, JesseBot ***@***.***>, wrote: > why would anyone set a limit to that ? It may sound simple and stupid, but couldn't I just set it to... 1Mio ? Or in my case, 64000 (to cover for ALL nc file opened at once, a worst case scenario) ? Where would the drawback be ? There are no stupid questions :) The limits are in place for mostly for security reasons, but sometimes also for restricting resource usage to accommodate hardware limitations. Increasing them to a known number of files that would generally be open is fine, but putting everything at unlimited may cause you to miss an intruder doing nefarious activities that require more than the average resource limit. The limits.conf file and ulimit command are designed as a bit of security sanity check, but in my opinion, the defaults tend to be a bit low for a kubernetes cluster with more than one major app and a prometheus stack haha (also sorry for the delay 🙏 ) — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>