Show limited mem and cpu **inside** containers

[nsoubelet]

Hello,

After several weeks of investigation of an issue which I don't really know how to solve, just decided to drop it here in case anyone else ever wonder about it.

Basically when using resource limits in k3s (I assume k8s behaves the same), inside the container the total available is always host's.
This happens in Docker and Containerd.
Just to give an example, we've got the following limits in a Pod:

resources:
    requests:
      cpu: "0.50"
      memory: "512Mi"
      ephemeral-storage: "512Mi"
    limits:
      cpu: "0.50"
      memory: "512Mi"
      ephemeral-storage: "512Mi"

However, when running cat /proc/meminfo inside the container I see the following values:

MemTotal:        4039316 kB
MemFree:          638464 kB
MemAvailable:    1766156 kB

... which is host's.

I am not saying that effectively containers in pod exceeds limits, it's just that I find surprising that the information given does not reflect container's limits.
I could not find too much people complaining about it so I wonder whether it is only me seeing this (a misconfiguration?) or this is something accepted in general.

Hope anyone is interested in the topic.

Thanks

[brandond]

There are a lot of discussions and articles about why /proc/meminfo doesn't show cgroup limits, here's one: https://ops.tips/blog/why-top-inside-container-wrong-memory/

[nsoubelet]

Thanks @brandond !
Yes, you are right. I actually found those discussions/articles however I never found the solution.
I do think there is a solution to that because, for example, Azure container run creates containers inside shared VMs and it works for them. Also GCP has something similar.
I do not think they use completely different and bespoken Kernels.

I just wanted to open a topic and discuss. I will keep investigating.

[brandond]

You can read the limits from within a container, just not from /proc/meminfo. See:
https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt (memory.limit_in_bytes)

Reading CPU limits is harder - unless you're using cpusets, the container doesn't really have a good idea of how much time it can use before it gets throttled by the kernel's scheduler.

[nsoubelet]

Thanks for replying @brandond
Problem I find here (real one) is that monitoring container resources inside the container is really hard if not impossible.
I mean, not thinking as a k8s admin, but as an user that only consumes compute capacity for containers but can't act outside those containers, like for example inspecting resources via kubectl.

I would love to see how Microsoft and Google have solved the problem. Again, if you create a container using https://docs.microsoft.com/en-us/azure/container-instances/ you see the real limits.
As far as I understand, Azure container instances places containers in already-existing VMs/BM, if so, maybe they have the key to the solution.

[brandond]

Azure Containers are not the same as Kubernetes pods, so looking for direct feature parity or identical behavior between the two probably isn't a productive exercise. I'm not sure how they've implemented it behind the scenes, but if /proc/meminfo shows the configured limits then I do suspect they are provisioning a lightweight VM for each container instance.

[nsoubelet]

Well, that is correct but the important detail here is that it is not a Kubernetes problem, it seems to be a runc issue.
Microsoft Azure Containers might not use K8s but for sure it uses containers and probably runc or another custom sandboxing strategy, that is, a runtimeClass in the k8s way of defining it.

If you run a container using Docker or Containerd with nothing more than the engine installed in a machine, problem will also be there since they use runc. As you you know, k8s relies on runc or another "runner" for scheduling workloads.

That's why it is still a bit surprising for me that the issue is still there after this years of huge communities of incredible engineers working on it...