Total utilization vastly exceeds sum of pods

[joshfrench]

I'm seeing this pattern reliably across a number of clusters. The total utilization doesn't seem to make any sense given the individual pod metrics. Requests and limits always add up to their respective totals which makes me wonder if the total cpu & mem are based on their own metric and not just summed. Do you have any insight into what those are actually measuring? Looking at the obvious suspects it doesn't seem to line up with the working set, etc.

Thanks!

CPU UTIL     MEMORY UTIL

275m (13%)   4729Mi (60%)
52m (2%)     102Mi (1%)
4m (0%)      58Mi (0%)
50m (2%)     219Mi (2%)
1m (0%)      18Mi (0%)
0m (0%)      173Mi (2%)
2m (0%)      23Mi (0%)
1m (0%)      70Mi (0%)
1m (0%)      7Mi (0%)
6m (0%)      45Mi (0%)
2m (0%)      30Mi (0%)

[almson]

I agree this is confusing. It doesn't even match what htop is showing. However, kubernetes separates the concepts of allocatable resources (excluding those used by the OS) and total resources. Since we're only really interested in allocatable resources, it would be more consistent if the sum of mem util would be the sum of the pods' mem util, excluding the OS. There could be a separate column or warning for the OS and its actual vs reserved resource usage.