High CPU load on x64 dockerhost system

[sxa]

Machine is frequently running with a load of around 60 while the test runs are in progress. The machine has 16 cores. Noting that we have over ten test containers on this system so we may need to revise the CPU caps (and make sure they are working as expected).

We could also look at reducing the priority of the Solaris vagrant machines since that is a low priority platform.

Noting also that the riscv64 builds are running with a particularly high CPU % in the containers:

CONTAINER ID   NAME                CPU %      MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS
36134aa4628d   distracted_galois   1176.88%   7.998GiB / 62.79GiB   12.74%    218MB / 515kB     6.38GB / 6.04GB   359

[sxa]

@Haroon-Khel If you get some gaps between JSF runs can you see if there's something wrong with the configuration of the containers on this host. We may need to drop the number we're running too :'(

[Haroon-Khel]

Taking a look

[Haroon-Khel]

At the moment we limit cpu usage to 0-3,

infrastructure/ansible/playbooks/AdoptOpenJDK_Unix_Playbook/roles/deploy_container/tasks/deploy.yml

Line 56 in 1d8d7ec

command: "{{ docker_run_command }} --restart unless-stopped -p {{ docker_port }}:22 --cpuset-cpus='0-3' --memory=6G --detach --name {{ docker_image | upper }}.{{ docker_port }}{{ arm32_suffix | upper }} aqa_{{ docker_image }}{{ arm32_suffix }}"

We could look at reducing this to 0-2

And yes theres certainly room for dropping the number of running containers outside of a release period, ive stopped https://ci.adoptium.net/computer/test-docker-ubuntu2004-x64-2/

[sxa]

At the moment we limit cpu usage to 0-3,

infrastructure/ansible/playbooks/AdoptOpenJDK_Unix_Playbook/roles/deploy_container/tasks/deploy.yml

Line 56 in 1d8d7ec

command: "{{ docker_run_command }} --restart unless-stopped -p {{ docker_port }}:22 --cpuset-cpus='0-3' --memory=6G --detach --name {{ docker_image | upper }}.{{ docker_port }}{{ arm32_suffix | upper }} aqa_{{ docker_image }}{{ arm32_suffix }}"

We could look at reducing this to 0-2

And yes theres certainly room for dropping the number of running containers outside of a release period, ive stopped https://ci.adoptium.net/computer/test-docker-ubuntu2004-x64-2/

Bearing in mind the machine only has 16 cores, so if we have "0-3" (Presumably 4 cores?) for each container, that's very quickly going to exceeed the amount of CPU available. I'd be tempted to drop them to 0-1 which I presume would be 2 cores each, and make sure we have no more than 8 containers (Bear in mind that this also doesn't include build work, or the Solaris work on the machine which will push it a bit higher at times)