pahole memory allocation problems

[jeffmerkey]

When building RHEL 9.4 running as a qemu-kvm vm under gnome-boxes, it is required to configure a swap file size in excess of 10 GB for the kvm image to prevent pahole from crashing and triggering the OOM Killer and failing linux kernel build. it requires excessive amounts of memory when parsing the BTF data from vmlinux. gcc linking with ld works but when pahole executes it consumes excessive memory during 'make bzImage modules'. It really needs to be rewritten with it's own internal lru or save data as intermediate files rather than just sucking up 6-7 GB of memory and crashing the system. Is there some workaround for this issue? Thanks.

[jeffmerkey]

I will look over the program and possibly suggest a fix for this.

[alan-maguire]

have you tested the "next" branch? It has multiple fixes around memory usage.

[jeffmerkey]

Hi,

I cloned the 'next' branch with:

git clone -b next --single-branch https://github.com/acmel/dwarves

I complied the pahole v1.27 program with the buildcmd.sh and moved it into a RHEL 9.4 /usr/bin and tested it. The memory usage was about 2 GB total for a build of the 5.14 redhat kernel.

This is an improvement over the pahole v1.25 version currently shipped with RHEL 9.4. There still needs to be some improvement in memory usage overall. It still requires a fairly large swap partition (over 4 GB) to prevent crashes. I am guessing there is a memory leak for pahole v1.25 which was addressed in v1.27.

So to summarize:

pahole v1.25 - requires 10 GB swap space to avoid OOM kill and crashing on RHEL 9.4 when running under gnome-boxes.
pahole v1.27 - requires 4-6 GB swap space to avoid OOM kill and crashing on RHEL 9.4 when running under gnome boxes.

Using intermediate tmp files like git does for storing data to disk so memory use is not excessive as is the case for pahole seems to be something that should be addressed with a better architectural model. rebuilding the kernel RPM for RHEL 9.4 uses even more memory and requires the full 10 GB to prevent crashes with pahole v1.25.

The problem gets worse the more processors you have, the greater the memory consumption. I discovered if you specify job mode during the make i.e. 'make -j4 bzImage modules' and have four or more processors the memory consumtion goes through the roof again and it uses 10 GB of swap the avoid crashes and OOM conditions.

Bottom line is invoking pahole for each processor during build in job mode which creates 4 or more pahole processes at once will send memory consumption through the roof. This program is poorly implemented when it comes to conserving memory. I would suggest that the program be optimized to store it's meta data in tmp files rather than using memory. gcc and ld work just fine in job mode and manage memory better but pahole is somewhat broken in this regard.

Bottom line, the more processors you have, the worse the problem. I just tested a run with pahole v1.27 on four processors and it consumed 2 GB for each pahole process (total of 8 GB). This seems broken to me.