Kernel 6.6: error initializing '/usr/lib/bpftune//route_table_tuner.so

[ptr1337]

Starting with the 6.6 Kernel I get following in journalctl/systemctl status:

Okt 30 21:37:21 ptr1337-cachyos systemd[1]: Started BPF-based auto-tuning of system parameters.
Okt 30 21:37:21 ptr1337-cachyos bpftune[5015]: bpftune works fully
Okt 30 21:37:21 ptr1337-cachyos bpftune[5015]: bpftune supports per-netns policy (via netns cookie)
Okt 30 21:37:22 ptr1337-cachyos bpftune[5015]: could not load skeleton: No such process
Okt 30 21:37:22 ptr1337-cachyos bpftune[5015]: error initializing '/usr/lib/bpftune//route_table_tuner.so: No such process

This is not reproduceable with 6.5.

[alan-maguire]

Thanks for the report! I'll dig into this more in a few days

[Suyash1989]

Nov 9 15:01:09 lab03 bpftune[38931]: bpftune works, but no BPF Type Format information (BTF) is available. This means kernel data structure offsets may not match those at compile-time, and tuners may not operate as expected. This mode of operation is unsupported, and failures are expected, so be warned.
Nov 9 15:01:09 lab03 bpftune[38931]: bpftune supports per-netns policy (via netns cookie)
Nov 9 15:01:09 lab03 bpftune[38931]: could not load skeleton: Kernel verifier blocks program loading
Nov 9 15:01:09 lab03 bpftune[38931]: error initializing '/usr/lib64/bpftune//tcp_conn_tuner.so: Unknown error 4007
Nov 9 15:01:20 lab03 bpftune[38931]: Scenario 'need to increase TCP buffer size(s)' occurred for tunable 'net.ipv4.tcp_rmem' in global ns. Need to increase buffer size(s) to maximize throughput
Nov 9 15:01:20 lab03 bpftune[38931]: Due to need to increase max buffer size to maximize throughput change net.ipv4.tcp_rmem(min default max) from (0 0 0) -> (0 0 0)

The error occurs with Kernel 5.14 on RHEL 9.2

[alan-maguire]

Starting with the 6.6 Kernel I get following in journalctl/systemctl status:

Okt 30 21:37:21 ptr1337-cachyos systemd[1]: Started BPF-based auto-tuning of system parameters.
Okt 30 21:37:21 ptr1337-cachyos bpftune[5015]: bpftune works fully
Okt 30 21:37:21 ptr1337-cachyos bpftune[5015]: bpftune supports per-netns policy (via netns cookie)
Okt 30 21:37:22 ptr1337-cachyos bpftune[5015]: could not load skeleton: No such process
Okt 30 21:37:22 ptr1337-cachyos bpftune[5015]: error initializing '/usr/lib/bpftune//route_table_tuner.so: No such process

This is not reproduceable with 6.5.

I reproduced this; looks like the root cause is fib6_age () is not present in kallsyms/BTF; since it is a static function I suspect it is being inlined. As it is needed for the route_table_tuner, the failure to load is the right thing; however it'd be good if the failure mode was a bit more informative.

[alan-maguire]

Nov 9 15:01:09 lab03 bpftune[38931]: bpftune works, but no BPF Type Format information (BTF) is available. This means kernel data structure offsets may not match those at compile-time, and tuners may not operate as expected. This mode of operation is unsupported, and failures are expected, so be warned. Nov 9 15:01:09 lab03 bpftune[38931]: bpftune supports per-netns policy (via netns cookie) Nov 9 15:01:09 lab03 bpftune[38931]: could not load skeleton: Kernel verifier blocks program loading Nov 9 15:01:09 lab03 bpftune[38931]: error initializing '/usr/lib64/bpftune//tcp_conn_tuner.so: Unknown error 4007 Nov 9 15:01:20 lab03 bpftune[38931]: Scenario 'need to increase TCP buffer size(s)' occurred for tunable 'net.ipv4.tcp_rmem' in global ns. Need to increase buffer size(s) to maximize throughput Nov 9 15:01:20 lab03 bpftune[38931]: Due to need to increase max buffer size to maximize throughput change net.ipv4.tcp_rmem(min default max) from (0 0 0) -> (0 0 0)

The error occurs with Kernel 5.14 on RHEL 9.2

looks like you don't have BTF; in that case we can't be sure the offsets we use are correct, so we can get unexpected behaviour as described in the log;

"3 bpftune[38931]: bpftune works, but no BPF Type Format information (BTF) is available. This means kernel data structure offsets may not match those at compile-time, and tuners may not operate as expected. This mode of operation is unsupported, and failures are expected, so be warned."

Can you confirm /sys/kernel/btf/vmlinux is absent?

In this case it looks like the offsets of the tcp memory values are wrong, and can't be fixed since we have no BTF, hence we get 0 values.

[Suyash1989]

[root@lab03 btf]# ls /sys/kernel/btf/ | grep vmlinux
vmlinux

I do have the file, so then can we say the offsets used are incorrect?

[alan-maguire]

[root@lab03 btf]# ls /sys/kernel/btf/ | grep vmlinux vmlinux

I do have the file, so then can we say the offsets used are incorrect?

Yeah, it looks like the BTF can't be read by bpftune. This usually happens when the tools used to create the kernel BTF are newer than those being used to read it - in that case the BTF cannot be parsed. If you can find a more up-to-date libbpf to build with that may help.

[0xAlcibiades]

Seeing the same thing on ubuntu hwe edge kernel 6.5.0-14-generic

Dec 26 21:52:58 subtle-bobcat bpftune[1158877]: bpftune works, but no BPF Type Format information (BTF) is available.  This means kernel data structure offsets may not match those at compile-time, and tuners may not operate as expected. This mod>
Dec 26 21:52:58 subtle-bobcat bpftune[1158877]: bpftune supports per-netns policy (via netns cookie)
Dec 26 21:53:00 subtle-bobcat bpftune[1158877]: could not load skeleton: Kernel verifier blocks program loading
Dec 26 21:53:00 subtle-bobcat bpftune[1158877]: error initializing '/usr/lib64/bpftune//tcp_conn_tuner.so: Unknown error 4007

[0xAlcibiades]

Got thi

[root@lab03 btf]# ls /sys/kernel/btf/ | grep vmlinux vmlinux
I do have the file, so then can we say the offsets used are incorrect?

Yeah, it looks like the BTF can't be read by bpftune. This usually happens when the tools used to create the kernel BTF are newer than those being used to read it - in that case the BTF cannot be parsed. If you can find a more up-to-date libbpf to build with that may help.

Indeed, the hwe-edge kernel was built using a newer version of libbpf than libbpf-devel provided, and that resolved.

[ptr1337]

Any updates on this?

[alan-maguire]

the original problem - inlining of fib6_age() - is hard to predict since it is very compiler-specific. we currently have a model of passing optional functions to the bpftuner load function that could potentially be extended to name required functions, and check kallsyms for these, but it feels kind of clunky. ideally we would like to rely on tracepoints in the kernel; the right answer in this case I think is to add a tracepoint to cover FIB[6] garbage collection; it will give us what we need.

[alan-maguire]

for now, the error message has been improved to better describe this:

Oct 25 17:16:03 bpf bpftune[1822269]: tuner 'route_table' failed to load, tracing target was not found; this can occur for unstable tracing targets like kernel functions.