Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add AttachStructOps() #476

Open
wants to merge 9 commits into
base: main
Choose a base branch
from

Conversation

ianchen0119
Copy link

@ianchen0119 ianchen0119 commented Jan 23, 2025

Hi @geyslan

The sched_ext gives the flexibility for customizing the os scheduler.
I'm trying to offload the scheduler behaviors from kernel space to user space in golang(The idea is inspired by the @arighi 's presentation).
Therefore, I made the PR for supporting the SCX eBPF program attachment.

The result shows below:

> sudo ./main
libbpf: loading object from scx_goland_core.bpf.o
libbpf: elf: section(3) struct_ops/scx_goland_core_select_cpu, size 320, link 0, flags 6, type=1
libbpf: sec 'struct_ops/scx_goland_core_select_cpu': found program 'scx_goland_core_select_cpu' at insn offset 0 (0 bytes), code size 40 insns (320 bytes)
libbpf: elf: section(4) .relstruct_ops/scx_goland_core_select_cpu, size 112, link 47, flags 40, type=9
libbpf: elf: section(5) struct_ops/scx_goland_core_enqueue, size 488, link 0, flags 6, type=1
libbpf: sec 'struct_ops/scx_goland_core_enqueue': found program 'scx_goland_core_enqueue' at insn offset 0 (0 bytes), code size 61 insns (488 bytes)
libbpf: elf: section(6) .relstruct_ops/scx_goland_core_enqueue, size 176, link 47, flags 40, type=9
libbpf: elf: section(7) struct_ops/scx_goland_core_dispatch, size 72, link 0, flags 6, type=1
libbpf: sec 'struct_ops/scx_goland_core_dispatch': found program 'scx_goland_core_dispatch' at insn offset 0 (0 bytes), code size 9 insns (72 bytes)
libbpf: elf: section(8) .relstruct_ops/scx_goland_core_dispatch, size 48, link 47, flags 40, type=9
libbpf: elf: section(9) struct_ops/scx_goland_core_running, size 112, link 0, flags 6, type=1
libbpf: sec 'struct_ops/scx_goland_core_running': found program 'scx_goland_core_running' at insn offset 0 (0 bytes), code size 14 insns (112 bytes)
libbpf: elf: section(10) .relstruct_ops/scx_goland_core_running, size 32, link 47, flags 40, type=9
libbpf: elf: section(11) struct_ops/scx_goland_core_stopping, size 136, link 0, flags 6, type=1
libbpf: sec 'struct_ops/scx_goland_core_stopping': found program 'scx_goland_core_stopping' at insn offset 0 (0 bytes), code size 17 insns (136 bytes)
libbpf: elf: section(12) .relstruct_ops/scx_goland_core_stopping, size 32, link 47, flags 40, type=9
libbpf: elf: section(13) struct_ops/scx_goland_core_enable, size 48, link 0, flags 6, type=1
libbpf: sec 'struct_ops/scx_goland_core_enable': found program 'scx_goland_core_enable' at insn offset 0 (0 bytes), code size 6 insns (48 bytes)
libbpf: elf: section(14) .relstruct_ops/scx_goland_core_enable, size 16, link 47, flags 40, type=9
libbpf: elf: section(15) struct_ops.s/scx_goland_core_init, size 32, link 0, flags 6, type=1
libbpf: sec 'struct_ops.s/scx_goland_core_init': found program 'scx_goland_core_init' at insn offset 0 (0 bytes), code size 4 insns (32 bytes)
libbpf: elf: section(16) .relstruct_ops.s/scx_goland_core_init, size 16, link 47, flags 40, type=9
libbpf: elf: section(17) struct_ops/scx_goland_core_exit, size 240, link 0, flags 6, type=1
libbpf: sec 'struct_ops/scx_goland_core_exit': found program 'scx_goland_core_exit' at insn offset 0 (0 bytes), code size 30 insns (240 bytes)
libbpf: elf: section(18) .relstruct_ops/scx_goland_core_exit, size 80, link 47, flags 40, type=9
libbpf: elf: section(19) license, size 4, link 0, flags 3, type=1
libbpf: license of scx_goland_core.bpf.o is GPL
libbpf: elf: section(20) .rodata, size 276, link 0, flags 2, type=1
libbpf: elf: section(21) .struct_ops.link, size 424, link 0, flags 3, type=1
libbpf: elf: section(22) .rel.struct_ops.link, size 128, link 47, flags 40, type=9
libbpf: elf: section(23) .data.uei_dump, size 1, link 0, flags 3, type=1
libbpf: elf: section(24) .data, size 1168, link 0, flags 3, type=1
libbpf: elf: section(25) .maps, size 32, link 0, flags 3, type=1
libbpf: elf: section(26) .bss, size 8, link 0, flags 3, type=8
libbpf: elf: section(37) .BTF, size 40802, link 0, flags 0, type=1
libbpf: elf: section(39) .BTF.ext, size 1556, link 0, flags 0, type=1
libbpf: elf: section(47) .symtab, size 2184, link 1, flags 0, type=2
libbpf: looking for externs among 91 symbols...
libbpf: collected 8 externs total
libbpf: extern (ksym) #0: symbol 48, name scx_bpf_consume___compat
libbpf: extern (ksym) #1: symbol 53, name scx_bpf_create_dsq
libbpf: extern (ksym) #2: symbol 41, name scx_bpf_dispatch___compat
libbpf: extern (ksym) #3: symbol 45, name scx_bpf_dispatch_vtime___compat
libbpf: extern (ksym) #4: symbol 40, name scx_bpf_dsq_insert
libbpf: extern (ksym) #5: symbol 44, name scx_bpf_dsq_insert_vtime
libbpf: extern (ksym) #6: symbol 47, name scx_bpf_dsq_move_to_local
libbpf: extern (ksym) #7: symbol 36, name scx_bpf_select_cpu_dfl
libbpf: map 'stats': at sec_idx 25, offset 0.
libbpf: map 'stats': found type = 6.
libbpf: map 'stats': found key_size = 4.
libbpf: map 'stats': found value_size = 8.
libbpf: map 'stats': found max_entries = 2.
libbpf: map 'scx_gola.rodata' (global data): at sec_idx 20, offset 0, flags 480.
libbpf: map 1 is "scx_gola.rodata"
libbpf: map '.data.uei_dump' (global data): at sec_idx 23, offset 0, flags 400.
libbpf: map 2 is ".data.uei_dump"
libbpf: map 'scx_gola.data' (global data): at sec_idx 24, offset 0, flags 400.
libbpf: map 3 is "scx_gola.data"
libbpf: map 'scx_gola.bss' (global data): at sec_idx 26, offset 0, flags 0.
libbpf: map 4 is "scx_gola.bss"
libbpf: struct_ops init: struct sched_ext_ops(type_id=405) scx_goland_core_ops found at offset 0
libbpf: sec '.relstruct_ops/scx_goland_core_select_cpu': collecting relocation for section(3) 'struct_ops/scx_goland_core_select_cpu'
libbpf: sec '.relstruct_ops/scx_goland_core_select_cpu': relo #0: insn #8 against 'scx_bpf_select_cpu_dfl'
libbpf: prog 'scx_goland_core_select_cpu': found extern #7 'scx_bpf_select_cpu_dfl' (sym 36) for insn #8
libbpf: sec '.relstruct_ops/scx_goland_core_select_cpu': relo #1: insn #15 against 'stats'
libbpf: prog 'scx_goland_core_select_cpu': found map 0 (stats, sec 25, off 0) for insn #15
libbpf: sec '.relstruct_ops/scx_goland_core_select_cpu': relo #2: insn #22 against '__SCX_DSQ_LOCAL'
libbpf: prog 'scx_goland_core_select_cpu': found data map 1 (scx_gola.rodata, sec 20, off 0) for insn 22
libbpf: sec '.relstruct_ops/scx_goland_core_select_cpu': relo #3: insn #25 against '__SCX_SLICE_DFL'
libbpf: prog 'scx_goland_core_select_cpu': found data map 1 (scx_gola.rodata, sec 20, off 0) for insn 25
libbpf: sec '.relstruct_ops/scx_goland_core_select_cpu': relo #4: insn #28 against 'scx_bpf_dsq_insert'
libbpf: prog 'scx_goland_core_select_cpu': found extern #4 'scx_bpf_dsq_insert' (sym 40) for insn #28
libbpf: sec '.relstruct_ops/scx_goland_core_select_cpu': relo #5: insn #33 against 'scx_bpf_dsq_insert'
libbpf: prog 'scx_goland_core_select_cpu': found extern #4 'scx_bpf_dsq_insert' (sym 40) for insn #33
libbpf: sec '.relstruct_ops/scx_goland_core_select_cpu': relo #6: insn #37 against 'scx_bpf_dispatch___compat'
libbpf: prog 'scx_goland_core_select_cpu': found extern #2 'scx_bpf_dispatch___compat' (sym 41) for insn #37
libbpf: sec '.relstruct_ops/scx_goland_core_enqueue': collecting relocation for section(5) 'struct_ops/scx_goland_core_enqueue'
libbpf: sec '.relstruct_ops/scx_goland_core_enqueue': relo #0: insn #6 against 'stats'
libbpf: prog 'scx_goland_core_enqueue': found map 0 (stats, sec 25, off 0) for insn #6
libbpf: sec '.relstruct_ops/scx_goland_core_enqueue': relo #1: insn #13 against 'fifo_sched'
libbpf: prog 'scx_goland_core_enqueue': found data map 1 (scx_gola.rodata, sec 20, off 0) for insn 13
libbpf: sec '.relstruct_ops/scx_goland_core_enqueue': relo #2: insn #17 against '__SCX_SLICE_DFL'
libbpf: prog 'scx_goland_core_enqueue': found data map 1 (scx_gola.rodata, sec 20, off 0) for insn 17
libbpf: sec '.relstruct_ops/scx_goland_core_enqueue': relo #3: insn #20 against 'scx_bpf_dsq_insert'
libbpf: prog 'scx_goland_core_enqueue': found extern #4 'scx_bpf_dsq_insert' (sym 40) for insn #20
libbpf: sec '.relstruct_ops/scx_goland_core_enqueue': relo #4: insn #26 against 'scx_bpf_dsq_insert'
libbpf: prog 'scx_goland_core_enqueue': found extern #4 'scx_bpf_dsq_insert' (sym 40) for insn #26
libbpf: sec '.relstruct_ops/scx_goland_core_enqueue': relo #5: insn #29 against '__SCX_SLICE_DFL'
libbpf: prog 'scx_goland_core_enqueue': found data map 1 (scx_gola.rodata, sec 20, off 0) for insn 29
libbpf: sec '.relstruct_ops/scx_goland_core_enqueue': relo #6: insn #32 against '.bss'
libbpf: prog 'scx_goland_core_enqueue': found data map 4 (scx_gola.bss, sec 26, off 0) for insn 32
libbpf: sec '.relstruct_ops/scx_goland_core_enqueue': relo #7: insn #43 against 'scx_bpf_dsq_insert_vtime'
libbpf: prog 'scx_goland_core_enqueue': found extern #5 'scx_bpf_dsq_insert_vtime' (sym 44) for insn #43
libbpf: sec '.relstruct_ops/scx_goland_core_enqueue': relo #8: insn #49 against 'scx_bpf_dsq_insert_vtime'
libbpf: prog 'scx_goland_core_enqueue': found extern #5 'scx_bpf_dsq_insert_vtime' (sym 44) for insn #49
libbpf: sec '.relstruct_ops/scx_goland_core_enqueue': relo #9: insn #54 against 'scx_bpf_dispatch___compat'
libbpf: prog 'scx_goland_core_enqueue': found extern #2 'scx_bpf_dispatch___compat' (sym 41) for insn #54
libbpf: sec '.relstruct_ops/scx_goland_core_enqueue': relo #10: insn #59 against 'scx_bpf_dispatch_vtime___compat'
libbpf: prog 'scx_goland_core_enqueue': found extern #3 'scx_bpf_dispatch_vtime___compat' (sym 45) for insn #59
libbpf: sec '.relstruct_ops/scx_goland_core_dispatch': collecting relocation for section(7) 'struct_ops/scx_goland_core_dispatch'
libbpf: sec '.relstruct_ops/scx_goland_core_dispatch': relo #0: insn #0 against 'scx_bpf_dsq_move_to_local'
libbpf: prog 'scx_goland_core_dispatch': found extern #6 'scx_bpf_dsq_move_to_local' (sym 47) for insn #0
libbpf: sec '.relstruct_ops/scx_goland_core_dispatch': relo #1: insn #4 against 'scx_bpf_dsq_move_to_local'
libbpf: prog 'scx_goland_core_dispatch': found extern #6 'scx_bpf_dsq_move_to_local' (sym 47) for insn #4
libbpf: sec '.relstruct_ops/scx_goland_core_dispatch': relo #2: insn #7 against 'scx_bpf_consume___compat'
libbpf: prog 'scx_goland_core_dispatch': found extern #0 'scx_bpf_consume___compat' (sym 48) for insn #7
libbpf: sec '.relstruct_ops/scx_goland_core_running': collecting relocation for section(9) 'struct_ops/scx_goland_core_running'
libbpf: sec '.relstruct_ops/scx_goland_core_running': relo #0: insn #1 against 'fifo_sched'
libbpf: prog 'scx_goland_core_running': found data map 1 (scx_gola.rodata, sec 20, off 0) for insn 1
libbpf: sec '.relstruct_ops/scx_goland_core_running': relo #1: insn #5 against '.bss'
libbpf: prog 'scx_goland_core_running': found data map 4 (scx_gola.bss, sec 26, off 0) for insn 5
libbpf: sec '.relstruct_ops/scx_goland_core_stopping': collecting relocation for section(11) 'struct_ops/scx_goland_core_stopping'
libbpf: sec '.relstruct_ops/scx_goland_core_stopping': relo #0: insn #1 against 'fifo_sched'
libbpf: prog 'scx_goland_core_stopping': found data map 1 (scx_gola.rodata, sec 20, off 0) for insn 1
libbpf: sec '.relstruct_ops/scx_goland_core_stopping': relo #1: insn #5 against '__SCX_SLICE_DFL'
libbpf: prog 'scx_goland_core_stopping': found data map 1 (scx_gola.rodata, sec 20, off 0) for insn 5
libbpf: sec '.relstruct_ops/scx_goland_core_enable': collecting relocation for section(13) 'struct_ops/scx_goland_core_enable'
libbpf: sec '.relstruct_ops/scx_goland_core_enable': relo #0: insn #1 against '.bss'
libbpf: prog 'scx_goland_core_enable': found data map 4 (scx_gola.bss, sec 26, off 0) for insn 1
libbpf: sec '.relstruct_ops.s/scx_goland_core_init': collecting relocation for section(15) 'struct_ops.s/scx_goland_core_init'
libbpf: sec '.relstruct_ops.s/scx_goland_core_init': relo #0: insn #2 against 'scx_bpf_create_dsq'
libbpf: prog 'scx_goland_core_init': found extern #1 'scx_bpf_create_dsq' (sym 53) for insn #2
libbpf: sec '.relstruct_ops/scx_goland_core_exit': collecting relocation for section(17) 'struct_ops/scx_goland_core_exit'
libbpf: sec '.relstruct_ops/scx_goland_core_exit': relo #0: insn #2 against 'uei'
libbpf: prog 'scx_goland_core_exit': found data map 3 (scx_gola.data, sec 24, off 0) for insn 2
libbpf: sec '.relstruct_ops/scx_goland_core_exit': relo #1: insn #4 against 'uei'
libbpf: prog 'scx_goland_core_exit': found data map 3 (scx_gola.data, sec 24, off 0) for insn 4
libbpf: sec '.relstruct_ops/scx_goland_core_exit': relo #2: insn #9 against 'uei'
libbpf: prog 'scx_goland_core_exit': found data map 3 (scx_gola.data, sec 24, off 0) for insn 9
libbpf: sec '.relstruct_ops/scx_goland_core_exit': relo #3: insn #15 against 'uei_dump_len'
libbpf: prog 'scx_goland_core_exit': found data map 1 (scx_gola.rodata, sec 20, off 0) for insn 15
libbpf: sec '.relstruct_ops/scx_goland_core_exit': relo #4: insn #19 against 'uei_dump'
libbpf: prog 'scx_goland_core_exit': found data map 2 (.data.uei_dump, sec 23, off 0) for insn 19
libbpf: struct_ops reloc scx_goland_core_ops: for 35 value 0 shdr_idx 3 rel->r_offset 0 map->sec_offset 0 name 40 ('scx_goland_core_select_cpu')
libbpf: struct_ops reloc scx_goland_core_ops: for 42 value 0 shdr_idx 5 rel->r_offset 8 map->sec_offset 0 name 682 ('scx_goland_core_enqueue')
libbpf: struct_ops reloc scx_goland_core_ops: for 46 value 0 shdr_idx 7 rel->r_offset 24 map->sec_offset 0 name 549 ('scx_goland_core_dispatch')
libbpf: struct_ops reloc scx_goland_core_ops: for 49 value 0 shdr_idx 9 rel->r_offset 48 map->sec_offset 0 name 629 ('scx_goland_core_running')
libbpf: struct_ops reloc scx_goland_core_ops: for 50 value 0 shdr_idx 11 rel->r_offset 56 map->sec_offset 0 name 589 ('scx_goland_core_stopping')
libbpf: struct_ops reloc scx_goland_core_ops: for 51 value 0 shdr_idx 13 rel->r_offset 144 map->sec_offset 0 name 788 ('scx_goland_core_enable')
libbpf: struct_ops reloc scx_goland_core_ops: for 52 value 0 shdr_idx 15 rel->r_offset 248 map->sec_offset 0 name 158 ('scx_goland_core_init')
libbpf: struct_ops reloc scx_goland_core_ops: for 54 value 0 shdr_idx 17 rel->r_offset 256 map->sec_offset 0 name 120 ('scx_goland_core_exit')
libbpf: object 'scx_goland_core': failed (-95) to create BPF token from '/sys/fs/bpf', skipping optional step...
libbpf: loaded kernel BTF from '/sys/kernel/btf/vmlinux'
libbpf: extern (func ksym) 'scx_bpf_consume___compat': resolved to vmlinux [19920]
libbpf: extern (func ksym) 'scx_bpf_create_dsq': resolved to vmlinux [19929]
libbpf: extern (func ksym) 'scx_bpf_dispatch___compat': resolved to vmlinux [19932]
libbpf: extern (func ksym) 'scx_bpf_dispatch_vtime___compat': resolved to vmlinux [19942]
libbpf: extern (func ksym) 'scx_bpf_select_cpu_dfl': resolved to vmlinux [19968]
libbpf: extern 'scx_bpf_dsq_insert' (weak): not resolved, defaulting to zero
libbpf: extern 'scx_bpf_dsq_insert_vtime' (weak): not resolved, defaulting to zero
libbpf: extern 'scx_bpf_dsq_move_to_local' (weak): not resolved, defaulting to zero
libbpf: struct_ops init_kern scx_goland_core_ops: type_id:405 kern_type_id:14126 kern_vtype_id:14223
libbpf: struct_ops init_kern scx_goland_core_ops: func ptr select_cpu is set to prog scx_goland_core_select_cpu from data(+0) to kern_data(+0)
libbpf: struct_ops init_kern scx_goland_core_ops: func ptr enqueue is set to prog scx_goland_core_enqueue from data(+8) to kern_data(+8)
libbpf: struct_ops init_kern scx_goland_core_ops: func ptr dispatch is set to prog scx_goland_core_dispatch from data(+24) to kern_data(+24)
libbpf: struct_ops init_kern scx_goland_core_ops: func ptr running is set to prog scx_goland_core_running from data(+48) to kern_data(+48)
libbpf: struct_ops init_kern scx_goland_core_ops: func ptr stopping is set to prog scx_goland_core_stopping from data(+56) to kern_data(+56)
libbpf: struct_ops init_kern scx_goland_core_ops: func ptr enable is set to prog scx_goland_core_enable from data(+144) to kern_data(+144)
libbpf: struct_ops init_kern scx_goland_core_ops: func ptr init is set to prog scx_goland_core_init from data(+248) to kern_data(+248)
libbpf: struct_ops init_kern scx_goland_core_ops: func ptr exit is set to prog scx_goland_core_exit from data(+256) to kern_data(+256)
libbpf: struct_ops init_kern scx_goland_core_ops: copy dispatch_max_batch 4 bytes from data(+264) to kern_data(+264)
libbpf: struct_ops init_kern scx_goland_core_ops: copy flags 8 bytes from data(+272) to kern_data(+272)
libbpf: struct_ops init_kern scx_goland_core_ops: copy timeout_ms 4 bytes from data(+280) to kern_data(+280)
libbpf: struct_ops init_kern scx_goland_core_ops: copy exit_dump_len 4 bytes from data(+284) to kern_data(+284)
libbpf: struct_ops init_kern scx_goland_core_ops: copy hotplug_seq 8 bytes from data(+288) to kern_data(+288)
libbpf: struct_ops init_kern scx_goland_core_ops: copy name 128 bytes from data(+296) to kern_data(+296)
libbpf: sec 'struct_ops/scx_goland_core_enqueue': found 1 CO-RE relocations
libbpf: CO-RE relocating [21] struct task_struct: found target candidate [146] struct task_struct in [vmlinux]
libbpf: prog 'scx_goland_core_enqueue': relo #0: <byte_off> [21] struct task_struct.scx.dsq_vtime (0:23:18 @ offset 856)
libbpf: prog 'scx_goland_core_enqueue': relo #0: matching candidate #0 <byte_off> [146] struct task_struct.scx.dsq_vtime (0:23:18 @ offset 856)
libbpf: prog 'scx_goland_core_enqueue': relo #0: patched insn #28 (LDX/ST/STX) off 856 -> 856
libbpf: sec 'struct_ops/scx_goland_core_running': found 2 CO-RE relocations
libbpf: prog 'scx_goland_core_running': relo #0: <byte_off> [21] struct task_struct.scx.dsq_vtime (0:23:18 @ offset 856)
libbpf: prog 'scx_goland_core_running': relo #0: matching candidate #0 <byte_off> [146] struct task_struct.scx.dsq_vtime (0:23:18 @ offset 856)
libbpf: prog 'scx_goland_core_running': relo #0: patched insn #8 (LDX/ST/STX) off 856 -> 856
libbpf: prog 'scx_goland_core_running': relo #1: <byte_off> [21] struct task_struct.scx.dsq_vtime (0:23:18 @ offset 856)
libbpf: prog 'scx_goland_core_running': relo #1: matching candidate #0 <byte_off> [146] struct task_struct.scx.dsq_vtime (0:23:18 @ offset 856)
libbpf: prog 'scx_goland_core_running': relo #1: patched insn #11 (LDX/ST/STX) off 856 -> 856
libbpf: sec 'struct_ops/scx_goland_core_stopping': found 4 CO-RE relocations
libbpf: prog 'scx_goland_core_stopping': relo #0: <byte_off> [21] struct task_struct.scx.slice (0:23:17 @ offset 848)
libbpf: prog 'scx_goland_core_stopping': relo #0: matching candidate #0 <byte_off> [146] struct task_struct.scx.slice (0:23:17 @ offset 848)
libbpf: prog 'scx_goland_core_stopping': relo #0: patched insn #8 (LDX/ST/STX) off 848 -> 848
libbpf: prog 'scx_goland_core_stopping': relo #1: <byte_off> [21] struct task_struct.scx.weight (0:23:6 @ offset 756)
libbpf: prog 'scx_goland_core_stopping': relo #1: matching candidate #0 <byte_off> [146] struct task_struct.scx.weight (0:23:6 @ offset 756)
libbpf: prog 'scx_goland_core_stopping': relo #1: patched insn #10 (LDX/ST/STX) off 756 -> 756
libbpf: prog 'scx_goland_core_stopping': relo #2: <byte_off> [21] struct task_struct.scx.dsq_vtime (0:23:18 @ offset 856)
libbpf: prog 'scx_goland_core_stopping': relo #2: matching candidate #0 <byte_off> [146] struct task_struct.scx.dsq_vtime (0:23:18 @ offset 856)
libbpf: prog 'scx_goland_core_stopping': relo #2: patched insn #13 (LDX/ST/STX) off 856 -> 856
libbpf: prog 'scx_goland_core_stopping': relo #3: <byte_off> [21] struct task_struct.scx.dsq_vtime (0:23:18 @ offset 856)
libbpf: prog 'scx_goland_core_stopping': relo #3: matching candidate #0 <byte_off> [146] struct task_struct.scx.dsq_vtime (0:23:18 @ offset 856)
libbpf: prog 'scx_goland_core_stopping': relo #3: patched insn #15 (LDX/ST/STX) off 856 -> 856
libbpf: sec 'struct_ops/scx_goland_core_enable': found 1 CO-RE relocations
libbpf: prog 'scx_goland_core_enable': relo #0: <byte_off> [21] struct task_struct.scx.dsq_vtime (0:23:18 @ offset 856)
libbpf: prog 'scx_goland_core_enable': relo #0: matching candidate #0 <byte_off> [146] struct task_struct.scx.dsq_vtime (0:23:18 @ offset 856)
libbpf: prog 'scx_goland_core_enable': relo #0: patched insn #4 (LDX/ST/STX) off 856 -> 856
libbpf: sec 'struct_ops/scx_goland_core_exit': found 6 CO-RE relocations
libbpf: CO-RE relocating [396] struct scx_exit_info: found target candidate [14117] struct scx_exit_info in [vmlinux]
libbpf: prog 'scx_goland_core_exit': relo #0: <byte_off> [396] struct scx_exit_info.reason (0:2 @ offset 16)
libbpf: prog 'scx_goland_core_exit': relo #0: matching candidate #0 <byte_off> [14117] struct scx_exit_info.reason (0:2 @ offset 16)
libbpf: prog 'scx_goland_core_exit': relo #0: patched insn #1 (LDX/ST/STX) off 16 -> 16
libbpf: prog 'scx_goland_core_exit': relo #1: <byte_off> [396] struct scx_exit_info.msg (0:5 @ offset 40)
libbpf: prog 'scx_goland_core_exit': relo #1: matching candidate #0 <byte_off> [14117] struct scx_exit_info.msg (0:5 @ offset 40)
libbpf: prog 'scx_goland_core_exit': relo #1: patched insn #12 (LDX/ST/STX) off 40 -> 40
libbpf: prog 'scx_goland_core_exit': relo #2: <byte_off> [396] struct scx_exit_info.dump (0:6 @ offset 48)
libbpf: prog 'scx_goland_core_exit': relo #2: matching candidate #0 <byte_off> [14117] struct scx_exit_info.dump (0:6 @ offset 48)
libbpf: prog 'scx_goland_core_exit': relo #2: patched insn #18 (LDX/ST/STX) off 48 -> 48
libbpf: prog 'scx_goland_core_exit': relo #3: <field_exists> [396] struct scx_exit_info.exit_code (0:1 @ offset 8)
libbpf: prog 'scx_goland_core_exit': relo #3: matching candidate #0 <field_exists> [14117] struct scx_exit_info.exit_code (0:1 @ offset 8)
libbpf: prog 'scx_goland_core_exit': relo #3: patched insn #22 (ALU/ALU64) imm 1 -> 1
libbpf: prog 'scx_goland_core_exit': relo #4: <byte_off> [396] struct scx_exit_info.exit_code (0:1 @ offset 8)
libbpf: prog 'scx_goland_core_exit': relo #4: matching candidate #0 <byte_off> [14117] struct scx_exit_info.exit_code (0:1 @ offset 8)
libbpf: prog 'scx_goland_core_exit': relo #4: patched insn #24 (LDX/ST/STX) off 8 -> 8
libbpf: prog 'scx_goland_core_exit': relo #5: <byte_off> [396] struct scx_exit_info.kind (0:0 @ offset 0)
libbpf: prog 'scx_goland_core_exit': relo #5: matching candidate #0 <byte_off> [14117] struct scx_exit_info.kind (0:0 @ offset 0)
libbpf: prog 'scx_goland_core_exit': relo #5: patched insn #26 (LDX/ST/STX) off 0 -> 0
libbpf: prog 'scx_goland_core_select_cpu': relo #5: poisoning insn #33 that calls kfunc 'scx_bpf_dsq_insert'
libbpf: prog 'scx_goland_core_enqueue': relo #4: poisoning insn #26 that calls kfunc 'scx_bpf_dsq_insert'
libbpf: prog 'scx_goland_core_enqueue': relo #9: poisoning insn #49 that calls kfunc 'scx_bpf_dsq_insert_vtime'
libbpf: prog 'scx_goland_core_dispatch': relo #1: poisoning insn #4 that calls kfunc 'scx_bpf_dsq_move_to_local'
libbpf: map 'stats': created successfully, fd=8
libbpf: map 'scx_gola.rodata': created successfully, fd=9
libbpf: map '.data.uei_dump': created successfully, fd=10
libbpf: map 'scx_gola.data': created successfully, fd=11
libbpf: map 'scx_gola.bss': created successfully, fd=12
libbpf: map 'scx_goland_core_ops': created successfully, fd=13
2025/01/23 22:19:31 local: 12, global: 0
2025/01/23 22:19:32 local: 4081, global: 4322
2025/01/23 22:19:33 local: 9852, global: 6612
2025/01/23 22:19:34 local: 15959, global: 12309
2025/01/23 22:19:35 local: 17101, global: 13003
2025/01/23 22:19:36 local: 23655, global: 17781
2025/01/23 22:19:37 local: 27279, global: 22053
^C2025/01/23 22:19:38 receive os signal
2025/01/23 22:19:38 scheduler exit

sameple code:

package main

import (
	"context"
	"log"
	"os"
	"os/signal"
	"sync"
	"syscall"
	"time"

	"encoding/binary"
	"unsafe"

	bpf "github.com/aquasecurity/libbpfgo"
)

func endian() binary.ByteOrder {
	var i int32 = 0x01020304
	u := unsafe.Pointer(&i)
	pb := (*byte)(u)
	b := *pb
	if b == 0x04 {
		return binary.LittleEndian
	}

	return binary.BigEndian
}

func main() {
	bpfModule, err := bpf.NewModuleFromFileArgs(bpf.NewModuleArgs{
		BPFObjPath:     "scx_goland_core.bpf.o",
		KernelLogLevel: 0,
	})
	if err != nil {
		panic(err)
	}
	defer bpfModule.Close()

	if err := bpfModule.BPFLoadObject(); err != nil {
		panic(err)
	}

	m := bpfModule

	iters := m.Iterator()
	for {
		m := iters.NextMap()
		if m == nil {
			break
		}
		if m.Type().String() == "BPF_MAP_TYPE_STRUCT_OPS" {
			if err := m.AttachStructOps(); err != nil {
				log.Printf("error: %v", err)
			}
		}
	}

	if statsMap, err := bpfModule.GetMap("stats"); err != nil {
		log.Printf("error: %v", err)
	} else {
		var wg sync.WaitGroup
		ctx, cancel := context.WithCancel(context.Background())
		signalChan := make(chan os.Signal, 1)
		signal.Notify(signalChan, syscall.SIGINT, syscall.SIGTERM)
		wg.Add(1)
		go func(ctx context.Context) {
			for true {
				select {
				case <-ctx.Done():
					wg.Done()
					return
				default:
					res := getStat(statsMap)
					log.Printf("local: %d, global: %d", res[0], res[1])
				}
				time.Sleep(1 * time.Second)
			}
		}(ctx)
		<-signalChan
		log.Println("receive os signal")
		cancel()
		wg.Wait()
		log.Println("scheduler exit")
	}
}

func getStat(m *bpf.BPFMap) []uint64 {
	cpuNum, err := bpf.NumPossibleCPUs()
	if err != nil {
		log.Fatal(err)
	}
	var cnts [][]uint64 = make([][]uint64, 2)
	cnts[0] = make([]uint64, cpuNum)
	cnts[1] = make([]uint64, cpuNum)
	stats := []uint64{0, 0}
	for i := 0; i < 2; i++ {
		v, err := m.GetValue(unsafe.Pointer(&i))
		if err != nil {
			log.Fatal(err)
		}
		for cpu := 0; cpu < cpuNum; cpu++ {
			n := v[cpu*8 : cpu*8+8]
			cnts[i][cpu] = endian().Uint64(n)
			stats[i] += cnts[i][cpu]
		}
	}
	return stats
}

@CLAassistant
Copy link

CLAassistant commented Jan 23, 2025

CLA assistant check
All committers have signed the CLA.

@geyslan
Copy link
Member

geyslan commented Jan 23, 2025

@ianchen0119 that's awesome. Tks for the contribution.

Would you mind creating a selftest to this feature? As soon as possible I'm returning to review it. Anything, just ask.

@geyslan geyslan self-requested a review January 23, 2025 14:43
@ianchen0119
Copy link
Author

Hi @geyslan

Sure, I will add the selftest in next few days.

Thank you.

@ianchen0119
Copy link
Author

Hi @geyslan

I just added the selftest for struct_ops, please help to reivew PR when you feel free.
image

Thank you!

@geyslan
Copy link
Member

geyslan commented Jan 27, 2025

@ianchen0119 thanks. Would you mind rebasing it? I believe you added not expected files. Regarding the Makefile, you can reuse the common, just copying the link to the folder of the new selftest.

@ianchen0119
Copy link
Author

ianchen0119 commented Jan 27, 2025

Hi @geyslan

Thanks for the comment.

I will rebase the PR later.
Moreover,
I believe that the added files are needed and not existed in the main branch if my understanding was correct.

@geyslan
Copy link
Member

geyslan commented Jan 27, 2025

I believe that the added files are needed and not existed in the main branch if my understanding was correct.

Any new type or definition you can add to https://github.com/aquasecurity/libbpfgo/blob/main/selftest/common/vmlinux.h. We keep it to minimal to reduce building time etc. Try adding only what's required to bpf_map__attach_struct_ops which is being wrapped by libbpfgo.

In your bpf.c include vmlinux and the helpers (e.g.: spinlocks), so I believe it'll be ok.

@ianchen0119
Copy link
Author

Hi @geyslan ,

I just updated my PR based on your comments.

Thank you!

@geyslan
Copy link
Member

geyslan commented Jan 29, 2025

I just updated my PR based on your comments.

I think your vmlinux.h is still commited, see the LOC changed: +151,024.

@ianchen0119
Copy link
Author

Hi @geyslan

I added the needed definition for compiling the scx program in vmlinux.h in the latest commit.
Is the change acceptable?

Thanks!

@ianchen0119
Copy link
Author

Hi @geyslan

It seems like the kernel version of the ci runner is v6.8.
However, the added test program requires the kernel v6.12.

Do you have any suggestions on this? Thanks!

@ianchen0119
Copy link
Author

@geyslan
To offload the scheduling policy from kernel space to user space, I must implement the golang API for user ring buffer map.
https://github.com/ianchen0119/scx/blob/feat/scx-goland/go/scx_goland_core/goland_core/obj.go#L47
I think I can also commit the needed changes in this branch before PR accepted.

@geyslan
Copy link
Member

geyslan commented Jan 31, 2025

However, the added test program requires the kernel v6.12.

Do you have any suggestions on this? Thanks!

Well, we can offload a runner (with newest kernel) for testing new features like it - we'll be able to do that in a few days.
Meanwhile you can create a new run script as

kern_version ge 5.8
for 6.12 and link your selftest/struct-ops/run.sh to it.

To offload the scheduling policy from kernel space to user space, I must implement the golang API for user ring buffer map.
I think I can also commit the needed changes in this branch before PR accepted.

You're currently bringing in the bpf_map__attach_struct_ops wrapper, right? So it would be nice to consider user_ring_buffer in a different PR, since it would require its own selftest as well. WDYT?

@ianchen0119
Copy link
Author

@geyslan

Well, we can offload a runner (with newest kernel) for testing new features like it - we'll be able to do that in a few days.
Meanwhile you can create a new run script as

The file run-6.12.sh has been added, thanks for your advice.

You're currently bringing in the bpf_map__attach_struct_ops wrapper, right? So it would be nice to consider user_ring_buffer in a different PR, since it would require its own selftest as well. WDYT?

No problem, I will open the new PR for support user_ring_buffer once the current PR merged.

@arighi
Copy link

arighi commented Jan 31, 2025

It seems like the kernel version of the ci runner is v6.8. However, the added test program requires the kernel v6.12.

Do you have any suggestions on this? Thanks!

JFYI, for the sched_ext selftests we use virtme-ng (https://github.com/arighi/virtme-ng). We also recompile the kernel, but in this case it'd be proobably easier to use a precompiled 6.12, for example:

vng -r v6.12.2 -- CMD

This would download a v6.12.2 from the Ubuntu mainline kernel builds, run the command CMD inside a guest with that kernel and print the stdout/stderr of the command back to the host, so you can catch potential errors. And you can run this also as a github action.

@geyslan
Copy link
Member

geyslan commented Feb 3, 2025

JFYI, for the sched_ext selftests we use virtme-ng (https://github.com/arighi/virtme-ng). We also recompile the kernel, but in this case it'd be proobably easier to use a precompiled 6.12, for example:

vng -r v6.12.2 -- CMD

@arighi That's awesome! @ianchen0119 perhaps you just need to change the specific run.sh to use vng.

@ianchen0119
Copy link
Author

@geyslan

No problem, I will do in next few day.

@@ -14,12 +14,14 @@ runs:
go-version: "${{ inputs.go-version }}"
- name: Install Compilers & Formatters
run: |
sudo add-apt-repository ppa:arighi/virtme-ng -y
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not maintaining this ppa anymore (the version of virtme-ng there is pretty old). You could simply apt install virtme-ng if you're using at least an Ubuntu Noble (24.04) image, or, even better, install vng via pip, pip install --break-system-packages virtme-ng. This is probably a safer / more reliable approach.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants