Stop using oldest-supported-numpy #111

Job	Run time
Tests with external liberfa	43s
[ ubuntu-latest ]	22s
[ macos-latest ]	2s
[ windows-latest ]	1s
Test with oldest supported versions of our dependencies [ ubuntu-20.04 ]	11s
Test with medium old supported versions of our dependencies [ ubuntu-22.04 ]	4s
Test with development versions of our dependencies [ ubuntu-latest ]	10s
Code style checks [ ubuntu-latest ]	12s
Documentation build [ ubuntu-latest ]	18s
build_and_publish / Load build targets	7s
build_and_publish / Build source distribution	1m 13s
build_and_publish / Build cp*-manylinux_i686 wheels	22m 35s
build_and_publish / Build cp*-manylinux_x86_64 wheels	2m 58s
build_and_publish / Build cp*-musllinux_x86_64 wheels	2m 52s
build_and_publish / Build cp*-manylinux_aarch64 wheels	49m 17s
build_and_publish / Build cp*-macosx_x86_64 wheels	6m 50s
build_and_publish / Build cp*-macosx_arm64 wheels	8m 5s
build_and_publish / Build cp-win wheels	12m 43s
build_and_publish / Upload	0s
	1h 48m 43s

Provide feedback