Setting the openfga properties breaks incus admin init --preseed

[rgl]

Required information

• Distribution: Debian
• Distribution version: 12
• The output of "incus info" or if that fails:
  • Kernel version: Linux incus 6.1.0-16-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.67-1 (2023-12-12) x86_64 GNU/Linux
  • LXC version:
  • Incus version: 0.4
  • Storage backend in use: btrfs

Issue description

Setting the openfga properties as (full example at https://github.com/rgl/incus-playground/blob/main/provision-incus.sh#L124-L133):

incus admin init --preseed <<EOF
config:
  openfga.api.url: https://$openfga_domain:8080
  openfga.api.token: abracadabra
  openfga.store.id: $(jq .store.id /vagrant/shared/openfga-incus.json)
...
EOF

breaks incus admin init --preseed with the error:

incus: Error: Failed to create storage pool "default": Post "http://unix.socket/1.0/storage-pools": EOF

Please note that configuring the openfga properties after the initial incus admin init --preseed works fine.

Steps to reproduce

1. At incus admin init --preseed time set the openfga properties as described in this issue description.
2. See the error.

Information to attach

• Any relevant kernel output (dmesg)
• Container log (incus info NAME --show-log)
• Container configuration (incus config show NAME --expanded)
• Main daemon log (at /var/log/incus/incusd.log)
• Output of the client with --debug
• Output of the daemon with --debug (alternatively output of incus monitor --pretty while reproducing the issue)

[stgraber]

Can you look for a stack trace in your journalctl -n 300 output or in /var/log/incus/incusd.log?

[rgl]

here are the logs:

root@incus:~# journalctl -u incus
dez 29 07:29:07 incus systemd[1]: Starting incus.service - Incus - Daemon...
dez 29 07:29:07 incus incusd[6282]: time="2023-12-29T07:29:07Z" level=warning msg=" - Couldn't find the CGroup hugetlb controller, hugepage limits will be ignored"
dez 29 07:29:07 incus incusd[6282]: time="2023-12-29T07:29:07Z" level=warning msg=" - Couldn't find the CGroup network priority controller, per-instance network priority will be ignored. Please use p>
dez 29 07:29:08 incus systemd[1]: Started incus.service - Incus - Daemon.

root@incus:~# 

PS after enabling --debug in incusd, I was able to see the stack trace:

time="2023-12-29T08:12:20Z" level=debug msg="Handling API request" ip=@ method=POST protocol=unix url=/1.0/storage-pools username=root
time="2023-12-29T08:12:20Z" level=debug msg="API Request\n\t{\n\t\t\"config\": {\n\t\t\t\"source\": \"/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_incus\"\n\t\t},\n\t\t\"description\": \"\",\n\t\t\"name\": \"default\",\n\t\t\"driver\": \"btrfs\"\n\t}" ip=@ method=POST protocol=unix url=/1.0/storage-pools username=root
time="2023-12-29T08:12:20Z" level=debug msg="Create started" clientType=normal config="map[source:/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_incus]" description= driver=btrfs pool=default
time="2023-12-29T08:12:21Z" level=debug msg="Success for operation" class=task description="Updating instance types" operation=a3eb02b4-cd4a-4a9c-a757-4bfebe13ea2f project=
time="2023-12-29T08:12:21Z" level=info msg="Done updating instance types"
time="2023-12-29T08:12:31Z" level=debug msg="Create finished" clientType=normal config="map[size: source:/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_incus volatile.initial_source:/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_incus]" description= driver=btrfs pool=default
time="2023-12-29T08:12:31Z" level=debug msg="Mount started" driver=btrfs pool=default
time="2023-12-29T08:12:31Z" level=debug msg="Mount finished" driver=btrfs pool=default
time="2023-12-29T08:12:31Z" level=debug msg="Marked storage pool local status as created" pool=default
time="2023-12-29T08:12:31Z" level=info msg="http: panic serving @: runtime error: invalid memory address or nil pointer dereference\ngoroutine 404 [running]:\nnet/http.(*conn).serve.func1()\n\t/usr/local/go/src/net/http/server.go:1868 +0xb9\npanic({0x1b43300?, 0x2eab120?})\n\t/usr/local/go/src/runtime/panic.go:920 +0x270\nmain.storagePoolsPost(0xc0004c0000?, 0xc000b34700)\n\t/build/incus/cmd/incusd/storage_pools.go:365 +0x7bd\nmain.(*Daemon).createCmd.func1.2({0x1f4ffc8?, 0xc0004a5900?, 0x30?})\n\t/build/incus/cmd/incusd/daemon.go:664 +0xcc\nmain.(*Daemon).createCmd.func1({0x21a7c48, 0xc000754fc0}, 0xc000b34600)\n\t/build/incus/cmd/incusd/daemon.go:675 +0x134e\nnet/http.HandlerFunc.ServeHTTP(0xc000b34500?, {0x21a7c48?, 0xc000754fc0?}, 0x1df1876?)\n\t/usr/local/go/src/net/http/server.go:2136 +0x29\ngithub.com/gorilla/mux.(*Router).ServeHTTP(0xc000140b40, {0x21a7c48, 0xc000754fc0}, 0xc000b34400)\n\t/root/go/pkg/mod/github.com/gorilla/[email protected]/mux.go:212 +0x1c5\nmain.(*httpServer).ServeHTTP(0xc00050d860, {0x21a7c48, 0xc000754fc0}, 0xc000b34400)\n\t/build/incus/cmd/incusd/api.go:379 +0xd0\nnet/http.serverHandler.ServeHTTP({0x21a2df0?}, {0x21a7c48?, 0xc000754fc0?}, 0x6?)\n\t/usr/local/go/src/net/http/server.go:2938 +0x8e\nnet/http.(*conn).serve(0xc0000fbc20, {0x21ab938, 0xc000b2f230})\n\t/usr/local/go/src/net/http/server.go:2009 +0x5f4\ncreated by net/http.(*Server).Serve in goroutine 51\n\t/usr/local/go/src/net/http/server.go:3086 +0x5cb"

PPS translating the \n into actual new lines, we get:

http: panic serving @: runtime error: invalid memory address or nil pointer dereference
goroutine 404 [running]:
net/http.(*conn).serve.func1()
	/usr/local/go/src/net/http/server.go:1868 +0xb9
panic({0x1b43300?, 0x2eab120?})
	/usr/local/go/src/runtime/panic.go:920 +0x270
main.storagePoolsPost(0xc0004c0000?, 0xc000b34700)
	/build/incus/cmd/incusd/storage_pools.go:365 +0x7bd
main.(*Daemon).createCmd.func1.2({0x1f4ffc8?, 0xc0004a5900?, 0x30?})
	/build/incus/cmd/incusd/daemon.go:664 +0xcc
main.(*Daemon).createCmd.func1({0x21a7c48, 0xc000754fc0}, 0xc000b34600)
	/build/incus/cmd/incusd/daemon.go:675 +0x134e
net/http.HandlerFunc.ServeHTTP(0xc000b34500?, {0x21a7c48?, 0xc000754fc0?}, 0x1df1876?)
	/usr/local/go/src/net/http/server.go:2136 +0x29
github.com/gorilla/mux.(*Router).ServeHTTP(0xc000140b40, {0x21a7c48, 0xc000754fc0}, 0xc000b34400)
	/root/go/pkg/mod/github.com/gorilla/[email protected]/mux.go:212 +0x1c5
main.(*httpServer).ServeHTTP(0xc00050d860, {0x21a7c48, 0xc000754fc0}, 0xc000b34400)
	/build/incus/cmd/incusd/api.go:379 +0xd0
net/http.serverHandler.ServeHTTP({0x21a2df0?}, {0x21a7c48?, 0xc000754fc0?}, 0x6?)
	/usr/local/go/src/net/http/server.go:2938 +0x8e
net/http.(*conn).serve(0xc0000fbc20, {0x21ab938, 0xc000b2f230})
	/usr/local/go/src/net/http/server.go:2009 +0x5f4
created by net/http.(*Server).Serve in goroutine 51
	/usr/local/go/src/net/http/server.go:3086 +0x5cb

[stgraber]

Thanks, that should make it easy to fix!

[stgraber]

@rgl could you share the full preseed file?

I've tried with your snippet as well as a similar snippet also including a storage pool definition, but this doesn't trigger the crash.

[rgl]

@stgraber you mean https://github.com/rgl/incus-playground/blob/main/provision-incus.sh#L133-L167?

[stgraber]

Oops, I missed that link earlier, sorry :)

[stgraber]

Got the fault reproduced here.