Re enabling iiso-offline-install-iscsi.bios

[jbtrystram]

Following the merge of coreos/coreos-assembler#3702

[dustymabe]

I assume this passes on every architecture now?

[jbtrystram]

the target container seems to hang when killed after the test finish, which cause kola to timeout the test. I am investigating this and trying to fix it, but it's hard to reproduce.

[jlebon]

Anything in the journal logs now that we have them?

I wonder if systemd in the target container isn't tearing down cleanly. One thing to try is to make the container directly run targetclid instead and pass --init when you podman run. Effectively, this replaces systemd with catatonit.

[jbtrystram]

Anything in the journal logs now that we have them?

journal.txt

edit : associated console log is there

[jlebon]

One thing I notice here looking at the logs for at least CI here is:

+ targetcli /backstores/block create name=coreos dev=/dev/disk/by-id/virtio-target
Rounding down aligned max_sectors from 4294967295 to 4294967288
g-io-error-quark: Could not connect: No such file or directory (1)
2024-01-23 16:44:49.613581372 +0000 UTC m=+0.578572190 container exec_died 2677d88dfe9cd25778370fdf1b043977a02a38874e30366fbe6b76010de7a007 (image=quay.io/jbtrystram/targetcli:latest, name=target, org.label-schema.license=GPLv2, org.label-
schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, PODMAN_SYSTEMD_UNIT=target.service, io.buildah.version=1.23.1, org.label-schema.build-date=20240116)
setup-targetcli.service: Main process exited, code=exited, status=255/EXCEPTION
setup-targetcli.service: Failed with result 'exit-code'.

which I think might be targetcli not being able to reach out to the targetclid socket? One way this could happen is if systemd in the container hasn't yet reached sockets.target before setup-targetcli.service starts running. I think this is something that changing the container to directly run targetclid would help. Alternatively, you can have the script wait until the socket is up before proceeding.

I have a suspicion that this is also the issue in the logs from #2816 (comment). The journal logs there look truncated, but stop just after that same line

+ targetcli /backstores/block create name=coreos dev=/dev/disk/by-id/virtio-target

I think what happened there is that OnFailure=emergency.target triggered, which in turn triggered coreos-test-entered-emergency-target.service which explicitly does systemctl poweroff, which is why we see the system power off.

Additionally, https://github.com/coreos/coreos-assembler/blob/1982e0fb3e67e157c9fa5e87abd0793d6088be8d/mantle/cmd/kola/testiso.go#L1015 should be testisocompletion (and the Butane config updated accordingly) so that coreos-test-entered-emergency-target.service can actually write to it so that we can detect failure correctly instead of hanging after QEMU exits and erroring with the more abstract "QEMU exited; timed out waiting for completion" error.

[jbtrystram]

which I think might be targetcli not being able to reach out to the targetclid socket?

yeah, good point. I added some healthcheck and updated the way the service is marked ready. I'll test that and report back !

Additionally, coreos/coreos-assembler@1982e0f/mantle/cmd/kola/testiso.go#L1015 should be testisocompletion ...

good catch, thanks ! Fixed.