failed: TASK [Get instance(s) ip address]

[mhagnumdw]

Error

The virtual machine does not start.

FAILED - RETRYING: [localhost]: Get instance(s) ip address (1 retries left).Result was: {
    "attempts": 10,
    "changed": true,
    "cmd": "virsh --connect qemu:///system domifaddr instance | grep 52:54:00:ab:cd: | awk '{ print $4 }' | sed 's/\\/.*//'\n",
    "delta": "0:00:00.028730",
    "end": "2022-06-16 19:10:10.203963",
    "invocation": {
        "module_args": {
            "_raw_params": "virsh --connect qemu:///system domifaddr instance | grep 52:54:00:ab:cd: | awk '{ print $4 }' | sed 's/\\/.*//'\n",
            "_uses_shell": true,
            "argv": null,
            "chdir": null,
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "stdin_add_newline": true,
            "strip_empty_ends": true,
            "warn": false
        }
    },
    "msg": "",
    "rc": 0,
    "retries": 11,
    "start": "2022-06-16 19:10:10.175233",
    "stderr": "",
    "stderr_lines": [],
    "stdout": "",
    "stdout_lines": []
}
<127.0.0.1> EXEC /bin/sh -c 'echo ~mhagnumdw && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/mhagnumdw/.ansible/tmp `"&& mkdir "` echo /home/mhagnumdw/.ansible/tmp/ansible-tmp-1655417412.2326565-58511-220510289016393 `" && echo ansible-tmp-1655417412.2326565-58511-220510289016393="` echo /home/mhagnumdw/.ansible/tmp/ansible-tmp-1655417412.2326565-58511-220510289016393 `" ) && sleep 0'
Using module file /home/mhagnumdw/projetos/ansible-setup-workstation/.venv/lib/python3.10/site-packages/ansible/modules/command.py
<127.0.0.1> PUT /home/mhagnumdw/.ansible/tmp/ansible-local-57192ogu4wtzr/tmpw43ce8a3 TO /home/mhagnumdw/.ansible/tmp/ansible-tmp-1655417412.2326565-58511-220510289016393/AnsiballZ_command.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/mhagnumdw/.ansible/tmp/ansible-tmp-1655417412.2326565-58511-220510289016393/ /home/mhagnumdw/.ansible/tmp/ansible-tmp-1655417412.2326565-58511-220510289016393/AnsiballZ_command.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/home/mhagnumdw/projetos/ansible-setup-workstation/.venv/bin/python /home/mhagnumdw/.ansible/tmp/ansible-tmp-1655417412.2326565-58511-220510289016393/AnsiballZ_command.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /home/mhagnumdw/.ansible/tmp/ansible-tmp-1655417412.2326565-58511-220510289016393/ > /dev/null 2>&1 && sleep 0'
failed: [localhost] (item=instance) => {
    "ansible_loop_var": "item",
    "attempts": 10,
    "changed": true,
    "cmd": "virsh --connect qemu:///system domifaddr instance | grep 52:54:00:ab:cd: | awk '{ print $4 }' | sed 's/\\/.*//'\n",
    "delta": "0:00:00.026336",
    "end": "2022-06-16 19:10:12.422628",
    "invocation": {
        "module_args": {
            "_raw_params": "virsh --connect qemu:///system domifaddr instance | grep 52:54:00:ab:cd: | awk '{ print $4 }' | sed 's/\\/.*//'\n",
            "_uses_shell": true,
            "argv": null,
            "chdir": null,
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "stdin_add_newline": true,
            "strip_empty_ends": true,
            "warn": false
        }
    },
    "item": {
        "cpu": 2,
        "cpu_model": "qemu64",
        "disk_size": "10G",
        "image_url": "https://download.fedoraproject.org/pub/fedora/linux/releases/34/Cloud/x86_64/images/Fedora-Cloud-Base-34-1.2.x86_64.qcow2",
        "memory_size": "1",
        "name": "instance",
        "ssh_port": 22
    },
    "msg": "",
    "rc": 0,
    "start": "2022-06-16 19:10:12.396292",
    "stderr": "",
    "stderr_lines": [],
    "stdout": "",
    "stdout_lines": []
}

Print

The virtual machine is stopped at boot.

Molecule vms folder

$ ls -lah /home/mhagnumdw/.local/molecule/default/vms/
total 2,9M
drwxr-xr-x. 1 mhagnumdw mhagnumdw  110 jun 16 19:09 .
drwxr-xr-x. 1 mhagnumdw mhagnumdw    6 jun 16 18:18 ..
-rw-r--r--. 1 qemu      qemu      366K jun 16 19:09 instance-cloud-init.iso
-rw-rw-r--. 1 mhagnumdw mhagnumdw    0 jun 16 19:09 .instance-expanded
-rw-r--r--. 1 qemu      qemu       16G jun 16 19:09 instance.qcow2

Virtual Machine xml

sudo virsh dumpxml instance

Click to see!

<domain type='kvm' id='6'>
  <name>instance</name>
  <uuid>b4e255d0-c7f2-48be-a49c-8b929ffc80f2</uuid>
  <memory unit='KiB'>1048576</memory>
  <currentMemory unit='KiB'>1048576</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-6.1'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <vmport state='off'/>
  </features>
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>qemu64</model>
    <feature policy='require' name='x2apic'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='lahf_lm'/>
    <feature policy='disable' name='svm'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/home/mhagnumdw/.local/molecule/default/vms/instance.qcow2' index='2'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/home/mhagnumdw/.local/molecule/default/vms/instance-cloud-init.iso' index='1'/>
      <backingStore/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <alias name='ide0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='piix3-uhci'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='ide' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:ab:cd:47'/>
      <source network='molecule-libvirt' portid='99ec45a6-5fd6-4126-8aa5-26690c66f065' bridge='molecule-br0'/>
      <target dev='vnet5'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/4'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/4'>
      <source path='/dev/pts/4'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-6-instance/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0' state='disconnected'/>
      <alias name='channel1'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    </channel>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'>
      <alias name='input1'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input2'/>
    </input>
    <graphics type='spice' port='5900' autoport='yes' listen='127.0.0.1'>
      <listen type='address' address='127.0.0.1'/>
      <image compression='off'/>
    </graphics>
    <sound model='ich6'>
      <alias name='sound0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <audio id='1' type='spice'/>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'>
    <label>system_u:system_r:svirt_t:s0:c54,c201</label>
    <imagelabel>system_u:object_r:svirt_image_t:s0:c54,c201</imagelabel>
  </seclabel>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+107:+107</label>
    <imagelabel>+107:+107</imagelabel>
  </seclabel>
</domain>

Molecule conf

Same as the README project homepage.

---
dependency:
  name: galaxy
driver:
  name: libvirt
platforms:
  - name: instance
    cpu: 2
    cpu_model: "qemu64"
    image_url: "https://download.fedoraproject.org/pub/fedora/linux/releases/34/Cloud/x86_64/images/Fedora-Cloud-Base-34-1.2.x86_64.qcow2"
    disk_size: "10G"
    ssh_port: 22
    memory_size: "1" # in GB
provisioner:
  name: ansible
verifier:
  name: ansible

Versions

$ molecule --version
molecule 3.6.1 using python 3.10 
    ansible:2.13.0
    delegated:3.6.1 from molecule
    docker:1.1.0 from molecule_docker requiring collections: community.docker>=1.9.1
    libvirt:0.0.5 from molecule_libvirt

Molecule command

molecule -vvv --debug converge

Can someone help me?

[apatard]

Was there some errors earlier ? did you look at the libvirt logs ? I think it should be something like /var/log/libvirt/qemu/instance.log

[mhagnumdw]

No error before TASK [Get instance(s) ip address].

No errors in Kvm log at /var/log/libvirt/qemu/instance.log

Is the size of the files I posted before really correct?

If you can test it in my test project:

git clone [email protected]:mhagnumdw/ansible-setup-workstation.git
cd ansible-setup-workstation
git checkout libvirt-kvm
pip install virtualenv
virtualenv .venv
source .venv/bin/activate
pip install molecule ansible-core molecule-libvirt libvirt-python
molecule -vvv --debug converge # problem here!
molecule destroy

Any ideas?

[mhagnumdw]

Good news.

I got the error in the Create cloud-init iso file task: "stderr": "I: -input-charset not specified, using utf-8 (detected in locale settings)\nTotal translation ...

So I set the charset in the task ($PROJECT_FOLDER/.venv/lib/python3.10/site-packages/molecule_libvirt/playbooks/create.yml), in the command mkisofs, being like this mkisofs -input-charset us-ascii -o .... The charset error is gone, but the rest of the error remains. I left it anyway.

Then I saw another error in the Expand original image on the new sized volume task that the virt-resize command was not found. In the case of Fedora I installed it with dnf install guestfs-tools and then everything started working.

Is it worth adding a check to see if the command exists?

[mhagnumdw]

Please feel free to close this issue.

[javanaut-de]

Same error occuring for me with virt-resize present ...

[goldyfruit]

@javanaut-de did you install the guestfs-tools as mentioned per @mhagnumdw ?

[javanaut-de]

@goldyfruit I did. I tried to get something to run with various parameters for some days now but no success so far.

The host is running on Ubuntu 22.04. I tried some basic variations of the examples, ubuntu 20 and 22 server cloudimg and the fedora version given in the example:

  - name: fedora34
    cpus: 2
    memory_size: "2" # in GB
    image_url: file:///var/lib/libvirt/images/images/Fedora-Cloud-Base-34-1.2.x86_64.qcow2
    qemu_user: libvirt-qemu

Stopping a test just after creation of the VM and enabling VNC I find that the booting process stopped with the following message:

Booting from Hard Disk...
Boot failed: not a bootable disk

No bootable device.

virt-df shows no partition present in the .qcow2 file.

While virt-filesystems --all --long showing:

Name      Type        VFS      Label  MBR  Size         Parent
/dev/sda  filesystem  unknown  -      -    16106127360  -
/dev/sda  device      -        -      -    16106127360  -

I managed to deploy some VMs sucessfully with a self-written ansible role before, so I guess I am missing something very basic here...

PS: Interesting. Modifying the create.yml playbook replacing the invokation of virt-resize and handling of the temp image with a similar task using qemu-resize (loosing the image_volume param in this example) as well as increasing the timeout in task "Get instance(s) ip address" seems to work:

    - name: Resize image with the desired size
      ansible.builtin.command:
        "qemu-img resize {{ libvirt_default_image_path }}/{{ item.name }}.qcow2
        {{ item.disk_size | default(default_disk_size) }}"
      args:
        chdir: "{{ libvirt_default_image_path }}"
      environment:
        LIBGUESTFS_BACKEND: direct
      loop: "{{ molecule_yml.platforms }}"
      loop_control:
        label: "{{ item.name }}"
      delegate_to: "{{ item.libvirt_host | default('localhost') }}"

[apatard]

Maybe something wrong with the qcow2 ?
For your problem of "un-bootable" VM, does this happen with every image ? Did you try also resizing manually, only to know if the issue is with the virt-resize invocation or with the image ? I also guess that the maintainers of molecule-libvirt would be happy to get more details on how to reproduce, for instance, with a link to the qcow2 files used.

I don't know the reason of using virt-resize instead of qemu-img but I guess that the advantage of using it is that virt-resize can also resize lvm/filesystems which can't be done with qemu-img afair. Of course, this also mean that I expect virt-resize to be really slower than qemu-img...

[javanaut-de]

With version 0.0.6 of the driver the error showed up always in the same way I think. I tried Ubuntu 20 and 22 cloud img in the first place which come as .qcow2 with file extensions .img. But also the fedora 34 cloud base img that is shown in the example configuration behaving the same. All images come as compressed qcow2 file according to "qemu-img info".

virt-resize resizes contained partitions and filesystems as well, whereas leaves this task to the user. This works for my use case as the cloud img distributions seems to auto-resize the partititions on first boot.

I don't know why virt-resize fails with my setup, but maybe I will find out. Could be something similar to this: https://serverfault.com/questions/976792/how-to-fix-partition-table-after-virt-resize-rearranges-it-kvm

[javanaut-de]

I have started to learn more about qemu/libvirt and to experiment a bit with the driver and some molecule scenarios. Some problems appearently correspond to permissions. For example I found virt-resize trying to read the hosts kernel image which is per default only readable for root. This prevents running VMs/performing tests under non-root user but can be resolved by allowing read acces per ACL. This can also be made persistent creating a kernel postinst script:

/etc/kernel/postinst.d/qemu

#! /bin/bash
setfacl -m u:<VM-User>:r /boot/vmlinuz-$(uname -r)