Port virtualbox scripts to VBoxManage CLI #625

stevemk14ebr · 2024-10-09T13:49:13Z

Ports to VBoxManage CLI, identical logic otherwise. Errors handled gracefully for the most part. Output:

stepheneckels@flarevm-build-2:~/source/repos/flare-vm$ python3 virtualbox/vbox-export-snapshots.py 
Starting operations on FLARE-VM
VM {b76d628b-737f-40a3-9a16-c5f66ad2cfcc} is already shut down (state: poweroff).
Restored 'FLARE-VM'
Found existing hostonlyif vboxnet0
Verified hostonly nic configuration correct
Power cycling before export...
VM {b76d628b-737f-40a3-9a16-c5f66ad2cfcc} is not running (state: poweroff). Starting VM...
VM {b76d628b-737f-40a3-9a16-c5f66ad2cfcc} started.
VM {b76d628b-737f-40a3-9a16-c5f66ad2cfcc} is not powered off. Shutting down VM...
VM {b76d628b-737f-40a3-9a16-c5f66ad2cfcc} is shut down (status: poweroff).
Power cycling done.
Exporting /usr/local/google/home/stepheneckels/EXPORTED VMS/FLARE-VM.20241009.dynamic.ova (this will take some time, go for an 🍦!)
Exported /usr/local/google/home/stepheneckels/EXPORTED VMS/FLARE-VM.20241009.dynamic.ova! 🎉
All operations on FLARE-VM successful ✅
Starting operations on FLARE-VM.full
VM {b76d628b-737f-40a3-9a16-c5f66ad2cfcc} is already shut down (state: poweroff).
Restored 'FLARE-VM.full'
Found existing hostonlyif vboxnet0
Changed nic1 to hostonly
Verified hostonly nic configuration correct
Power cycling before export...
VM {b76d628b-737f-40a3-9a16-c5f66ad2cfcc} is not running (state: poweroff). Starting VM...
VM {b76d628b-737f-40a3-9a16-c5f66ad2cfcc} started.
VM {b76d628b-737f-40a3-9a16-c5f66ad2cfcc} is not powered off. Shutting down VM...
VM {b76d628b-737f-40a3-9a16-c5f66ad2cfcc} is shut down (status: poweroff).
Power cycling done.
Exporting /usr/local/google/home/stepheneckels/EXPORTED VMS/FLARE-VM.20241009.full.dynamic.ova (this will take some time, go for an 🍦!)
Exported /usr/local/google/home/stepheneckels/EXPORTED VMS/FLARE-VM.20241009.full.dynamic.ova! 🎉
All operations on FLARE-VM.full successful ✅
Starting operations on FLARE-VM.EDU
VM {b76d628b-737f-40a3-9a16-c5f66ad2cfcc} is already shut down (state: poweroff).
Restored 'FLARE-VM.EDU'
Found existing hostonlyif vboxnet0
Changed nic1 to hostonly
Verified hostonly nic configuration correct
Power cycling before export...
VM {b76d628b-737f-40a3-9a16-c5f66ad2cfcc} is not running (state: poweroff). Starting VM...
VM {b76d628b-737f-40a3-9a16-c5f66ad2cfcc} started.
VM {b76d628b-737f-40a3-9a16-c5f66ad2cfcc} is not powered off. Shutting down VM...
VM {b76d628b-737f-40a3-9a16-c5f66ad2cfcc} is shut down (status: poweroff).
Power cycling done.
Exporting /usr/local/google/home/stepheneckels/EXPORTED VMS/FLARE-VM.20241009.EDU.ova (this will take some time, go for an 🍦!)
Exported /usr/local/google/home/stepheneckels/EXPORTED VMS/FLARE-VM.20241009.EDU.ova! 🎉
All operations on FLARE-VM.EDU successful ✅
Done. Exiting...

…e power cycles

Ana06

Thanks for the work @stevemk14ebr! I need to still test the code locally, but I have added some questions and improvement suggestions already. It is good to see what we can do with VBoxManage and how it allows us to remove the virtualbox dependency. The disadvantage is that it is less flexible, as it does not export everything in the API (for example, it seems it is not possible to access the max number of adapters which would allow us to write simpler code as in the previous version) and that we need to create a subprocess everytime we want to run a command. The new code using VBoxManage also looks longer and more complicated, but we may be able to simplify it a bit.

What about keeping both the version using the virtualbox library and the new one using VBoxManage until we have tested and migrated everything else?

Also, I think we need some documentation in /virtualbox/README.md.

Ana06 · 2024-10-10T12:08:48Z

virtualbox/vbox-export-snapshots.py

+    except subprocess.CalledProcessError as e:
+        # exit code is an error
+        print(f"Error running VBoxManage command: {e} ({e.stderr})")
+    raise Exception(f"Error running VBoxManage command")


why is it needed to catch the exception to print and error and re-reise it? I see the same pattern in other functions as well.

Style choice, this throws a pretty error to the top level main to print out. I can change if you think there's a more pythonic style

I have seen in other Python code that an exception is re-triggered to add extra details or format the exception differently, but without a print that duplicates similar information. The print apart from duplicating the information, can make the output difficult to digest in this case, as {e.stderr} is rendering the output that is likely to be the long help message from VBoxManage. I think we should remove the try-catch, as the re-triggered exception is almost the same:

Original exception: Command '['VBoxManage', 'list2', 'list', 'hostonlyifs']' returned non-zero exit status 2

Retriggered exception: Error running VBoxManage command: Command '['VBoxManage', 'list2', 'list', 'hostonlyifs']' returned non-zero exit status 2.

Suggested change

except subprocess.CalledProcessError as e:

# exit code is an error

print(f"Error running VBoxManage command: {e} ({e.stderr})")

raise Exception(f"Error running VBoxManage command")

Also, if we re-raise the exception, I think we should use a more concrete typ of exception like RuntimeError.

virtualbox/vbox-export-snapshots.py

Ana06 · 2024-10-10T12:43:21Z

virtualbox/vbox-export-snapshots.py

-            session.unlock_machine()
-            print(f"Restored '{snapshot_name}' and changed its adapter(s) to host-only")
-
+            vm_uuid = get_vm_uuid(VM_NAME)


do we need to get the UUID? It seems like the commands work with the VM_NAME (we may need to enclose the entire name in double quotes to avoid issues with spaces), or am I missing something?

We could rely on the VM_NAME alone, but I use the UUID so that we can support multiple VMs of the same name and be sure we refer to the same VM consistently for all operations

virtualbox/vbox-export-snapshots.py

stevemk14ebr · 2024-10-10T15:35:42Z

for example, it seems it is not possible to access the max number of adapters which would allow us to write simpler code as in the previous version

we can, the vminfo command lists all 8 adapters (the max) and any unset adapters have the value 'none'. The code doesn't need to check the max adapters because it lists all of them, even if they're unset, so we always loop all 8 adapters.

What about keeping both the version using the virtualbox library and the new one using VBoxManage until we have tested and migrated everything else

I have no issues with not merging these PRs (I will send more for the other two scripts) until we are ready to drop the virtualbox package dependency entirely. I would not want to keep two version around though, that goes against the spirit of doing this work. While the code does appear more complex, the port was actually quite straightforward, there is just a lot of logic to parse the text CLI output and handle the errors nicely. Some things are different than the virtualbox package for sure, but there are not any glaring things missing from the CLI. In the long term this should be very easy to maintain as the CLI does not often change. More importantly though on some setup the python .so that virtualbox uses is not build/included, and the package is unmaintained for +1 year at this time, so we should not rely on it anymore.

Ana06

I have tested vbox-export-snapshots.py and I it fails because the network interface does not have a name. I though this happened in the previous version when exporting the VM, but it seems like setting it could be the issue and because you start the VM (what I was not doing in the previous version) it fails even before exporting it:

Starting operations on FLARE-VM
VM {40138663-f254-412b-8776-10a7cc08daea} is already shut down (state: poweroff).
Restored 'FLARE-VM'
VM {40138663-f254-412b-8776-10a7cc08daea} is already shut down (state: poweroff).
Found existing hostonlyif vboxnet0
Changed nic1
Nic configuration verified correct
Power cycling before export...
VM {40138663-f254-412b-8776-10a7cc08daea} is not running (state: poweroff). Starting VM...
Error running VBoxManage command: Command '['VBoxManage', 'startvm', '{40138663-f254-412b-8776-10a7cc08daea}', '--type', 'gui']' returned non-zero exit status 1. (VBoxManage: error: Nonexistent host networking interface, name '' (VERR_INTERNAL_ERROR)
VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component ConsoleWrap, interface IConsole
)
Error checking VM state: Error running VBoxManage command
Unexpectedly failed doing operations on FLARE-VM. Exiting...
Done. Exiting...

I reported what I think was a bug in https://www.virtualbox.org/ticket/22158. But what really confuses me is that it seems it does work for you. 😕

virtualbox/vbox-adapter-check.py

virtualbox/vbox-export-snapshots.py

Ana06 · 2024-11-06T15:22:09Z

virtualbox/vbox-export-snapshots.py

+    except subprocess.CalledProcessError as e:
+        # exit code is an error
+        print(f"Error running VBoxManage command: {e} ({e.stderr})")
+    raise Exception(f"Error running VBoxManage command")


I have seen in other Python code that an exception is re-triggered to add extra details or format the exception differently, but without a print that duplicates similar information. The print apart from duplicating the information, can make the output difficult to digest in this case, as {e.stderr} is rendering the output that is likely to be the long help message from VBoxManage. I think we should remove the try-catch, as the re-triggered exception is almost the same:

Original exception: Command '['VBoxManage', 'list2', 'list', 'hostonlyifs']' returned non-zero exit status 2

Retriggered exception: Error running VBoxManage command: Command '['VBoxManage', 'list2', 'list', 'hostonlyifs']' returned non-zero exit status 2.

Suggested change

except subprocess.CalledProcessError as e:

# exit code is an error

print(f"Error running VBoxManage command: {e} ({e.stderr})")

raise Exception(f"Error running VBoxManage command")

Ana06 · 2024-11-06T15:36:21Z

virtualbox/vbox-export-snapshots.py

+    except subprocess.CalledProcessError as e:
+        # exit code is an error
+        print(f"Error running VBoxManage command: {e} ({e.stderr})")
+    raise Exception(f"Error running VBoxManage command")


Also, if we re-raise the exception, I think we should use a more concrete typ of exception like RuntimeError.

virtualbox/vbox-export-snapshots.py

…amic vms

…all adapters

Co-authored-by: Ana María Martínez Gómez <[email protected]>

stevemk14ebr added 4 commits October 8, 2024 20:57

Migrate vbox export snapshots to VBoxManage CLI

ca44732

update logs

0fbcac2

Create hostonlyif if it doesnt exist. Verify nic configuration. Reduc…

919d0af

…e power cycles

Better logging

35b3c30

Ana06 reviewed Oct 10, 2024

View reviewed changes

stevemk14ebr added 2 commits October 11, 2024 15:18

Apply review changes

64fd32f

Port vbox-adapter-check

52a4e47

stevemk14ebr changed the title ~~Port vbox-export-snapshots to VBoxManage CLI~~ Port virtualbox scripts to VBoxManage CLI Oct 11, 2024

stevemk14ebr added 2 commits October 14, 2024 17:43

Use controlvm if VM is running to change nic

8107c51

Port vbox-clean-snapshots

9ae7cf8

Ana06 reviewed Nov 6, 2024

View reviewed changes

stevemk14ebr and others added 8 commits November 25, 2024 21:09

Address saved VMState throwing error on poweroff. Don't scan just dyn…

daf4793

…amic vms

adjust export comment

c97048c

Dont use nic in user message, only show message once rather than for …

5ec96f4

…all adapters

Remove found existing hostonlyif output

f2e70cc

Update virtualbox/vbox-adapter-check.py

cb4d64d

Co-authored-by: Ana María Martínez Gómez <[email protected]>

Fix print typo

52a0a56

Create common library

8f57726

Remove superflous comment

a82cead

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Port virtualbox scripts to VBoxManage CLI #625

Port virtualbox scripts to VBoxManage CLI #625

stevemk14ebr commented Oct 9, 2024 •

edited

Loading

Ana06 left a comment •

edited

Loading

Ana06 Oct 10, 2024

stevemk14ebr Oct 10, 2024

Ana06 Nov 6, 2024

Ana06 Nov 6, 2024

Ana06 Oct 10, 2024

stevemk14ebr Oct 10, 2024

stevemk14ebr commented Oct 10, 2024 •

edited

Loading

Ana06 left a comment

Ana06 Nov 6, 2024

Ana06 Nov 6, 2024

Port virtualbox scripts to VBoxManage CLI #625

Are you sure you want to change the base?

Port virtualbox scripts to VBoxManage CLI #625

Conversation

stevemk14ebr commented Oct 9, 2024 • edited Loading

Ana06 left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stevemk14ebr commented Oct 10, 2024 • edited Loading

Ana06 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stevemk14ebr commented Oct 9, 2024 •

edited

Loading

Ana06 left a comment •

edited

Loading

stevemk14ebr commented Oct 10, 2024 •

edited

Loading