Apple M1 / Hypervisor.framework support

[flying-sausages]

What are you trying to do?
Run Multipass to launch instances on top of Apple Silicon devices

What's your proposed solution?
Restrict multipass on AS to use the Virtualisation API intrdouced in Big Sur only and ensure ARM builds of ubuntu are being installed

Additional context
I have 0 experience in any of the underlying technologies that are used here. I know 1.5.0 has Big Sur support on Intel through the API above, and so I'm wondering what are the extra steps that need to be taken care in order to allow users to have ARM-based VMs.

ED: Latest build

1.8.0 build 787

[Saviq]

Hi @flying-sausages, this is definitely something we want to get going, but it will take some time. The API you mention is quite low level, the likes of hyperkit and qemu are what we'd need to implement support for it first. Did you see evidence that the hardware even supports virtualization already? From what we've read the dev kits did not, and I've not seen anyone confirming they have a VM running under Apple Silicon.

[flying-sausages]

I've seen a success story here which I think could be helpful https://forums.macrumors.com/threads/ubuntu-linux-virtualized-on-m1-success.2270365/

[flying-sausages]

Another useful link from the same thread https://github.com/JacopoMangiavacchi/M1-Linux-SSH

[flying-sausages]

And another https://github.com/evansm7/vftool

[RyanMorash]

The M1 does have virtualization support. There have been a set of patches made to enable Hypervisor.framework (hvf) support to QEMU for aarch64: https://patchwork.kernel.org/project/qemu-devel/list/?series=391797

Also, there are now two different virtualization frameworks on macOS. There continues to be the lower-level Hypervisor.framework which is currently utilized by hyperkit via machyve/xhyve. Virtualization.framework is a new higher-level set of APIs which allows you to create VMs using the Virtio spec without having to deal with a separate hypervisor (such as xhyve)

[Saviq]

Yeah thanks @RyanMorash @flying-sausages for references, we're monitoring this space and will jump on it whenever things settle. One notorious source of problems for us is networking (NAT, bridging, DHCP, DNS), which hyperkit (through Hypervisor.framework) mostly solves and we'd love to continue using that. If, however, qemu gains support for all the bits'n'pieces we need, we may well go that route.

[flying-sausages]

https://gist.github.com/niw/e4313b9c14e968764a52375da41b4278#file-readme-md looks like there's QEMU support through some patches? Lively discussion happening underneath that.

[RyanMorash]

Docker is moving away from using HyperKit (which they originally created for Docker Desktop) and switching to Virtualization.framework: https://www.docker.com/blog/download-and-try-the-tech-preview-of-docker-desktop-for-m1/

[ChangheeOh]

I have tried to create VM with vmcli(https://github.com/gyf304/vmcli) and almost succeeded to run Ubuntu arm64 image.
vmcli is also based on MacOS Virtualization.framework and it also have several issues and not yet stable in my opinion.
I wish it could help multipass team to enhance it to support M1 Mac.

[nottrobin]

It seems docker have made good progress: https://docs.docker.com/docker-for-mac/apple-m1/

[Saviq]

@nottrobin yeah we know everything we need now, just need to roadmap it - hoping for early next cycle.

[andy-isoc]

Is there a good place (other than this issue!) to watch for updates on this and when it might be released?

[bassu]

There is also ACVM (https://github.com/KhaosT/ACVM) GUI qemu launcher (based on patches from Alexander Graf to support Hypervisor.framework syscalls on M1 macOS).

But I suppose if multipass uses Hyperkit then it might take some time!

[sudo8com]

The complete guide for MacOS Apps Optimized for M1 Apple Silicon Macs
https://isapplesiliconready.com/

[flying-sausages]

This open-source implementation works quite well, wonder if it can be of any inspiration. https://github.com/utmapp/UTM

[misterproton]

@Saviq can you update us on this issue? No new Intel Macs will be designed, so us Mac Multipass users are kind of in the cold here.

[Saviq]

Hey @misterproton, all, we've scheduled the work on M1 support for the development cycle that just started. We'll be starting the work on it within a couple weeks. Stay tuned!

[misterproton]

@Saviq do you have any update or ETA for this one?

[Saviq]

Hi @misterproton we're actively working on it now. We should have a first build by July.

[rugwirobaker]

👀

[flying-sausages]

Happy to hear! Is there a chance this will show up in the 1.7.0 RCs?

[townsend2010]

Hey @flying-sausages!

No, it definitely won't be in the 1.7.0 release, but the plan is to make some beta packages available once I get this in a relatively good working state, most likely some time in July. I'm actively working on it 😃

[rugwirobaker]

How is going so far, are we still on track(no pressure😉)?

[lethargosapatheia]

I tried vftool, but I'm probably doing something wrong:

$ ./vftool -k ../../../vm-test/ubuntu-20.04-server-cloudimg-arm64-vmlinuz-generic -i ../../../vm-test/ubuntu-20.04-server-cloudimg-arm64-initrd-generic ../../../vm-test/ubuntu-20.04-server-cloudimg-arm64.img -p 2 -m 4096 -a "console=hvc0 root=/dev/vda1"
2021-09-28 22:58:15.770 vftool[42158:751629] vftool (v0.3 10/12/2020) starting
2021-09-28 22:58:15.771 vftool[42158:751629] +++ kernel at ../../../vm-test/ubuntu-20.04-server-cloudimg-arm64-vmlinuz-generic, initrd at ../../../vm-test/ubuntu-20.04-server-cloudimg-arm64-initrd-generic, cmdline 'console=hvc0', 1 cpus, 512MB memory
2021-09-28 22:58:15.771 vftool[42158:751629] +++ fd 3 connected to /dev/ttys001
2021-09-28 22:58:15.771 vftool[42158:751629] +++ Waiting for connection to:  /dev/ttys001
2021-09-28 23:08:54.649 vftool[42158:751629] +++ Configuration validated.
2021-09-28 23:08:54.650 vftool[42158:751629] +++ canStart = 1, vm state 0
2021-09-28 23:08:54.696 vftool[42158:759748] --- VM start error: Error Domain=VZErrorDomain Code=1 "The virtual machine failed to start." UserInfo={NSLocalizedFailure=Internal Virtualization error., NSLocalizedFailureReason=The virtual machine failed to start.}

The last lines appear only after I execute screen to connect to the tty.

[townsend2010]

Hey all,

We've dropped the first Release Candidate for Apple M1 support. It can be found at:
https://multipass-ci.s3.amazonaws.com/multipass-1.8.0-rc.849%2Bgb6625c10.mac-Darwin.pkg

This is a Universal package that supports both Arm and Intel Macs.

The Intel side of things will now support a QEMU driver, but is not enabled by default. In order to enable the QEMU driver, you need to do:
$ sudo multipass set local.driver=qemu

Please note that if you currently have Hyperkit-based instances, those will not be available when using the QEMU driver, but they will be preserved and can be used again when setting the driver to hyperkit. We have plans to have a conversion utility to allow using the hyperkit instances with the qemu driver, but we aren't there yet 😄

Lastly, there have a been a few reports during the beta testing that launching instances will suddenly start timing out where before it would work. There seems to be something going on with the Apple virtualized networking that we haven't been able to solve yet. We are tracking that in #2265 and if you run in to that, please report it there.

Thank you, good luck, and let us know how it goes!

[franz2256]

Hello,

I tested it and worked fine on a mac mini. Did you know if the bridge option to access the vm from an outside computer is implemented now ? thank you for your work

[townsend2010]

Hi @franz2256,

Thanks for trying it out and reporting your success.

Did you know if the bridge option to access the vm from an outside computer is implemented now ?

Not yet and won't be in the initial release, but we plan on making that available in later release.

[gluxon]

Hey @townsend2010, thanks a lot for the RC! I tested it out an an ARM iMac and everything worked well.

[bboykin87]

I was also able to get an instance running and open a shell successfully using the RC build that was posted here. Thanks!

M1 MBP

[PrernaWeb]

Can confirm I have multiple instances up and running using the RC build on an M1 MacBook Air. Nice work guys.

[louielin]

I can't mount folder at this version.

multipass   1.8.0-rc.849+gb6625c10.mac
multipassd  1.8.0-rc.849+gb6625c10.mac

mount failed: source "{Path}" is not readable

When I give the access right for multipassd by Settings -> Security & Privacy -> Privacy -> Full Disk Access and run multipass mount again.

It would still show the error message and auto cancel the access right in Settings -> Security & Privacy -> Privacy -> Full Disk Access

[tgerov]

I tested the latest RC and it works well on the MacBook Pro M1, GJ.

[m-bers]

Since this runs on QEMU and not hyperkit, any chance we could get image launching support for the Apple Silicon release?
#1260

[Boes-man]

Hi @franz2256,

Thanks for trying it out and reporting your success.

Did you know if the bridge option to access the vm from an outside computer is implemented now ?

Not yet and won't be in the initial release, but we plan on making that available in later release.

Hello, can you please clarify. With this rc there is no network bridge and it is also not planned for the ga release? Its one of the key reasons I use multipass. Perhaps its only not availalble for these M1 architectures, but will continue for the current architectures (intel). Thanks.

[Saviq]

Hello, can you please clarify. With this rc there is no network bridge and it is also not planned for the ga release? Its one of the key reasons I use multipass. Perhaps its only not availalble for these M1 architectures, but will continue for the current architectures (intel). Thanks.

It's not there on Intel macOS today (unless you're using VirtualBox), either - but nothing changes there. The statement above was about Apple Silicon only.

We plan to close the gap in the coming months.

[Saviq]

This is released in v1.8.0 now!

[Boes-man]

Hello, can you please clarify. With this rc there is no network bridge and it is also not planned for the ga release? Its one of the key reasons I use multipass. Perhaps its only not availalble for these M1 architectures, but will continue for the current architectures (intel). Thanks.

It's not there on Intel macOS today (unless you're using VirtualBox), either - but nothing changes there. The statement above was about Apple Silicon only.

We plan to close the gap in the coming months.

Thanks @Saviq , fyi I use intel macOS + multipass only (no virtualbox). Multipass creates a network bridge to the VM and I can ssh/ping/etc straight to the VM's IP from my laptop. Super useful and a major reason I support multipass (and microk8s). Losing this feature would be a bummer :)

Congratulations on the 1.8.0 release!

[kenji21]

Take care that could use only a single IP (it is in multipass documentation), the origin is from vmnet

"The interface receives a private IPv4 address via DHCP. IPv4 traffic originating from the guest operating system must use the private IPv4 address. Packets sent from a different IPv4 address are dropped by the system."

I have asked on apple dev forums if there is a way to use multiple IPs : https://developer.apple.com/forums/thread/693146

[Saviq]

Thanks @Saviq , fyi I use intel macOS + multipass only (no virtualbox). Multipass creates a network bridge to the VM and I can ssh/ping/etc straight to the VM's IP from my laptop. Super useful and a major reason I support multipass (and microk8s). Losing this feature would be a bummer :)

Ah yes! That bridged interface :)

We will certainly keep that if at all possible. What we meant by "bridging" was additional networks that you can configure with --network … - but not (yet) on macOS :)

Congratulations on the 1.8.0 release!

Thanks! :)

[maberer]

Thanks @Saviq for your hard work!

So can we suggest a change to https://isapplesiliconready.com/app/Multipass ?

Is multipass a Universal app now?

[Saviq]

Is multipass a Universal app now?

It is, yeah!

[maberer]

Great. Submitted a change to https://isapplesiliconready.com/app/Multipass

[mlb5000]

@Saviq is there a place to track --networks support for non VirtualBox setups? As it stands, I believe multi-NIC support is impossible on Apple Silicon, blocking compatibility with my particular network setup for my K8S cluster.

Edit: actually, I believe this is a larger issue for me as a software engineer on the M1 platform as a whole. Without networks support, I don't think I could expose any resources launched by multipass on my local network, which would be a big limitation.

[Saviq]

@mlb5000 in progress, you should be able to try the package from here:

https://github.com/canonical/multipass/pull/2462/checks?check_run_id=5375689050

[mlb5000]

@Saviq oh dang, thanks! I will try to do that this week and report back.

[mlb5000]

@Saviq can confirm the Networks option is there now. Is there documentation somewhere for how to add a network/launch option to an existing instance? I'd like to either add --network bridged or a specific network interface to my existing microk8s-vm instance since I'm not sure I can have the MicroK8S installer pass the network options into multipass during the install process.

[eminchen]

I having error after installing and launching multipass:

multipass start
[2022-03-24T14:21:43.129] [error] [primary] process error occurred Crashed program: qemu-system-aarch64; error: Process crashed
[2022-03-24T14:21:43.132] [error] [primary] error: program: qemu-system-aarch64; error: Process crashed
start failed: cannot connect to the multipass socket
Please ensure multipassd is running and '/var/run/multipass_socket' is accessible

Any tips how to troubleshoot it?