Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

lima-vm: vz error with the pkgx version, not with the brew one #7853

Open
tannevaled opened this issue Nov 7, 2024 · 14 comments
Open

lima-vm: vz error with the pkgx version, not with the brew one #7853

tannevaled opened this issue Nov 7, 2024 · 14 comments
Labels
bug Something isn't working help wanted Extra attention is needed

Comments

@tannevaled
Copy link
Contributor

Nested virtualization under M3+ masOS 15+

Template file : nested.yaml

minimumLimaVersion: "1.0.0"
images:
# Try to use release-yyyyMMdd image if available. Note that release-yyyyMMdd will be removed after several months.
- location: "https://cloud-images.ubuntu.com/releases/24.04/release-20241004/ubuntu-24.04-server-cloudimg-amd64.img"
  arch: "x86_64"
  digest: "sha256:fad101d50b06b26590cf30542349f9e9d3041ad7929e3bc3531c81ec27f2c788"
- location: "https://cloud-images.ubuntu.com/releases/24.04/release-20241004/ubuntu-24.04-server-cloudimg-arm64.img"
  arch: "aarch64"
  digest: "sha256:e380b683b0c497d2a87af8a5dbe94c42eb54548fa976167f307ed8cf3944ec57"
# Fallback to the latest release image.
# Hint: run `limactl prune` to invalidate the cache
- location: "https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-amd64.img"
  arch: "x86_64"
- location: "https://cloud-images.ubuntu.com/releases/24.04/release/ubuntu-24.04-server-cloudimg-arm64.img"
  arch: "aarch64"

mounts:
- location: "~"
- location: "/tmp/lima"
  writable: true

vmType: vz
nestedVirtualization: true

VM creation

phymath@mba-10838921 Documents % pkgx limactl create --name nested --tty=false ./nested.yaml
INFO[0000] Terminal is not available, proceeding without opening an editor 
WARN[0000] vmType vz: ignoring [User]                   
INFO[0000] Attempting to download the image              arch=aarch64 digest="sha256:e380b683b0c497d2a87af8a5dbe94c42eb54548fa976167f307ed8cf3944ec57" location="https://cloud-images.ubuntu.com/releases/24.04/release-20241004/ubuntu-24.04-server-cloudimg-arm64.img"
INFO[0000] Using cache "/Users/phymath/Library/Caches/lima/download/by-url-sha256/1f3456e7d7c2bc0a8f8993d1308a3a4c124d703f56bf4cf6dfca50eb5f11f1c3/data" 
INFO[0000] Converting "/Users/phymath/.lima/nested/basedisk" (qcow2) to a raw disk "/Users/phymath/.lima/nested/diffdisk" 
3.50 GiB / 3.50 GiB [---------------------------------------] 100.00% 1.62 GiB/s
INFO[0002] Expanding to 100GiB                          
INFO[0002] Attempting to download the nerdctl archive    arch=aarch64 digest="sha256:fe085381a09aa240ae5d1e0bbef1beccfb7c1d6dbb98bdc55bd416581d46ebc8" location="https://github.com/containerd/nerdctl/releases/download/v2.0.0/nerdctl-full-2.0.0-linux-arm64.tar.gz"
INFO[0002] Using cache "/Users/phymath/Library/Caches/lima/download/by-url-sha256/1699e54a52757df863155fca76f8a77b50f05d993edca23421798af6635156f0/data" 
INFO[0002] Run `limactl start nested` to start the instance.

VM start (pkgx version)

phymath@mba-10838921 Documents % pkgx limactl start nested                             
INFO[0000] Using the existing instance "nested"         
INFO[0000] Starting the instance "nested" with VM driver "vz" 
WARN[0000] vmType vz: ignoring [User]                   
INFO[0000] [hostagent] hostagent socket created at /Users/phymath/.lima/nested/ha.sock 
INFO[0000] [hostagent] Starting VZ (hint: to watch the boot progress, see "/Users/phymath/.lima/nested/serial*.log") 
FATA[0001] exiting, status={Running:false Degraded:false Exiting:true Errors:[] SSHLocalPort:0} (hint: see "/Users/phymath/.lima/nested/ha.stderr.log") 
phymath@mba-10838921 Documents % grep features /Users/phymath/.lima/nested/ha.stderr.log
{"level":"debug","msg":"Failed to detect CPU features. Assuming that AES acceleration is available on this Apple silicon.","time":"2024-11-07T15:33:54+01:00"}

VM start (brew version)

phymath@mba-10838921 Documents % limactl start nested    
INFO[0000] Using the existing instance "nested"         
INFO[0000] Starting the instance "nested" with VM driver "vz" 
WARN[0000] vmType vz: ignoring [User]                   
INFO[0000] [hostagent] hostagent socket created at /Users/phymath/.lima/nested/ha.sock 
INFO[0000] [hostagent] Starting VZ (hint: to watch the boot progress, see "/Users/phymath/.lima/nested/serial*.log") 
INFO[0001] SSH Local Port: 57631                        
INFO[0001] [hostagent] [VZ] - vm state change: running  
INFO[0001] [hostagent] Waiting for the essential requirement 1 of 2: "ssh" 
INFO[0011] [hostagent] Waiting for the essential requirement 1 of 2: "ssh" 
INFO[0011] [hostagent] The essential requirement 1 of 2 is satisfied 
INFO[0011] [hostagent] Waiting for the essential requirement 2 of 2: "user session is ready for ssh" 
INFO[0011] [hostagent] The essential requirement 2 of 2 is satisfied 
INFO[0011] [hostagent] Waiting for the optional requirement 1 of 2: "systemd must be available" 
INFO[0011] [hostagent] Guest agent is running           
INFO[0011] [hostagent] Not forwarding UDP 127.0.0.54:53 
INFO[0011] [hostagent] Not forwarding UDP 127.0.0.53:53 
INFO[0011] [hostagent] Not forwarding UDP 192.168.5.15:68 
INFO[0011] [hostagent] Not forwarding TCP 127.0.0.53:53 
INFO[0011] [hostagent] Not forwarding TCP 127.0.0.54:53 
INFO[0011] [hostagent] Not forwarding TCP [::]:22       
INFO[0011] [hostagent] The optional requirement 1 of 2 is satisfied 
INFO[0011] [hostagent] Waiting for the optional requirement 2 of 2: "containerd binaries to be installed" 
INFO[0023] [hostagent] The optional requirement 2 of 2 is satisfied 
INFO[0023] [hostagent] Waiting for the guest agent to be running 
INFO[0023] [hostagent] Waiting for the final requirement 1 of 1: "boot scripts must have finished" 
INFO[0026] [hostagent] Forwarding TCP from 127.0.0.1:44307 to 127.0.0.1:44307 
INFO[0035] [hostagent] The final requirement 1 of 1 is satisfied 
INFO[0035] READY. Run `limactl shell nested` to open the shell. 
phymath@mba-10838921 Documents % limactl shell nested
phymath@lima-nested:/Users/phymath/Documents$ kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used
phymath@mba-10838921 Documents % pkgx limactl --version
limactl version 1.0.0
phymath@mba-10838921 Documents % limactl --version
limactl version 1.0.0
phymath@mba-10838921 Documents % which limactl
/opt/homebrew/bin/limactl
@jhheider
Copy link
Contributor

jhheider commented Nov 7, 2024

Does ha.stderr.log show anything interesting? It looks like it's trying to start ssh and failing. Maybe we need openssh.org as a companion? I don't see any particular major differences with https://formulae.brew.sh/formula/lima, unless we're not getting the templates installed, maybe?

@jhheider
Copy link
Contributor

jhheider commented Nov 7, 2024

I get {"level":"fatal","msg":"nested virtualization is not supported on this device","time":"2024-11-07T14:53:01-05:00"}, so that suggests we need a build flag or library. Might even be a qemu issue.

@jhheider
Copy link
Contributor

jhheider commented Nov 7, 2024 •
edited
Loading

actually:

{"level":"debug","msg":"ResolveVMType: resolved VMType \"vz\" (explicitly specified in []*LimaYAML{o,y,d}[1])","time":"2024-11-07T14:56:59-05:00"}
{"level":"debug","msg":"Creating iso file /Users/jacob/.lima/nested/cidata.iso","time":"2024-11-07T14:56:59-05:00"}
{"level":"debug","msg":"Using /var/folders/8x/k382fgcs59vfffl1dgq015gh0000gn/T/diskfs_iso115028808 as workspace","time":"2024-11-07T14:56:59-05:00"}
{"level":"debug","msg":"Failed to detect CPU features. Assuming that AES acceleration is available on this Apple silicon.","time":"2024-11-07T14:57:00-05:00"}
{"level":"debug","msg":"OpenSSH version 9.8.1 detected","time":"2024-11-07T14:57:00-05:00"}
{"level":"debug","msg":"AES accelerator seems available, prioritizing [email protected] and [email protected]","time":"2024-11-07T14:57:00-05:00"}
{"level":"info","msg":"hostagent socket created at /Users/jacob/.lima/nested/ha.sock","time":"2024-11-07T14:57:00-05:00"}
{"level":"info","msg":"Starting VZ (hint: to watch the boot progress, see \"/Users/jacob/.lima/nested/serial*.log\")","time":"2024-11-07T14:57:00-05:00"}
{"level":"debug","msg":"Start udp DNS listening on: 127.0.0.1:60329","time":"2024-11-07T14:57:00-05:00"}
{"level":"debug","msg":"Using search domains: [jacobsdomain.arpa]","time":"2024-11-07T14:57:00-05:00"}
{"level":"debug","msg":"Start tcp DNS listening on: 127.0.0.1:64960","time":"2024-11-07T14:57:00-05:00"}
{"level":"debug","msg":"Kernel file \"/Users/jacob/.lima/nested/kernel\" not found","time":"2024-11-07T14:57:00-05:00"}
{"level":"debug","msg":"Using EFI Boot Loader","time":"2024-11-07T14:57:00-05:00"}
{"level":"fatal","msg":"nested virtualization is not supported on this device","time":"2024-11-07T14:57:00-05:00"}

it finds AES, but doesn't find the kernel. that seems like it's potentially a problem.

@jhheider
Copy link
Contributor

jhheider commented Nov 7, 2024

i note that nestedVirtualization is off in pkgx limactl info. Can you diff the info output of the two? it might be that we need some build flags.

@jhheider
Copy link
Contributor

jhheider commented Nov 7, 2024

our qemu is missing --enable-fdt=system. checking to see if that might be contributory.

@jhheider
Copy link
Contributor

jhheider commented Nov 7, 2024

interestingly, it works with pkgx limactl~0.23.2 start nested, though it complains about the nestedVirtualization key. so, it seems like it's either something to do with our build for v1, or something that v0 isn't checking in qemu (but i don't see what it might be). more exploration definitely needed.

@jhheider jhheider added bug Something isn't working help wanted Extra attention is needed labels Nov 7, 2024
@tannevaled
Copy link
Contributor Author

on osx it does not use qemu but vz.
i will have acces to the M3 hardware next week to test what you asked.

@jhheider
Copy link
Contributor

jhheider commented Nov 8, 2024

cool. yeah, always good to have users involved in testing. i reviewed both their release process and the homebrew build and didn't see differences of note, though there's clearly something.

@tannevaled
Copy link
Contributor Author

i note that nestedVirtualization is off in pkgx limactl info. Can you diff the info output of the two? it might be that we need some build flags.

nestedvirtualization is off even when using the brew limactl binary. when diffing the info output only the templates location differ.

@tannevaled
Copy link
Contributor Author

on what kind of hardware is it built? M3- or M3+ ?

@tannevaled
Copy link
Contributor Author

cf https://github.com/lima-vm/lima/blob/master/.github/workflows/release.yml and lima-vm/lima#2767 the compiling environment is important to enable the new features

@jhheider
Copy link
Contributor

M1. We should be able to control that with build flags, hopefully.

@jhheider
Copy link
Contributor

So, this appears to be the relevant code: https://github.com/lima-vm/lima/blob/5ac7de0bf9e45e403f1af08d1f2f998bb8d04d58/Makefile#L24-L31

Both build machines are using the 14.5 SDK, so it shouldn't be disabling vz.

@jhheider
Copy link
Contributor

I find no difference in behavior building with sdk 14.5 or 15.1 locally (though, I also have the brew binary fail to nest on my M2). the entitlements look correct too:

$ codesign -d --entitlements - ~/.pkgx/lima-vm.io/v1.0.1/bin/limactl
Executable=/Users/jacob/.pkgx/lima-vm.io/v1.0.1/bin/limactl
[Dict]
	[Key] com.apple.security.network.client
	[Value]
		[Bool] true
	[Key] com.apple.security.network.server
	[Value]
		[Bool] true
	[Key] com.apple.security.virtualization
	[Value]
		[Bool] true

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working help wanted Extra attention is needed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tannevaled @jhheider