virtualization_test fails at second step -- irecovery -f iBSS.vma2.RELEASE.patched.img4

[steven-michaud]

The first step (virtualization_test -r AVPBooter.vmapple2.patched.bin) works fine, and afterwards I'm able to get plausible results from irecovery -q. But the second step (recovery -f iBSS.vma2.RELEASE.patched.img4) seems to hang something inside the VM (irecovery -q now fails after a timeout), and I need to CTRL-C to stop virtualization_test.

I get the same results on macOS 12.6.7, 13.4.1 and 14 Beta 2. My guess is that there's something wrong with the irecovery that I'm using, or maybe with the libusb that it's using. My copy of it comes from Homebrew (brew install libirecovery), and ultimately from libimobiledevice. My libusb also comes from Homebrew. Both seem quite up to date.

So which irecovery did you use in your tests? And if possible please let me know the version of libusb that it used.

I'm going to be spending a lot of time debugging this problem. But I'd appreciate any information you can give me (if possible in addition to what I've already asked for). Also let me know if you think I'm on the right trail.

Ultimately I'm hoping to use Apple's Virtualization framework's forceDFU setting to hack their VMs enough to work around this design flaw. I've patched UTM to support forceDFU on macOS 12 and up, and it seems to work very much like your virtualizaton_test. irecovery -q returns plausible results. And it even gets into the same "hang" whenever I try to use irecovery -f on it.

[steven-michaud]

Something I forgot to mention above: I had to patch your virtualization_test to get it working on macOS 13 and up. I'll do a pull request once I've had a chance to polish my patch up a bit.

[NyanSatan]

Hello!

I used iRecovery built from the latest (back then) source code. Also I built it against IOKit, and not libusb

It was a long time ago since the last time I played with this, but if I remember correctly, AVPBooter fails to boot the next stage when one uses APTicket supplied within an IPSW instead of the one provided via TSS

Speaking of the macOS 13+ issues - yea, as far as I know they made some previously private API non-private, so it will fail to compile

[steven-michaud]

Thanks for the quick response! I'll try building libirecovery (and its irecovery component) against the IOKit.

For my tests on macOS 12 I used an apticket.der that I copied from a macOS 12.0.1 virtual machine that I'd just set up. Aside from AVPBooter.vmapple2.bin, I got all my test files either from that VM's IPSW file or from the VM itself. I extracted their payloads and patched them using your original patches. Then I put the parts back together and tacked this apticket.der onto the end. I used img4tool to perform these transformations. My AVPBooter.vmapple2.bin I got from the macOS 12.6.7 installation I've been working on. I patched it in a disassembler following your instructions.

My tests on other versions of macOS used appropriately different copies of all these test files. I went to some trouble to find all these files, including for the ramdisk. If I can get everything working, I'll write a detailed gist and post a link to it here.

[steven-michaud]

I've been using the Ghidra disassembler, with an extension for it that adds support for iBoot files.

[steven-michaud]

This problem is very deep -- possibly deeper than I'll be able to penetrate.

It turns out the Homebrew irecovery already links against IOKit. But building it to link against libusb makes no difference -- I get exactly the same failures.

I got the clever idea of using Apple Configurator to try to "recover" the VM run by virtualization_test. That failed, though the same operation worked with my macOS 12.0.1 VM running in UTM, booted into DFU mode. I'd hoped virtualization_test's logging would shed light on things. But it only got to the end of iBoot Stage2 before the failure happened.

I'll keep playing with this, but I don't know how far I'll get. I'm going to try to rewrite virtualization_test to make it behave more like UTM, hoping that this will allow Apple Configurator to finish "recovering" it.

For whoever's interested, you can also get relevant log messages by running the following command in Terminal before you start doing things in DFU mode. /Library/Apple/System/Library/PrivateFrameworks/MobileDevice.framework (on macOS 12) is heavily involved in this business. Among other things, it sends the lpolrestore command (and many others) to the machines (running in DFU mode) that Apple Configurator is trying to restore.

log stream --info --debug --predicate='sender == "MobileDevice"'

I don't know how you were able to get virtualization_test working. Possibly macOS 12.6.7's security's been tightened enough to break what you did in 12.0.1. Or maybe it's just down to some simple, accidental difference between the steps you took and those I took.

[NyanSatan]

The next stage after iBootStage2 is kernel, that doesn’t really log anything unless booted with special boot-args

The possible cause of failure to restore is that you probably didn’t provide a main storage file to it, thus there was nowhere to restore root filesystem and stuff

lpolrestore is needed indeed for iBootStage1. If you don’t provide one, it will fail to boot the Stage 2

Also if you’re restoring, then you should try to do it via idevicerestore utility. On failure it can yield an entire restored log from device

And I’m not sure I understand what your final goal is. If you can share this, maybe I can help you more efficiently

[steven-michaud]

I've now progressed considerably beyond my last comment, but I still haven't discovered anything useful to me.

I found all the files I need in one place -- a PersonalizedBundleRestore bundle that exists temporarily (on my system in /var/folders/yd/g3g0mtsn43v2gykm6q4846100000gn/T/com.apple.configurator.xpc.DeviceService/TemporaryItems/) while the Apple Configurator is restoring a UTM VM booted into DFU mode (using my patch mentioned above). All these files are for macOS 12.0.1, so I extracted their payloads, applied your patches to them, and rebuilt the img4 files according to your instructions. Everything worked as you reported. But it sort of stopped part way through.

Once I got your original steps working, I tried them with my macOS 12.0.1 VM booted into DFU mode. Once again things stopped part way through. But after 30 seconds or so the VM booted normally, with no effect from what had happened in DFU mode. There wasn't even any trace of it in the VM's system logs (what you see running log show) -- which were entirely "normal", and didn't look like the logs from your virtualization_test. This VM definitely has its own main storage. So not having passed any to your virtualization_test isn't the only explanation for it having stalled out. (Edit: Though the lack of main storage does explain why Apple Configurator wasn't able to "restore" virtualization_test while it was running in DFU mode.)

I told you above what my end goal is -- to find a workaround for Apple's refusal to support third party kernel extensions in its macOS guest VMs (those that use the Virtualization framework). Yes, that's pretty vague. And I'm not sure it's even possible. But your work with DFU mode is the best lead I have. The other approaches I tried (messing with files in the VM or in the IPSW directly) didn't work out.

I'm slowly working through the changes made by your patches (beside the one you documented). I don't yet fully understand them, but I've made some interesting discoveries. For example I found out that your kernelcache patch changes the AppleImage4 and AppleMobileFileIntegrity kernel extensions. Your work is very interesting, and I'm sure I'll learn a lot from it. I may even do a writeup on it, filling in the details you left out.

I'll be even happier if my work on your stuff helps me accomplish my original goal. But that's a long shot.

[steven-michaud]

One thing I forgot to mention above:

Before testing with my macOS 12.0.1 VM in DFU mode, I followed instructions in this document to allow me to copy a patched AVPBooter.vmapple2.bin over the original copy in /System/Library/Frameworks/Virtualization.framework/Resources.

And another:

I dug through Apple's XNU kernel source code to decipher your special boot args. This showed me that they're needed to enable "serial" output past iBoot Stage2, and to tell the kernel about the ramdisk device (/dev/md0). So I started using them.

Edit: Actually the rd boot arg tells the kernel what the "root disk" is -- in this case the ramdisk.

[steven-michaud]

Your virtualization_test does work as expected, so I'm going to close this bug. But I'll keep adding comments if I find information that's especially interesting.

[steven-michaud]

You might like to know that I've finally figured out my original problem -- how to load third-party kernel extensions on Virtualization framework macOS VMs.

I said nice things about your Virtual-iBoot-Fun project, but I found a different way to patch image4_validate_property_callback(). I also needed another iBoot patch to get things working, but that's it -- just two iBoot patches.

[dariaphoebe]

This is fantastic. Thank you for all your work

[steven-michaud]

You're most welcome. I'm glad you found it useful.