Add support for Hyperlight KVM guest debugging using gdb #111

dblnz · 2024-12-13T17:54:02Z

The current implementation supports only 4 hardware breakpoints.
There might be some bugs, I am still testing
There are some modifications I plan on doing but shouldn't have big impact on this solution

devigned

Super excited to see this landing. Nice work! My feedback mostly consists of nits and questions.

docs/how-to-debug-a-hyperlight-guest.md

src/hyperlight_host/src/hypervisor/gdb/mod.rs

devigned · 2025-01-21T22:54:06Z

src/hyperlight_host/src/hypervisor/gdb/mod.rs

+    <T as Target>::Error:
+        std::fmt::Debug + Send + From<io::Error> + From<DebugMessage> + From<TryRecvError>,
+{
+    // TODO: Address multiple sandboxes scenario


Rather than have a code comment, please create a new issue in the repo to track the feature.

I'll do that after I gather all the feedback as this might be addressed before merging

devigned · 2025-01-21T23:04:39Z

src/hyperlight_host/src/hypervisor/gdb/target.rs

+
+    /// Translates the guest address to physical address
+    fn translate_gva(&self, gva: u64) -> Result<u64, GdbTargetError> {
+        // TODO: Properly handle errors


What would you like to do based on this TODO?

A related nit, it would be nice to have some preservation of the data in the underlying error, so as the error bubbles up, we can determine the root cause.

This was a left over comment from the development phase, I added some error handling in the meantime.
I would like to maybe get an idea as to how to redo the error handling to be easier to track where errors originated

devigned · 2025-01-22T12:25:30Z

src/hyperlight_host/src/hypervisor/hypervisor_handler.rs

@@ -88,7 +88,8 @@ impl HypervisorHandler {
 #[derive(Clone)]
 struct HvHandlerExecVars {
    join_handle: Arc<Mutex<Option<JoinHandle<Result<()>>>>>,
-    shm: Arc<Mutex<Option<SandboxMemoryManager<GuestSharedMemory>>>>,
+    #[allow(clippy::type_complexity)] // TODO: Change this type
+    shm: Arc<Mutex<Option<Arc<Mutex<SandboxMemoryManager<GuestSharedMemory>>>>>>,


Do you intend on changing this prior to PR merging?

I would like to change it.
Cloning SandboxMemoryManager directly might be a better idea, but the GuestSharedMemory type does not implement Clone, I need to check what implementing that would mean.

Does KVMDriver need a clone of SandboxMemoryManager or does a mutable borrow maybe suffice? GuestSharedMemory was purposefully not Clone because it was intended to reflect unique ownership at any time of the guest side access to the shared memory.

devigned · 2025-01-22T12:27:38Z

src/hyperlight_host/src/hypervisor/kvm.rs

+        let mut target = HyperlightKvmSandboxTarget::new(mgr, vcpu_fd, entrypoint, hyp_conn);
+        let _ = target
+            .set_entrypoint_bp()
+            .map_err(|_| new_error!("Cannot set entrypoint breakpoint"))?;


Is there useful information in the underlying error?

It is a KVM generated error, I need to think about how to propagate the errors

- it adds a function to spawn the GDB thread - adds an empty implementation for a gdb target that is to be used with the gdbstub crate - adds `gdb` feature that can be enabled at compile time Signed-off-by: Doru Blânzeanu <[email protected]>

Signed-off-by: Doru Blânzeanu <[email protected]>

- the `execution_variables` cover now the scenario the locks were meant for Signed-off-by: Doru Blânzeanu <[email protected]>

- VcpuFd is needed for register read/write and setting debug settings - MemoryManager is needed for read/write memory access Signed-off-by: Doru Blânzeanu <[email protected]>

- this avoids the guest being terminated for timeout Signed-off-by: Doru Blânzeanu <[email protected]>

- this is needed to be able to be notified when the vcpu is stopped and when to resume Signed-off-by: Doru Blânzeanu <[email protected]>

- the hypervisor signals the gdb thread with a message that the vcpu stopped and it waits for a signal to resume Signed-off-by: Doru Blânzeanu <[email protected]>

Signed-off-by: Doru Blânzeanu <[email protected]>

- add `KvmDebug` struct that abstracts the details of kvm guest debug and offers an simple API for setting breakpoints - add breakpoint at entrypoint Signed-off-by: Doru Blânzeanu <[email protected]>

Signed-off-by: Doru Blânzeanu <[email protected]>

- gdb debugger now stops at entrypoint and is able to read addresses, read registers and add or remove breakpoints Signed-off-by: Doru Blânzeanu <[email protected]>

Signed-off-by: Doru Blânzeanu <[email protected]>

- this will make it simpler for other hypervisors support to be added Signed-off-by: Doru Blânzeanu <[email protected]>

Signed-off-by: Doru Blânzeanu <[email protected]>

- add CI check using `gdb` feature Signed-off-by: Doru Blânzeanu <[email protected]>

Signed-off-by: Doru Blânzeanu <[email protected]>

danbugs · 2025-01-27T16:51:23Z

src/hyperlight_host/src/hypervisor/hypervisor_handler.rs

@@ -88,7 +88,8 @@ impl HypervisorHandler {
 #[derive(Clone)]
 struct HvHandlerExecVars {
    join_handle: Arc<Mutex<Option<JoinHandle<Result<()>>>>>,
-    shm: Arc<Mutex<Option<SandboxMemoryManager<GuestSharedMemory>>>>,
+    #[allow(clippy::type_complexity)] // TODO: Change this type
+    shm: Arc<Mutex<Option<Arc<Mutex<SandboxMemoryManager<GuestSharedMemory>>>>>>,


Does KVMDriver need a clone of SandboxMemoryManager or does a mutable borrow maybe suffice? GuestSharedMemory was purposefully not Clone because it was intended to reflect unique ownership at any time of the guest side access to the shared memory.

danbugs · 2025-01-27T17:06:11Z

src/hyperlight_host/src/hypervisor/hypervisor_handler.rs

+        #[cfg(gdb)]
+        match self.communication_channels.from_handler_rx.recv() {
+            Ok(msg) => match msg {
+                HandlerMsg::Error(e) => Err(e),
+                HandlerMsg::FinishedHypervisorHandlerAction => Ok(()),
+            },
+            Err(_) => Err(HyperlightError::HypervisorHandlerMessageReceiveTimedout()),
+        }
+
+        #[cfg(not(gdb))]


Disabling the timeout might be helpful for scenarios other than debugging too. Might want to put this under a different feature flag or even as a runtime option. Though, that's not too related to this PR, so maybe leaving it like this is OK for now 👍

dblnz added the enhancement New feature or request label Dec 13, 2024

dblnz self-assigned this Dec 13, 2024

dblnz marked this pull request as draft December 13, 2024 17:54

dblnz force-pushed the gdb-support-latest branch 2 times, most recently from be21b85 to ca61def Compare December 19, 2024 11:41

devigned requested changes Jan 22, 2025

View reviewed changes

dblnz added 22 commits January 22, 2025 17:16

add basic gdb target for a HLSandbox

34f6813

- it adds a function to spawn the GDB thread - adds an empty implementation for a gdb target that is to be used with the gdbstub crate - adds `gdb` feature that can be enabled at compile time Signed-off-by: Doru Blânzeanu <[email protected]>

create gdb thread when KVM driver is created

c11143e

Signed-off-by: Doru Blânzeanu <[email protected]>

add gdb event loop execution

edadd8f

Signed-off-by: Doru Blânzeanu <[email protected]>

remove unnecessary locks

25b424d

- the `execution_variables` cover now the scenario the locks were meant for Signed-off-by: Doru Blânzeanu <[email protected]>

pass the necessary handles to the target

5fa8355

- VcpuFd is needed for register read/write and setting debug settings - MemoryManager is needed for read/write memory access Signed-off-by: Doru Blânzeanu <[email protected]>

disable guest timeouts when gdb features is enabled

63dfb87

- this avoids the guest being terminated for timeout Signed-off-by: Doru Blânzeanu <[email protected]>

add communication channel between gdb and hypervisor

9fb7b2f

- this is needed to be able to be notified when the vcpu is stopped and when to resume Signed-off-by: Doru Blânzeanu <[email protected]>

vcpu state synchronization between hypervisor and gdb client

bc222a5

- the hypervisor signals the gdb thread with a message that the vcpu stopped and it waits for a signal to resume Signed-off-by: Doru Blânzeanu <[email protected]>

add target internal state that keeps track whether the vcpu is paused

38de187

Signed-off-by: Doru Blânzeanu <[email protected]>

add entrypoint breakpoint support

f589255

- add `KvmDebug` struct that abstracts the details of kvm guest debug and offers an simple API for setting breakpoints - add breakpoint at entrypoint Signed-off-by: Doru Blânzeanu <[email protected]>

add stop reason transmission to gdb

2a144ca

Signed-off-by: Doru Blânzeanu <[email protected]>

fixup kvm vcpu run

e92f1cd

Signed-off-by: Doru Blânzeanu <[email protected]>

add read and write registers support

bbfcad1

Signed-off-by: Doru Blânzeanu <[email protected]>

add read/write address and text section offset support

a45d5e0

Signed-off-by: Doru Blânzeanu <[email protected]>

add hw breakpoint support

b917859

- gdb debugger now stops at entrypoint and is able to read addresses, read registers and add or remove breakpoints Signed-off-by: Doru Blânzeanu <[email protected]>

add gdb resume capabilities

f8a0e12

Signed-off-by: Doru Blânzeanu <[email protected]>

add gdb step capabilities

95615ed

Signed-off-by: Doru Blânzeanu <[email protected]>

rename target file to kvm specific name

11d18c7

Signed-off-by: Doru Blânzeanu <[email protected]>

change gdb feature to support generics for future improvements

1322701

- this will make it simpler for other hypervisors support to be added Signed-off-by: Doru Blânzeanu <[email protected]>

fix issues for windows and mshv

35c4f60

Signed-off-by: Doru Blânzeanu <[email protected]>

add documentation about how to debug a guest binary

791aaf8

- add CI check using `gdb` feature Signed-off-by: Doru Blânzeanu <[email protected]>

add sw breakpoint support

16492de

Signed-off-by: Doru Blânzeanu <[email protected]>

dblnz force-pushed the gdb-support-latest branch from 4a44b2e to 16492de Compare January 22, 2025 15:40

fix documentation and remove unnecessary comments

2a36f06

Signed-off-by: Doru Blânzeanu <[email protected]>

dblnz marked this pull request as ready for review January 22, 2025 16:07

dblnz changed the title ~~Initial support for gdb debugging in Hyperlight KVM guest~~ Add support for Hyperlight KVM guest debugging using gdb Jan 22, 2025

danbugs reviewed Jan 28, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for Hyperlight KVM guest debugging using gdb #111

Add support for Hyperlight KVM guest debugging using gdb #111

dblnz commented Dec 13, 2024

devigned left a comment

devigned Jan 21, 2025

dblnz Jan 22, 2025

devigned Jan 21, 2025

dblnz Jan 22, 2025 •

edited

Loading

devigned Jan 22, 2025

dblnz Jan 22, 2025

danbugs Jan 27, 2025

devigned Jan 22, 2025

dblnz Jan 22, 2025

danbugs Jan 27, 2025

danbugs Jan 27, 2025

Add support for Hyperlight KVM guest debugging using gdb #111

Are you sure you want to change the base?

Add support for Hyperlight KVM guest debugging using gdb #111

Conversation

dblnz commented Dec 13, 2024

devigned left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dblnz Jan 22, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dblnz Jan 22, 2025 •

edited

Loading