Debugger not exiting on exception when set to not debug

[adam-fowler]

I am currently implementing a vscode.TestController extension for Swift and using vscode.debug.startDebugging to trigger running the test launch configuration. You can set up debug and non-debug profiles for a TestController. In the situation where I am running a non-debug profile I am setting noDebug in the DebugSessionOptions when calling startDebugging

vscode.debug.startDebugging(this.folderContext.workspaceFolder, testBuildConfig, {
    noDebug: !shouldDebug,
})

This is all fine until one of the tests crashes. In this situation CodeLLDB stops and waits for input from the user which is slightly confusing as I told the debugger I didn't want to debug the test executable.

Is there anyway to get CodeLLDB to exit when it hits an exception in this case? Or am I going to have to run the test executable outside of the debugger?