Mixed mode (C#/C++) callstack on unhandled exceptions

[pixeltris]

If an unhandled exception occurs the crash reporter is invoked which displays the callstack info. This callstack doesn't include C# frames (and trashes the topmost C++ frames) which can make that callstack info basically useless.

To get the full callstack we need the FGenericPlatformCrashContext / FGenericPlatformStackWalk classes (and their platform specific equivalents) to have callbacks so we can implement a custom stack walker. There currently aren't any suitable callbacks available which would mean we would need several function hooks per platform if we wanted to implement this without modifying the engine source.

Even if there were callbacks available, the implementation wouldn't be trivial as we have to support .NET Framework, .NET Core and Mono on at least Windows, Mac and Linux. Each platform has a different way of getting the callstack and on top of that each platform may have several different ways of getting the callstack depending on requirements (e.g. RtlCaptureStackBackTrace has issues on functions outside of known modules which doesn't work well with JITed functions). There are also issues with calling convention differences between X86 and X64 which again increases the likelihood that the APIs we need to use would be different between architectures.

.NET Framework / Windows solution

On .NET Framework (and possibly .NET Core) under Windows there is an API for getting mixed mode callstack info using CLRDataCreateInstance / ICLRDataTarget and associated functions. This might be one easy-ish way of getting the full callstack.

https://github.com/dotnet/coreclr/blob/master/src/debug/createdump/crashinfo.cpp
https://www.codeproject.com/Articles/371137/A-Mixed-Mode-Stackwalk-with-the-IDebugClient-Inter
http://blog.steveniemitz.com/building-a-mixed-mode-stack-walker-part-1/

There is also some useful information here for getting raw callstack info (this would still need to be paired with something to get the C# function names). https://stackoverflow.com/questions/34501602/fast-capture-stack-trace-on-windows-64-bit-mixed-mode

Generic solution

One possible way of dealing with it could be a stack walker similar to the one in Unreal Engine (which might have to be modified / completely manual if there are platforms which don't have APIs capable of walking functions outside of well known modules (JITed functions)).

Once we have the raw callstack addresses we can use the regular symbol lookup functions in conjunction with C# function lookup via reflection (non generic functions only). This reflection based lookup would use MethodInfo.MethodHandle.GetFunctionPointer() to create some crude managed function table lookup without having to look inside the .NET Framework, .NET Core or Mono internals. This might need to be combined with a simpler dissembler such as hde32 / hde64 to determine function length.

Temporary solution

Calls such as FMessage.Log(ELogVerbosity.Fatal, "error"); crash UE4, trash the callstack and cannot be handled by AppDomain.CurrentDomain.UnhandledException to print out the C# callstack. try/catch blocks in C# seem to work (when not debugging) so try/catch blocks should be where possible until we have better exception handling. Log the C# callstack somewhere and then somehow invoke the crash reporter (this could be done by setting GIsCriticalError to false and then causing another fatal error without a C# try/catch block).

NOTE: As far as I'm aware there isn't any way to provide useful information to the crash reporter. UE_LOG fatal seems to put the error message all on one line which isn't good for outputting the C# callstack. There is NewReportEnsure but after the first try/catch that function doesn't seem to work. Maybe just use the C# FMessage.OpenDialog() method.