Mono issues on Linux

[pixeltris]

Unreal ignores all unused signals in FUnixPlatformMisc::SetCrashHandler which messes with the mono signal setup such as this. Ignoring unused signals seems unnecessary as handlers wont get called unless explicitly hooked up? There is a workaround for mono sub-processes but this doesn't help with embedding mono. Perhaps @RCL might know the reasoning for ignoring signals and a possible workaround?

C# threads crash when they exit. There can be various errors but the most common is an assert on mono_bitset_test_fast. The only info I have found on this is an issue in libTAS which mentions something about thread recycling. This threading issue doesn't happen on Mac or Windows.

[RCL]

Ignoring signals that are not handled explicitly was done after we have run into a number of cases when the engine was terminated due to an unexpected signal without triggering the crash handler.

Arguably it is overly zealous and at least realtime signals can be left alone, provided that their default handlers do not terminate the engine. Also, maybe the plugin itself could reset a couple of signals to SIG_DFL to satisfy mono needs?

[pixeltris]

Resetting 3 signals before loading mono seems to work. I'm not too familiar with signals so I'm not sure if it matters which signals I reset. Also there is a fair amount of signal code in mono so there might be other issues which pop up. I'll have to play around with it a bit more once I fix the threading issue.

[RCL]

Judging by the mono code your change should work great. You can reset SA_ONSTACK from the flags if you wish, but it shouldn't really matter (mono will not carry it over).

As for the second issue you mention, I don't have the full context, but the comment in the linked issue seems to explain it pretty well. Mono is apparently using a global thread-local variable to store an id of the thread (its own 16-bit id, not system's) as an optimization (as opposed to storing the same id in a TLS key), and apparently mono can reuse the same (from system's / libc's POV) thread for another logical thread that should have gotten another mono's id but it doesn't. This looks like a bug in mono to me, and probably in places like https://github.com/mono/mono/blob/master/mono/utils/mono-threads.c#L530 there should be an #else clause that would reset tls_small_id back to -1.

[pixeltris]

After some debugging it looks like an issue with my setup. I have been using vmware for testing linux and it looks like there is an issue with __thread causing tls_small_id to be initialized to 0 instead of -1, giving all threads the same small_id. This shouldn't happen on normal systems. The thread reuse issue is unique to libTAS as they hook threading functions and reuse threads.

I'll be sure to use a regular linux setup in future testing.

[pixeltris]

__thread tls_small_id being initialized to 0 rather than -1 also seems to occur on real systems. I tried creating another dynamically linked library which exposes a function that returns a __thread value initialized to -1 and it works as expected. I'm not sure what could possibly be causing the issue for Mono.