Avoid using thread_local to work-around mingw bug

[peterbell10]

Ref #1, #2, msys2/MINGW-packages#10164

The mingw issue suggests that thread-local storage is buggy in mingw, so avoiding thread_local might help. I don't have a mingw setup to test though.

cc @carlkl, @rgommers

[rgommers]

Cc @matthew-brett also

[mreineck]

Thanks a lot for this, @peterbell10 ! I don't have access to a MinGW installation either, so let's see what @carlkl says.

[mreineck]

If I read the changes correctly, the evolution of this multi-threading code in https://github.com/mreineck/ducc/blob/ducc0/src/ducc0/infra/threading.cc should be unaffected by this problem; do you agree @peterbell10 ?

Pity we can't switch to this implementation yet, at least for scipy.

[peterbell10]

Yes, because you use pass scheduler explicitly instead of via thread-local storage it should be equivalent.

[carlkl]

@peterbell10, I will test this as soon as I can with msys2 scipy.

[matthew-brett]

As a data point, I tried naively copying pocketfft_hdronly.h here to the Scipy source tree, and compiling with

• -DPOCKETFFT_PTHREADS and pthreads linking - https://github.com/matthew-brett/scipy/runs/4447908937
• Without - https://github.com/matthew-brett/scipy/runs/4440857238

but in both cases, Scipy.fft still hangs.

[matthew-brett]

Can I ask - what is the best way of replicating the hang without compiling Scipy? I have not observed a hang on my machine when using mingw-w64 to compile the demo from this repo : #1 (comment) - even running the demo hundreds of times.

[carlkl]

I now tested this PR with scipy. Sorry for the long time ...
With MT scipy still hangs unfortunately. I extracted a stacktrace during the hang: pocketffthang.log.txt (reformatted and commented). However, I don't really get any more insight from it though. It seems to hang on shutdown of the thread pool, but why?
Maybe the experts?

[mati865]

This stacktrace looks like issue LLD had with mingw-w64 GCC: https://reviews.llvm.org/D102944
Replacing exit with _Exit might help.

To clarify:

The mingw issue suggests that thread-local storage is buggy in mingw

Emutls that GCC uses is buggy on Windows but it's totally unrelated to mingw-w64, TLS in Clang+mingw-w64 works just fine.

[carlkl]

I don't think that this hang is related to DLL unloading but I may be wrong. In my opinion it has to do with threads shutdown. And there is no exit in pocketfft code.
EDIT: I'm using the latest and greatest UCRT64 gcc-11.2 (msys2)

[carlkl]

I have to correct myself: The hang happens at process termination. @mati865, is this related to https://sourceforge.net/p/mingw-w64/bugs/859/?

[carlkl]

as reference: msys2/MINGW-packages#10164 (comment). Seems to be the __cxa_thread_atexit hell... Still investigating.

[mati865]

Sorry foe the late reply, this notification got lost in my backlog.

I have to correct myself: The hang happens at process termination. @mati865, is this related to https://sourceforge.net/p/mingw-w64/bugs/859/?

I think this is similar issue that might require another fix.