Is there a way to hook malloc from within the process without risking a race condition?

[simonvanbernem]

I have an application where I want to hook malloc to do more specialized error checking of allocations than the c-runtime hooks allow for. Detours seems to fit perfectly, and I got it working nicely, but I think there is a race condition looming:

I currently only call DetourUpdateThread on the current thread (which doesn't even seem to be necessary?). But there are other threads running that I haven't created and don't control, which might be calling malloc right when I commit the detours transaction, which could lead to bad things happening, if I understand this correctly.

One solution for discovering all threads in the process (to then call DetourUpdateThread on each) I have seen uses CreateToolhelp32Snapshot, and then iterates through all threads and checks if they belong to the current process. This is "probably fine" I guess, but between iterating the list of threads and committing the transaction, new threads could have spawned in my process, right? And there is no way to tell windows "don't let any new threads be created until I'm done here" right? If those threads somehow made it into malloc, we are back to square one.

That leaves me with the question: Is there a way to hook malloc using detours that 100% guarantees that nothing bad will happen, not relying on hopefully no other (possibly newly created) thread being in malloc right when the transaction is commited?

[sylveon]

I guess, but between iterating the list of threads and committing the transaction, new threads could have spawned in my process, right? And there is no way to tell windows "don't let any new threads be created until I'm done here" right?

If you're creating the hook in DllMain, this isn't an issue because the loader lock prevents any thread from starting up until your DllMain function completes. This is the safest place to create a hook in.

[simonvanbernem]

I am not building a dll, just a normal executable. Is there a place to put the hooking that gives the same guarantees in that case?

[sylveon]

I do not believe there is a similar thing for executables. You can however implement that part of your application in a DLL to get access to a DllMain function, and keep the rest of the code in the executable.