BUG: miss breakpoint for multi-thread program

[WuBingzheng]

Steps:

1. memleax set breakpoints at the entry of malloc and free at beginning;
2. one thread runs in free;
3. memleax catches it, set a breakpoint at the exit of free, and deletes the breakpoint at the entry of free temporarily to let the thread continue;
4. if another thread runs in free now, memleax will miss it because the breakpoint is deleted!!! Then this memory block record will live for ever in memleax, and will be considered as a memory leak.

Similarly, if this happens at malloc, we will miss a memory block allocation.

[WuBingzheng]

Maybe I could study how GDB works later.

[alvislin]

Hello Wu,

I would like to share you my quick & dirty fix for multi-thread related issue.
I found that all of the routines started with "lui gp, xxxx".
We can just skip executing of the breakpoint object code by emulating the result of these "lui gp,xxxx" functions. Thus, we can adjust PC & GP registers via ptrace syscall, instead of putting the original object code back.

00028850 :
28850: 3c1c0008 lui gp,0x8
28854: 279c1a70 addiu gp,gp,6768
28858: 0399e021 addu gp,gp,t9
2885c: 27bdff98 addiu sp,sp,-104

Since these breakpoints will never be removed, there is no chance of any race-condition to happen.

On the other hand, I changed the main loop to stay for several seconds to handling waitpid() returned events after remove all breakpoints. It seems to remedy the multi-thread program detach problem.

[WuBingzheng]

Sounds like a good idea. Have you tried this?

lui is a MIPS instruction. I will look into whether x86-64 is similar.
For a quick look, routines begin with pushq %rbp by -O0, and subq $8, %rsp by -O2.

[alvislin]

Wu, thanks for your great tools. I ported the tool to MIPS32 little-endian OpenWRT based platform.
After my modifications, it works fine to detect mpdaemon and Amazon AVS device SDK memory leakages.

Meanwhile, the insertion of the 2nd breakpoint could/shall also be avoided.
Here is my solution,

1. modified the return address register (to exactly the same address of the 1st breakpoint) when 1st breakpoint hit.
2. the system will hit the same breakpoint again when malloc() or free() is done
3. this time, modify the return address register again (to the real caller address) and move the PC to that address

I believed that insert breakpoint in the return address is somewhat dangerous.
The inserted breakpoint instruction could be installed in a delayed branch slot, which may mess-up the execution of the attached program.
I did experience some stranger SIGSEGV fault on the attached program before the modification above.

[WuBingzheng]

Great! I will try this later too.

I am not familiar with debugger tech or assembly instructions. I just write memleax with basic knowledge from school, so there should be a lot room for improvement.
Thanks.

[WuBingzheng]

BTW, I write another tool to detect memory leak libleak.

memleax uses ptrace and assembly instructions, which make it complex, CPU arch dependent, and impact on performance (a TRAP on each malloc/free invoke).

libleak uses LD_PRELOAD. I think it's better than memleax. I even want to stop maintaining memleax.

You may try it.