Segmentation fault

[domenkozar]

Reported in cachix/cachix#75 (comment)

I have a core dump with following bt:

(gdb) thread apply all bt
                                            
Thread 6 (Thread 0x7fc8d7fff700 (LWP 3033)):                                                                         
#0  0x00007fc92578b7eb in bt_skip_func ()                                                                                 
   from /nix/store/chf54cl12ifswf6swh7kxpif477drihi-xz-5.2.3/lib/liblzma.so.5                                             
#1  0x00007fc92578c558 in lzma_mf_bt4_skip ()                                                                   
   from /nix/store/chf54cl12ifswf6swh7kxpif477drihi-xz-5.2.3/lib/liblzma.so.5
#2  0x00007fc92578ef48 in lzma_lzma_optimum_normal ()
   from /nix/store/chf54cl12ifswf6swh7kxpif477drihi-xz-5.2.3/lib/liblzma.so.5
#3  0x00007fc92578d64d in lzma_lzma_encode ()
   from /nix/store/chf54cl12ifswf6swh7kxpif477drihi-xz-5.2.3/lib/liblzma.so.5
#4  0x00007fc925794d95 in lzma2_encode ()
   from /nix/store/chf54cl12ifswf6swh7kxpif477drihi-xz-5.2.3/lib/liblzma.so.5
#5  0x00007fc92578adf4 in lz_encode ()
   from /nix/store/chf54cl12ifswf6swh7kxpif477drihi-xz-5.2.3/lib/liblzma.so.5                                                
#6  0x00007fc925782e8a in block_encode ()                                                                       
   from /nix/store/chf54cl12ifswf6swh7kxpif477drihi-xz-5.2.3/lib/liblzma.so.5
#7  0x00007fc925784555 in stream_encode ()  
   from /nix/store/chf54cl12ifswf6swh7kxpif477drihi-xz-5.2.3/lib/liblzma.so.5                                       
#8  0x00007fc925780551 in lzma_code ()                                                                                  
   from /nix/store/chf54cl12ifswf6swh7kxpif477drihi-xz-5.2.3/lib/liblzma.so.5                                                   
#9  0x00000000004b8ba2 in ?? ()                                                                                         
#10 0x00000000004b7cf6 in ?? ()                                                                                     
#11 0x0000000000000004 in ?? ()                                                                                  
#12 0x0000000000000000 in ?? ()                                                                                     
                                                                                                                     
Thread 5 (LWP 3223):                                                                                             
#0  0x00007fc924460697 in munmap ()
   from /nix/store/2kcrj1ksd2a14bm5sky182fv2xwfhfap-glibc-2.26-131/lib/libc.so.6
#1  0x00007fc9251e5569 in __free_stacks ()
   from /nix/store/2kcrj1ksd2a14bm5sky182fv2xwfhfap-glibc-2.26-131/lib/libpthread.so.0
#2  0x00007fc9251e56aa in __deallocate_stack ()
   from /nix/store/2kcrj1ksd2a14bm5sky182fv2xwfhfap-glibc-2.26-131/lib/libpthread.so.0
#3  0x00007fc9251e6539 in start_thread ()
   from /nix/store/2kcrj1ksd2a14bm5sky182fv2xwfhfap-glibc-2.26-131/lib/libpthread.so.0
#4  0x00007fc92446557f in clone ()
   from /nix/store/2kcrj1ksd2a14bm5sky182fv2xwfhfap-glibc-2.26-131/lib/libc.so.6

Thread 4 (Thread 0x7fc9157fa700 (LWP 3024)):
#0  0x00007fc9251efb1d in read ()
   from /nix/store/2kcrj1ksd2a14bm5sky182fv2xwfhfap-glibc-2.26-131/lib/libpthread.so
.0
#1  0x0000000000bce904 in ?? ()
#2  0x00007fc9251e62a7 in start_thread ()
---Type <return> to continue, or q <return> to quit---
  wfhfap-glibc-2.26-131/lib/libpthread.so.0
#3  0x00007fc92446557f in clone () from /nix/store/2kcrj1ksd2a14bm5sky182fv2xwfhfap-
glibc-2.26-131/lib/libc.so.6

Thread 3 (Thread 0x7fc925f0a740 (LWP 3016)):
#0  0x00007fc925780315 in lzma_next_end () from /nix/store/chf54cl12ifswf6swh7kxpif4
77drihi-xz-5.2.3/lib/liblzma.so.5
#1  0x00007fc925784453 in stream_encoder_end () from /nix/store/chf54cl12ifswf6swh7k
xpif477drihi-xz-5.2.3/lib/liblzma.so.5
#2  0x00007fc9257801d2 in lzma_next_end.part () from /nix/store/chf54cl12ifswf6swh7k
xpif477drihi-xz-5.2.3/lib/liblzma.so.5
#3  0x00007fc9257806ed in lzma_end () from /nix/store/chf54cl12ifswf6swh7kxpif477dri
hi-xz-5.2.3/lib/liblzma.so.5
#4  0x0000000000bbf6f3 in ?? ()
#5  0x0000000000bbf756 in ?? ()
#6  0x0000000000bba63b in ?? ()
#7  0x0000000000bbaa95 in ?? ()
#8  0x0000000000b4a0f9 in ?? ()
#9  0x0000000000000000 in ?? ()

Thread 2 (LWP 3032):
#0  0x00007fc9251e6366 in start_thread () from /nix/store/2kcrj1ksd2a14bm5sky182fv2x
wfhfap-glibc-2.26-131/lib/libpthread.so.0
#1  0x00007fc92446557f in clone () from /nix/store/2kcrj1ksd2a14bm5sky182fv2xwfhfap-
glibc-2.26-131/lib/libc.so.6
Thread 1 (Thread 0x7fc90d7fa700 (LWP 3031)):
#0  0x00007fc92578b7eb in bt_skip_func () from /nix/store/chf54cl12ifswf6swh7kxpif47
7drihi-xz-5.2.3/lib/liblzma.so.5
#1  0x00007fc92578c558 in lzma_mf_bt4_skip () from /nix/store/chf54cl12ifswf6swh7kxp
if477drihi-xz-5.2.3/lib/liblzma.so.5
#2  0x00007fc92578ef48 in lzma_lzma_optimum_normal () from /nix/store/chf54cl12ifswf
6swh7kxpif477drihi-xz-5.2.3/lib/liblzma.so.5
#3  0x00007fc92578d64d in lzma_lzma_encode () from /nix/store/chf54cl12ifswf6swh7kxp
if477drihi-xz-5.2.3/lib/liblzma.so.5
#4  0x00007fc925794d95 in lzma2_encode () from /nix/store/chf54cl12ifswf6swh7kxpif47
7drihi-xz-5.2.3/lib/liblzma.so.5
#5  0x00007fc92578adf4 in lz_encode () from /nix/store/chf54cl12ifswf6swh7kxpif477dr
ihi-xz-5.2.3/lib/liblzma.so.5
#6  0x00007fc925782e8a in block_encode () from /nix/store/chf54cl12ifswf6swh7kxpif47
7drihi-xz-5.2.3/lib/liblzma.so.5
#7  0x00007fc925784555 in stream_encode () from /nix/store/chf54cl12ifswf6swh7kxpif4
77drihi-xz-5.2.3/lib/liblzma.so.5
#8  0x00007fc925780551 in lzma_code () from /nix/store/chf54cl12ifswf6swh7kxpif477dr
ihi-xz-5.2.3/lib/liblzma.so.5
#9  0x00000000004b8ba2 in ?? ()
#10 0x00000000004b7cf6 in ?? ()
#11 0x0000000000000000 in ?? ()

Possibly relevant to #4?

[hvr]

hrm... so this has only ever been triggered when a unix signal was delivered?

[domenkozar]

Yes and it's not that easy to reproduce (needed like 50 tries) with 4 concurrent lzma compressions going on.

[domenkozar]

Reproduced and resolved in cachix/cachix#75 (comment)

[hvr]

@domenkozar interesting... so do you think there's something that can be done in lzma to workaround the issue for older GHCs?

[domenkozar]

I don't think so, my gut feeling says this was a bug in GHC runtime system for handling SIGINT. But maybe @nh2 can chip in, he has been working on that part.

[nh2]

@domenkozar I'm very suspicious of it.

My RTS SIGINT fixes are mostly about IO, but these functions don't do IO:

https://github.com/haskell-hvr/lzma-clib/blob/1b7cc365bdab2c67ccb4ab328af1701021e44d92/src/liblzma/lz/lz_encoder_mf.c#L450

https://github.com/haskell-hvr/lzma-clib/blob/1b7cc365bdab2c67ccb4ab328af1701021e44d92/src/liblzma/lz/lz_encoder_mf.c#L516

So I would not be surprised if the crash is a use-after-free when accessing pointers; perhaps that free is done from Haskell?

@domenkozar Can you build xz with debugging symbols so we can see exactly which access is crashing?

[domenkozar]

Well, ctrl-c is raised in IO monad so it could be that the gist is in another thread. Did any of RTS SIGINT fixes hit GHC 8.4.x?

[hvr]

Btw, I tried reproducing the repro locally w/ GHC 8.2.2 (but w/o Nix), but failed to reproduce it... didn't segfaulted for me on several iterations. Could you provide me something equivalent of a cabal freeze file of the install-plan as used by Nix, so that I can try to see to try a configuration that's a bit closer to the one on Nix you where observing the segfaults?

[domenkozar]

@hvr sure :)

Dependencies with segfault: https://gist.github.com/domenkozar/20cdff299fbd160e2c9b872ac21fd20d
Dependencies without segfault: https://gist.github.com/domenkozar/6c88c670aab547781c7998526375ef7e

[domenkozar]

Yes, I can confirm that taking not-segfaulting nixpkgs commit and using GHC 8.2.2 still means no segfault, so that only leaves one of the library bumps fixed it or that it's a false positive.

[domenkozar]

Difference in packages seems to be:

$ diff nonsegfaulting segfaulting 
2,3c2,3
< async-2.2.1
< async-2.2.1-doc
---
> async-2.1.1.1
> async-2.1.1.1-doc
16,17c16,17
< exceptions-0.10.0
< exceptions-0.10.0-doc
---
> exceptions-0.8.3
> exceptions-0.8.3-doc
49,50c49,50
< primitive-0.6.3.0
< primitive-0.6.3.0-doc
---
> primitive-0.6.4.0
> primitive-0.6.4.0-doc
59,60c59,60
< transformers-compat-0.6.2
< transformers-compat-0.6.2-doc
---
> transformers-compat-0.5.1.4
> transformers-compat-0.5.1.4-doc

[domenkozar]

In my local testing shows it's async bump.