Question: potential deadlock with goroutine stuck on internal lock? #15
Comments
|
Hi. |
|
I found one bug in the library, hopefully fixed it in 1595213 |
|
I just implemented a deadlock detector with easy to prove correctness: |
|
I'm facing the same problem while debugging Mercure: I'm using the same version of the library. Thanks for writing such an useful tool btw! |
|
v0.2.1 (1595213) This blocking occurs in the caching layer, which is heavily loaded, about 100,000 requests per second. |
|
But in this case, there is a very small timeout, while in my real code there is 5 minutes. package main
import (
"time"
"github.com/sasha-s/go-deadlock"
)
func main() {
deadlock.Opts.DeadlockTimeout = 1 * time.Second
var mx deadlock.Mutex
fn := func() {
mx.Lock()
mx.Unlock()
}
for i := 0; i < 100_000; i++ {
go func() {
for {
fn()
}
}()
}
select {}
}Perhaps the problem is not in this library, but in some other place. But in my real case, mutex protected counting the number in a slice and adding to a slice. // copy code from https://github.com/jaegertracing/jaeger/blob/748bf213d1e5a03ebc4da63601473be3bbe193f4/examples/hotrod/pkg/tracing/mutex.go with my fixes
type Mutex struct {
...
waiters []string
waitersLock deadlock.Mutex // <--
}
func (sm *Mutex) Lock(ctx context.Context, name string) {
activeSpan := opentracing.SpanFromContext(ctx) // "activeSpan == noopSpan{}" in production
sm.waitersLock.Lock() // <-- "Previous place where the lock was grabbed"
if waiting := len(sm.waiters); waiting > 0 && activeSpan != nil {
activeSpan.LogFields(...) // <-- func (n noopSpan) LogFields(fields ...log.Field) {}
}
sm.waiters = append(sm.waiters, name)
sm.waitersLock.Unlock() // <-- "Have been trying to lock it again for more than 5m0s"
sm.realLock.Lock()
sm.holder = name
sm.waitersLock.Lock()
behindLen := len(sm.waiters) - 1
sm.waitersLock.Unlock()
if activeSpan != nil {
activeSpan.LogFields(...)
}
} |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hey, first of all thanks for the hard work on this great lib!
I'm having trouble interpreting the output below. It suggests
goroutine 77264holdslock 0xc4202a60e0for a long time, preventing others (likegoroutine 77325and many more) from acquiring it.However, the output suggests that
goroutine 77264actually got stuck during unlock:raft.go:688is a deferredmu.Unlock(), anddeadlock.go:330is actually is a lock acquire statement in this lib.Does this mean that the (potential) deadlock is coming from this lib in this case? What would make
goroutine 77264get stuck on that internal lock? (I reproduced the same output with1m30slock timeout.)POTENTIAL DEADLOCK: Previous place where the lock was grabbed goroutine 77264 lock 0xc4202a60e0 ../raft/raft.go:618 raft.(*Raft).AppendEntriesRPCHandler { rf.mu.Lock() } <<<<< /usr/local/Cellar/go/1.9/libexec/src/runtime/asm_amd64.s:509 runtime.call32 { CALLFN(·call32, 32) } /usr/local/Cellar/go/1.9/libexec/src/reflect/value.go:434 reflect.Value.call { call(frametype, fn, args, uint32(frametype.size), uint32(retOffset)) } /usr/local/Cellar/go/1.9/libexec/src/reflect/value.go:302 reflect.Value.Call { return v.call("Call", in) } ../labrpc/labrpc.go:478 labrpc.(*Service).dispatch { function.Call([]reflect.Value{svc.rcvr, args.Elem(), replyv}) } ../labrpc/labrpc.go:402 labrpc.(*Server).dispatch { return service.dispatch(methodName, req) } ../labrpc/labrpc.go:229 labrpc.(*Network).ProcessReq.func1 { r := server.dispatch(req) } Have been trying to lock it again for more than 30s goroutine 77325 lock 0xc4202a60e0 ../raft/raft.go:618 raft.(*Raft).AppendEntriesRPCHandler { rf.mu.Lock() } <<<<< /usr/local/Cellar/go/1.9/libexec/src/runtime/asm_amd64.s:509 runtime.call32 { CALLFN(·call32, 32) } /usr/local/Cellar/go/1.9/libexec/src/reflect/value.go:434 reflect.Value.call { call(frametype, fn, args, uint32(frametype.size), uint32(retOffset)) } /usr/local/Cellar/go/1.9/libexec/src/reflect/value.go:302 reflect.Value.Call { return v.call("Call", in) } ../labrpc/labrpc.go:478 labrpc.(*Service).dispatch { function.Call([]reflect.Value{svc.rcvr, args.Elem(), replyv}) } ../labrpc/labrpc.go:402 labrpc.(*Server).dispatch { return service.dispatch(methodName, req) } ../labrpc/labrpc.go:229 labrpc.(*Network).ProcessReq.func1 { r := server.dispatch(req) } Here is what goroutine 77264 doing now goroutine 77264 [semacquire]: sync.runtime_SemacquireMutex(0xc420084744, 0xa900000000) /usr/local/Cellar/go/1.9/libexec/src/runtime/sema.go:71 +0x3d sync.(*Mutex).Lock(0xc420084740) /usr/local/Cellar/go/1.9/libexec/src/sync/mutex.go:134 +0x14c github.com/sasha-s/go-deadlock.(*lockOrder).postUnlock(0xc420084740, 0x1288e00, 0xc4202a60e0) .../src/github.com/sasha-s/go-deadlock/deadlock.go:330 +0x3f github.com/sasha-s/go-deadlock.postUnlock(0x1288e00, 0xc4202a60e0) .../src/github.com/sasha-s/go-deadlock/deadlock.go:167 +0x5f github.com/sasha-s/go-deadlock.(*Mutex).Unlock(0xc4202a60e0) .../src/github.com/sasha-s/go-deadlock/deadlock.go:97 +0x7f raft.(*Raft).AppendEntriesRPCHandler(0xc4202a60e0, 0xc420beb840, 0xc421834620) .../src/raft/raft.go:688 +0xcc8 ...The text was updated successfully, but these errors were encountered: