Fixes for go races and miner crashes #42
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
1. Fixed race in gRPC client send/recv/close handler: Write at 0x00c01e77fc90 by goroutine 6616: google.golang.org/grpc.(*clientStream).CloseSend() Previous read at 0x00c01e77fc90 by goroutine 6615: google.golang.org/grpc.(*clientStream).SendMsg() 2. Fixed race in gRPC client error handler: This is a workaround for current implementation to set RLock for non-atomic error handler, some parts are atomic already. It is also necessary to store new value 'isError' otherwise concurrent connections can block each other but without lock no atomic read/write is guaranteed. The gRPC client implementation with a shared error handler between send/recv should be reworked.
The miner routines are asynchroneous and in stability tests the
miner regularly crashes when you set very low limit of blocks to
be mined, it will fail with:
2024-02-22 12:18:23.209 [CRT] KSMN: Exiting: Fatal error in goroutine `blocksLoop 7`: Writing to the logger when it's not running
panic: Writing to the logger when it's not running
goroutine 39 [running]:
github.com/karlsen-network/karlsend/infrastructure/logger.(*Logger).printf(0xfe4180, 0x5, {0xad163c, 0x4}, {0xad725b, 0xb}, {0xc0000837c0, 0x1, 0x1})
/home/runner/work/karlsend/infrastructure/logger/logs.go:185 +0x385
github.com/karlsen-network/karlsend/infrastructure/logger.(*Logger).Writef(...)
/home/runner/work/karlsend/infrastructure/logger/logs.go:144
github.com/karlsen-network/karlsend/infrastructure/logger.(*Logger).Criticalf(0xc0000b61e0?, {0xad725b?, 0x0?}, {0xc0000837c0?, 0x0?, 0x0?})
/home/runner/work/karlsend/infrastructure/logger/logs.go:127 +0x4c
github.com/karlsen-network/karlsend/util/panics.exit.func1()
/home/runner/work/karlsend/util/panics/panics.go:58 +0xa6
created by github.com/karlsen-network/karlsend/util/panics.exit in goroutine 28
/home/runner/work/karlsend/util/panics/panics.go:57 +0x14a
Root cause for this issue is that in simnet with very low
difficulty we can mine and submit blocks very fast and while
submitting an already mined block, the miner will find several
others.
The limit was only applied for submitting blocks, but not for
mining blocks. It is fixed now.
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #42 +/- ##
==========================================
+ Coverage 61.42% 61.43% +0.01%
==========================================
Files 646 646
Lines 31882 31898 +16
==========================================
+ Hits 19583 19597 +14
- Misses 9447 9449 +2
Partials 2852 2852 ☔ View full report in Codecov by Sentry. |
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a stability improvement pull request. In the Go node implementation there are several races which are detected by
go -raceand have been fixed.1. Fixed race in gRPC client
send,recvandclosehandler:2. Fixed race in gRPC client error handler:
This is a workaround for current implementation to set
RLockfor non-atomic error handler, some parts are atomic already. It is also necessary to store new valueisErrorotherwise concurrent connections can block each other but without lock no atomicreadandwriteis guaranteed. The gRPC client implementation with a shared error handler betweensendandrecvshould be reworked.3. Fixed
karlsenminercrash:The miner routines are asynchronous and in stability tests the miner regularly crashes when you set very low limit of blocks to be mined. If running the miner
karlsenminer --simnet -s localhost:9000 --mine-when-not-synced --miningaddr karlsensim:qre0nh3wemr8c4zqlwsd5tch78dx93mmx2u2pvrugheny7ja8k6t682pvpx83 --numblocks 1, it will fail with:Root cause for this issue is that in
simnetwith very low difficulty we can mine and submit blocks very fast and while submitting an already mined block, the miner will find several others. It can be easily reproduced via:The limit was only applied for submitting blocks, but not for mining blocks. It is fixed now.