Miner process terminated , log:Segmentation fault (core dumped)

[BBQFIL]

Checklist

• This is not a security-related bug/issue. If it is, please follow please follow the security policy.
• This is not a question or a support request. If you have any lotus related questions, please ask in the lotus forum.
• This is not a new feature request. If it is, please file a feature request instead.
• This is not an enhancement request. If it is, please file a improvement suggestion instead.
• I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
• I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
• I did not make any code changes to lotus.

Lotus component

• lotus daemon - chain sync
• lotus miner - mining and block production
• lotus miner/worker - sealing
• lotus miner - proving(WindowPoSt)
• lotus miner/market - storage deal
• lotus miner/market - retrieval deal
• lotus miner/market - data transfer
• lotus client
• lotus JSON-RPC API
• lotus message management (mpool)
• Other

Lotus Version

lotus version 1.14.0+mainnet+git.1fd563436.dirty

Describe the Bug

Miner was proving when the error occurred,

Logging Information

DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 39546561
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 97158835
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 64423511
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 120079858
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 79820898
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 130475993
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 116090731
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 11586903
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 65871671
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 71562507
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 49035453
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 21001216
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 122253593
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 63390453
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 64806662
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 44817386
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 39237429
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 71630760
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 22131130
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 121016676
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 8505983
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 17518448
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 49644182
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 125147902
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 52969038
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 32182648
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 128959723
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 26210356
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 4625843
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 81051697
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 74189358
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 18315100
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 132917291
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 73054755
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 30532204
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 51865702
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 102695752
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 13975762
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 60475941
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 38750369
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 32159864
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 28195058
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 64991501
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 107869184
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 36459859
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 49296373
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 28709378
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 8063276
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 74751996
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 126550808
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 55643233
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 44159754
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 28627512
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 83854453
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 40553903
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 45046763
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 11001773
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 29463039
 DEBUG merkletree::merkle > generated partial_tree of row_count 4 and len 585 with 8 branches for proof at 30828650
 INFO storage_proofs_core::compound_proof > vanilla_proofs:finish
 INFO storage_proofs_core::compound_proof > snark_proof:start
 INFO bellperson::groth16::prover > Bellperson 0.18.2 is being used!
 INFO bellperson::groth16::prover > synthesis time: 67.57258379s
 INFO bellperson::groth16::prover > starting proof timer
 DEBUG bellperson::gpu::locks > Acquiring priority lock at "/tmp/bellman.priority.lock" ...
 DEBUG bellperson::gpu::locks > Priority lock acquired!
 DEBUG bellperson::groth16::prover > get h
 INFO bellperson::gpu::locks > GPU is available for FFT!
 DEBUG bellperson::gpu::locks > Acquiring GPU lock at "/tmp/bellman.gpu.lock" ...
 DEBUG bellperson::gpu::locks > GPU lock acquired!
 INFO bellperson::gpu::program > Using kernel on CUDA.
 INFO bellperson::gpu::program > Using kernel on CUDA.
 INFO bellperson::gpu::fft > FFT: 2 working device(s) selected. 
 INFO bellperson::gpu::fft > FFT: Device 0: GeForce RTX 3090
 INFO bellperson::gpu::fft > FFT: Device 1: GeForce RTX 3090
 INFO bellperson::domain > GPU FFT kernel instantiated!
 WARN	watchdog	[email protected]/watchdog.go:338	system-driven watchdog triggering GC; 136767688704/135100835840 bytes (used/threshold)
 INFO	watchdog	[email protected]/watchdog.go:357	watchdog is forcing GC
 INFO	watchdog	[email protected]/watchdog.go:368	watchdog-triggered GC finished; took: 95.893401ms; current heap allocated: 758885784 bytes
Segmentation fault (core dumped)
[BBQ@Miner1 ~/logs]$ 

CPU:AMD EPYC 7402 24-Core Processor

### Repo Steps

_No response_

[rjan90]

Hey @BBQFIL!

Thanks for the report, I converted it into a discussion to further find out what is wrong before we potentially move it back as an issue.

First of all, it seems like you have done some code-modifications, can you clarify a bit more around those? So we can remove that as a possibility for the seg-fault. If so, it would be ideal if you could revert to a stable release like https://github.com/filecoin-project/lotus/releases/tag/v1.15.0, and see if you experience this issue there as well.