It may fix sync stuck issue at runtime

[hexcola]

Describe the bug
It's hard to catch up with the current chain height, and messages are jammed in mpool.

To Reproduce
It happened accidentally. After analyzing millions of line of lotus log and miner log by timeline, I noticed, the retry of failed proof commit will cause a sync latency issue.

Screenshots

sync status:
worker 0:
        Base:  [bafy2bzacedgaruy6wmsdckntkmqe52epczvoabjzixksprikepmpixcjn3oai]
        Target: [bafy2bzaceblqd2lqipdniic7rnzlkz2226z5jqpsvbh3hd5vwik4jjq46coyi] (49283)
        Height diff:   2718
        Stage: error
        Height: 0
        Elapsed: 3.080831914s
        Error: collectChain syncMessages: message processing failed: validating block bafy2bzacecr57r7xgupckunqspnnop4pcvj2i7foibjmhgrusgsixi63migei: parent state root did not match computed state (bafy2bzacecgcsqloew3bu6igaxfooepjwnn224mvmztdgvunbdwkirkye6gji != bafy2bzacedgzwtrdgml7eygsu6kqk67kguefb2vxp4ulc2hs6nfaonwryzl2a)
worker 1:
        Base:  [bafy2bzacedgaruy6wmsdckntkmqe52epczvoabjzixksprikepmpixcjn3oai]
        Target: [bafy2bzacecj4x54qti74fmfapvnvfx4zkjl2sq5p6c2kcnzkxkajym6m3lqes bafy2bzacedzxlhfs5wisgsgtyzipo7yclhkj2yrt3yv5yb5quibs2xrat2m7e bafy2bzaceb4sl7agweyfz62d6wcfa4xnriconett5jpy7wossh3wzl4amareg bafy2bzacecaajexpaqnymuznhq4hb6jga2tsd4xpq6adjbiuxci4ndnzavhee bafy2bzacec75klcdewuyo5xwbddtdty24ji6oi47hhft4kmeyds5qgqolsomk] (49292)
        Height diff:   2727
        Stage: error
        Height: 49292
        Elapsed: 42.994603ms
        Error: chain contained block marked previously as bad ([bafy2bzacecj4x54qti74fmfapvnvfx4zkjl2sq5p6c2kcnzkxkajym6m3lqes bafy2bzacedzxlhfs5wisgsgtyzipo7yclhkj2yrt3yv5yb5quibs2xrat2m7e bafy2bzaceb4sl7agweyfz62d6wcfa4xnriconett5jpy7wossh3wzl4amareg bafy2bzacecaajexpaqnymuznhq4hb6jga2tsd4xpq6adjbiuxci4ndnzavhee bafy2bzacec75klcdewuyo5xwbddtdty24ji6oi47hhft4kmeyds5qgqolsomk], bafy2bzacecr57r7xgupckunqspnnop4pcvj2i7foibjmhgrusgsixi63migei) (reason: parent state root did not match computed state (bafy2bzacecgcsqloew3bu6igaxfooepjwnn224mvmztdgvunbdwkirkye6gji != bafy2bzacedgzwtrdgml7eygsu6kqk67kguefb2vxp4ulc2hs6nfaonwryzl2a))

Version (run lotus version):
Any version may have this issue.

How to solve it at runtime?

⚠️ Note: it's not the right way to solve the problem, you on your risk.
You may lose your seal work.
You may lose your seal work.
You may lose your seal work.

Just monitor your log, as long as you saw a proof failed to commit, you kick it out, from what I noticed, no matter how many times it tries, it will fail eventually, no exception.

here is the bash script that will run automatically.

remember to change this variable:

• <YOUR_MINER_LOG_PATH>
• <WHERE_YOU_WANT_SAVE_REMOVE_HISTORY>

#!/bin/bash
tail  -n1000 -f <YOUR_MINER_LOG_PATH>.log | while read line
do
        sector_id=`echo "${line}"|grep -E "proof failed | invalid proof | comm_d do not match" |awk -F ")" '{print $1}' |awk '{print $6}'`
        if [[ "$sector_id" != "" ]];then
        date "+%F %H:%M:%S" >><WHERE_YOU_WANT_SAVE_REMOVE_HISTORY>.log
        echo  $sector_id >><WHERE_YOU_WANT_SAVE_REMOVE_HISTORY>.log
         lotus-miner sectors  remove  --really-do-it $sector_id
        fi
done

[huazhuangnan]

hello, I have the same, what CPU are you？

[ericnjin]

Same to me...
lns@lns-TRX40-AORUS-XTREME:~$ lotus version
Daemon: 0.7.0+git.232cc320+api0.14.0
Local: lotus version 0.7.0+git.232cc320

lns@lns-TRX40-AORUS-XTREME:~$ lotus sync status
sync status:
worker 0:
Base: [bafy2bzacedgaruy6wmsdckntkmqe52epczvoabjzixksprikepmpixcjn3oai]
Target: [bafy2bzaced66ze5ivlemxmnf3cfcgxglmcwqdb4nuqv7i5rt7s5luhjg2be7c] (50046)
Height diff: 3481
Stage: error
Height: 50046
Elapsed: 55.683778ms
Error: chain contained block marked previously as bad ([bafy2bzaced66ze5ivlemxmnf3cfcgxglmcwqdb4nuqv7i5rt7s5luhjg2be7c], bafy2bzacecr57r7xgupckunqspnnop4pcvj2i7foibjmhgrusgsixi63migei) (reason: parent state root did not match computed state (bafy2bzacecgcsqloew3bu6igaxfooepjwnn224mvmztdgvunbdwkirkye6gji != bafy2bzacedgzwtrdgml7eygsu6kqk67kguefb2vxp4ulc2hs6nfaonwryzl2a))
worker 1:
Base: [bafy2bzacedgaruy6wmsdckntkmqe52epczvoabjzixksprikepmpixcjn3oai]
Target: [bafy2bzaced66ze5ivlemxmnf3cfcgxglmcwqdb4nuqv7i5rt7s5luhjg2be7c] (50046)
Height diff: 3481
Stage: error
Height: 0
Elapsed: 3.40410599s
Error: collectChain syncMessages: message processing failed: validating block bafy2bzacecr57r7xgupckunqspnnop4pcvj2i7foibjmhgrusgsixi63migei: parent state root did not match computed state (bafy2bzacecgcsqloew3bu6igaxfooepjwnn224mvmztdgvunbdwkirkye6gji != bafy2bzacedgzwtrdgml7eygsu6kqk67kguefb2vxp4ulc2hs6nfaonwryzl2a)
worker 2:
Base: [bafy2bzacedgaruy6wmsdckntkmqe52epczvoabjzixksprikepmpixcjn3oai]
Target: [bafy2bzacedbxgl2lr4aectbmwjc6wgveeo7slujl7feu2xyk2pqmu4jooihkq] (50025)
Height diff: 3460
Stage: error
Height: 50025
Elapsed: 64.253339ms
Error: chain contained block marked previously as bad ([bafy2bzacedbxgl2lr4aectbmwjc6wgveeo7slujl7feu2xyk2pqmu4jooihkq], bafy2bzacecr57r7xgupckunqspnnop4pcvj2i7foibjmhgrusgsixi63migei) (reason: parent state root did not match computed state (bafy2bzacecgcsqloew3bu6igaxfooepjwnn224mvmztdgvunbdwkirkye6gji != bafy2bzacedgzwtrdgml7eygsu6kqk67kguefb2vxp4ulc2hs6nfaonwryzl2a))

[huazhuangnan]

why？

[SgtCoin]

Archiving due to age.