Doris 2.0.0, 3 fe nodes, 1 master, 1 follower, and 1 observer all crashed at the same time and cannot be restored

[48N6E]

After crashing at the same time, close the follower and observer, refer to https://doris.apache.org/zh-CN/docs/2.0/admin-manual/maint-monitor/metadata-operation/?_highlight=metadata_failure_recovery#%E6%95%85%E9%9A%9C%E6%81%A2%E5%A4%8D Fault recovery in, but master startup prompt
2024-07-24 01:42:40,774 INFO (stateListener|97) [BDBHA.fencing():78] start fencing, epoch number is 139
2024-07-24 01:42:40,795 INFO (stateListener|97) [Env.replayJournal():2446] replayed journal id is 1989095, replay to journal id is 2079319
2024-07-24 01:42:40,796 ERROR (stateListener|97) [BDBJournalCursor.():83] Can not find the key:1989096, fail to get journal cursor. will exit.

May I ask how to solve this

[ixzc]

please make sure your machine's clock is synchronized

[48N6E]

Yes,Three nodes at the same time,Now i change cluster to the 1 follower ,2 observer

[maruyue]

1 master 1 follower 1 observer，is not a good config.
you could config 1 master， 1-2 obsersers. The config is very stable.

[maruyue]

Try not to use metadata_failure_recovery. This is for disaster recovery and should not be used unless absolutely necessary. Once used, it may cause metadata truncation, metadata loss, and metadata brain split.

[48N6E]

Try not to use metadata_failure_recovery. This is for disaster recovery and should not be used unless absolutely necessary. Once used, it may cause metadata truncation, metadata loss, and metadata brain split.

Tky for your advice.