Internal watchdog functionality

[shawn-zil]

An application layer restart for internal nodes. This is in-contrast to #2004 that tried to use a OS-level restart mechanism.

This internal watchdog mechanism is more modest. Since a service may run multiple nodes, this mechanism restarts only the affected node and not the entire service. It can be used alongside any infra-level restarts that detect and restart the entire service.

2024-12-13T02:54:08.912071Z  INFO zilliqa::consensus: 487: We are on view: 1 but we are not a validator, so we are waiting.
2024-12-13T02:54:09.422499Z  INFO zilliqa::consensus: 487: We are on view: 1 but we are not a validator, so we are waiting.
2024-12-13T02:54:09.536248Z  WARN zilliqa::node_launcher: 234: WDT node stuck at self_highest=0 remote_highest=7025
2024-12-13T02:54:09.536304Z  INFO zilliqa::node_launcher: 385: WDT restarting 33468.
2024-12-13T02:54:09.536850Z  INFO zilliqa::db: 235: PRAGMA journal_mode="memory" journal_size_limit=33554432 synchronous=1 temp_store=2 page_size=32768 cache_size=8192
2024-12-13T02:54:09.573453Z  INFO zilliqa::node_launcher: 267: WDT period: 30s
2024-12-13T02:54:09.589972Z  INFO zilliqa::consensus: 487: We are on view: 1 but we are not a validator, so we are waiting.
2024-12-13T02:54:10.100579Z  INFO zilliqa::consensus: 487: We are on view: 1 but we are not a validator, so we are waiting.

I've also checked that the joinset is stable, does not grow with multiple restarts.

The definition of stuck node is:

• Has not changed its own highest canonical block number in a while. This is an internal-check. If N historical samples are the same, then this condition is met.
• Has a canonical block number that is smaller than the network. This is an external-check to ensure that we’re behind. This will help to mitigate against timeouts.

[github-actions]

Bencher Report

Branch	1917-internal-watchdog
Testbed	self-hosted

Click to view all benchmark results

Benchmark	Latency	Benchmark Result nanoseconds (ns) (Result Δ%)	Upper Boundary nanoseconds (ns) (Limit %)
process-empty/process-empty	📈 view plot 🚷 view threshold	8,433,200.00 (-7.95%)	10,637,765.68 (79.28%)
produce-full/produce-full	📈 view plot 🚷 view threshold	2,250,100,000.00 (-0.96%)	3,138,061,085.10 (71.70%)

🐰 View full continuous benchmarking report in Bencher