Sync error handling

[Zacholme7]

Issue Addressed

#119

Proposed Changes

This PR introduces a way to handle rpc errors and signal if sync is stalled.

If a websocket goes down or an rpc endpoint is having issues,OPERATIONAL_STATUS is set to false. The rest of the application can be conditioned on this value to determine if the execution layer is having sync issues.

If there is an rpc error, there is nothing we can do until the endpoint is operational again. The simplest way to test this is just to continuously poll for a block number with exponential backoffs. There is no longer a set number of retries. It just keeps retrying until it is valid again.

[dknopik]

I retested it locally. Could not get it to crash now. Nice!

But I noticed that right now, this error condition does not properly apply any backoff:

anchor/anchor/eth/src/sync.rs

Lines 413 to 414 in b5ffff4

	// If we get here, the stream ended (likely due to disconnect)
	error!("WebSocket stream ended, reconnecting...");

This got me thinking once more about the approach. How do you feel about, instead of having to apply troubleshoot_rpc at multiple appropriate locations throughout the file, just returning an error in the live and historical functions and doing the reconnect logic at the top level? This also handles cases where we want to fall back to historical sync logic after being offline for a considerable amount of time, and helps us assert that we really do not crash out anymore if something fails. There might be some disadvantages I am not thinking of right now though.

[Zacholme7]

@dknopik Threw together a quick POC. I think I that approach works very nice. Take a look when you have a sec and if thats along the lines of what you were thinking ill clean it up and make sure it works.

[dknopik]

I feel like it would be bit cleaner to create the Arc within the new method.

[dknopik]

Threw together a quick POC. I think I that approach works very nice. Take a look when you have a sec and if thats along the lines of what you were thinking ill clean it up and make sure it works.

yea, seems good! This is what I meant. :​)