Request Node Monitoring System

[MantisClone]

Problem

Sometimes, the Request Node gets into an error state and silently fails to persist all new requests.

Proposed Solution

Build a monitoring system to detect when a new CID is written to IPFS and when that same CID persists on-chain. If the CID is not persisted on-chain within a certain timeout, the monitoring system raises an alert in Slack to let us know that a request creation has failed.

Rejected Solution

We considered making the monitoring system actually create requests, once every hour. But this would be expensive over time and would require us to filter out "test" requests from our Request Scan and Metrics.