discuss: subgraph failures

[thelostone-mc]

Scenario:

The protocol's frontend products rely on subgraph as a source of data. If the service goes down -> we should be able to relay that information to the users.

Long term solution:
We could build a fail-safe layer to rely on contract events and directly populate the data
This is tedious cause chains like phantom have a limit of the event queries which can be done (time based limit)
So as adopt more chains -> it might get tricker building a custom solution
I would suggest deferring this to a later date if we notice failures popping up

Short term solution to check if subgraph is responsive

Every Chain has it's own subgraph.
We could build an integration to https://uptimerobot.com/ where the steps would be

For every subgraph URL

• hit an empty query to check if response is 200
• trigger email / discord notification if response isn't
• build a similar service into the frontend on RM + GM and show indexing error modal if response is not 200

Short term solution to check if subgraph is delayed in indexing

For every subgraph URL

• hit an empty query to get latest indexed block number
• compare it with the latest block on chain
• if there is a dealy report as a warning (This might get spammy)

I don't think we should report any issue on the front end at this point in time here for slow indexing (cause it might get spammy)

[thelostone-mc]

Explore Datadog synthetic

[vacekj]

This issue still exists even with our new indexer. All the points still apply, and since this is a major moving part that all of our apps depend on that currently doesn't have good redundancy or failsafe options, I would recommend we work on this sooner rather than later.

[vacekj]

Closing as stale