discuss: datadog and logging failures

[hmrtn]

Scenario:

If our logging to datadog fails and users face an error - it may become very difficult to identify the issue(s) that need to be addressed.

Long term solution:

Short term solution to check if datadog is responsive

We can check the status of datadog here.

If datadog is down, we should have a script that is automatically triggered to revert to manual logging on our servers.

WIP

[thelostone-mc]

Points of failure:

Round Manager

• Program Creation
• Round Creation

Grant Explorer

• Finalize ballot
• Add to ballot

[vacekj]

We are tracking all errors in sentry as well as datadog, so a failure would have to occur in both at the same time for us to miss issues. I think this redundancy is enough, closing. If this unlikely scenario happens, we can add in another service like hotjar to do session monitoring.