Configure monitoring and alerting for cloud.gov app logs and metrics

[mogul]

User Story

In order to meet the monitoring and alerting needs for data.gov applications running in cloud.gov, the data.gov team wants to monitor logs and telemetry from cloud.gov apps and alert when suspicious conditions arise.

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

• WHEN an application in a space in the gsa-datagov organization on cloud.gov logs a "test" error
  THEN the data.gov team is alerted that the test was logged.
• [something about the specific kings of monitoring and alerting we want to do goes here; should it be a separate issue to configure specific alerts for each app??]

Background

• This Terraform provider explicitly supports OpenDistro features in AWS ES

Security Considerations (required)

Relates to section SI-4(5) of our SSP.

Sketch

The Loki+Grafana path:

• Drain logs to Loki via syslog -> Fluent Bit (or promtail)
• Configure Grafana to alert to Slack

The AWS ES path:

• Practice configuring monitoring and alerting directly in the Kibana UI; this is probably enough to meet the immediate needs of the story.
• Bonus points:
  • Use the ES API to siphon out our configured monitors and alerts.
  • Use the ODfE Terraform provider to configure our monitoring and alerts via CI. (This is to make Contingency Planning and auditing easier.)

[adborden]

We're considering this as an improvement to monitoring/alerting and have other controls in place to meet compliance requirements.

[mogul]

Closing as a dupe of #3062