Skip to content

Alerts Event Standard Operating Procedure

Jump to bottom
Aaron D Borden edited this page Jan 9, 2019 · 17 revisions

Data.gov Escalation Path

Whoever gets the alert would follow this process:

  • Alert received (via slack/email/twitter)
  • Drop an @here note on slack w/ alert content and context #datagov-devsecops channel: "Looking into this"
  • Hit the "Acknowledge" button on the New Relic incident page
  • Open Issue in Data.gov datagov-deploy with link to alert
  • Others acknowledge/verify and offer support
    • Include as additional information in github issue
  • Assign person(s) most capable to solve issue with @user
  • Get written confirmation as a comment that assigned member is handling the issue and they have needed support to resolve.
    • If after hours or during weekend, the assigned person should be phoned to have this immediately brought to their attention if there is no response on the issue in 10 minutes. This step is not complete until there is written confirmation that work is underway.
  • All project team members are responsible for monitoring progress and providing updates/context to github issue with more information as it is received.

Note: If at all possible, there should be two sets of eyes to review the code on any emergency changes pushed straight to production.

Once Resolved:

  • Ensure that issue is fully up to date with all information learned about the problem

  • Schedule retroactive meeting to review response (and update this document with lessons learned) and conduct root cause analysis.

    • write up additional github issue and submit PR if it was preventable via code
    • prioritize and resolve github issues

Runbook

This section covers procedures for common alerts and events.

"Restart required" notification from unattended-upgrades

unattended-upgrades is an apt package that installs the latest OS packages and security updates. Usually packages are updated and services are restarted without any additional action. If this notification is received, a reboot is required to complete the upgrade.

  • ansible-playbook actions/reboot.yml --limit $host

Note: jumpboxes are not included in the reboot playbook and should be rebooted manually:

  • sudo reboot
  • wait for jumpbox to restart and ensure connectivity.
Clone this wiki locally