Bat Team Tips

Production Incidents

When we get a production incident from PagerDuty or DataDog alert, we typically go through the following.

Check Logs and Monitors

• See notes and links in the DataDog alert
• https://monitor.cf.ds.va.gov/
• https://app.datadoghq.com/dashboard/lists
  • triggered monitors
  • Be aware of scheduled downtime
  • List of production machines and AWS zones
• Cloudwatch logs: directly from AWS or via Kibana

Take action

• Follow the Bat Team manual on responding to production incidents
• If needed, refer to First Responder Manual

Support Issues

When we get a support issue in the #appeals-batteam Slack channel, we typically go through the following.

Replicate in Caseflow

• Log into Citrix Access Gateway (CAG) to access Caseflow in production: https://appeals.cf.ds.va.gov
• Impersonate as the user using their CSS_ID at https://appeals.cf.ds.va.gov/test/users
• Browse to the URL relevant to the problem provided by the Support Team

Locate the appeal in Rails console

• ssm into a production machine:

ssm caseflow prod auto
sudo su -c "source /opt/caseflow-certification/caseflow-certification_env.sh; cd /opt/caseflow-certification/src; bin/rails c"

• Get the appeal/case id from the URL if possible
  • URL format for queue is https://appeals.cf.ds.va.gov/queue/appeals/<vacols_id or appeal_uuid>
• Retrieve an AMA appeal
  • Given appeal uuid: Appeal.find_by(uuid: "b6a684f8-37f0-4856-a0ec-426d138545f6")
  • Given a docket number for an AMA appeal, it is in the format #{receipt_date.strftime('%y%m%d')}-#{id} (see def docket_number in appeal.rb), so just use the id number after the hyphen to grab the appeal: Appeal.find(id)
• Retrieve a Legacy appeal
  • Given VACOLS id: LegacyAppeal.find_by(vacols_id: 2717083)
  • Given docket number:

docket_number = "1853273A"
vacols_id = VACOLS::Folder.find_by(tinum: docket_number).ticknum
appeal = LegacyAppeal.find_by(vacols_id: vacols_id)

• Print the task tree for an appeal

puts appeal.structure_render(:id, :status, :updated_at)

To find similar real task trees, check out Task Tree docs

Locate the Intake models in Rails console

• Similarly to AMA Appeals, HLRs and SCs are identified by UUID:

sc = SupplementalClaim.find_by(uuid: "77cdc5af-cc7b-4662-acd6-0ddfc3ed0136")

• Given a VBMS Claim ID, we can also retrieve Caseflow's corresponding EPE:

epe = EndProductEstablishment.find_by(reference_id: CLAIM_ID)

• Once you have any relevant model (even just a Veteran object), print the intake structure:

puts IntakeRenderer.render(obj)

Read more about the Intake Structure Renderer on its wiki page.

Keep the Intake Data Model handy, as a guide to the complicated relationships between intake models.

Search for existing solutions

Search on:

• Bat Team quick ref
• Slack messages
• Previous Bat Team journals on Bat Team handoffs page

Find expertise

• Ask Tech Lead of relevant team/product:
  • Echo (Queue, Reader): JC
  • Foxtrot (Intake, Decision Reviews): Yang
  • Tango (Hearings): Tomas
• Ask Engineering Lead: Lowell

Sentry Alerts

When we investigate a Sentry alert, we typically do the following.

Inspect the Sentry alert

• Remember to enable socks proxy sockson <mfa_pin>. See utility scripts and
• Click on the alert link in Slack which loads https://sentry.ds.va.gov/department-of-veterans-affairs/caseflow/issues/... in a browser
• Click the Comments tab to check for any previous work or ticket reference
• Click the Events tab to check for frequency and/or patterns

Search for existing solutions

• Scroll up in Slack for comments and reactions to similar alerts
• Use the search bars to look for existing GitHub issues
• See the other "Search for existing solutions" section above

Creating a GitHub issue/ticket

• See the relevant section in the Bat Team manual.
• Post ticket in Sentry alert's Comment tab and in a Slack thread for the alert.

All the Caseflow Sentry alerts.