O+M 2023-09-01

[hkdctol]

As part of day-to-day operation of Data.gov, there are many Operation and Maintenance (O&M) responsibilities. Instead of having the entire team watching notifications and risking some notifications slipping through the cracks, we have created an O&M Triage role. One person on the team is assigned the Triage role which rotates each sprint. This is not meant to be a 24/7 responsibility, only East Coast business hours. If you are unavailable, please note when you will be unavailable in Slack and ask for someone to take on the role for that time.

Check the O&M Rotation Schedule for future planning.

Miscs

• Watch for user email requests
• Watch in #datagov-alerts and Vulnerable dependency notifications (daily email reports) for critical alerts.
• Monitor and improve Data.gov O&M Dashboard

Acceptance criteria

You are responsible for all O&M responsibilities this week. We've highlighted a few so they're not forgotten. You can copy each checklist into your daily report.

Daily Checklist

Check Production State/Actions

• README

Note: Catalog Auto Tasks
You will need to update the chart values manually. Click the Action link in each issue and grab the values from monitor task output and check runtime.

• DB-Solr Sync
• Check Harvesting Emails
• New Relic Alerts Triaged
• Triage DMARC Report from Google

Weekly Checklist

• Audit Log (more info on AU-3 and AU-6 Log auditing)
• Tracking Update
• Check Catalog Solr
• Catalog Dupe Check
• Invicti Scan

[nickumia-reisys]

Day 1 Summary

• Solr follower 2 restarted 1:24p on 8/25
• Solr follower 1 restarted 9:23 on 8/27
• Triaged Sitemap Failure, Sitemap Generation Failure catalog.data.gov#990 (comment)
• Logged tracking update
• Logged DB-Solr sync
• USDA Harvest got interrupted on Saturday, but it has been fine since then
  | Error loading json content: ("Connection broken: InvalidChunkLength(got length b'', 0 bytes read)", InvalidChunkLength(got length b'', 0 bytes read)).

[nickumia-reisys]

Day 2 Summary

• Catalog production issue
• No critical harvesting errors
• https://github.com/GSA/datagov-account-management/issues/227

[nickumia-reisys]

Day 3/4 Summary

• HUD JSON failed with access issue to data.json, re-running the harvest was successful. Probably a network issue.
  
• Inventory prod issue:
  • Automated CKAN Job Error Condition catalog.data.gov#1051
  • Egress Check Failed catalog.data.gov#1050
• Support issue with NASA Data.json.
• Discussion on automated ckan jobs.
• Support issue with Philadelphia staging harvest.
• Helped @FuhuXia get these out
  • Forward User-Agent from AWS CloudFront to catalog app #4059
  • Limit catalog-proxy traffic to only from Cloudfront #4413

[nickumia-reisys]

Day 5 Summary

• Catalog 30 min outage 8/31 @ 9:16p
• Catalog solr follower 2 restarted 8/31 @ 9:10p
• Catalog solr follower 1 restarted 8/31 @ 9:53p
• FEMA-R10 Harvest source gave 500 on 9/1 harvest: - Unable to get content for URL: https://hazards.fema.gov/filedownload/metadata/R10/: HTTPError('500 Server Error: Internal Server Error for url: https://hazards.fema.gov/filedownload/metadata/R10/')
• inventory-proxy access forbidden logs
• catalog-fetch failure to create package?
• Catalog 500 Error on /member/authenticate #4443
• Catalog 500 Error on specific dataset? #4444