O+M 2023-09-29

[hkdctol]

As part of day-to-day operation of Data.gov, there are many Operation and Maintenance (O&M) responsibilities. Instead of having the entire team watching notifications and risking some notifications slipping through the cracks, we have created an O&M Triage role. One person on the team is assigned the Triage role which rotates each sprint. This is not meant to be a 24/7 responsibility, only East Coast business hours. If you are unavailable, please note when you will be unavailable in Slack and ask for someone to take on the role for that time.

Check the O&M Rotation Schedule for future planning.

Acceptance criteria

You are responsible for all O&M responsibilities this week. We've highlighted a few so they're not forgotten. You can copy each checklist into your daily report.

Daily Checklist

• Check Production State/Actions

Note: Catalog Auto Tasks
You will need to update the chart values manually. Click the Action link in each issue and grab the values from monitor task output and check runtime.

• DB-Solr Sync
• Check Harvesting Emails
• New Relic Alerts Triaged
• Triage DMARC Report from Google

Weekly Checklist

• Audit Log (more info on AU-3 and AU-6 Log auditing)
• Tracking Update
  • NOTE: This job will consistently timeout, but it is processing results ((more details)[https://github.com/change tracking update from nightly job to weekly #4345])
• Check Catalog Solr
• Catalog Dupe Check

Monthly Checklist

• Invicti Scan

Reference

• Watch for user email requests
• Watch in #datagov-alerts and Vulnerable dependency notifications (daily email reports) for critical alerts.
• Monitor and improve Data.gov O&M Dashboard
• Update and revise Data.gov O&M Tasks

[Jin-Sun-tts]

Monday 09/25

https://github.com/GSA/data.gov/

Check Catalog Auto Tasks

• DB-Solr Sync:
  0 packages need to be removed from Solr
  152 packages need to be updated/added to Solr
  225 packages without harvest_object need to be mannually deleted
  Finished 529s

• Tracking Update:
  2023-09-24T07:40:53.92+0000 [APP/TASK/ckangeodatagovtracking-update-6288764933-1/0] OUT 2023-09-24 07:40:53,926 INFO [ckanext.geodatagov] 135537 package indexes to be rebuilt starting from 2023-09-15 00:00:00

Check Harvesting Emails

• Catalog:
• 9 harvesting job reported at 11am EST

Other

• New Relic Alerts

Checked catalog, inventory production, works fine.

Also checked Solr leader and followers, all work as normal.

[Jin-Sun-tts]

Tuesday 09/26

https://github.com/GSA/data.gov/

Check Catalog Auto Tasks

• DB-Solr Sync:
  0 packages need to be removed from Solr
  3 packages need to be updated/added to Solr
  225 packages without harvest_object need to be mannually deleted
  Finished 514s

Check Harvesting Emails

• Catalog:
• 7 harvesting job reported at 1:30pm EST

Other

• New Relic Alerts

Checked catalog, inventory production, works fine.

Also checked Solr leader and followers, all work as normal.

• Added user as Editor in Inventory

three users in opm #538 #539 #540
one user in Ntia #541

[Jin-Sun-tts]

Wednesday 09/27

https://github.com/GSA/data.gov/

Check Catalog Auto Tasks

• DB-Solr Sync:
  0 packages need to be removed from Solr
  2 packages need to be updated/added to Solr
  225 packages without harvest_object need to be mannually deleted
  Finished 492s

Check Harvesting Emails

• Catalog:
• 10 harvesting job reported at 2:00pm EST

Other

• New Relic Alerts
  double checked the new formatted login information (either from API or saml) is logged in the log

Checked catalog, inventory production, works fine.

Also checked Solr leader and followers, all work as normal.

• Added user as Editor in Inventory

one user in DOL #425

[Jin-Sun-tts]

Thursday 09/28

https://github.com/GSA/data.gov/

Check Catalog Auto Tasks

• DB-Solr Sync:
  0 packages need to be removed from Solr
  2 packages need to be updated/added to Solr
  226 packages without harvest_object need to be mannually deleted
  Finished 484s

Check Harvesting Emails

• Catalog:
• 6 harvesting job reported at 1:56pm EST
  most of them are related identifier, keyword missing errors.

Other

• New Relic Alerts

Checked catalog, inventory production, works fine.

Also checked Solr leader and followers, all work as normal.

• issues:

Automated CKAN Job Error Condition#1085
harvester run failed in all three environments: memory_in_mb exceeds organization memory quota
ckan harvester run - prod

Egress Check Failed#1086
catalog-admin - development - BadDomainTest
2023-09-26T20:52:15.14+0000 [APP/TASK/egress-check-6318109245-2/0] ERR bash: line 1: BadDomainTest: command not found

[Jin-Sun-tts]

Friday 09/29

https://github.com/GSA/data.gov/

Check Catalog Auto Tasks

• DB-Solr Sync:
  0 packages need to be removed from Solr
  1 packages need to be updated/added to Solr
  232 packages without harvest_object need to be mannually deleted
  Finished 486s

Check Harvesting Emails

• Catalog:
• 96 harvesting job error emails reported at 12:20am EST

Other

• New Relic Alerts

Checked catalog, inventory production, works fine.

Also checked Solr leader and followers, all work as normal.

Weekly Checklist

• Audit Log (more info on AU-3 and AU-6 Log auditing)
• Tracking Update - Done on Monday 09/25
• Check Catalog Solr
• Catalog Dupe Check
  

Run de-dup for above organizations.