Maintenance

Care and Feeding for Sal

Periodically, Sal will perform maintenance tasks in the background to minimize stale or dead data. Some maintenance tasks, however, must be manually initiated. This page describes both.

Postgres Optimization

Postgres' defaults for maintenance are far too conservative. You will need to edit postgresql.conf with these settings below^1.

autovacuum_analyze_scale_factor = 0.01
# This means the 1% of the table needs to change to trigger autovacuum.
 
autovacuum_max_workers = 1
# The default is 3. We set this to 1 to allow maximum resources for each worker, so it can complete it's work quickly and move onto the next table.
 
autovacuum_naptime = 30
# The delay between autovacuum runs in seconds. This is half the default - we want autovacuum to run as often as possible.
 
autovacuum_vacuum_cost_limit = 10000
# The 'cost' of autovacuuming is calculated using several factors (see the article linked for a good explanation) - we want autovacuum to happen as much as possible, so this is high.
 
autovacuum_vacuum_scale_factor = 0.1
# % of dead tuples to tolerate before triggering an autovacuum
 
maintenance_work_mem = 10485760
# The amount of memory to assign to mantinenance in Kb. We have assigned ~10Gb, as we have lots of memory on our RDS instance and can spare it. It should be set to the maximum amount of memory you can spare, as the maintenance will run much quicker if it can load more of the table into memory rather than having to read it from disk every time.

Periodic Maintenance

Sal uses supervisord to schedule ongoing maintenance tasks. You can configure a custom frequency for these tasks with the environment variable MAINT_FREQUENCY (default is 300 seconds).

Periodic maintenance consists of:

• Search maintenance:
  • Clearing searches older than 30 days old.
  • Clearing SearchFieldCache objects.
  • Rebuilding Fact caches for searching.
• Server maintenance
  • Deleting inactive/undeployed Machine records (if you have that setting configured).
  • Deleting outdated (based on historical_retention) PluginScriptSubmissions.
  • Deleting outdated (based on historical_retention) UpdateHistorys and UpdateHistoryItems.
  • Deleting outdated (based on historical_retention) HistoricalFacts.

Manual Maintenance

At times, you may want to run some manual maintenance tasks. To do so, you need to shell into the docker container and run ./manage.py <name of task>. (If you run just manage.py you'll get a list of available management commands, most of which are provided by Django).

• application_maintenance: Deletes Application objects which have no installs within your current machine pool from the application inventory system. This can't be automated because it's possible that it interferes with machine checkins / inventory submissions.