Solr on ECS EFS

EFS Backup/Restore Procedures

EFS Backups are taken daily and automatically thanks to https://github.com/GSA/datagov-brokerpak-solr/pull/40

EFS Restores can be completed using the following procedure (originally created in https://github.com/GSA/data.gov/issues/3869):

• Turn off Harvesting and make sure no changes are happening in Solr and/or DB
• Follow AWS Documentation for restoring an EFS Backup
  • In practice, there is no need to update Access Policy roles, and EFS can be restored by simply clicking restore on the relevant backup in the backup manager (Step #4).
• Wait for backup to restore
• Perform an update-service (see below) on the solr instance referencing the restore script as the "setupLink" in the service configuration,
  • https://github.com/GSA/catalog.data.gov/blob/main/solr/restore_backups.sh
  • https://github.com/GSA/inventory-app/blob/main/solr/restore_backups.sh
• Wait for the update to complete and verify that Solr is started by inspecting the logs in the AWS Console.

Gold Nugget: See this slack discussion for information about update-service.

cf update-service catalog-solr -c "\
{
    "solrImageRepo": unchanged,
    "solrImageTag": unchanged,
    "solrMem": unchanged,
    "solrCpu": unchanged,
    "setupLink": "https://raw.githubusercontent.com/GSA/catalog.data.gov/main/solr/restore_backups.sh",
    "efsProvisionedThroughput": unchanged
}

Solr Credential Rotation

This guide details how to reset credentials (username and password) for Solr services bound to CKAN applications. These credentials have admin privileges, so it's crucial to reset them if compromised.

Finding Solr Service Information:

Identify Solr Service Details: Use the cf env APP-NAME command for each CKAN application (catalog-web, catalog-gather, catalog-fetch, catalog-admin, inventory). This will reveal Solr service information, including the ECS task name, leader and follower URLs, and current username/password.

Resetting Credentials:

• Unbind and Rebind Service: To generate new credentials for an app, use cf unbind-service APP-NAME SOLR-SERVICE-NAME followed by cf bind-service APP-NAME SOLR-SERVICE-NAME. This unbinds and then rebinds the service, providing fresh credentials.

• Create Username List: Compile a list of usernames currently used by all CKAN applications to access the Solr service.

• Identify Initial Credentials: Locate the initial username and password for each Solr service. You can find these embedded within the command script of the solr-admin-init service task definition in the AWS ECS console (accessible through your AWS SSB account). Add this one to the compiled list.

• Delete Unused Users: Access each Solr service URL and log in to the Solr console. Navigate to the "Security" tab and delete any user not included in your compiled list.