Reset Customer Local Registry

It is possible for a user's registry's AWS access to be revoked. The error message that usually accompanies this includes a wonderful message to check /debug/health and usually surfaces in Rollbar with something like:

create-instance-container: Create container failed: Error: Status 503 trying to pull repository 3697270/56eca0b9b515da1f00e5147c: "{"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /deb

In order to reset it, we have a script available to reset the registry configuration. The script does check to make sure the registry is bad before it restarts it, so you should be safe to run it on an entire environment (though this could be very disruptive).

Get the Vault Token

In order to keep things more secure, we get keys from Vault. To run this play, you need a Vault token for your environment. Look in $ENVIRONMENT-hosts/variables for vault_auth_token, and that value should be what you can use (as $VAULT_TOKEN below).

With Known Bad Hosts

If you know the IP addresses of the docks with bad registries, you can run the following:

ENVIRONMENT=gamma # replace w/ your env
IPS=10.0.0.1,10.0.0.2 # replace w/ your bad hosts
ansible-playbook \
  -i "$ENVIRONMENT-hosts" \
  --limit "$IPS,$ENVIRONMENT-consul-a,$ENVIRONMENT-consul-b,$ENVIRONMENT-consul-c"
  -e vault_token="$VAULT_TOKEN" \
  oneOffScripts/resetRegistry.yml 

Unknown Hosts

If you do not know the hosts, you can omit the limit statement and do the following:

ENVIRONMENT=gamma # replace w/ your env
ansible-playbook \
  -i "$ENVIRONMENT-hosts" \
  -e vault_token="$VAULT_TOKEN" \
  oneOffScripts/resetRegistry.yml