mac maintenance

Mac Maintenance

These are the manual steps to maintain the macs. Until we fully automate setup and maintenance, we need to ssh into each mac to do maintenance.

The list of machines is here. Please keep the statuses updated on that page.

For each of these, consider quarantining the machine before performing maintenance.

Updating python packages

There's three steps here:

1. Update the ronin_puppet repo with the changes.
2. Run Puppet on the signing workers
3. Restart the daemons

Both of these are documented below.

Step 1: Creating the PR

Dependency changes

Dependencies are hash pinned in Puppet (todo, add link). To update these, you must create a PR to the ronin_puppet repo as follows:

cd ronin_puppet/modules/signing_worker/files
# choose which worker type(s) you will update: ff-prod, tb-prod, dep
# run this script for each one that needs to be updated
bash update-requirements.sh dep > requirements.dep.txt

Then commit the changes and create a PR. .

Updating scriptworker or scriptworker-scripts

These repositories are revision pinned for each worker type in the scriptworker_config section of common.yaml. Update them and create a ronin-puppet PR to change them.

Updating the widevine repository

This repository is not revision pinned, but is also not updated automatically. To pick up changes to it, update that repository, and then delete the widevine directory on the signing workers, eg:

# ff-prod or tb-prod
rm -rf /builds/scriptworker/widevine
# dep
rm -rf /builds/{dep1,dep2,tb-dep}/widevine

When you run Puppet (see below), it will repopulate and reinstall widevine with the latest code.

Step 2: Run Puppet

As root on each signing worker:

cd ~
git clone https://github.com/mozilla-platform-ops/ronin_puppet/ ronin_puppet
cd ronin_puppet
mkdir -p data/secrets
cp ~/vault.yaml data/secrets/vault.yaml
FQDN=$(hostname)
cat <<EOF > manifests/nodes/nodes.pp
node '${FQDN}' {
    include ::roles_profiles::roles::mac_v3_signing
}
EOF
cd /
puppet apply --modulepath=/var/root/ronin_puppet/modules/:/var/root/ronin_puppet/r10k_modules/ --hiera_config=/var/root/ronin_puppet/hiera.yaml --logdest=console /var/root/ronin_puppet/manifests

Step 3: Restart the daemons

As root on the signing worker:

# for ff-prod or tb-prod
launchctl unload /Library/LaunchDaemons/org.mozilla.scriptworker.cltbld.plist && launchctl load /Library/LaunchDaemons/org.mozilla.scriptworker.cltbld.plist
launchctl unload /Library/LaunchDaemons/org.mozilla.notarization_poller.poller.plist && launchctl load /Library/LaunchDaemons/org.mozilla.notarization_poller.poller.plist
# for dep
launchctl unload /Library/LaunchDaemons/org.mozilla.scriptworker.depbld1.plist && launchctl load /Library/LaunchDaemons/org.mozilla.scriptworker.depbld1.plist
launchctl unload /Library/LaunchDaemons/org.mozilla.scriptworker.depbld2.plist && launchctl load /Library/LaunchDaemons/org.mozilla.scriptworker.depbld2.plist
launchctl unload /Library/LaunchDaemons/org.mozilla.scriptworker.tbbld.plist && launchctl load /Library/LaunchDaemons/org.mozilla.scriptworker.tbbld.plist

Wiping secrets

Wipe secrets if someone may gain access to the box: when moving the machine or retiring the machine.

When wiping secrets, you want to nuke ed25519_privkey (prod only), *.keychain, and widevine*.crt. Use rm -P to overwrite them multiple times.