mac maintenance

Mac Maintenance

These are the manual steps to maintain the macs. Until we fully automate setup and maintenance, we need to ssh into each mac to do maintenance.

The list of machines is here. Please keep the statuses updated on that page.

For each of these, consider quarantining the machine before performing maintenance.

Updating python packages

Prod, puppetized

On puppetized fx and tb prod boxes, scriptworker is running out of /builds/scriptworker and the poller is running out of /builds/scriptworker/poller. They share a single virtualenv in /builds/scriptworker/virtualenv.

Config files are at /builds/scriptworker/script_config.yaml, /builds/scriptworker/scriptworker.yaml, and /builds/scriptworker/poller/poller.yaml.

Certs are in /builds/scriptworker/certs.

# scriptworker / iscript / notarization_poller
sudo -u cltbld -i
cd /builds/scriptworker
. virtualenv/bin/activate
pip install ...

# TODO scriptworker-scripts update

# restart scriptworker / iscript
sudo -u root -i
launchctl stop org.mozilla.scriptworker && launctl start org.mozilla.scriptworker

# restart notarization_poller
sudo -u root -i
launchctl stop org.mozilla.notarization_poller && launctl start org.mozilla.notarization_poller

Prod, non-puppetized

As of this writing 2020.09.16, this is only the TB boxes, and only because :dhouse is poking at tb2.

Scriptworker is installed in /builds/scriptworker. Only some non-puppetized machines have pollers, but those are in /builds/scriptworker/poller. The virtualenv is in /builds/scriptworker.

Config files are at /builds/scriptworker/script_config.yaml, /builds/scriptworker/scriptworker.yaml, and /builds/scriptworker/poller/poller.yaml if the poller is installed.

Certs are in /builds/scriptworker/ and /home/cltbld.

# scriptworker / iscript / notarization_poller
sudo -u cltbld -i
cd /builds/scriptworker
. bin/activate
pip install ...

# TODO scriptworker-scripts update

# restart scriptworker / iscript
# look for SCREEN in processes.
ps -ef | grep -i screen
# Kill that process if it doesn't belong to you.
sudo kill PID

# start scriptworker in screen.
screen
sudo -u cltbld -i
cd /builds/scriptworker
bin/scriptworker script_config.yaml

# restart notarization_poller
screen
sudo -u poller -i
cd /builds/scriptworker/poller
../bin/notarization_poller poller.yaml

Dep, non-puppetized

As of this writing 2020.09.16, puppet is broken on dep boxes, so both dep boxes are manually set up and configured.

Scriptworker is running out of /builds/dep1, /builds/dep2, and /builds/tb-dep.

TODO

Wiping secrets

Wipe secrets if someone may gain access to the box: when moving the machine or retiring the machine.

When wiping secrets, you want to nuke ed25519_privkey (prod only), *.keychain, and widevine*.crt. Use rm -P to overwrite them multiple times.