Incomplete Transactions and MgoPurge

Symptom

cannot find transaction ObjectIdHex("..")

Solution

Before running the tool, please sanity check the system:

1. Shut down machine agents on all state servers: Step 1;
2. Make sure that MongoDB replicaset is in good state: Step 2.

Run the mgopurge tool on any state server: Step 3. You should get output of what the tool is doing. If you encounter any further problems, this output may be of value, please keep it handy.

After the tool is run, restart machine agents: Step 4.

These instructions are adapted from mgopurge original one to cater for HA as well as different series.

---

Step 1

On each state server, run: sudo stop jujud-machine-* (for trusty) sudo systemctl stop jujud-machine-* (for xenial)

I tend to do this from /var/lib/juju/tools/machine-* just to make sure I know what machine I am on.

---

Step 2

On each state server:

a. If you do not have a mongodb-clients, please install it by running: sudo apt-get install mongodb-clients

b. Get MongoDB password by running: sudo grep oldpassword /var/lib/juju/agents/machine-*/agent.conf | cut -d' ' -f2

c. Login into MongoDB by running: mongo 127.0.0.1:37017/juju--authenticationDatabase admin --ssl --username "admin" --password ""

d. Find out each MongoDB opinion on the health and status of replicaset by running: rs.status()

There should be one PRIMARY node on one of the state severs and the rest should be SECONDARY.

---

Step 3

https://github.com/mjs/mgopurge

---

Step 4

On each state server, run: sudo start jujud-machine-* (for trusty) sudo systemctl start jujud-machine-* (for xenial)

As for [1], I tend to do this from /var/lib/juju/tools/machine-* just to make sure I know what machine I am on.