Incomplete Transactions and MgoPurge

Symptom

cannot find transaction ObjectIdHex("..")

Solution

Before running the tool, please sanity check the system:

1. Shut down machine agents on all state servers: stop machine agent;
2. Make sure that MongoDB replicaset is in good state: Step 2.

Run the mgopurge tool on any state server: Step 3. You should get output of what the tool is doing. If you encounter any further problems, this output may be of value, please keep it handy.

After the tool is run, restart machine agents: start machine agent.

These instructions are adapted from mgopurge original one to cater for HA as well as different series.

---

Step 2

On each state server:

a. If you do not have a mongodb-clients, please install it by running: sudo apt-get install mongodb-clients

b. Get MongoDB password by running: sudo grep oldpassword /var/lib/juju/agents/machine-*/agent.conf | cut -d' ' -f2

c. Login into MongoDB by running: mongo 127.0.0.1:37017/juju--authenticationDatabase admin --ssl --username "admin" --password ""

d. Find out each MongoDB opinion on the health and status of replicaset by running: rs.status()

There should be one PRIMARY node on one of the state severs and the rest should be SECONDARY.

---

Step 3

https://github.com/mjs/mgopurge