Add per task report upload metrics (0.6 backport)

[inahga]

Supports #2293

Backport of #2508 and #2537.

This PR should not be merged until #2553 is merged and in a release. In other words, these PRs should not be in the same release.

[inahga]

Using this for load testing, since 0.7 is WIP. In reality I think this will need to be two PRs over two releases, to account for DB migration.

[inahga]

Load test results:

Each trial is the last 15 minutes of our standard load test run at 100QPS. The task is Time Interval Prio3Count. This should exercise the slowest path--validation that requires DB access and a successful report.

System details:

• CPU: AMD Ryzen 9 7950X 16-Core Processor (16C/32T)
• Memory: 64GB DDR5
• Disk: Samsung SSD 980 PRO 1TB NVMe
• rustc 1.75.0-alpine

100QPS Baseline (Janus release 0.6.9)

Trial 1:

Trial 2:

n.b. there is no transaction error graph since this transaction can't fail due to serialization errors.

100QPS 1 shard

Trial 1:

Trial 2:

100QPS 32 shards

Trial 1:

Trial 2:

Overall the results look to be non-regressive, as long as the shard count is greater than 1 to avoid serialization errors.

Note that these graphs record the total memory usage of the postgresql container, but I think that turned out to be meaningless as it includes the memory used for buffer. Either way, I don't observe any strange behavior wrt memory.

Note that some graphs contain a period of high latency--this is noise from the kubernetes cluster HPA not having scaled the aggregator deployment to a happy st ate.

[inahga]

I'll need to break this PR into two to facilitate a zero downtime PostgreSQL migration.

[inahga]

I'll need to break this PR into two to facilitate a zero downtime PostgreSQL migration.

Actually, I don't think this is true.

Our rollout cadence is like so:

1. Roll out new version of Janus.
2. Execute DB migrations.

Note that we don't wait for 1's success during deployment. The pods will remain in a crash loop until the database has been updated. Meanwhile, traffic won't be shifted to the new deployment.

[divergentdave]

I think it would be preferable to do this in two deploys as follows:

1. Update the supported schema versions to (2, 1), and add the new table
2. Update the supported schema versions to (2), and make the rest of the code changes depending on the task_upload_counters table

If we did this in a single deploy, note that the old ReplicaSet wouldn't be able to start any new pods until the schema migration was applied. If we encountered issues during the deploy, this would complicate the response.

[inahga]

If we did this in a single deploy, note that the old ReplicaSet wouldn't be able to start any new pods until the schema migration was applied. If we encountered issues during the deploy, this would complicate the response.

Fair enough. I was relying on the "old replica set won't start new pods" behavior to make this work, but that is indeed chaotic.