bin/list-build-logs does not list recent builds

[Atry]

$ bin/list-build-logs
2021-06-07-12:25:39	2021/06/07/w_i-xxxxxxxxx
2021-06-07-12:22:59	2021/05/28/w_i-xxxxxxxxx
2021-06-07-12:22:53	2021/05/25/w_i-xxxxxxxxx
...
2021-02-21-20:54:06	2021/02/22/w_i-xxxxxxxxx
2021-02-21-20:53:15	2021/02/22/w_i-xxxxxxxxx
2021-02-21-20:44:51	2021/02/22/w_i-xxxxxxxxx

The latest build is 2021-06-07. Builds in the recent half year is missing.

[fredemmott]

#265 for a quick workaround (which might be nice anyway)

Root cause:

+ python3 awslogs-agent-setup.py -n -r us-west-2 -c /tmp/tmp.GeCt8OJ0Rk
ESC[91mERROR: This script only supports python version 2.6 - 3.5ESC[0m

The right fix is to migrate to this:

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/download-cloudwatch-agent-commandline.html

[fredemmott]

we should also add /var/log/kern.log to the configuration to catch OOMs etc

[Atry]

Shall we modify bin/list-build-logs and bin/tail-build-log to using S3, or simply remove bin/list-build-logs and bin/tail-build-log?

[fredemmott]

The right fix is to migrate to this:

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/download-cloudwatch-agent-commandline.html

This is still accurate :)

S3 is a bit of a nasty fallback; the right fix is to get the actual logging system working again.

[fredemmott]

ClouWatch gives us:

• better organization and retention management
• streaming: logs are uploaded as they go
  • we can watch as they go
  • if a machine hard-crashes or is otherwise unable to upload to S3, we still have partial logs