Does etcd-snapshot prune not work on S3?

[lindhe]

I'm on v1.28.10+rke2r1.

In /etc/rancher/rke2/config.yaml, I have set etcd-snapshot-retention: 20. I was under the impression that this should cause periodic pruning of snapshots both on disk and on S3. But I recently discovered that I have around 700 snapshots in my S3 bucket, so that's clearly not working for me.

While RKE2 docs are not clear on what effect pruning should have on S3, the K3s docs indicate that it should prune on S3 too:

These commands will perform as expected whether the etcd snapshots are stored locally or in an S3 compatible object store.

Additionally, I found #5216 that seems to confirm my belief that prune should work towards S3 as well as local disk.

At first, I found that I hadn't granted Allow on s3:DeleteObject in my bucket policy, so I assumed that was the issue. But even after granting that, it still will not prune on S3.

I am able to delete individual snapshots like this:

[admin@rancher-node011 ~]$ sudo rke2 etcd-snapshot delete  on-demand-rancher-node011-1720524031
WARN[0000] Unknown flag --etcd-snapshot-schedule-cron found in config.yaml, skipping
WARN[0000] Unknown flag --tls-san found in config.yaml, skipping
INFO[0003] Snapshot on-demand-rancher-node011-1720524031 deleted.
INFO[0003] Snapshot on-demand-rancher-node011-1720524031 deleted.

But when running prune, nothing happens:

[admin@rancher-node011 ~]$ sudo rke2 etcd-snapshot prune --debug
WARN[0000] Unknown flag --etcd-snapshot-schedule-cron found in config.yaml, skipping
WARN[0000] Unknown flag --tls-san found in config.yaml, skipping

Considering that delete works, and the fact that I do not get an error when running prune, it feels like my bucket policy is not the problem. But I don't know, since it's not documented what the bucket policy should be.

Any idea what the issue could be?

[brandond]

It will prune s3 if s3 is configured - either explicitly on the CLI, or in the config file.

A few recent releases of RKE2 were affected by k3s-io/k3s#10292, however that has been resolved in the most recent release.

[lindhe]

Ah, interesting! That might be what's been causing issues for me. I see that v1.28.11+rke2r1 was recently released, I'll try that and see if things improve! Thanks!