Velero backup deletion is not deleting objects in kopia repository

[mergwyn]

What steps did you take and what happened:

I've been running Velero and Kopia for some time now, and have noticed that there seems to be an increasing number of snapshots in the Kopia repository.

If I look at the Velero backups my old backup is from Feb 9th 2025, however my oldest Kopia snapshot is from April 7th 2024!

It is strange, that whilst there are many old snapshots, recent act

I've looked and it seems like Kopia maintanance is being run.

What did you expect to happen:

The following information will help us better understand what's going on:

If you are using velero v1.7.0+:
Please use velero debug --backup <backupname> --restore <restorename> to generate the support bundle, and attach to this issue, more options please refer to velero debug --help

If you are using earlier versions:
Please provide the output of the following commands (Pasting long output into a GitHub gist or other pastebin is fine.)

• kubectl logs deployment/velero -n velero
• velero backup describe <backupname> or kubectl get backup/<backupname> -n velero -o yaml
• velero backup logs <backupname>
• velero restore describe <restorename> or kubectl get restore/<restorename> -n velero -o yaml
• velero restore logs <restorename>

As this issue is not to do with indivudly backups I have not included this information. Hopefully, this is more relevant:

Output of velero backup list:

velero_backup_get.txt

Output of command: kopia repo status

kopia_repo_status.txt

Output of command: kopia maintenance info --json

kopia_maintenance_info_--json.txt

Output of command: kopia snapshot list --all

kopia_snapshot_list_--all.txt

Output of command: kopia content stats

kopia_content_stats.txt

Output of command: kopia blob stats

kopia_blob_stats.txt

Output of command: kopia index list --json

kopia_index_list_--json.txt

Output of command: kopia content list --deleted-only

kopia_content_list_--deleted-only.txt

Output of command: kopia index epoch list

kopia_index_epoch_list.txt

Anything else you would like to add:

Environment:

• Velero version (use velero version):

Client:
	Version: v1.15.0
	Git commit: 1d4f1475975b5107ec35f4d19ff17f7d1fcb3edf
Server:
	Version: v1.15.2

• Velero features (use velero client config get features):
  features: <NOT SET>
• Kubernetes version (use kubectl version):

WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"26", GitVersion:"v1.26.0", GitCommit:"b46a3f887ca979b1a5d14fd39cb1af43e7e5d12d", GitTreeState:"clean", BuildDate:"2022-12-08T19:58:30Z", GoVersion:"go1.19.4", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.7
Server Version: version.Info{Major:"1", Minor:"32", GitVersion:"v1.32.1+k3s1", GitCommit:"6a322f122729e0e668ca67fd9f0e993541bdce49", GitTreeState:"clean", BuildDate:"2025-01-28T18:27:08Z", GoVersion:"go1.23.4", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.26) and server (1.32) exceeds the supported minor version skew of +/-1

• Kubernetes installer & version:

k3s version v1.32.1+k3s1 (6a322f12)
go version go1.23.4

• Cloud provider or hardware configuration:
  I use a combination of local Minio and idrivee2 s3

minio version RELEASE.2024-08-17T01-24-54Z (commit-id=72cff79c8a7cc59bccb591995e3c3ed6aa2f4cd5)
Runtime: go1.22.6 linux/amd64
License: GNU AGPLv3 - https://www.gnu.org/licenses/agpl-3.0.html
Copyright: 2015-2024 MinIO, Inc.

• OS (e.g. from /etc/os-release):
  PRETTY_NAME="Ubuntu 24.04.2 LTS"
  NAME="Ubuntu"
  VERSION_ID="24.04"
  VERSION="24.04.2 LTS (Noble Numbat)"
  VERSION_CODENAME=noble
  ID=ubuntu
  ID_LIKE=debian
  HOME_URL="https://www.ubuntu.com/"
  SUPPORT_URL="https://help.ubuntu.com/"
  BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
  PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
  UBUNTU_CODENAME=noble
  LOGO=ubuntu-logo

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "I would like to see this bug fixed as soon as possible"
• 👎 for "There are more important bugs to focus on right now"