DataUpload path seems to be incorrect

[ScOut3R]

What steps did you take and what happened:
I'm executing a CSI snapshot backup coupled with a DataMover step. The process succeeds end to end, however the resulting backup does not contain the data from the PVC, but an AWS IAM token.

The status.path has the following:

/host_pods/6a774d15-b3f1-419a-89b5-07c1f08c8ab6/volumes/kubernetes.io~projected/aws-iam-token

This is consistent across different backups.

What did you expect to happen:

I expected that the backup will contain the actual data from the source PVC.

The following information will help us better understand what's going on:

bundle-2023-10-27-14-21-51.tar.gz

Anything else you would like to add:
The backup storage location is an S3 bucket and using IRSA authentication/authorization.

Environment:

• Velero version (use velero version): v1.12.0
• Velero features (use velero client config get features): <NOT SET>
• Kubernetes version (use kubectl version): 1.26.9

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "I would like to see this bug fixed as soon as possible"
• 👎 for "There are more important bugs to focus on right now"

[qiuming-best]

this is because you backup one volume named aws-iam-token

[ScOut3R]

this is because you backup one volume named aws-iam-token

The aws-iam-token volume is attached to the velero and related pods, not to the pods that I am backup up with their PVCs.

According to the Backup resource there are two volumes marked for backup:

  Operation for persistentvolumeclaims k6/grafana:
    Backup Item Action Plugin:  velero.io/csi-pvc-backupper
    Operation ID:               du-5df9a8dc-739a-4d17-a49c-5c747f203ef8.c076ddbd-731b-4309fed94
    Items to Update:
                           datauploads.velero.io velero/k6-daily-20231027040615-46r2s
    Phase:                 Completed
    Progress:              1044 of 1044 complete (Bytes)
    Progress description:  Completed
    Created:               2023-10-27 14:06:41 +1000 AEST
    Started:               2023-10-27 14:06:41 +1000 AEST
    Updated:               2023-10-27 14:07:03 +1000 AEST
  Operation for persistentvolumeclaims k6/influxdb-data-influxdb-0:
    Backup Item Action Plugin:  velero.io/csi-pvc-backupper
    Operation ID:               du-5df9a8dc-739a-4d17-a49c-5c747f203ef8.80ec7df7-b1ee-4180fe2ec
    Items to Update:
                           datauploads.velero.io velero/k6-daily-20231027040615-cczgt
    Phase:                 Completed
    Progress:              1044 of 1044 complete (Bytes)
    Progress description:  Completed
    Created:               2023-10-27 14:06:46 +1000 AEST
    Started:               2023-10-27 14:06:46 +1000 AEST
    Updated:               2023-10-27 14:07:14 +1000 AEST

There are two DataUpload instances created afterwards, but they both backup the aws-iam-token volume, instead of the actual data volume.

My assumption is that the aws-iam-token volume is attached alongside the actual PVC to the DataUploader pod because of IRSA and the DataPath manager picks the aws-iam-token's path since that's the first volume on the pod.

I'm not familiar enough with Go to validate this by going through the source code, but I'll give it a try.

[qiuming-best]

@ScOut3R I think maybe you are right, the aws-iam-token volume is attached alongside the actual PVC to the DataUploader pod automatically, which makes DataUploader choose the wrong volume.

DataUploader will always choose the first Volume, here is the code snippet.

[ScOut3R]

Thank you for resolving it so quickly @qiuming-best @Lyndon-Li!