pvc restored but not the data

[mfmessaoudi]

I'm trying to do a backup/restore of a pod using a pvc.
It seems the restore operation is creating a new pvc, but I do not find the content.

k get pvc

NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
myclaim Bound pvc-4bcbd0c3-2ce5-4f86-9a2b-184228eea75d 1Gi RWO my-sc 14h

origin pod in test namespace:

k -n test exec -it mypod -- ls /var/www/html

lost+found testfile1 testfile2

velero backup create sav --include-namespaces test

velero restore create restore-sav --from-backup sav --include-namespaces test --namespace-mappings test:restore-sav

k -n restore-sav get pvc

NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
myclaim Bound pvc-62b426dd-c63a-4582-99a2-8a06fc8b4760 1Gi RWO my-sc 29m

k -n restore-sav exec -it mypod -- ls /var/www/html

lost+found

I should see testfile1 testfile2 in the restored pod.

The following information will help us better understand what's going on:
bundle-2024-01-15-09-12-05.tar.gz

• velero deploy log:
  deploy_velero.log

• velero backup describe <backupname> or kubectl get backup/<backupname> -n velero -o yaml

I cannot display logs of backup/restore, I think it's because minio is internal, I tried with port-forward still not working, I don't know how to do it..

The following error occurred while trying to access the URL : http://minio.minio.svc.cluster.local:9000/backup/restores/restore-sav/restore-restore-sav-logs.gz?

Unable to determine IP address from hostname minio.minio.svc.cluster.local

velero get backup

NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR
bkp0 Completed 0 2 2024-01-14 20:36:35 +0100 CET 29d default
bkp01 PartiallyFailed 2 0 2024-01-14 21:45:58 +0100 CET 29d default
fs-backup PartiallyFailed 99 11 2024-01-14 23:02:12 +0100 CET 29d default
sav Completed 0 2 2024-01-15 08:42:30 +0100 CET 29d default

velero get restore

NAME BACKUP STATUS STARTED COMPLETED ERRORS WARNINGS CREATED SELECTOR
csi-restore bkp0 Completed 2024-01-14 20:51:07 +0100 CET 2024-01-14 20:51:09 +0100 CET 0 1 2024-01-14 20:51:07 +0100 CET
fs-restore fs-backup Completed 2024-01-14 23:23:06 +0100 CET 2024-01-14 23:23:09 +0100 CET 0 1 2024-01-14 23:23:06 +0100 CET
restore-sav sav Completed 2024-01-15 08:47:42 +0100 CET 2024-01-15 08:47:42 +0100 CET 0 1 2024-01-15 08:47:42 +0100 CET

kv get po

NAME READY STATUS RESTARTS AGE
backup-driver-766fc79884-r8ngz 1/1 Running 0 2d16h
datamgr-for-vsphere-plugin-4kkmk 1/1 Running 0 2d16h
datamgr-for-vsphere-plugin-dxwrg 1/1 Running 0 2d16h
datamgr-for-vsphere-plugin-q7d6j 1/1 Running 0 2d16h
velero-5b9444cff-sdkvj 1/1 Running 0 2d16h

kv logs backup-driver-766fc79884-r8ngz
backup_driver.log

APIGroup:0xc0009465a0 Kind:PersistentVolumeClaim BackupRepository:br-e2da238a-ab65-4cb5-92eb-d5fdd642f2b4 CloneCancel:false} Status:{Phase:Failed Message:cloneFromSnapshot PreCloneFromSnapshot: Failed for PVC default/mongodb-persistent-storage-claim-mongod-0 because StorageClassName is not set ResourceHandle:nil CompletionTimestamp:2024-01-13 13:05:33 +0000 UTC}}" controller=BackupDriverController logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/backupdriver/backup_driver_controller.go:258"
time="2024-01-15T08:29:09Z" level=error msg="cloneFromSnapshot PreCloneFromSnapshot: Failed for PVC default/mongodb-persistent-storage-claim-mongod-0 because StorageClassName is not set" controller=BackupDriverController logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/backupdriver/backup_driver_controller.go:296"
time="2024-01-15T08:29:09Z" level=error msg="cloneFromSnapshot default/ad6b5baf-4a6d-49ca-82b0-eb62c8fb7d7c failed: cloneFromSnapshot PreCloneFromSnapshot: Failed for PVC default/mongodb-persistent-storage-claim-mongod-0 because StorageClassName is not set" controller=BackupDriverController logSource="/go/src/github.com/vmware-tanzu/velero-plugin-for-vsphere/pkg/backupdriver/backup_driver_controller_base.go:619"

Environment:

• velero version

Client:
Version: v1.12.1
Git commit: 5c4fdfe
Server:
Version: v1.12.1

• velero client config get features

features: enableCSI

• Kubernetes version v1.26.9
• Kubernetes installer & version:
• Cloud provider or hardware configuration: vsphere
• OS (e.g. from /etc/os-release): rhel8

Another question I cannot send the snapshot to minio,

kv describe vsl

Name: minio-snapshot
Namespace: velero
Labels:
Annotations:
API Version: velero.io/v1
Kind: VolumeSnapshotLocation
Metadata:
Creation Timestamp: 2024-01-13T20:01:05Z
Generation: 3
Managed Fields:
API Version: velero.io/v1
Fields Type: FieldsV1
fieldsV1:
f:spec:
.:
f:config:
.:
f:region:
f:provider:
f:status:
Manager: velero
Operation: Update
Time: 2024-01-13T20:01:05Z
Resource Version: 15201763
UID: d9ca3290-a173-4523-beaa-bce7f170ee90
Spec:
Config:
Region: minio
Provider: aws
Status:
Events:

kv describe backupstoragelocation

Name: default
Namespace: velero
Labels: component=velero
Annotations:
API Version: velero.io/v1
Kind: BackupStorageLocation
Metadata:
Creation Timestamp: 2024-01-12T00:21:14Z
Generation: 9455
Managed Fields:
API Version: velero.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:labels:
.:
f:component:
f:spec:
.:
f:config:
.:
f:region:
f:s3ForcePathStyle:
f:s3Url:
f:default:
f:objectStorage:
.:
f:bucket:
f:provider:
Manager: velero
Operation: Update
Time: 2024-01-12T00:21:14Z
API Version: velero.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:lastSyncedTime:
f:lastValidationTime:
f:phase:
Manager: velero-server
Operation: Update
Time: 2024-01-15T08:36:02Z
Resource Version: 15761798
UID: e498c486-e9ed-45fa-882d-9f41d3014886
Spec:
Config:
Region: minio
s3ForcePathStyle: true
s3Url: http://minio.minio.svc.cluster.local:9000
Default: true
Object Storage:
Bucket: backup
Provider: aws
Status:
Last Synced Time: 2024-01-15T08:36:02Z
Last Validation Time: 2024-01-15T08:35:44Z
Phase: Available
Events:

Do I need to set the url config same as BSL ?

• 👍 for "I would like to see this bug fixed as soon as possible"

[mfmessaoudi]

I've just made a test using the option --existing-resource-policy=update
in fact the warning disappeared but I still don't have the data in the restored pvc
please find attached restore log downloaded from minio: https://gist.github.com/mfmessaoudi/92c582079922eaf2ca7e96d14a59149d

[mfmessaoudi]

Using the option --namespace-mappings does not restore the content to pvc, it's creating new empty one.
When I do :
kubectl -n velero get downloads => no downloads happening

after deleting the pod and its pvc, making a restore in the same namespace, I can see the download of the ivd/fcd , and the data as well:

kv get downloads

NAME AGE
download-6cf44a9b-1691-4dbc-9fc5-40aa7c9e2516-12d71d14-22b3-40d9-b4c4-f0c49c5e8b71 11s

Something wrong with the option ? any idea ?
Thanks

[allenxu404]

Based on the information provided, seems that Velero failed to take volume snapshot with vsphere plugin. This failure could be due to an environment configuration issue.

For simplicity, you could use file system backup instead to backup PV by specifying --default-volumes-to-fs-backup flag when running velero backup command. Please refer to this doc for more info.

[mfmessaoudi]

I had the VSL plugin set to "aws", fixed it to vsphere, in my previous note, the backup/restore is working fine without using --namespace-mappings. When I use this option, velero creates the pvc in the target namespace but empty.
The snapshot is working correctly:

k describe snapshot snap-9c16f8e9-8320-4252-bc2a-7ffcf4be3c12

Name: snap-9c16f8e9-8320-4252-bc2a-7ffcf4be3c12
Namespace: default
Labels: velero.io/backup-name=ns-default-backup
velero.io/exclude-from-backup=true
Annotations:
API Version: backupdriver.cnsdp.vmware.com/v1alpha1
Kind: Snapshot
Metadata:
Creation Timestamp: 2024-01-15T20:49:58Z
Generation: 1
Managed Fields:
API Version: backupdriver.cnsdp.vmware.com/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:labels:
.:
f:velero.io/backup-name:
f:velero.io/exclude-from-backup:
f:spec:
.:
f:backupRepository:
f:resourceHandle:
.:
f:apiGroup:
f:kind:
f:name:
Manager: velero-plugin-for-vsphere
Operation: Update
Time: 2024-01-15T20:49:58Z
API Version: backupdriver.cnsdp.vmware.com/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:progress:
f:svcSnapshotName:
Manager: velero-plugin-for-vsphere
Operation: Update
Subresource: status
Time: 2024-01-15T20:49:58Z
API Version: backupdriver.cnsdp.vmware.com/v1alpha1
Fields Type: FieldsV1
fieldsV1:
f:status:
f:completionTimestamp:
f:metadata:
f:phase:
f:snapshotID:
Manager: backup-driver-server
Operation: Update
Subresource: status
Time: 2024-01-15T20:50:32Z
Resource Version: 15970740
UID: fc99b640-c209-4b19-9e62-c209739dabe4
Spec:
Backup Repository: br-e2da238a-ab65-4cb5-92eb-d5fdd642f2b4
Resource Handle:
API Group:
Kind: PersistentVolumeClaim
Name: myclaim
Status:
Completion Timestamp: 2024-01-15T20:50:32Z
Metadata: CosLCgdteWNsYWltEgAaB2RlZmF1bHQiACokYzg2NzM2N2MtYjA3Zi00NTFjLTgzYTctNzMyMWRkZjM5MmYyMggxNTg1MjA0NDgAQggIivGUrQYQAGK3Agowa3ViZWN0bC5rdWJlcm5ldGVzLmlvL2xhc3QtYXBwbGllZC1jb25maWd1cmF0aW9uEoICeyJhcGlWZXJzaW9uIjoidjEiLCJraW5kIjoiUGVyc2lzdGVudFZvbHVtZUNsYWltIiwibWV0YWRhdGEiOnsiYW5ub3RhdGlvbnMiOnt9LCJuYW1lIjoibXljbGFpbSIsIm5hbWVzcGFjZSI6ImRlZmF1bHQifSwic3BlYyI6eyJhY2Nlc3NNb2RlcyI6WyJSZWFkV3JpdGVPbmNlIl0sInJlc291cmNlcyI6eyJyZXF1ZXN0cyI6eyJzdG9yYWdlIjoiMUdpIn19LCJzdG9yYWdlQ2xhc3NOYW1lIjoiYXBwLXNjIiwidm9sdW1lTW9kZSI6IkZpbGVzeXN0ZW0ifX0KYiYKH3B2Lmt1YmVybmV0ZXMuaW8vYmluZC1jb21wbGV0ZWQSA3llc2IrCiRwdi5rdWJlcm5ldGVzLmlvL2JvdW5kLWJ5LWNvbnRyb2xsZXISA3llc2JHCi12b2x1bWUuYmV0YS5rdWJlcm5ldGVzLmlvL3N0b3JhZ2UtcHJvdmlzaW9uZXISFmNzaS52c3BoZXJlLnZtd2FyZS5jb21iQgoodm9sdW1lLmt1YmVybmV0ZXMuaW8vc3RvcmFnZS1wcm92aXNpb25lchIWY3NpLnZzcGhlcmUudm13YXJlLmNvbXIca3ViZXJuZXRlcy5pby9wdmMtcHJvdGVjdGlvbnoAigGhAgoZa3ViZWN0bC1jbGllbnQtc2lkZS1hcHBseRIGVXBkYXRlGgJ2MSIICIrxlK0GEAAyCEZpZWxkc1YxOuEBCt4BeyJmOm1ldGFkYXRhIjp7ImY6YW5ub3RhdGlvbnMiOnsiLiI6e30sImY6a3ViZWN0bC5rdWJlcm5ldGVzLmlvL2xhc3QtYXBwbGllZC1jb25maWd1cmF0aW9uIjp7fX19LCJmOnNwZWMiOnsiZjphY2Nlc3NNb2RlcyI6e30sImY6cmVzb3VyY2VzIjp7ImY6cmVxdWVzdHMiOnsiLiI6e30sImY6c3RvcmFnZSI6e319fSwiZjpzdG9yYWdlQ2xhc3NOYW1lIjp7fSwiZjp2b2x1bWVNb2RlIjp7fX19QgCKAbgCChdrdWJlLWNvbnRyb2xsZXItbWFuYWdlchIGVXBkYXRlGgJ2MSIICIvxlK0GEAAyCEZpZWxkc1YxOvoBCvcBeyJmOm1ldGFkYXRhIjp7ImY6YW5ub3RhdGlvbnMiOnsiZjpwdi5rdWJlcm5ldGVzLmlvL2JpbmQtY29tcGxldGVkIjp7fSwiZjpwdi5rdWJlcm5ldGVzLmlvL2JvdW5kLWJ5LWNvbnRyb2xsZXIiOnt9LCJmOnZvbHVtZS5iZXRhLmt1YmVybmV0ZXMuaW8vc3RvcmFnZS1wcm92aXNpb25lciI6e30sImY6dm9sdW1lLmt1YmVybmV0ZXMuaW8vc3RvcmFnZS1wcm92aXNpb25lciI6e319fSwiZjpzcGVjIjp7ImY6dm9sdW1lTmFtZSI6e319fUIAigGYAQoXa3ViZS1jb250cm9sbGVyLW1hbmFnZXISBlVwZGF0ZRoCdjEiCAiL8ZStBhAAMghGaWVsZHNWMTpVClN7ImY6c3RhdHVzIjp7ImY6YWNjZXNzTW9kZXMiOnt9LCJmOmNhcGFjaXR5Ijp7Ii4iOnt9LCJmOnN0b3JhZ2UiOnt9fSwiZjpwaGFzZSI6e319fU3RhdHVzEmEKDVJlYWRXcml0ZU9uY2USEhIQCgdzdG9yYWdlEgUKAzFHaRoocHZjLWM4NjczNjdjLWIwN2YtNDUxYy04M2E3LTczMjFkZGYzOTJmMioGYXBwLXNjMgpGaWxlc3lzdGVtGigKBUJvdW5kEg1SZWFkV3JpdGVPbmNlGhAKB3N0b3JhZ2USBQoDMUdp
Phase: Uploaded
Progress:
Snapshot ID: pvc:default/myclaim:aXZkOmMxM2EwYjFkLTM0NjMtNDQ1ZS1hYjI5LTE0MzVhYWJlZWIzMzo2Y2Y0NGE5Yi0xNjkxLTRkYmMtOWZjNS00MGFhN2M5ZTI1MTY
Svc Snapshot Name:
Events:

[allenxu404]

May I know which cloud provider are you using? I'm not familiar with the intended use of Snapshot and downloads. I assume these are the evidences to demonstrate that the PV has been successfully backed up using volume snapshot.

The symptom of this issue is that PV's data can be successful restored when deleting the pod and pvc then restoring them in the same namespace. However, when attempting to restore them into a new namespace using flag --namespace-mappings, the PV's data is lost. Is my understanding of the issue correct?

[mfmessaoudi]

Kubernetes is actually installed in vsphere.
Yes you are right, deleting the resources (here I deleted the namespace) and restore, it's working perfectly.
But when using --namespace-mappings, the pvc created in the new namespace has no data.

I think it's the same behaviour as this issue: vmware-tanzu/velero-plugin-for-vsphere#410

Thanks

[allenxu404]

Yes, It appears to be an issue specific to vSphere, as when I tried to reproduce it on EKS the operation worked as expected.

[xing-yang]

cc @varunsrinivasan2

[reasonerjt]

@mfmessaoudi
The vsphere plugin is not maintained by velero developer.
I have informed the vsphere-plugin developer to take a look.

By the way, if you are a VMware's customer I'd suggest you contact GSS and open a jira ticket to have better support.

[reasonerjt]

@xing-yang could you clarify if this has been fixed already?

[xing-yang]

Yes, it's fixed by this PR: vmware-tanzu/velero-plugin-for-vsphere#565

[reasonerjt]

Closed as fixed.