velero restore completed with Warning: could not restore, Velero restore completed with warning ConfigMap "kube-root-ca.crt" already exists

[lipingxue]

What steps did you take and what happened:

Env:

Velero: velero/velero:v1.12.0-rc.1

Velero Plugin for CSI: velero/velero-plugin-for-csi:v0.6.0-rc.1

Velero Plugin for AWS: harbor-repo.vmware.com/cnsdpvelero/velero-plugin-for-aws:v1.1.0

Kubernetes: v1.26.5

1. velero backup a namespace with one pvc
2. delete the namespace, and then run velero restore from the backup created in step 1.

The velero restore completed with warning : could not restore, ConfigMap "kube-root-ca.crt" already exists. Warning: the in-cluster version is different than the backed-up version.
namespace and pvc have been restored successfully.

velero restore describe test-restore-mover
Name:         test-restore-mover
Namespace:    velero
Labels:       <none>
Annotations:  <none>
 
Phase:                       Completed
Total items to be restored:  6
Items restored:              6
 
Started:    2023-09-12 22:01:39 +0000 UTC
Completed:  2023-09-12 22:02:48 +0000 UTC
 
Warnings:
  Velero:     <none>
  Cluster:    <none>
  Namespaces:
    test-ns-2:  could not restore, ConfigMap "kube-root-ca.crt" already exists. Warning: the in-cluster version is different than the backed-up version
 
Backup:  test-backup-mover
 
Namespaces:
  Included:  all namespaces found in the backup
  Excluded:  <none>
 
Resources:
  Included:        *
  Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io, csinodes.storage.k8s.io, volumeattachments.storage.k8s.io, backuprepositories.velero.io
  Cluster-scoped:  auto
 
Namespace mappings:  <none>
 
Label selector:  <none>
 
Restore PVs:  auto
 
Existing Resource Policy:   <none>
ItemOperationTimeout:       4h0m0s
 
Preserve Service NodePorts:  auto
 
Restore Item Operations:  1 of 1 completed successfully, 0 failed (specify --details for more information)

What did you expect to happen:
velero restore should completed without warning. The warning message may confuse the user.

The following information will help us better understand what's going on:

If you are using velero v1.7.0+:
Please use velero debug --backup <backupname> --restore <restorename> to generate the support bundle, and attach to this issue, more options please refer to velero debug --help

If you are using earlier versions:
Please provide the output of the following commands (Pasting long output into a GitHub gist or other pastebin is fine.)

• kubectl logs deployment/velero -n velero
• velero backup describe <backupname> or kubectl get backup/<backupname> -n velero -o yaml
• velero backup logs <backupname>
• velero restore describe <restorename> or kubectl get restore/<restorename> -n velero -o yaml
• velero restore logs <restorename>

Anything else you would like to add:

Environment:

• Velero version (use velero version):
• Velero features (use velero client config get features):
• Kubernetes version (use kubectl version):
• Kubernetes installer & version:
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):

Vote on this issue!

This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.

• 👍 for "I would like to see this bug fixed as soon as possible"
• 👎 for "There are more important bugs to focus on right now"

[yanggangtony]

@lipingxue
Hello , would you tell us , what your config of the Kind: Restore.
Because there is a config item to set existingResourcePolicy, it is : none or update.

The existingResourcePolicy meanning is Users can choose to overwrite or patch the existing resources during restore by setting this policy.

[reasonerjt]

Based on my impression this is working as expected.

[kaovilai]

velero restore should completed without warning. The warning message may confuse the user.

I disagree kube-root-ca.crt is a cluster-admin managed configmap. User of velero who decided to backup the whole cluster including namespaces contianing kube-root-ca.crt should be knowledgeable enough to know and aware of if kube-root-ca.crt was restored or not.

If an item is in the backup, and restore skips because it already exists, user should know about it.

[sseago]

Velero warns any time it did not restore a resource that 1) already exists in the cluster and 2) backup version differs from in-cluster version.

This warning is useful for users because it will point out to them that they may need to look at these items to see whether they want to re-attempt restore with an Update policy or modify them manually, or whether they can ignore the differences. It's a warning, not an error, which means Velero restored everything as expected, but the warnings indicate areas that a user may want to verify.

If your use case does not require verifying these "already in the cluster but different" items, you can always ignore the warnings, but if we remove the warnings, then users who need to verify this will have no way of knowing what they need to look at.

[xiaozuo7]

@lipingxue Hello , would you tell us , what your config of the Kind: Restore. Because there is a config item to set existingResourcePolicy, it is : none or update.

The existingResourcePolicy meanning is Users can choose to overwrite or patch the existing resources during restore by setting this policy.

I have the same problem，and my Existing Resource Policy is none

[xing-yang]

cc @xing-yang

[lipingxue]

@lipingxue Hello , would you tell us , what your config of the Kind: Restore. Because there is a config item to set existingResourcePolicy, it is : none or update.

The existingResourcePolicy meanning is Users can choose to overwrite or patch the existing resources during restore by setting this policy.
@reasonerjt See the following. restore failed with "warning", and existingResourcePolicy is "none".

Name:         restore-2
Namespace:    velero
Labels:       <none>
Annotations:  <none>

Phase:                       Completed
Total items to be restored:  87
Items restored:              87

Started:    2023-11-06 13:36:35 +0000 UTC
Completed:  2023-11-06 13:38:13 +0000 UTC

Warnings:
  Velero:     <none>
  Cluster:  could not restore, CustomResourceDefinition "kubeadmcontrolplanetemplates.controlplane.cluster.x-k8s.io" already exists. Warning: the in-cluster version is different than the backed-up version
            could not restore, CustomResourceDefinition "vsphereclustertemplates.infrastructure.cluster.x-k8s.io" already exists. Warning: the in-cluster version is different than the backed-up version
            could not restore, CustomResourceDefinition "vspheremachinetemplates.infrastructure.cluster.x-k8s.io" already exists. Warning: the in-cluster version is different than the backed-up version
  Namespaces:
    test-ns-label-tqnhefw:  could not restore, ConfigMap "kube-root-ca.crt" already exists. Warning: the in-cluster version is different than the backed-up version
                            could not restore, Package "vsphere-csi.tanzu.vmware.com.3.0.2+vmware.1-tkg.2-20231102" already exists. Warning: the in-cluster version is different than the backed-up version

Backup:  dual-stack-ipv6-run3

Namespaces:
  Included:  all namespaces found in the backup
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        kubeadmcontrolplanetemplates.controlplane.cluster.x-k8s.io, vsphereclustertemplates.infrastructure.cluster.x-k8s.io, vspheremachinetemplates.infrastructure.cluster.x-k8s.io, nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io, csinodes.storage.k8s.io, volumeattachments.storage.k8s.io, backuprepositories.velero.io
  Cluster-scoped:  auto

Namespace mappings:  <none>

Label selector:  <none>

Restore PVs:  auto

Existing Resource Policy:   <none>
ItemOperationTimeout:       4h0m0s

Preserve Service NodePorts:  auto

[kaovilai]

@xiaozuo7 @lipingxue this is a feature, not a bug. If you wanna override what's exists in cluster with what's in the backup then you can set existingresourcepolicy to update.

[kaovilai]

It is a warning and should not be muted. One needs to know that velero did not attempt to override/update if not set explicitly.

[kaovilai]

If you don't wanna risk overriding them but still want it in backup then explicitly exclude them from restore.

[xiaozuo7]

It is a warning and should not be muted. One needs to know that velero did not attempt to override/update if not set explicitly.

Thanks for the reply. BTW, I would like to ask if there is an additional parameter option to skip or ignore the warning if it already exists.

[sseago]

@xiaozuo7 There is no paramter to suppress restore warnings. I don't think that would be a good idea. The reason it's a warning and not an error is that the restore is not considered failed. This is potentially useful information -- it's up to the user whether to ignore the warnings, or look at them to determine relevance. In most cases, I'd say that users will ignore them unless something goes wrong (problem with post-restore application, etc.), then looking at the warnings would be the first thing to do.

[kaovilai]

skip or ignore the warning

that means you don't care that the item is restored.. so excluding known problematic items from restore would give similar result to "ignore the warning".

[varunsrinivasan2]

This warning was observed with Velero vSphere Plugin when restoring a namespace. Since the kube-controller-manager automatically recreates the kube-root-ca.crt ConfigMap upon restoration of a namespace, Velero will fail to restore the resource and issue the warning.

The warning is consistent between both plugins.

[xing-yang]

Thanks @varunsrinivasan2 for verifying this. This issue can be closed now.

[reasonerjt]

Thanks!