Error patching backup location's last-synced time. <backupStorageLocation.name> not found

[johnjbateman]

What steps did you take and what happened:
Installed latest velero helm chart, deployed to AKS. Backup location status is Unknown. Errors in the pod logs.

What did you expect to happen:
Backup location is ready

The output of the following commands will help us better understand what's going on:

helm install:

/usr/local/bin/helm upgrade --install --values /azp/agent/_work/1/s/override.yaml --set credentials.secretContents.cloud=AZURE_STORAGE_ACCOUNT_ACCESS_KEY=*** --wait --debug velero vmware-tanzu/velero

velero backup-location get
NAME    PROVIDER   BUCKET/PREFIX   PHASE     LAST VALIDATED   ACCESS MODE
azure   azure      velero/backup   Unknown   Unknown          ReadWrite

velero backup get
NAME                                     STATUS            ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
velero-eus-np-aks-daily-20210323135142   PartiallyFailed   11       0          2021-03-23 09:51:42 -0400 EDT   29d       azure              <none>
velero-eus-np-aks-daily-20210323133906   Failed            0        0          2021-03-23 09:39:09 -0400 EDT   29d       azure              <none>

trying to get the logs of the backup:

velero backup logs velero-eus-np-aks-daily-20210323135142

An error occurred: timed out waiting for download URL

kubectl get backupstoragelocations.velero.io
NAME    AGE
azure   30m

override.yaml:

configuration:
  provider: azure
  backupStorageLocation:
    name: azure
    bucket: velero
    prefix: backup
    config:
      resourceGroup: eus-np-backups-rg
      storageAccount: eusnpvelerosa
  volumeSnapshotLocation:
    name: azure
    config:
      resourceGroup: eus-np-backups-rg
      storageAccount: eusnpvelerosa
      apiTimeout: 1m0s
credentials:
  useSecret: true
  name: velero-accesskey
  extraSecretRef: azure-credentials
initContainers:
- name: velero-plugin-for-microsoft-azure
  image: velero/velero-plugin-for-microsoft-azure:v1.2.0
  imagePullPolicy: IfNotPresent
  volumeMounts:
    - mountPath: /target
      name: plugins
schedules:
  eus-np-aks-daily:
    schedule: "0 0 * * *"
    template:
      includeClusterResources: true
      includedNamespaces:
      - '*'
      includedResources:
      - '*'
      storageLocation: azure
      snapshotVolumes: true      
      ttl: 720h0m0s

velero pod logs:

I0323 14:24:38.145606       1 request.go:621] Throttling request took 1.040869219s, request: GET:https://10.0.0.1:443/apis/apiextensions.k8s.io/v1?timeout=32s
time="2021-03-23T14:24:39Z" level=error msg="Error patching backup location's last-synced time" backupLocation=azure controller=backup-sync error="backupstoragelocations.velero.io \"azure\" not found" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/controller/backup_sync_controller.go:311" error.function="github.com/vmware-tanzu/velero/pkg/controller.(*backupSyncController).run" logSource="pkg/controller/backup_sync_controller.go:311"
time="2021-03-23T14:25:09Z" level=error msg="Error patching backup location's last-synced time" backupLocation=azure controller=backup-sync error="backupstoragelocations.velero.io \"azure\" not found" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/controller/backup_sync_controller.go:311" error.function="github.com/vmware-tanzu/velero/pkg/controller.(*backupSyncController).run" logSource="pkg/controller/backup_sync_controller.go:311"
time="2021-03-23T14:25:40Z" level=error msg="Error patching backup location's last-synced time" backupLocation=azure controller=backup-sync error="backupstoragelocations.velero.io \"azure\" not found" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/controller/backup_sync_controller.go:311" error.function="github.com/vmware-tanzu/velero/pkg/controller.(*backupSyncController).run" logSource="pkg/controller/backup_sync_controller.go:311"
time="2021-03-23T14:26:10Z" level=error msg="Error patching backup location's last-synced time" backupLocation=azure controller=backup-sync error="backupstoragelocations.velero.io \"azure\" not found" error.file="/go/src/github.com/vmware-tanzu/velero/pkg/controller/backup_sync_controller.go:311" error.function="github.com/vmware-tanzu/velero/pkg/controller.(*backupSyncController).run" logSource="pkg/controller/backup_sync_controller.go:311"

Anything else you would like to add:
Followed #168, didn't help.

Environment:

• helm version (use helm version): version.BuildInfo{Version:"v3.5.2", GitCommit:"167aac70832d3a384f65f9745335e9fb40169dc2", GitTreeState:"dirty", GoVersion:"go1.15.7"
• helm chart version and app version:
  • helm2 (use helm list):
  • helm3 (use helm list -n <YOUR NAMESPACE>): velero-2.15.0 1.5.3
• Kubernetes version (use kubectl version): Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.1", GitCommit:"c4d752765b3bbac2237bf87cf0b1c2e307844666", GitTreeState:"clean", BuildDate:"2020-12-18T12:09:25Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-16T20:43:08Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}
• Kubernetes installer & version: v1.19.3
• Cloud provider or hardware configuration: AKS
• OS (e.g. from /etc/os-release): Ubuntu 18.04.5 LTS

[johnjbateman]

Closing. fixed right after I posted this.

[headyj]

@johnjbateman I have the exact same issue with the timeout, except I'm using EKS. Could you please tell me how you fixed this issue?

[johnjbateman]

@johnjbateman I have the exact same issue with the timeout, except I'm using EKS. Could you please tell me how you fixed this issue?

Forgive me. I cannot find what I did to fix this. IIRC it was a bone-head mistake. For reference though, here is my end override.yaml:

configuration:
  provider: azure
  backupStorageLocation:
    name: azure
    bucket: velero
  volumeSnapshotLocation:
    name: azure
    config:
      apiTimeout: 1m0s
credentials:
  useSecret: true
  name: velero-accesskey
  extraSecretRef: azure-credentials
initContainers:
- name: velero-plugin-for-microsoft-azure
  image: velero/velero-plugin-for-microsoft-azure:v1.2.0
  imagePullPolicy: IfNotPresent
  volumeMounts:
    - mountPath: /target
      name: plugins
schedules:
  daily:
    schedule: "0 0 * * *"
    template:
      includeClusterResources: false
      includedNamespaces:
      - '*'
      includedResources:
      - '*'
      storageLocation: azure
      snapshotVolumes: true      
      ttl: 720h0m0s