Create volume snapshots in parallel

[jglick]

Experimental patch to run volume snapshotter plugins in parallel, at least for the volume creation. Necessary for vmware-tanzu/velero-plugin-for-aws#90 to be practical since it must wait for an EBS snapshot to be complete before initiating a request to copy it to another region, and this often blocks for about 2m37s (apparently regardless of the size of the delta, or number of snapshots currently pending).

[dsu-igeek]

We're redoing architecture to support this feature, this is not the direction for handling this.

[jglick]

We're redoing architecture to support this feature

Good news! Do you have anything I can look at yet?

[jglick]

#4202 (comment) and https://github.com/vmware-tanzu/velero-plugin-for-csi/blob/64cbe394b7bc0bea1a8e88d31452e841e1c673b2/internal/util/util.go#L156-L187 suggests that this is a general need. (Also correspondingly during restore: openshift/velero-plugin-for-aws#2)

[jglick]

In fact something like this patch is necessary even for the AWS plugin as currently designed (with no cross-zone/region functionality): API calls such as DescribeVolumes (example) which are normally near-instantaneous (only waiting for one HTTP request to complete) can nonetheless fail with RequestLimitExceeded under load, as I found when I tried running backups on a cluster with ~100 PVs, and as others have apparently seen as well: #1135 (comment). The remedy is to pause and retry (example): vmware-tanzu/velero-plugin-for-aws@b5d7c52 is working reliably for me under load, but requires this core patch to be practical. Note that even implementing #3533 would not suffice for that case, as the issue is not that the backup has been successfully initiated and the volume snapshotter plugin was merely waiting for infrastructure to mark it as complete, but rather than the snapshot cannot even be initiated without waiting.

[jglick]

see #2888

[jglick]

$ COSIGN_EXPERIMENTAL=1 cosign verify ghcr.io/jglick/velero:concurrent-snapshot

Verification for ghcr.io/jglick/velero:concurrent-snapshot --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - Any certificates were verified against the Fulcio roots.

[{"critical":{"identity":{"docker-reference":"ghcr.io/jglick/velero"},"image":{"docker-manifest-digest":"sha256:b8c86e7c129cd40ae2079f11a79b2f52ed2e28e4f1677a8ba7125eb1aaa95151"},…

[jglick]

Possibly superseded by facilities in https://github.com/vmware-tanzu/velero/blob/main/design/Implemented/general-progress-monitoring.md