velero incremental backup for snapshot of pv's not working - Azure

[sanzzz]

i started using helm chart for velero backup and set volumesnaphotlocation object with incremental: true
when i schdule the velero backup it always takes a full backup with volume size (example: 8GB)
second run also shows as full backup 8GB.

how to setup the configuration such that next run should just be differential which is much less than 8GB snapshot.
I dont see command line for incremental and schedule. yaml also doesnt work. Please let me know.

[kaovilai]

set volumesnaphotlocation object with incremental: true

What object store plugin are you using? How does one set volumesnapshotlocation with incremental when it's not a configurable option per documentation?
https://github.com/vmware-tanzu/velero-plugin-for-aws/blob/main/volumesnapshotlocation.md

[kaovilai]

Actions taken by the volumesnapshotlocation at least for aws is to trigger EBS snapshot.

per stackoverflow this is already incremental in nature.

[sanzzz]

https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure/blob/main/volumesnapshotlocation.md is the plugin . Also regular helmchart has the support https://github.com/vmware-tanzu/helm-charts/blob/main/charts/velero/values.yaml#L385

whenever i create new backup after first backup it is treating as new backup job and creating new snapshot with full backup. When i create it using schedule.yaml every 4 hrs it creating new snapshot, instead of creating snapshot with just the differential.

https://github.com/vmware-tanzu/helm-charts/blob/main/charts/velero/templates/schedule.yaml

[sanzzz]

is there a manual velero command to create backup with incremental

like velero backup create mybackup <--incremental> something like this?

[kaovilai]

Please add azure to your issue title.

[sanzzz]

Added. Thanks

[kaovilai]

is there a manual velero command to create backup with incremental

like velero backup create mybackup <--incremental> something like this?

No. Incremental flag you are referencing are for volumesnapshot plugins which in this case is the azure plugin. The plugin tells Azure to take incremental backup per your settings.

You could consider velero file-system-backup meanwhile where velero have full control over incrementality.

https://velero.io/docs/v1.13/file-system-backup/#:~:text=An%20incremental%20backup,difference%20is%20small.

[anshulahuja98]

to summarize
your end goal is to take snapshots of Azure Disks attached to your cluster and you are not sure if the snapshots are incremental or full.
Is that correct?

Can you show/validate on Azure, just open any snapshot created by velero and see if it says if the snapshot is incremental or full.

when you say 8GB backup size -> is this the size that velero tells you it backed up? or where else are you validating this info

[sanzzz]

just trying to understand why incremental flag exist https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure/blob/main/volumesnapshotlocation.md and is not working, not sure if i am using it right

[sanzzz]

to summarize your end goal is to take snapshots of Azure Disks attached to your cluster and you are not sure if the snapshots are incremental or full. Is that correct?

Can you show/validate on Azure, just open any snapshot created by velero and see if it says if the snapshot is incremental or full.

when you say 8GB backup size -> is this the size that velero tells you it backed up? or where else are you validating this info

That's right - end goal is to tak snapshot of PV's (Azure disks). When i take the backup using https://github.com/vmware-tanzu/velero-plugin-for-microsoft-azure/blob/main/volumesnapshotlocation.md uisng velero backup commanline or schduel using helm schedule.yaml it takes the backup full on every run
snapshot is showing as incremental for first and subsequent backups
it always shows 8GB for all snapshots - that makes me think its taking a full backup everytime and it cost a lot if a pv with 500 GB is created and every run adds lots of cost

[anshulahuja98]

"snapshot is showing as incremental for first and subsequent backups" -> if snapshot is showing incremental on azure, be rest assured your are charged for only the deltas.
The size of snapshot indicate more about what will be the size of restored disk and not the billing of that particular snapshot.

You can read the first paragraph on this page which talks about the same -> https://learn.microsoft.com/en-us/azure/virtual-machines/disks-incremental-snapshots?tabs=azure-cli

[sanzzz]

just fyi, i am running "velero backup create mybackup" that is creating backup and then i cannot use this same backup name next time, so if i change the backup name it is taking it as a new backup and creating full snapshot again conidering as differnt backup. Just finding how does it links to exising snapshot and take incrementa after tha.

[anshulahuja98]

you don't need to manually link snapshots to be incremental.
if you snapshot the same disk multiple times no matter with what name / or at what time, the snapshots will know that what is the base on top of which they are incrmeental
Just make sure that incremental is always set -> and you can cross check this on azure.

[sanzzz]

"snapshot is showing as incremental for first and subsequent backups" -> if snapshot is showing incremental on azure, be rest assured your are charged for only the deltas. The size of snapshot indicate more about what will be the size of restored disk and not the billing of that particular snapshot.

You can read the first paragraph on this page which talks about the same -> https://learn.microsoft.com/en-us/azure/virtual-machines/disks-incremental-snapshots?tabs=azure-cli

does this mean though every snapshot showing as 8GB in size, but its only taking differential backup . How can i validate that , can you please help?
Also, does it take full backup first time though snapshot showing as incremental?

[anshulahuja98]

Also, does it take full backup first time though snapshot showing as incremental? -> YES

does this mean though every snapshot showing as 8GB in size, but its only taking differential backup . How can i validate that , can you please help? -> this is more of an azure abstraction, might be clear from billing of the snapshots, I am personally not aware if there is any way to validate otherwise.
You can read this further to get more confidence: https://azure.microsoft.com/en-us/blog/announcing-general-availability-of-incremental-snapshots-of-managed-disks/

[anshulahuja98]

You can directly contact azure support for more details.
From velero POV, I think since we are able to validate that snapshots are marked as incremental on azure, there is no other work required.

[sanzzz]

You can directly contact azure support for more details. From velero POV, I think since we are able to validate that snapshots are marked as incremental on azure, there is no other work required.

I will check with azure support. Looks like its showing as 8GB for every snapshot makes me belive it is considering each new backup as new job and creaing full bakup. I will check on this and get back. Thank very much for your time on this.

[anshulahuja98]

Again, I'll reiterate -> as long as it shows incremental, you can be rest assured it is incremental in nature. Size only indicates the size of the disk when you restore it. ( "diskSizeBytes": 1073741824,). It is not the snapshot size, but the disk size.
To get fine grain details on how to validate the actual consumption, that is the only part you can check with azure support.