Ensuring the integrity of full snapshot before uploading it to the object store.

[ishan16696]

What this PR does / why we need it:
It has been observed that while doing the restoration from full snapshot sometimes backup-restore failed to restore the etcd due to sometimes full snapshots got corrupted or missing a hash.
This PR is try to minimise the occurrence of such scenarios by verifying the integrity of full snapshot before uploading it to the object store.

Which issue(s) this PR fixes:
Fixes #778

Special notes for your reviewer:
This is how it's done:

Release note:

Ensure the integrity of the full snapshot prior to uploading it to the object store, thereby reducing the potential restoration failures.

[ishan16696]

Performance tests have been performed with etcd of 7GB database size:

~ > etcdctl endpoint status --endpoints=http://127.0.0.1:2379 --cluster -w table
+-----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|       ENDPOINT        |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| http://localhost:2379 | 8e9e05c52164694d |   3.5.9 |  7.0 GB |      true |      false |         3 |      35952 |              35952 |        |
+-----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

Backups taken by backup-restore with their sizes on disk.

25M	Full-00000000-02000001-1726772100.gz
79M	Full-00000000-05532798-1726864342.gz
140M	Full-00000000-09083821-1726946137.gz
185M	Full-00000000-11680572-1727038676.gz
200K	Incr-11680573-11691572-1727038777.gz
200K	Incr-11691573-11702572-1727038778.gz
200K	Incr-11702573-11712987-1727038807.gz
68K	Incr-11712988-11716517-1727068649.gz

Then a big full-snapshot is triggered by backup-restore:

INFO[0536] Taking scheduled full snapshot for time: 2024-09-23 11:00:00.014743 +0530 IST  actor=snapshotter
{"level":"info","ts":"2024-09-23T11:00:00.038+0530","caller":"clientv3/maintenance.go:200","msg":"opened snapshot stream; downloading"}
INFO[0536] Total time taken by Snapshot API: 0.002619 seconds.  actor=snapshotter
INFO[0536] checking the full snapshot integrity with the help of SHA256  actor=snapshotter
{"level":"info","ts":"2024-09-23T11:00:06.902+0530","caller":"clientv3/maintenance.go:208","msg":"completed snapshot read; closing"}
INFO[0547] full snapshot SHA256 hash has been successfully verified.  actor=snapshotter
INFO[0547] start compressing the snapshot using gzip Compression Policy  actor=compressor
INFO[0547] Total time taken in full snapshot compression: 0.001206 seconds.  actor=snapshotter
INFO[0547] Successfully opened snapshot reader on etcd   actor=snapshotter
INFO[0578] Total written bytes: 6982844448               actor=compressor
INFO[0578] Total time to save full snapshot: 41.202852 seconds.  actor=snapshotter
INFO[0578] Successfully saved full snapshot at: Full-00000000-11716517-1727069410.gz  actor=snapshotter
INFO[0578] Applied watch on etcd from revision: 11716518  actor=snapshotter

Then restoration is triggered:

INFO[0007] Received start initialization request.        actor=backup-restore-server
INFO[0007] Updating status from New to Progress          actor=backup-restore-server
INFO[0007] Setting status to : 503                       actor=backup-restore-server
INFO[0007] Validation failBelowRevision:                 actor=backup-restore-server
INFO[0007] Validation mode: full                         actor=backup-restore-server
INFO[0007] Checking for data directory structure validity...
INFO[0007] Data directory structure invalid.
INFO[0007] Finding latest set of snapshot to recover from...
INFO[0007] Removing directory(default.etcd.part).
INFO[0007] Restoring from base snapshot: Full-00000000-11716517-1727069410.gz  actor=restorer
INFO[0007] start decompressing the snapshot with gzip compressionPolicy  actor=de-compressor
INFO[0009] Responding to status request with: Progress   actor=backup-restore-server
{"level":"warn","ts":"2024-09-23T13:04:23.060+0530","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"passthrough:///http://localhost:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = \"transport: Error while dialing dial tcp [::1]:2379: connect: connection refused\""}
ERRO[0010] failed to get status of etcd endPoint: http://localhost:2379 with error: context deadline exceeded
INFO[0012] successfully fetched data of base snapshot in 4.938676375 seconds [CompressionPolicy:gzip]  actor=restorer
{"level":"warn","ts":"2024-09-23T13:04:27.060+0530","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"passthrough:///http://localhost:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = \"transport: Error while dialing dial tcp [::1]:2379: connect: connection refused\""}
ERRO[0014] failed to get status of etcd endPoint: http://localhost:2379 with error: context deadline exceeded
INFO[0016] Responding to status request with: Progress   actor=backup-restore-server
{"level":"warn","ts":"2024-09-23T13:04:31.061+0530","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"passthrough:///http://localhost:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: connection error: desc = \"transport: Error while dialing dial tcp [::1]:2379: connect: connection refused\""}
ERRO[0018] failed to get status of etcd endPoint: http://localhost:2379 with error: context deadline exceeded
{"level":"info","ts":1727076872.963309,"caller":"membership/cluster.go:392","msg":"added member","cluster-id":"cdf818194e3a8c32","local-member-id":"0","added-peer-id":"8e9e05c52164694d","added-peer-peer-urls":["http://localhost:2380"]}
INFO[0020] No delta snapshots present over base snapshot.  actor=restorer
INFO[0020] Removing directory(default.etcd).
INFO[0020] Successfully restored the etcd data directory.
INFO[0020] Successfully initialized data directory for etcd.  actor=backup-restore-server

• Full snapshot is successfully taken with integrity check. ✅
• Successfully restored the etcd from the full snapshot. ✅
• Snapshot-Compaction is also working as expected. ✅

[anveshreddy18]

/assign

[anveshreddy18]

Thanks for the PR @ishan16696.

PR looks good, I just have 2 small suggestions. PTAL

[anveshreddy18]

I tested it and things are working good.
LGTM

[seshachalam-yv]

/assign

[renormalize]

Thanks for this much needed PR @ishan16696!

Just a few comments from my side.

Thanks.

[renormalize]

Another suggestion for better error handling, as discussed.

[renormalize]

Tiny nit, but otherwise the PR looks in great condition.
Thanks.

[renormalize]

Thanks @ishan16696 for filling the gaps in my understanding in validation, and addressing all review comments!

[renormalize]

@ishan16696 could you rebase your PR on master? This will fix the failing integration tests.
Thanks.

[shreyas-s-rao]

@ishan16696 thank you so much for addressing my comments, and also for adding tests. I have one small concern regarding closure of ReadClosers, but other than that, looks good.

[shreyas-s-rao]

@ishan16696 thanks for making the requested change.
/lgtm

[seshachalam-yv]

Thanks for addressing my review comments
/lgtm