add_data.sh - verify checksum of downloaded files

[anthonyfok]

As a follow up to Issue #66 / PR #82, it would be nice if:

• Verify checksum of downloaded files
• Check if a pre-compressed git repo exists, and if not, fall back to the original Git LFS files
• Allow repeated runs of add_data.sh, i.e. try to resume an interrupted download. (need to think about clearing and re-importing the PostGIS database? Probably outside of the scope of this issue.)

Previous notes (copied from #66):

• Check for type: file vs dir
• --no-act or --dry-run or something similar for add_data.sh to "predict" how much files would be downloaded before download.
• Get sha256 checksum (Git LFS only)

$ jq -r '.content' response.json | base64 -d
version https://git-lfs.github.com/spec/v1
oid sha256:7d2487268dad05f83a0c5be6dcf90419e0bb53cf34706362baae21ac7841c2b5
size 234453739

• Use a repo for compressed cached copies?
  • Encrypted, compressed cached private data on a (public or private?) repo?
  • See if the compressed files are small enough (100MB or less) to avoid Git LFS altogether.
  • See: Uploading Large Files to GitHub. 3 ways to avoid getting error messages… | by Eden Au | Towards Data Science
    https://towardsdatascience.com/uploading-large-files-to-github-dbef518fa1a

[anthonyfok]

For (my) reference, curl author Daniel Steinberg explains the retry options in depth:
https://daniel.haxx.se/blog/2020/03/24/curl-ootw-retry-max-time/

--retry-connrefused may be a good candidate to add too. It was added in 7.52.0. curl included Ubuntu 18.04 and above is new enough.

[anthonyfok]

Idea: Use git partial clone and sparse-checkout to get just the pointers to *.csv in the original repo to quickly get the SHA256 checksums. Something like the following:

git clone --filter=blob:none --sparse [email protected]:OpenDRR/openquake-inputs.git
cd openquake-inputs
git sparse-checkout add '*.csv'