Fix github actions for large file storage #224
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
end users do not need tests; developers can always access to the tests in the github repo and releases
|
Got an email from github on warning of the bandwidth usage of lfs (see below), it looks lfs is not a great solution for us. So I converted this PR to draft and will explore other solutions. |
only automatically do unit tests
now if http response code is not "200 OK", then the website is down.
The hooks will make sure creation and deletion only run once in mater process when parallel testing is enabled
This was referenced Mar 18, 2024
Merged
Merged
gcroci2
approved these changes
Mar 21, 2024
Co-authored-by: Giulia Crocioni <[email protected]>
Co-authored-by: Giulia Crocioni <[email protected]>
Co-authored-by: Giulia Crocioni <[email protected]>
This stack of pull requests is managed by Graphite. Learn more about stacking. Join @CunliangGeng and the rest of your teammates on |
Closed
Merge activity
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We used large file storage (lfs) for handling large test dataset (zip files).
The github action " actions/checkout" does not use lfs automatically, leading to failed tests, and so we have to set it explicitly (see the usage).
However, it's not free to use lfs and there is storage and bandwidth limit (see the comment), which we do not want.
This PR stopped the use of lfs and implemented an alternative way to handle large files. The approach is to put the large files in Zenodo, and download it before running tests if there is no local cache.
Major changes: