downloader import and development #11
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Pre-containerization python downloader code. Everything hard-coded, this will be converted to a containerized version.
Prelim dockerfile and requirements for build testing; not tested at all, just saved to transfer from machine to machine.
Now works from /data instead of from hard-coded directory. The assumption is that /data is an externally mounted-and-managed directory.
Add downloader to build checks. Untested as yet.
Tieing in /data directory to the build.
Adding entries for downloader in parallel with the other containered components.
Quick fix; statically fixes the name of the json file. Doesn't fix the real logic of extracting the name from the incoming packet (which would future-proof this).
craigsteffen
force-pushed
the
downloader_initial
branch
from
cad72bb to
3c2b7f5
Compare
Downloader working and tested in local mode. Seems to interact correctly with rabbitmq.
Downloader script now gets URI from shell environment. This MUST be set or the downloader won't work.
main() runs in a "while True" loop. Cleaning up a bunch of stuff after that, which will never run but was bothering the syntax validator.
Removed a catch clause that probably wasn't doing anything and probably wans't needed. Fixed a variable error that crept in from cut and paste. Fixed grouping so that basic_ack and worker=None were at the right level in the main cleanup loop.
Cleaned out some deprecated URI definition code that wasn't used any more.
All print() statements replaced with (mostly) logging.debug, or in two cases, logging.info. Tested outside of container.
Flushing wget output so it's not cluttering up the command line output.
…uc-cdr into downloader_initial
Apparently /dev/null doesn't play well in containers. Replacing with ordinary file name.
Tweaking mount points and making download agree with pipeline.
Cleaning up wget output in easiest way.
Apparently redirecting output, even to a benign file, causes heartburn for apptainer. Removing redirects.
Double-check merge by merging main into us first.
About to merge new release with downloader.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The beginning of the branch is pulling in the hard-coded version of Craig's original downloader.
Further commits have very preliminary versions of the Docker files. So far they build, the container runs, but the directories haven't been hooked up yet so that it actually writes files to disk. Further commits will enable those features.
We want to have it at least rudimentarily working with at least one tested download before merging this commit, but the PR is here for tracking and discussion.