Implement a progress output handler for ftp and aspera downloads #36
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fix format check for sequence data
…wserTools into aspera_status
Improved checks for Aspera files and binary Handle known Python 3 Mac installation problem impacting HTTPS URLs Don't re-download run/analysis file if exists (with MD5 check) Replace (rather than append to) sequence files if they exist Download WGS set if WGS-only assembly or assembly contains WGS scaffolds (even without WGS flag) Added flag to extract WGS scaffolds from WGS file into wgs_scaffolds file Added some progress information to enaDataGet assembly fetch
Issue 35 fix Removal of wrongly committed files
|
I've had a quick look at this and can't authorised the pull request at this time due to the use of pexpect. We made a conscious decision to use only libraries in a standard python installation. This may change with future deployment strategies that have better dependency management, but for now it's a key feature that people can install python and then use these scripts out of the box without worrying about anything else. |
alakob
force-pushed
the
master
branch
2 times, most recently
from
e587330 to
dcebaf3
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I haven't implemented the "queue" or "file" handlers yet, but here's a basic implementation of passing a handler function to those functions that would have otherwise just printed to stdout. Previously, especially for aspera transfers, the output is swallowed and can't be hoovered up by a process that watches stdout, so progress can't be monitored in any situation other than running the scripts on the command line. I've implemented code on the assumption that no handler will be provided, so the aspera and ftp output will go to stdout. More canned handlers could be added in future, or users can supply their own handlers.
For example, in a Django app, the aspera progress output wasn't discoverable by a websocket. Now, pexpect is used to gather up the output produced by aspera at each tick, and can be outputted to stdout as text or a JSON fragment that can be used in a webapp.
It could do with a bit more work, but I hope it's useful!