Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for streaming cli output? #7

Open
zstumgoren opened this issue Feb 24, 2017 · 4 comments
Open

Add support for streaming cli output? #7

zstumgoren opened this issue Feb 24, 2017 · 4 comments
Milestone

Comments

@zstumgoren
Copy link
Contributor

zstumgoren commented Feb 24, 2017

Refactor the S3 wrapper class to return real-time/streaming output from the CLI process rather than returning all output at tail of end processing. Will need to switch from subprocess.check_output to a custom use of Popen, likely with polling etc.

@zstumgoren zstumgoren added this to the 0.2 milestone Feb 24, 2017
@zstumgoren
Copy link
Contributor Author

zstumgoren commented Mar 14, 2017

fwiw, simply printing stdout at tail end of subprocess call appears to work fine. Streaming output would be most useful to avoid excessive memory usage and delayed CLI feedback when uploading very large files and/or a large number of files (on the order of hundreds or thousands).

@zstumgoren zstumgoren changed the title Streaming cli output Add support for streaming cli output? Mar 14, 2017
@zstumgoren
Copy link
Contributor Author

We agreed to hold off on implementation for now since it's a "nice-to-have". Bumping to 0.3.0 release and adding a Help Wanted label, in case someone else out there decides to poke at this.

@zstumgoren zstumgoren modified the milestones: 0.2, 0.3 Mar 17, 2017
@myersjustinc
Copy link

Started to refresh my memory on Popen and such, and got at least some success with the following:

proc = subprocess.Popen(
    ['./long.sh'], stdout=sys.stdout, stderr=subprocess.STDOUT,
    encoding='utf-8')
proc.wait()
if proc.returncode != 0:
    print('\n*** Error ***\n')

where long.sh is a bash script with an assortment of echo and sleep commands.

Doesn't seem ideal, though, and doesn't play as nicely with datakit_data.s3's logger.

The "right" way to do it probably is going to involve asyncio or something similar, but wanted to make note of what I've looked into so far.

@zstumgoren
Copy link
Contributor Author

+1 Yep, I had kicked around some Popen implementations as well on first pass, but punted until we had time to vet solutions more thoroughly. We can include asyncio as a possible solution as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants