FIX: shell commands with excessive stdout no longer freeze

[ZLLentz]

In the PyDMShellCommand widget, we've been using subprocess.PIPE for the stdout and stderr arguments to subprocess.Popen when we don't want these streams to show in the caller's terminal.

Unfortunately, this creates a problem. If the subprocess generates a certain amount of stdout or stderr, the pipe "fills up" so to speak, and then the subprocess is stuck waiting on being allowed to write more to stdout. The subprocess will then "freeze" until the calling process reads enough of the stdout buffer to allow the subprocess to finish writing data.

I'm not sure if this affects all flavors of subprocesses, but this definitely affects shell commands that open chatty Python processes.

One simple fix is to replace PIPE with DEVNULL, which causes the Popen object to drop these outputs on the floor rather than send them to the parent process. See https://docs.python.org/3/library/subprocess.html#subprocess.DEVNULL.

[ZLLentz]

This breaks two pydm unit tests that intend to check the process attribute's stdout stream. I'm not immediately sure what to do about this. I'll try to figure something out.

[ZLLentz]

I thought about this just a little bit more, and fixing the tests is trivial, but maybe this leads into some more broad design questions:

• Are there any active users who rely on the subprocess.PIPE in their .py PyDM screens to pull stdout from subprocesses launched from these buttons and process them in custom code?
• In lieu of specific live examples, are there good reasons (aside from testing) to keep subprocess.PIPE as an option?
• Is there then value in refactoring this widget to have full control over what we use for stdout and stderr? I could imagine us being able to select None, PIPE, or DEVNULL on both stdout and stderr. With some simplified naming, this might even be more intuitive than the existing redirectCommandOutput boolean which I think has confused people in the past. This would also allow people to propagate stderr from the subprocess to the main process's terminal, which hasn't ever been an option.

I am looking for insight from the the core PyDM team. I'm happy to implement something here that is more future-thinking than the small change currently in the PR diff, which does fix my team's problem but doesn't consider the above questions.

[jbellister-slac]

So I'm not aware of any users who rely on the subprocess.PIPE functionality, maybe we'll see some response in slack.

If we did want to keep this functionality, I think your proposal of None/PIPE/DEVNULL sounds perfect for it. As you say that is much clearer than the redirectCommandOuptut boolean which is not clear what setting it to true will actually do without looking at the code.

In lieu of specific live examples, are there good reasons (aside from testing) to keep subprocess.PIPE as an option?

Only one I can think of is the obvious one of someone out there using it already. But I think I'd be fine erring on the side of just going with what you've done in this PR and if someone out there does still want subprocess.PIPE and let's us know later then we can take another pass at this with the solution you suggested.

[ZLLentz]

In that case, I'll check back in with slack on Monday to see if there's any response, and if not I'll fix the tests and move on for now.

edit: I'll implement the full option redesign if that ends up being the easiest way to fix the test suite anyway