Using subprocess.Popen for Process with Large Output

by

You’re doing blocking reads to two files; the first needs to complete before the second starts. If the application writes a lot to stderr, and nothing to stdout, then your process will sit waiting for data on stdout that isn’t coming, while the program you’re running sits there waiting for the stuff it wrote to stderr to be read (which it never will be–since you’re waiting for stdout).

There are a few ways you can fix this.

The simplest is to not intercept stderr; leave stderr=None. Errors will be output to stderr directly. You can’t intercept them and display them as part of your own message. For commandline tools, this is often OK. For other apps, it can be a problem.

Another simple approach is to redirect stderr to stdout, so you only have one incoming file: set stderr=STDOUT. This means you can’t distinguish regular output from error output. This may or may not be acceptable, depending on how the application writes output.

The complete and complicated way of handling this is select (http://docs.python.org/library/select.html). This lets you read in a non-blocking way: you get data whenever data appears on either stdout or stderr. I’d only recommend this if it’s really necessary. This probably doesn’t work in Windows.

More Related Contents:

  1. Read streaming input from subprocess.communicate()
  2. How do I use subprocess.Popen to connect multiple processes by pipes?
  3. Passing Variables to Subprocess.Popen
  4. ImageMagick not authorized to convert PDF to an image
  5. How do I hide the console when I use os.system() or subprocess.call()?
  6. How can I run an external command asynchronously from Python?
  7. catching stdout in realtime from subprocess
  8. How to get exit code when using Python subprocess communicate method?
  9. Displaying subprocess output to stdout and redirecting it
  10. Python subprocess.call a bash alias
  11. Pipe subprocess standard output to a variable [duplicate]
  12. Python read from subprocess stdout and stderr separately while preserving order
  13. How do I get all of the output from my .exe using subprocess and Popen?
  14. Is it possible to run function in a subprocess without threading or writing a separate file/script.
  15. What’s a good equivalent to subprocess.check_call that returns the contents of stdout?
  16. Cross-platform subprocess with hidden window
  17. Python read file as stream from HDFS
  18. Python subprocess: callback when cmd exits
  19. ‘yes’ reporting error with subprocess communicate()
  20. How do I pass large numpy arrays between python subprocesses without saving to disk?
  21. wait process until all subprocess finish? [duplicate]
  22. OSError: [Errno 8] Exec format error
  23. running a command as a super user from a python script
  24. Run multiple programs sequentially in one Windows command prompt?
  25. how to kill (or avoid) zombie processes with subprocess module
  26. Piping output of subprocess.Popen to files
  27. What’s the difference between Python’s subprocess.call and subprocess.run
  28. Sending ^C to Python subprocess objects on Windows
  29. Execute Commands Sequentially in Python?
  30. How to install and import Python modules at runtime

Leave a Comment