Python read from subprocess stdout and stderr separately while preserving order

by

The code in your question may deadlock if the child process produces enough output on stderr (~100KB on my Linux machine).

There is a communicate() method that allows to read from both stdout and stderr separately:

from subprocess import Popen, PIPE

process = Popen(command, stdout=PIPE, stderr=PIPE)
output, err = process.communicate()

If you need to read the streams while the child process is still running then the portable solution is to use threads (not tested):

from subprocess import Popen, PIPE
from threading import Thread
from Queue import Queue # Python 2

def reader(pipe, queue):
    try:
        with pipe:
            for line in iter(pipe.readline, b''):
                queue.put((pipe, line))
    finally:
        queue.put(None)

process = Popen(command, stdout=PIPE, stderr=PIPE, bufsize=1)
q = Queue()
Thread(target=reader, args=[process.stdout, q]).start()
Thread(target=reader, args=[process.stderr, q]).start()
for _ in range(2):
    for source, line in iter(q.get, None):
        print "%s: %s" % (source, line),

See:

More Related Contents:

  1. How to replicate tee behavior in Python when using subprocess?
  2. “subprocess.Popen” – checking for success and errors
  3. Temporarily Redirect stdout/stderr
  4. catching stdout in realtime from subprocess
  5. Displaying subprocess output to stdout and redirecting it
  6. Python: How to get stdout after running os.system? [duplicate]
  7. Interactive input/output using Python
  8. Intercepting stdout of a subprocess while it is running
  9. Multiple inputs and outputs in python subprocess communicate
  10. subprocess readline hangs waiting for EOF
  11. How do I get ‘real-time’ information back from a subprocess.Popen in python (2.5)
  12. Running an interactive command from within Python
  13. Redirect subprocess stderr to stdout
  14. Piping output of subprocess.Popen to files
  15. How can I capture the stdout output of a child process?
  16. Python subprocess output to stdout
  17. multiprocessing: How can I ʀᴇʟɪᴀʙʟʏ redirect stdout from a child process?
  18. A non-blocking read on a subprocess.PIPE in Python
  19. How to hide output of subprocess
  20. Retrieving the output of subprocess.call() [duplicate]
  21. Constantly print Subprocess output while process is running
  22. How to print to stderr in Python?
  23. subprocess: deleting child processes in Windows
  24. Ensuring subprocesses are dead on exiting Python program
  25. Force another program’s standard output to be unbuffered using Python
  26. Passing variables to a subprocess call [duplicate]
  27. In Python 2.5, how do I kill a subprocess?
  28. Why does subprocess.Popen() with shell=True work differently on Linux vs Windows?
  29. Execute Commands Sequentially in Python?
  30. How to install and import Python modules at runtime

Leave a Comment