Read streaming input from subprocess.communicate()

by

To get subprocess’ output line by line as soon as the subprocess flushes its stdout buffer:

#!/usr/bin/env python2
from subprocess import Popen, PIPE

p = Popen(["cmd", "arg1"], stdout=PIPE, bufsize=1)
with p.stdout:
    for line in iter(p.stdout.readline, b''):
        print line,
p.wait() # wait for the subprocess to exit

iter() is used to read lines as soon as they are written to workaround the read-ahead bug in Python 2.

If subprocess’ stdout uses a block buffering instead of a line buffering in non-interactive mode (that leads to a delay in the output until the child’s buffer is full or flushed explicitly by the child) then you could try to force an unbuffered output using pexpect, pty modules or unbuffer, stdbuf, script utilities, see Q: Why not just use a pipe (popen())?

Here’s Python 3 code:

#!/usr/bin/env python3
from subprocess import Popen, PIPE

with Popen(["cmd", "arg1"], stdout=PIPE, bufsize=1,
           universal_newlines=True) as p:
    for line in p.stdout:
        print(line, end='')

Note: Unlike Python 2 that outputs subprocess’ bytestrings as is; Python 3 uses text mode (cmd’s output is decoded using locale.getpreferredencoding(False) encoding).

More Related Contents:

  1. How do I use subprocess.Popen to connect multiple processes by pipes?
  2. ImageMagick not authorized to convert PDF to an image
  3. How do I write to a Python subprocess’ stdin?
  4. How to use subprocess popen Python [duplicate]
  5. live output from subprocess command
  6. Python spawn off a child subprocess, detach, and exit
  7. Python subprocess.call a bash alias
  8. subprocess.call() arguments ignored when using shell=True w/ list [duplicate]
  9. Python subprocess.Popen() error (No such file or directory)
  10. How to kill a python child process created with subprocess.check_output() when the parent dies?
  11. Cross-platform subprocess with hidden window
  12. Python subprocess and user interaction
  13. How to suppress or capture the output of subprocess.run()?
  14. Python subprocess .check_call vs .check_output
  15. How to run a background process and do *not* wait?
  16. How to parallelize file downloads?
  17. Sending a password over SSH or SCP with subprocess.Popen
  18. Communicate multiple times with a process without breaking the pipe?
  19. Python – how to execute shell commands with pipe, but without ‘shell=True’?
  20. how to kill (or avoid) zombie processes with subprocess module
  21. What can lead to “IOError: [Errno 9] Bad file descriptor” during os.system()?
  22. Launch a python script from another script, with parameters in subprocess argument
  23. Python: subprocess call with shell=False not working
  24. Python subprocess.Popen as different user on Windows
  25. Bash style process substitution with Python’s Popen
  26. Execute Subprocess in Background
  27. Run subprocess and print output to logging
  28. subprocess call ffmpeg (command line)
  29. Wait the end of subprocesses with multiple parallel jobs
  30. How can I capture the stdout output of a child process?

Leave a Comment