catching stdout in realtime from subprocess

by

Some rules of thumb for subprocess.

  • Never use shell=True. It needlessly invokes an extra shell process to call your program.
  • When calling processes, arguments are passed around as lists. sys.argv in python is a list, and so is argv in C. So you pass a list to Popen to call subprocesses, not a string.
  • Don’t redirect stderr to a PIPE when you’re not reading it.
  • Don’t redirect stdin when you’re not writing to it.

Example:

import subprocess, time, os, sys
cmd = ["rsync.exe", "-vaz", "-P", "source/" ,"dest/"]

p = subprocess.Popen(cmd,
                     stdout=subprocess.PIPE,
                     stderr=subprocess.STDOUT)

for line in iter(p.stdout.readline, b''):
    print(">>> " + line.rstrip())

That said, it is probable that rsync buffers its output when it detects that it is connected to a pipe instead of a terminal. This is the default behavior – when connected to a pipe, programs must explicitly flush stdout for realtime results, otherwise standard C library will buffer.

To test for that, try running this instead:

cmd = [sys.executable, 'test_out.py']

and create a test_out.py file with the contents:

import sys
import time
print ("Hello")
sys.stdout.flush()
time.sleep(10)
print ("World")

Executing that subprocess should give you “Hello” and wait 10 seconds before giving “World”. If that happens with the python code above and not with rsync, that means rsync itself is buffering output, so you are out of luck.

A solution would be to connect direct to a pty, using something like pexpect.

More Related Contents:

  1. Displaying subprocess output to stdout and redirecting it
  2. Interactive input/output using Python
  3. Intercepting stdout of a subprocess while it is running
  4. Multiple inputs and outputs in python subprocess communicate
  5. subprocess readline hangs waiting for EOF
  6. Python read from subprocess stdout and stderr separately while preserving order
  7. How do I get ‘real-time’ information back from a subprocess.Popen in python (2.5)
  8. Running an interactive command from within Python
  9. How to replicate tee behavior in Python when using subprocess?
  10. Piping output of subprocess.Popen to files
  11. “subprocess.Popen” – checking for success and errors
  12. How can I capture the stdout output of a child process?
  13. Python subprocess output to stdout
  14. Using module ‘subprocess’ with timeout
  15. Python: execute cat subprocess in parallel
  16. Difference between subprocess.Popen and os.system
  17. Subprocess.Popen: cloning stdout and stderr both to terminal and variables
  18. How to redirect stdout and stderr to logger in Python
  19. Python subprocess.Popen() error (No such file or directory)
  20. How to kill a python child process created with subprocess.check_output() when the parent dies?
  21. Python subprocess and user interaction
  22. How to suppress or capture the output of subprocess.run()?
  23. How to run a background process and do *not* wait?
  24. Sending a password over SSH or SCP with subprocess.Popen
  25. Python – how to execute shell commands with pipe, but without ‘shell=True’?
  26. What can lead to “IOError: [Errno 9] Bad file descriptor” during os.system()?
  27. Launch a python script from another script, with parameters in subprocess argument
  28. Python: subprocess call with shell=False not working
  29. Execute Commands Sequentially in Python?
  30. How to install and import Python modules at runtime

Leave a Comment