read subprocess stdout line by line

by

I think the problem is with the statement for line in proc.stdout, which reads the entire input before iterating over it. The solution is to use readline() instead:

#filters output
import subprocess
proc = subprocess.Popen(['python','fake_utility.py'],stdout=subprocess.PIPE)
while True:
  line = proc.stdout.readline()
  if not line:
    break
  #the real code does filtering here
  print "test:", line.rstrip()

Of course you still have to deal with the subprocess’ buffering.

Note: according to the documentation the solution with an iterator should be equivalent to using readline(), except for the read-ahead buffer, but (or exactly because of this) the proposed change did produce different results for me (Python 2.5 on Windows XP).

More Related Contents:

  1. Running shell command and capturing the output
  2. A non-blocking read on a subprocess.PIPE in Python
  3. How to hide output of subprocess
  4. Python subprocess readlines() hangs
  5. Interactive input/output using Python
  6. Run command and get its stdout, stderr separately in near real time like in a terminal
  7. Launch a completely independent process
  8. Multiple inputs and outputs in python subprocess communicate
  9. Python C program subprocess hangs at “for line in iter”
  10. subprocess: deleting child processes in Windows
  11. How do I get ‘real-time’ information back from a subprocess.Popen in python (2.5)
  12. Use subprocess to send a password
  13. Kill a running subprocess call
  14. How do I run another script in Python without waiting for it to finish? [duplicate]
  15. How to replicate tee behavior in Python when using subprocess?
  16. Ensuring subprocesses are dead on exiting Python program
  17. Subprocess timeout failure
  18. When to use Shell=True for Python subprocess module [duplicate]
  19. ‘^’ is ignored by Python – how to escape ‘^’ character in Popen Windows?
  20. Break the function after certain time
  21. What’s the difference between subprocess Popen and call (how can I use them)?
  22. Python subprocess in parallel
  23. Passing variables to a subprocess call [duplicate]
  24. In Python 2.5, how do I kill a subprocess?
  25. Why does subprocess.Popen() with shell=True work differently on Linux vs Windows?
  26. Python Subprocess: Too Many Open Files
  27. How can I spawn new shells to run Python scripts from a base Python script?
  28. Handling directories with spaces Python subprocess.call()
  29. Opening a process with Popen and getting the PID
  30. Subprocess check_output returned non-zero exit status 1

Leave a Comment