live output from subprocess command

by

TLDR for Python 3:

import subprocess
import sys

with open("test.log", "wb") as f:
    process = subprocess.Popen(your_command, stdout=subprocess.PIPE)
    for c in iter(lambda: process.stdout.read(1), b""):
        sys.stdout.buffer.write(c)
        f.buffer.write(c)

You have two ways of doing this, either by creating an iterator from the read or readline functions and do:

import subprocess
import sys

# replace "w" with "wb" for Python 3
with open("test.log", "w") as f:
    process = subprocess.Popen(your_command, stdout=subprocess.PIPE)
    # replace "" with b'' for Python 3
    for c in iter(lambda: process.stdout.read(1), ""):
        sys.stdout.write(c)
        f.write(c)

or

import subprocess
import sys

# replace "w" with "wb" for Python 3
with open("test.log", "w") as f:
    process = subprocess.Popen(your_command, stdout=subprocess.PIPE)
    # replace "" with b"" for Python 3
    for line in iter(process.stdout.readline, ""):
        sys.stdout.write(line)
        f.write(line)

Or you can create a reader and a writer file. Pass the writer to the Popen and read from the reader

import io
import time
import subprocess
import sys

filename = "test.log"
with io.open(filename, "wb") as writer, io.open(filename, "rb", 1) as reader:
    process = subprocess.Popen(command, stdout=writer)
    while process.poll() is None:
        sys.stdout.write(reader.read())
        time.sleep(0.5)
    # Read the remaining
    sys.stdout.write(reader.read())

This way you will have the data written in the test.log as well as on the standard output.

The only advantage of the file approach is that your code doesn’t block. So you can do whatever you want in the meantime and read whenever you want from the reader in a non-blocking way. When you use PIPE, read and readline functions will block until either one character is written to the pipe or a line is written to the pipe respectively.

More Related Contents:

  1. How do I execute a program or call a system command?
  2. Running shell command and capturing the output
  3. Using sudo with Python script
  4. Python: execute cat subprocess in parallel
  5. Execute terminal command from python in new terminal window?
  6. How do I pipe a subprocess call to a text file?
  7. subprocess.call() arguments ignored when using shell=True w/ list [duplicate]
  8. File not found error when launching a subprocess containing piped commands
  9. Process substitution not allowed by Python’s subprocess with shell=True?
  10. python getoutput() equivalent in subprocess [duplicate]
  11. ‘yes’ reporting error with subprocess communicate()
  12. Understanding python subprocess.check_output’s first argument and shell=True [duplicate]
  13. real time subprocess.Popen via stdout and PIPE
  14. When to use Shell=True for Python subprocess module [duplicate]
  15. Log exception with traceback in Python
  16. Python: subprocess call with shell=False not working
  17. Why does subprocess.Popen() with shell=True work differently on Linux vs Windows?
  18. Run subprocess and print output to logging
  19. Cannot find the file specified when using subprocess.call(‘dir’, shell=True) in Python
  20. What is the subprocess.Popen max length of the args parameter?
  21. Interactive input/output using Python
  22. Run command and get its stdout, stderr separately in near real time like in a terminal
  23. subprocess: deleting child processes in Windows
  24. What is the point of setLevel in a python logging handler?
  25. How to replicate tee behavior in Python when using subprocess?
  26. ‘^’ is ignored by Python – how to escape ‘^’ character in Popen Windows?
  27. Python subprocess in parallel
  28. Lazy logger message string evaluation
  29. Sending ^C to Python subprocess objects on Windows
  30. How to install and import Python modules at runtime

Leave a Comment