Subprocess.Popen: cloning stdout and stderr both to terminal and variables

by

To capture and display at the same time both stdout and stderr from a child process line by line in a single thread, you could use asynchronous I/O:

#!/usr/bin/env python3
import asyncio
import os
import sys
from asyncio.subprocess import PIPE

@asyncio.coroutine
def read_stream_and_display(stream, display):
    """Read from stream line by line until EOF, display, and capture the lines.

    """
    output = []
    while True:
        line = yield from stream.readline()
        if not line:
            break
        output.append(line)
        display(line) # assume it doesn't block
    return b''.join(output)

@asyncio.coroutine
def read_and_display(*cmd):
    """Capture cmd's stdout, stderr while displaying them as they arrive
    (line by line).

    """
    # start process
    process = yield from asyncio.create_subprocess_exec(*cmd,
            stdout=PIPE, stderr=PIPE)

    # read child's stdout/stderr concurrently (capture and display)
    try:
        stdout, stderr = yield from asyncio.gather(
            read_stream_and_display(process.stdout, sys.stdout.buffer.write),
            read_stream_and_display(process.stderr, sys.stderr.buffer.write))
    except Exception:
        process.kill()
        raise
    finally:
        # wait for the process to exit
        rc = yield from process.wait()
    return rc, stdout, stderr

# run the event loop
if os.name == 'nt':
    loop = asyncio.ProactorEventLoop() # for subprocess' pipes on Windows
    asyncio.set_event_loop(loop)
else:
    loop = asyncio.get_event_loop()
rc, *output = loop.run_until_complete(read_and_display(*cmd))
loop.close()

More Related Contents:

  1. Calling a python script with input within a python script using subprocess
  2. How to use subprocess popen Python [duplicate]
  3. Popen waiting for child process even when the immediate child has terminated
  4. Intercepting stdout of a subprocess while it is running
  5. Python subprocess.Popen() error (No such file or directory)
  6. Popen with conflicting executable/path
  7. How do I get ‘real-time’ information back from a subprocess.Popen in python (2.5)
  8. How to write to stdout AND to log file simultaneously with Popen?
  9. How do I get all of the output from my .exe using subprocess and Popen?
  10. link several Popen commands with pipes
  11. Is it possible to run function in a subprocess without threading or writing a separate file/script.
  12. Python subprocess and user interaction
  13. Keep a subprocess alive and keep giving it commands? Python
  14. Python output to Console within Subprocess from the child scricpt
  15. How to suppress or capture the output of subprocess.run()?
  16. Launch a shell command with in a python script, wait for the termination and return to the script
  17. How to get output from subprocess.Popen(). proc.stdout.readline() blocks, no data prints out
  18. How to run a background process and do *not* wait?
  19. real time subprocess.Popen via stdout and PIPE
  20. How to parallelize file downloads?
  21. Sending a password over SSH or SCP with subprocess.Popen
  22. What’s the difference between subprocess Popen and call (how can I use them)?
  23. python subprocess Popen environment PATH?
  24. How can I specify working directory for a subprocess
  25. Bash style process substitution with Python’s Popen
  26. Why does subprocess.Popen() with shell=True work differently on Linux vs Windows?
  27. How to do multiple arguments with Python Popen?
  28. subprocess.wait() not waiting for Popen process to finish (when using threads)?
  29. subprocess.Popen in different console
  30. What is the subprocess.Popen max length of the args parameter?

Leave a Comment