python subprocess with timeout and large output (>64K)

by

As opposed to all the warnings in the subprocess documentation then directly reading from process.stdout and process.stderr has provided a better solution.

By better I mean that I can read output from a process that exceeds 2^16 bytes without having to temporarily store the output on disk.

The code follows:

import fcntl
import os
import subprocess
import time

def nonBlockRead(output):
    fd = output.fileno()
    fl = fcntl.fcntl(fd, fcntl.F_GETFL)
    fcntl.fcntl(fd, fcntl.F_SETFL, fl | os.O_NONBLOCK)
    try:
        return output.read()
    except:
        return ''

def cmd(cmdline, timeout=60):
    """
    Execute cmdline, limit execution time to 'timeout' seconds.
    Uses the subprocess module and subprocess.PIPE.

    Raises TimeoutInterrupt
    """

    p = subprocess.Popen(
        cmdline,
        bufsize = bufsize, # default value of 0 (unbuffered) is best
        shell   = False, # not really needed; it's disabled by default
        stdout  = subprocess.PIPE,
        stderr  = subprocess.PIPE
    )

    t_begin = time.time() # Monitor execution time
    seconds_passed = 0

    stdout=""
    stderr=""

    while p.poll() is None and seconds_passed < timeout: # Monitor process
        time.sleep(0.1) # Wait a little
        seconds_passed = time.time() - t_begin

        # p.std* blocks on read(), which messes up the timeout timer.
        # To fix this, we use a nonblocking read()
        # Note: Not sure if this is Windows compatible
        stdout += nonBlockRead(p.stdout)
        stderr += nonBlockRead(p.stderr)

    if seconds_passed >= timeout:
        try:
            p.stdout.close()  # If they are not closed the fds will hang around until
            p.stderr.close()  # os.fdlimit is exceeded and cause a nasty exception
            p.terminate()     # Important to close the fds prior to terminating the process!
                              # NOTE: Are there any other "non-freed" resources?
        except:
            pass

        raise TimeoutInterrupt

    returncode  = p.returncode

    return (returncode, stdout, stderr)

More Related Contents:

  1. Using module ‘subprocess’ with timeout
  2. Kill or terminate subprocess when timeout?
  3. Read streaming input from subprocess.communicate()
  4. How do I use subprocess.Popen to connect multiple processes by pipes?
  5. Passing Variables to Subprocess.Popen
  6. ImageMagick not authorized to convert PDF to an image
  7. How do I hide the console when I use os.system() or subprocess.call()?
  8. How can I run an external command asynchronously from Python?
  9. catching stdout in realtime from subprocess
  10. How to get exit code when using Python subprocess communicate method?
  11. Displaying subprocess output to stdout and redirecting it
  12. How to set timeout on python’s socket recv method?
  13. Pipe subprocess standard output to a variable [duplicate]
  14. Python read from subprocess stdout and stderr separately while preserving order
  15. How do I get all of the output from my .exe using subprocess and Popen?
  16. Is it possible to run function in a subprocess without threading or writing a separate file/script.
  17. What’s a good equivalent to subprocess.check_call that returns the contents of stdout?
  18. How to set Selenium Python WebDriver default timeout?
  19. Cross-platform subprocess with hidden window
  20. Python read file as stream from HDFS
  21. Python subprocess: callback when cmd exits
  22. ‘yes’ reporting error with subprocess communicate()
  23. How do I pass large numpy arrays between python subprocesses without saving to disk?
  24. wait process until all subprocess finish? [duplicate]
  25. OSError: [Errno 8] Exec format error
  26. running a command as a super user from a python script
  27. Run multiple programs sequentially in one Windows command prompt?
  28. Piping output of subprocess.Popen to files
  29. Timeout a function (windows)?
  30. What is the global default timeout

Leave a Comment