Wait the end of subprocesses with multiple parallel jobs

by

If your platform is not Windows, you could probably select against the stdout pipes of your subprocesses. Your app will then block until either:

  • One of the registered file descriptors has an I/O event (in this case, we’re interested in a hangup on the subprocess’s stdout pipe)
  • The poll times out

Non-fleshed-out example using epoll with Linux 2.6.xx:

import subprocess
import select

poller = select.epoll()
subprocs = {} #map stdout pipe's file descriptor to the Popen object

#spawn some processes
for i in xrange(5):
    subproc = subprocess.Popen(["mylongrunningproc"], stdout=subprocess.PIPE)
    subprocs[subproc.stdout.fileno()] = subproc
    poller.register(subproc.stdout, select.EPOLLHUP)

#loop that polls until completion
while True:
    for fd, flags in poller.poll(timeout=1): #never more than a second without a UI update
        done_proc = subprocs[fd]
        poller.unregister(fd)
        print "this proc is done! blah blah blah"
        ...  #do whatever
    #print a reassuring spinning progress widget
    ...
    #don't forget to break when all are done

More Related Contents:

  1. Read streaming input from subprocess.communicate()
  2. How do I use subprocess.Popen to connect multiple processes by pipes?
  3. Passing Variables to Subprocess.Popen
  4. ImageMagick not authorized to convert PDF to an image
  5. How do I hide the console when I use os.system() or subprocess.call()?
  6. How can I run an external command asynchronously from Python?
  7. catching stdout in realtime from subprocess
  8. How to get exit code when using Python subprocess communicate method?
  9. Displaying subprocess output to stdout and redirecting it
  10. Python subprocess.call a bash alias
  11. Pipe subprocess standard output to a variable [duplicate]
  12. Python read from subprocess stdout and stderr separately while preserving order
  13. How do I get all of the output from my .exe using subprocess and Popen?
  14. Is it possible to run function in a subprocess without threading or writing a separate file/script.
  15. What’s a good equivalent to subprocess.check_call that returns the contents of stdout?
  16. Cross-platform subprocess with hidden window
  17. Python read file as stream from HDFS
  18. Python subprocess: callback when cmd exits
  19. ‘yes’ reporting error with subprocess communicate()
  20. How do I pass large numpy arrays between python subprocesses without saving to disk?
  21. wait process until all subprocess finish? [duplicate]
  22. OSError: [Errno 8] Exec format error
  23. running a command as a super user from a python script
  24. Run multiple programs sequentially in one Windows command prompt?
  25. how to kill (or avoid) zombie processes with subprocess module
  26. Piping output of subprocess.Popen to files
  27. Capture “Segmentation fault” message for a crashed subprocess: no out and err after a call to communicate()
  28. WindowsError [error 5] Access is denied
  29. Killing child process when parent crashes in python
  30. Opening a process with Popen and getting the PID

Leave a Comment