Python threading multiple bash subprocesses?

by

You don’t need threads to run subprocesses in parallel:

from subprocess import Popen

commands = [
    'date; ls -l; sleep 1; date',
    'date; sleep 5; date',
    'date; df -h; sleep 3; date',
    'date; hostname; sleep 2; date',
    'date; uname -a; date',
]
# run in parallel
processes = [Popen(cmd, shell=True) for cmd in commands]
# do other things here..
# wait for completion
for p in processes: p.wait()

To limit number of concurrent commands you could use multiprocessing.dummy.Pool that uses threads and provides the same interface as multiprocessing.Pool that uses processes:

from functools import partial
from multiprocessing.dummy import Pool
from subprocess import call

pool = Pool(2) # two concurrent commands at a time
for i, returncode in enumerate(pool.imap(partial(call, shell=True), commands)):
    if returncode != 0:
       print("%d command failed: %d" % (i, returncode))

This answer demonstrates various techniques to limit number of concurrent subprocesses: it shows multiprocessing.Pool, concurrent.futures, threading + Queue -based solutions.

You could limit the number of concurrent child processes without using a thread/process pool:

from subprocess import Popen
from itertools import islice

max_workers = 2  # no more than 2 concurrent processes
processes = (Popen(cmd, shell=True) for cmd in commands)
running_processes = list(islice(processes, max_workers))  # start new processes
while running_processes:
    for i, process in enumerate(running_processes):
        if process.poll() is not None:  # the process has finished
            running_processes[i] = next(processes, None)  # start new process
            if running_processes[i] is None: # no new processes
                del running_processes[i]
                break

On Unix, you could avoid the busy loop and block on os.waitpid(-1, 0), to wait for any child process to exit.

More Related Contents:

  1. Using module ‘subprocess’ with timeout
  2. Kill a running subprocess call
  3. Python Subprocess.Popen from a thread
  4. Trying to fix tkinter GUI freeze-ups (using threads)
  5. subprocess.wait() not waiting for Popen process to finish (when using threads)?
  6. Read streaming input from subprocess.communicate()
  7. Passing Variables to Subprocess.Popen
  8. ImageMagick not authorized to convert PDF to an image
  9. How to run functions in parallel?
  10. threading.Timer – repeat function every ‘n’ seconds
  11. Intercepting stdout of a subprocess while it is running
  12. Using a global variable with a thread
  13. multiprocessing vs multithreading vs asyncio in Python 3
  14. Process substitution not allowed by Python’s subprocess with shell=True?
  15. Proper use of mutexes in Python
  16. How to get output from subprocess.Popen(). proc.stdout.readline() blocks, no data prints out
  17. Overriding python threading.Thread.run()
  18. What is the difference between multiprocessing and subprocess?
  19. Python Threading String Arguments
  20. Terminate multiple threads when any thread completes a task
  21. Why does passing variables to subprocess.Popen not work despite passing a list of arguments?
  22. How to catch exception output from Python subprocess.check_output()?
  23. How can I specify working directory for a subprocess
  24. Return value from thread
  25. Using multiprocessing.Process with a maximum number of simultaneous processes
  26. Launch IPython notebook with selected browser
  27. Threading in python using queue
  28. Execute Python function in Main thread from call in Dummy thread
  29. Why does Paramiko hang if you use it while loading a module?
  30. Python: Why is threaded function slower than non thread

Leave a Comment