Parallel execution of shell processes

by

Edit – I modified the script to optionally display the output of each process

Here is a native batch solution that reliably runs a list of commands in parallel, never launching more than n processes at a time.

It even has a mechanism built in to distribute the processes to specific CPUs or remote machines via PSEXEC, but I haven’t tested that feature.

The trick to make this work is to START each command through a CMD process that redirects either stdout or an undefined handle to a lock file. The process will maintain an exclusive lock on the file until it terminates. It doesn’t matter how the process terminates (normal exit, crash, killed process), the lock will be released as soon as it does.

The master script can test if the process is still active by attempting to redirect to the same lock file. The redirection will fail if the process is still active, succeed if it has terminated.

By default, the script ignores the output of each process. If started with the /O option as the 1st parameter, then it displays the output of each process, without interleaving.

My demo sets the process limit to 4, and simply runs a series of PING commands of varying length.

I’ve tested this on XP, Vista, and Windows 7.

@echo off
setlocal enableDelayedExpansion

:: Display the output of each process if the /O option is used
:: else ignore the output of each process
if /i "%~1" equ "/O" (
  set "lockHandle=1"
  set "showOutput=1"
) else (
  set "lockHandle=1^>nul 9"
  set "showOutput="
)

:: The list of commands could come from anywhere such as another file
:: or the output of another command. For this demo I will list the
:: commands within this script - Each command is prefixed with :::
::: ping /n 05 ::1
::: ping /n 20 ::1
::: ping /n 10 ::1
::: ping /n 15 ::1
::: ping /n 07 ::1
::: ping /n 05 ::1
::: ping /n 20 ::1
::: ping /n 10 ::1
::: ping /n 15 ::1
::: ping /n 07 ::1

:: Define the maximum number of parallel processes to run.
:: Each process number can optionally be assigned to a particular server
:: and/or cpu via psexec specs (untested).
set "maxProc=4"

:: Optional - Define CPU targets in terms of PSEXEC specs
::           (everything but the command)
::
:: If a CPU is not defined for a proc, then it will be run on the local machine.
:: I haven't tested this feature, but it seems like it should work.
::
:: set cpu1=psexec \\server1 ...
:: set cpu2=psexec \\server1 ...
:: set cpu3=psexec \\server2 ...
:: etc.

:: For this demo force all CPU specs to undefined (local machine)
for /l %%N in (1 1 %maxProc%) do set "cpu%%N="

:: Get a unique base lock name for this particular instantiation.
:: Incorporate a timestamp from WMIC if possible, but don't fail if
:: WMIC not available. Also incorporate a random number.
  set "lock="
  for /f "skip=1 delims=-+ " %%T in ('2^>nul wmic os get localdatetime') do (
    set "lock=%%T"
    goto :break
  )
  :break
  set "lock=%temp%\lock%lock%_%random%_"

:: Initialize the counters
  set /a "startCount=0, endCount=0"

:: Clear any existing end flags
  for /l %%N in (1 1 %maxProc%) do set "endProc%%N="

:: Launch the commands in a loop
:: Modify the IN () clause as needed to retrieve the list of commands
  set launch=1
  for /f "tokens=* delims=:" %%A in ('findstr /b ":::" "%~f0"') do (
    if !startCount! lss %maxProc% (
      set /a "startCount+=1, nextProc=startCount"
    ) else (
      call :wait
    )
    set cmd!nextProc!=%%A
    if defined showOutput echo -------------------------------------------------------------------------------
    echo !time! - proc!nextProc!: starting %%A
    2>nul del %lock%!nextProc!
    %= Redirect the lock handle to the lock file. The CMD process will     =%
    %= maintain an exclusive lock on the lock file until the process ends. =%
    start /b "" cmd /c %lockHandle%^>"%lock%!nextProc!" 2^>^&1 !cpu%%N! %%A
  )
  set "launch="

:wait
:: Wait for procs to finish in a loop
:: If still launching then return as soon as a proc ends
:: else wait for all procs to finish
  :: redirect stderr to null to suppress any error message if redirection
  :: within the loop fails.
  for /l %%N in (1 1 %startCount%) do 2>nul (
    %= Redirect an unused file handle to the lock file. If the process is    =%
    %= still running then redirection will fail and the IF body will not run =%
    if not defined endProc%%N if exist "%lock%%%N" 9>>"%lock%%%N" (
      %= Made it inside the IF body so the process must have finished =%
      if defined showOutput echo ===============================================================================
      echo !time! - proc%%N: finished !cmd%%N!
      if defined showOutput type "%lock%%%N"
      if defined launch (
        set nextProc=%%N
        exit /b
      )
      set /a "endCount+=1, endProc%%N=1"
    )
  )
  if %endCount% lss %startCount% (
    1>nul 2>nul ping /n 2 ::1
    goto :wait
  )

2>nul del %lock%*
if defined showOutput echo ===============================================================================
echo Thats all folks^^!

Here is output from a sample run that ignores process output

12:24:07.52 - proc1: starting  ping /n 05 ::1
12:24:07.52 - proc2: starting  ping /n 20 ::1
12:24:07.53 - proc3: starting  ping /n 10 ::1
12:24:07.54 - proc4: starting  ping /n 15 ::1
12:24:11.60 - proc1: finished  ping /n 05 ::1
12:24:11.60 - proc1: starting  ping /n 07 ::1
12:24:16.66 - proc3: finished  ping /n 10 ::1
12:24:16.66 - proc3: starting  ping /n 05 ::1
12:24:17.68 - proc1: finished  ping /n 07 ::1
12:24:17.68 - proc1: starting  ping /n 20 ::1
12:24:20.72 - proc3: finished  ping /n 05 ::1
12:24:20.72 - proc3: starting  ping /n 10 ::1
12:24:21.75 - proc4: finished  ping /n 15 ::1
12:24:21.75 - proc4: starting  ping /n 15 ::1
12:24:26.82 - proc2: finished  ping /n 20 ::1
12:24:26.82 - proc2: starting  ping /n 07 ::1
12:24:29.86 - proc3: finished  ping /n 10 ::1
12:24:32.89 - proc2: finished  ping /n 07 ::1
12:24:35.92 - proc4: finished  ping /n 15 ::1
12:24:36.93 - proc1: finished  ping /n 20 ::1
Thats all folks!

Here is the output if run with the /O option showing process output

-------------------------------------------------------------------------------
12:24:51.02 - proc1: starting  ping /n 05 ::1
-------------------------------------------------------------------------------
12:24:51.02 - proc2: starting  ping /n 20 ::1
-------------------------------------------------------------------------------
12:24:51.03 - proc3: starting  ping /n 10 ::1
-------------------------------------------------------------------------------
12:24:51.04 - proc4: starting  ping /n 15 ::1
===============================================================================
12:24:55.10 - proc1: finished  ping /n 05 ::1

Pinging ::1 with 32 bytes of data:
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms

Ping statistics for ::1:
    Packets: Sent = 5, Received = 5, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms
-------------------------------------------------------------------------------
12:24:55.10 - proc1: starting  ping /n 07 ::1
===============================================================================
12:25:00.17 - proc3: finished  ping /n 10 ::1

Pinging ::1 with 32 bytes of data:
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms

Ping statistics for ::1:
    Packets: Sent = 10, Received = 10, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms
-------------------------------------------------------------------------------
12:25:00.19 - proc3: starting  ping /n 05 ::1
===============================================================================
12:25:01.22 - proc1: finished  ping /n 07 ::1

Pinging ::1 with 32 bytes of data:
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms

Ping statistics for ::1:
    Packets: Sent = 7, Received = 7, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms
-------------------------------------------------------------------------------
12:25:01.23 - proc1: starting  ping /n 20 ::1
===============================================================================
12:25:04.27 - proc3: finished  ping /n 05 ::1

Pinging ::1 with 32 bytes of data:
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms

Ping statistics for ::1:
    Packets: Sent = 5, Received = 5, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms
-------------------------------------------------------------------------------
12:25:04.28 - proc3: starting  ping /n 10 ::1
===============================================================================
12:25:05.30 - proc4: finished  ping /n 15 ::1

Pinging ::1 with 32 bytes of data:
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms

Ping statistics for ::1:
    Packets: Sent = 15, Received = 15, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms
-------------------------------------------------------------------------------
12:25:05.32 - proc4: starting  ping /n 15 ::1
===============================================================================
12:25:10.38 - proc2: finished  ping /n 20 ::1

Pinging ::1 with 32 bytes of data:
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms

Ping statistics for ::1:
    Packets: Sent = 20, Received = 20, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms
-------------------------------------------------------------------------------
12:25:10.40 - proc2: starting  ping /n 07 ::1
===============================================================================
12:25:13.44 - proc3: finished  ping /n 10 ::1

Pinging ::1 with 32 bytes of data:
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms

Ping statistics for ::1:
    Packets: Sent = 10, Received = 10, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms
===============================================================================
12:25:16.48 - proc2: finished  ping /n 07 ::1

Pinging ::1 with 32 bytes of data:
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms

Ping statistics for ::1:
    Packets: Sent = 7, Received = 7, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms
===============================================================================
12:25:19.52 - proc4: finished  ping /n 15 ::1

Pinging ::1 with 32 bytes of data:
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms

Ping statistics for ::1:
    Packets: Sent = 15, Received = 15, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms
===============================================================================
12:25:20.54 - proc1: finished  ping /n 20 ::1

Pinging ::1 with 32 bytes of data:
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms

Ping statistics for ::1:
    Packets: Sent = 20, Received = 20, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms
===============================================================================
Thats all folks!

More Related Contents:

  1. How to wait all batch files to finish before exiting?
  2. How to run a PowerShell script from a batch file
  3. What is the proper way to test if a parameter is empty in a batch file?
  4. Start multiple tasks in parallel and wait for them in windows?
  5. Windows batch: sleep [duplicate]
  6. Execute multiple batch files concurrently and monitor if their process is completed
  7. Several ways to call a windows batch file from another one or from prompt. Which one in which case?
  8. How to expand a CMD shell variable twice (recursively)
  9. How do I get current date/time on the Windows command line in a suitable format for usage in a file/folder name?
  10. Symbol equivalent to NEQ, LSS, GTR, etc. in Windows batch files
  11. Windows XP or later Windows: How can I run a batch file in the background with no window displayed?
  12. How can I send the stdout of one process to multiple processes using (preferably unnamed) pipes in Unix (or Windows)?
  13. XCOPY still asking (F = file, D = directory) confirmation
  14. How to prevent auto-closing of console after the execution of batch file
  15. Get current batchfile directory
  16. Redirecting Output from within Batch file
  17. Creating a file name as a timestamp in a batch job
  18. Detect whether current Windows version is 32 bit or 64 bit
  19. How to check if a service is running via batch file and start it, if it is not running?
  20. How can I run a windows batch file but hide the command window?
  21. How to list all folder with size via batch file
  22. How to wait for a process to terminate to execute another process in batch file
  23. Access clipboard in Windows batch file
  24. Rename all files in a directory with a Windows batch script
  25. Batch file encoding
  26. How do I increment a folder name using Windows batch?
  27. Batch goto loses errorlevel
  28. Escape percent signs in given variables
  29. Out of a git console: how do I execute a batch file and then return to git console?
  30. UTF-16 to UTF-8 conversion (for scripting in Windows)

Leave a Comment