What does the –ntasks or -n tasks does in SLURM?

by

The --ntasks parameter is useful if you have commands that you want to run in parallel within the same batch script.
This may be two separate commands separated by an & or two commands used in a bash pipe (|).

For example

Using the default ntasks=1

#!/bin/bash

#SBATCH --ntasks=1

srun sleep 10 & 
srun sleep 12 &
wait

Will throw the warning:

Job step creation temporarily disabled, retrying

The number of tasks by default was specified to one, and therefore the second task cannot start until the first task has finished.
This job will finish in around 22 seconds. To break this down:

sacct -j515058 --format=JobID,Start,End,Elapsed,NCPUS

        JobID               Start                 End    Elapsed      NCPUS
------------ ------------------- ------------------- ---------- ----------
515058       2018-12-13T20:51:44 2018-12-13T20:52:06   00:00:22          1
515058.batch 2018-12-13T20:51:44 2018-12-13T20:52:06   00:00:22          1
515058.0     2018-12-13T20:51:44 2018-12-13T20:51:56   00:00:12          1
515058.1     2018-12-13T20:51:56 2018-12-13T20:52:06   00:00:10          1

Here task 0 started and finished (in 12 seconds) followed by task 1 (in 10 seconds). To make a total user time of 22 seconds.

To run both of these commands simultaneously:

#!/bin/bash

#SBATCH --ntasks=2

srun --ntasks=1 sleep 10 & 
srun --ntasks=1 sleep 12 &
wait

Running the same sacct command as specified above

    sacct -j 515064 --format=JobID,Start,End,Elapsed,NCPUS
    JobID               Start                 End    Elapsed      NCPUS
    ------------ ------------------- ------------------- ---------- ----------
    515064       2018-12-13T21:34:08 2018-12-13T21:34:20   00:00:12          2
    515064.batch 2018-12-13T21:34:08 2018-12-13T21:34:20   00:00:12          2
    515064.0     2018-12-13T21:34:08 2018-12-13T21:34:20   00:00:12          1
    515064.1     2018-12-13T21:34:08 2018-12-13T21:34:18   00:00:10          1

Here the total job taking 12 seconds. There is no risk of jobs waiting for resources as the number of tasks has been specified in the batch script and therefore the job has the resources to run this many commands at once.

Each task inherits the parameters specified for the batch script. This is why --ntasks=1 needs to be specified for each srun task, otherwise each task uses --ntasks=2 and so the second command will not run until the first task has finished.

Another caveat of the tasks inheriting the batch parameters is if --export=NONE is specified as a batch parameter. In this case --export=ALL should be specified for each srun command otherwise environment variables set within the sbatch script are not inherited by the srun command.

Additional notes:
When using bash pipes, it may be necessary to specify –nodes=1 to prevent commands either side of the pipes running on separate nodes.
When using & to run commands simultaneously, the wait is vital. In this case, without the wait command, task 0 would cancel itself, given task 1 completed successfully.

More Related Contents:

  1. Make Bash handle variables like Batch
  2. Some troubles with using sed and awk [closed]
  3. Why would a correct shell script give a wrapped/truncated/corrupted error message? [duplicate]
  4. Add a new element to an array without specifying the index in Bash
  5. Lookup shell variables by name, indirectly [duplicate]
  6. How do I use su to execute the rest of the bash script as that user?
  7. How to iterate over associative arrays in Bash
  8. Why start a shell command with a backslash?
  9. How to process each output line in a loop?
  10. How to delete files older than X hours
  11. bash: force exec’d process to have unbuffered stdout
  12. Bash: Strip trailing linebreak from output
  13. Find and replace filename recursively in a directory
  14. How can bash script do the equivalent of Ctrl-C to a background task?
  15. What does it mean in shell when we put a command inside dollar sign and parentheses: $(command)
  16. How does bash tab completion work?
  17. How to add leading zeros for for-loop in shell? [duplicate]
  18. Running a Bash script over ssh
  19. Bash script store command output into variable
  20. How to resolve symbolic links in a shell script
  21. How to remove double-quotes in jq output for parsing json files in bash?
  22. Forward function declarations in a Bash or a Shell script?
  23. Expand variables in sed
  24. Exclude a string from wildcard search in a shell
  25. Looping through find output in Bash where file name contains white spaces
  26. Using groovy, how do you pipe multiple shell commands?
  27. Assign output to variable in Bash [duplicate]
  28. How does one change the language of the command line interface of Git?
  29. Can I use sed to manipulate a variable in bash?
  30. How to check if gcc has failed, returned a warning, or succeeded in Bash?

Leave a Comment