Job Dependency

Job Dependency

Use sbatch --dependency=<dependency_list> option if you need your jobs to run in a specific order.

DEPENDENCY	DESCRIPTION
after	This job can begin execution after the specified jobs have begun execution.
afterany	This job can begin execution after the specified jobs have terminated.
afternotok	This job can begin execution after the specified jobs have terminated in some failed state (non-zero exit code, node failure, timed out, etc).
afterok	This job can begin execution after the specified jobs have successfully executed (ran to completion with an exit code of zero).
expand	Resources allocated to this job should be used to expand the specified job. The job to expand must share the same QOS (Quality of Service) and partition. Gang scheduling of resources in the partition is also not supported.
singleton	This job can begin execution after any previously launched jobs sharing the same job name and user have terminated.

Example: “diamond workflow”

Four jobs A,B,C,D. Job A runs first. Jobs B and C depend on completition of job A. Job D depends on jobs B and C.

Submit job A first:

$ sbatch A
Submitted batch job 17703

Jobs B and C depend on successful completion of job A.

$sbatch -d afterok:17703 B
Submitted batch job 17704

$ sbatch -d afterok:17703 C
Submitted batch job 17705

Job D depends on successful completion of both jobs B and C

sbatch -d afterok:17704:17705 D
Submitted batch job 17706

Check your running queue

$squeue -u <username>

JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
17706   compute        D nikolout PD       0:00      1 (Dependency)
17704   compute        B nikolout  R       1:27      1 node260
17705   compute        C nikolout  R       1:27      1 node260

Job A has already completed so jobs B and C are already running. Job D is on pending state waiting B and C to end.

Chain dependencies

Automating dependency steps:

jobA=`sbatch A | awk '{print $4}'`
jobB=`sbatch -d afterok:$jobA B | awk '{print $4}'`
jobC=`sbatch -d afterok:$jobA C | awk '{print $4}'`
jobD=`sbatch -d afterok:$jobB:$jobC D | awk '{print $4}'`

Or, you can submit your dependent jobs from within the job that they depend on.

#!/bin/bash -l
#SBATCH -n 1
#SBATCH -t 00:01:00
#SBATCH -J A

sbatch --dependency=afterok:$SLURM_JOB_ID B
srun EXE ARGS

#!/bin/bash -l
#SBATCH -n 1
#SBATCH -t 00:01:00
#SBATCH -J B
srun EXE ARGS

NOTE Job B will only be considered for scheduling when its dependency has completed.