{"id":8988,"date":"2025-03-11T15:22:52","date_gmt":"2025-03-11T15:22:52","guid":{"rendered":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/?page_id=8988"},"modified":"2026-04-15T11:16:12","modified_gmt":"2026-04-15T10:16:12","slug":"s-commands","status":"publish","type":"page","link":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/batch-slurm\/s-commands\/","title":{"rendered":"Slurm Batch Commands (sbatch, squeue, scancel, sacct)"},"content":{"rendered":"

Batch Commands<\/h2>\n

Your applications should be run in the batch system. You\u2019ll need a jobscript<\/em> (a plain text file) describing your job – its CPU, memory and possibly GPU requirements, and also the commands you actually want the job to run.<\/p>\n

Further details on how to write jobscripts are in the sections on serial jobs<\/a>, parallel jobs<\/a>, job-arrays<\/a> and GPU jobs<\/a>.<\/p>\n

You\u2019ll then use one or more of the following batch system commands to submit<\/em> your job to the system and check on its status. These commands should be run from the CSF’s login nodes.<\/p>\n

Job submission using sbatch<\/h2>\n
\n
sbatch jobscript<\/em><\/code><\/dt>\n
Submit a job to the batch system, usually by submitting a jobscript<\/em>. Alternatively you can specify job options on the sbatch<\/code> command-line. We recommend using a jobscript because this allows you to easily reuse your jobscript every time you want to run the job. Remembering the command-line options you used (possibly months ago) is much more difficult.<\/p>\n

The sbatch<\/code> command will return a unique job-ID number if it accepts the job. You can use this in other commands (see below) and, when requesting support about a job, you should include this number in the details you send in.<\/p>\n

For example, when submitting a job you will see a message similar to:<\/p>\n

\r\n[mabcxyz1@login1[csf3] ~]$ sbatch myjobscript<\/em><\/strong>\r\nSubmitted batch job 373\r\n<\/pre>\n
Adding flags to the command-line will override the same flag in the jobscript, if present, or add a setting if not currently specified in the jobscript. This is a convenient way to submit the job again but with a different setting to what you’ve put in the jobscript. Command-line flags must come before<\/em> the name of the jobscript. This example submits a job with a 2-day wallclock, which might be different to what you’ve specified in the jobscript.<\/p>\n
\r\n[mabcxyz1@login1[csf3] ~]$ sbatch -t 2-0 myjobscript<\/em><\/strong>\r\nSubmitted batch job 374\r\n<\/pre>\n<\/dd>\n<\/dl>\n
sbatch flags<\/h3>\n
The sbatch<\/code> command has an extensive list of command-line flags<\/a>. In most cases you can also specify these in your jobscripts using one of:<\/p>\n
\r\n#SBATCH -f value<\/em>             # short form flag (e.g., -p)\r\n#SBATCH --flag=value<\/em>         # long form flag (e.g., --partition)\r\n<\/pre>\n
We provide details of a few flags here:<\/p>\n
\n
-p, --partition=partition<\/em><\/code><\/dt>\n
Slurm partition (think of this as a queue) that contains a certain type of hardware. Your job will run in this partition. For example, a GPU partition or the multicore AMD job partition or the high memory partition. See the CSF3 partitions<\/a> page.<\/dd>\n
-n,--ntasks=NUM<\/em><\/code><\/dt>\n
Number of tasks, which can usually be thought of as the number of cores to be used by your job. You should use this flag if running MPI parallel code, but it can also be used for other parallel jobs. See the CSF parallel jobs<\/a> page.<\/dd>\n
-c,--cpus-per-task=NUM<\/em><\/code><\/dt>\n
Number of cores (which Slurm refers to as CPUs) per task<\/em> (see -n<\/code> above). Used to specify the number of cores that each MPI rank should use, so you should use this when running mixed-mode MPI+OpenMP applications. But can also be used when running OpenMP parallel applications. See the CSF parallel jobs<\/a> page.<\/dd>\n
-G, --gpus=NUM<\/em><\/code><\/dt>\n
Number of GPUs. Note that the GPU type (A100, L40S, and so on) is determined by the GPU partition you submit to. See the CSF3 GPU jobs<\/a> page.<\/dd>\n
-t,--time=wallclock<\/em><\/code><\/dt>\n
Wallclock time limit for your job. See the CSF3 time limits<\/a> page.<\/dd>\n
-a,--array=start<\/em>-end<\/em><\/code><\/dt>\n
Job array to run multiple copies of the same job. See CSF3 job arrays<\/a> page for further options and examples.<\/dd>\n
--mail-type=flag<\/em><\/code><\/dt>\n
Specify when you want to receive an email about the job. The flag can be one or more (a comma-separated list) of NONE<\/code>, BEGIN<\/code>, END<\/code>, FAIL<\/code>, REQUEUE<\/code>, ALL<\/code>. Please consider how many jobs you are submitting and how many emails you might receive based on you choice of flags.<\/dd>\n
--mail-user=emailaddress<\/em><\/code><\/dt>\n
Specify the email address to receive emails sent from Slurm. This can be your Manchester address or an external address.<\/p>\n
To avoid needing to specify this flag on every job, you can place your email address in a file named .forward<\/code> (yes, with a “dot” at the start of the name). For example, to create this file, run on the login node the command:<\/p>\n
\r\necho my.name<\/em>@manchester.ac.uk > ~\/.forward\r\n        #\r\n        # Change my.name to YOUR own correct email address!\r\n        # This can be a comma-separated list of addresses.\r\n<\/pre>\n
Then you can submit jobs using only the --mail-type<\/code> flag (see above.)\n<\/dd>\n
-d, --dependency=dependencylist<\/em><\/code><\/dt>\n
See the Job Dependencies<\/a> section.<\/dd>\n
--parsable<\/code><\/dt>\n
For scripting purposes, you may prefer just to receive the jobid number from the sbatch<\/code> command. Add the --parsable<\/code> flag to achieve this:<\/p>\n
sbatch --parsable<\/strong> myjobscript<\/em>\r\n12345\r\n<\/pre>\n<\/dd>\n
Error messages<\/h3>\n
When submitting a job, if you see the following errors, something is wrong:<\/p>\n
sbatch: error: Batch job submission failed: No partition specified or system default partition\r\n<\/pre>\n
You must<\/strong> specify a partition, even for serial jobs. Add to your jobscript: #SBATCH -p partitionname<\/em><\/code>.<\/p>\n
sbatch: error: Batch job submission failed: Requested time limit is invalid (missing or exceeds some limit)\r\n<\/pre>\n
You must<\/strong> specify a “wallclock” time limit for your job. The maximum permitted is usually 7 days (or 4 days for GPU and HPC Pool jobs.) Add to your jobscript: #SBATCH -t timelimt<\/em><\/code>.<\/dd>\n
Job Status using squeue<\/h2>\n
\n
squeue<\/code><\/dt>\n
Report the current status of your<\/em> jobs in the batch system (queued\/waiting, running, in error, finished). Note that if you see no jobs listed when you run squeue<\/code> it means you have no jobs in the system – they have all finished or you haven’t submitted any!<\/dd>\n<\/dl>\n
Examples<\/h3>\n
In this example squeue<\/code> returns no output<\/em> which means you have no jobs<\/em> in the queue, either running or waiting:<\/p>\n
[mabcxyz1@login1[csf3] ~]$ squeue<\/strong>\r\n[mabcxyz1@login1[csf3] ~]$\r\n<\/pre>\n
In this example squeue<\/code> shows we have two jobs running (one using 1 core, the other using 8 cores) and one job waiting (it will use 16 cores when it runs.):<\/p>\n
[mabcxyz1@login1[csf3] ~]$ squeue<\/strong>\r\nJOBID PRIORITY  PARTITION NAME         USER     ST SUBMIT_TIME    START_TIME     TIME NODES CPUS NODELIST(REASON)\r\n  372 0.0000005 multicore mymulticore  mabcxyz1 R  08\/03\/25 13:02 08\/03\/25 13:32 2:04     1    8 node1260\r\n  371 0.0000005 serial    simple.x     mabcxyz1 R  09\/03\/25 14:58 09\/03\/25 15:02 8:22     1    1 node603\r\n  403 0.0000003 himem     mypythonjob  mabcxyz1 PD 11\/03\/25 09:25 N\/A            0:00     1    4 (Resources)\r\n   #                          #                 #                 ###                          #\r\n   #                          #                 #                  #                           # Number of\r\n   #                          #                 #                  #                           # CPU cores\r\n   #                          #                 #                  #\r\n   #                          #                 #                  # If running: date & time the job started\r\n   #                          #                 #                  # If waiting: N\/A\r\n   #                          #                 #\r\n   #                          #                 # R   - job is running\r\n   #                          #                 # PD  - job is queued waiting\r\n   #                          #                 # CG  - Completing (contact us, may indicate an error)\r\n   #                          #\r\n   #                          # Usually the name of your jobscript\r\n   #\r\n   # Every job is given a unique job ID number\r\n   # (please tell us this number if requesting support<\/strong>)\r\n<\/pre>\n
Reasons for Pending Jobs<\/h3>\n
If your job is queued you might see one of the following reasons in the squeue<\/code> output:<\/p>\n
\n
(QOSMaxCpuPerUserLimit)<\/code><\/dt>\n
You have reached a global limit for the type of CPU resources you need. Free at the point of use users may see this more frequently than members of contributing groups due to the max-cores-in-use limit applied to all f@pou users.<\/dd>\n
(AssocGrpGRES)<\/code><\/dt>\n
Your group has reached the maximum number of GPUs they can use.<\/dd>\n
(Resources)<\/code><\/dt>\n
This usually means all of the requested resources (e.g., CPUs, GPUs) are in use by other jobs – the CSF is simply very busy!<\/dd>\n
(Priority)<\/code><\/dt>\n
Similar to (Resources)<\/code>. This usually indicates there is a high demand for a particular resource and it will take some time for them to be released. e.g. a 168 core job requires a whole AMD node to run.<\/dd>\n
(MaxCpuPerAccount)<\/code><\/dt>\n
The group you are a part of has reached a global limit.<\/dd>\n
(DependencyNeverSatisfied)<\/code><\/dt>\n
The job was waiting on an earlier job to achieve some state – e.g., finish successfully. But if that job failed, then the dependency could not be met, so the current job was not allowed to start. To check what happened to the earlier job, have a look at the current job’s dependency info:<\/p>\n
\r\nsqueue\r\nJOBID PRIORITY  PARTITION NAME         USER     ST SUBMIT_TIME    START_TIME TIME  NODES CPUS NODELIST(REASON)\r\n  372<\/strong> 0.0000005 multicore mymulticore  mabcxyz1 PD<\/strong> 08\/03\/25 13:02 N\/A        0:00  1     8    (DependencyNeverSatisfied<\/strong>)\r\n\r\nscontrol show job 372<\/strong> | grep Depend\r\n  JobState=PENDING Reason=DependencyNeverSatisfied Dependency=afterok:361(failed)<\/strong>\r\n<\/pre>\n
The afterok:361(failed)<\/code> tells you that the current job (372) was given a dependency on job 361 and that job 361 needed to finish successfully (afterok<\/code>) before 372 would be allowed to run. But instead job 361 failed<\/code>. So job 372’s dependency will never be satisfied and so it will never run.<\/p>\n
You should remove this job from the queue (see scancel<\/code> below).<\/dd>\n
(ReqNodeNotAvail, Reserved for maintenance)<\/code><\/dt>\n
The resources you have requested have been flagged for maitenance and as such are temporarily unavailable for your job. Your job will queue until the resource becomes available again. For significant\/lenghthy maintenance work we will always advise all users in advance by email.<\/dd>\n<\/dl>\n
Changing the squeue output format<\/h3>\n
You can modify the list of fields (columns) output by the squeue<\/code> command by setting the $SQUEUE_FORMAT<\/code> or $SQUEUE_FORMAT2<\/code> environment variables. In fact, the default set of columns you see is given by the first variable – it has a default value when you login to the CSF. To see the value, run:<\/p>\n
echo $SQUEUE_FORMAT\r\n%.15i %9p %9P %15j %8u %2t %14V %14S %10M %.6D %.5C %R\r\n<\/pre>\n
For more information on the two SQUEUE_FORMAT<\/code> env vars and the column codes you can use, run man squeue<\/code>.<\/p>\n
GPU job status using gpusqueue \/ gpustat<\/h3>\n
The Research Infrastructure team provide the following extra command:<\/p>\n
\n
gpusqueue<\/code> \/ gpustat<\/code><\/dt>\n
To see a list of GPU jobs you have in the system, you can use the custom gpusqueue<\/code> command. This runs squeue<\/code> but only shows GPU jobs and adds in some extra columns to show the types of GPUs requested:<\/p>\n
\r\n# Show GPU job information\r\ngpusqueue\r\n<\/pre>\n<\/dd>\n<\/dl>\n
To monitor running GPU jobs, please see the nvitop<\/a> command.<\/p>\n
Delete a Job using scancel<\/h2>\n
\n
scancel jobid<\/em><\/code><\/dt>\n
To remove your<\/em> job from the batch system early, either to terminate a running job before it finishes or to simply remove a queued job before it has started running.<\/dd>\n<\/dl>\n
Also use this if your job goes in to an error state or you decide you don’t want a job to run.<\/p>\n
Note that if your job is in the CG<\/code> state, please leave it in the queue if requesting support<\/strong>. It is easier for us to diagnose the error if we can see the job. We may ask you to scancel<\/code> the job once we have looked at it – there is usually no way to fix an existing job.<\/p>\n
For example, maybe you realise you’ve given a job the wrong input parameters causing it to produce junk results. You don’t need to leave it running to completion (which might be hours or days). Instead you can kill<\/em> the job using scancel. You need to know the job-ID number of the job:<\/p>\n
[mabcxyz1@login1[csf3] ~]$ scancel 12345<\/em><\/strong>\r\n<\/pre>\n
The job will eventually be deleted (it may take a minute or two for this to happen). Use squeue<\/code> to check your list of jobs.<\/p>\n
To delete all<\/em> of your jobs:<\/p>\n
# Delete all of your jobs. Use with CAUTION!\r\n[mabcxyz1@login1[csf3] ~]$ scancel -u $USER<\/strong>\r\n<\/pre>\n
Please also see the Deleteing Job Arrays<\/a> notes.<\/p>\n
Get Info About Finished Jobs<\/h3>\n
\n
sacct -j jobid<\/em><\/code><\/dt>\n
Once your job has finished you can use this command to get a summary of information for wall-clock time, max memory consumption and exit status amongst many other statistics about the job. This is useful for diagnosing why a job failed.<\/p>\n
A lot of information about a job is available using this command. To see a list of every possible field, run sacct -e<\/code>. To have a “long” list of fields automatically displayed when querying a job, use sacct --long -j jobid<\/em><\/code>. Run man sacct<\/code> for further info about his command.<\/p>\n
For info about the extern<\/code> step that now appears in the sacct<\/code> output, please see this FAQ answer<\/a>.<\/dd>\n<\/dl>\n
Further Information<\/h2>\n
Our own documentation throughout this site provides lots of examples of writing jobscripts and how to submit jobs. SGE also comes with a set of comprehensive man pages<\/em>. Some of the most useful ones are:<\/p>\n