{"id":9123,"date":"2025-03-17T17:58:43","date_gmt":"2025-03-17T17:58:43","guid":{"rendered":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/?page_id=9123"},"modified":"2026-03-27T11:04:12","modified_gmt":"2026-03-27T11:04:12","slug":"partitions","status":"publish","type":"page","link":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/batch-slurm\/partitions\/","title":{"rendered":"Slurm Partitions (Compute Resources)"},"content":{"rendered":"

In Slurm, jobs are submitted to partitions<\/em> – think of these as job queues for different types of jobs.<\/p>\n

In your jobscript, you must<\/em> specify the partition<\/em>, number of cores<\/em> and max “wallclock<\/em>” time your job is allowed to run for (up to a permitted maximum time.) There is now no<\/em> default wallclock time – you must specify it.<\/p>\n

You may also need to specify other flags depending on which parition you submit to. For example, if using one of the GPU partitions, you’ll need to say how many GPUs you want. The HPC Pool and H200 partitions also require an account code.<\/p>\n

The table below provides an overview of the partitions, and the compute resources, currently available in the upgraded CSF3 cluster (Slurm).<\/p>\n

The Slurm jobscript template<\/a> (shown further down this page) gives you a starting point for your jobscript. Much more detail on Slurm jobscript options is available in the SGE to Slurm reference<\/a>.<\/p>\n

You will also need to use new commands on the login node: Slurm uses sbatch<\/code>, squeue<\/code>, scancel<\/code> and srun<\/code> commands, (not qsub<\/code>, qstat<\/code>, qdel<\/code> and qrsh<\/code> as used in SGE.)<\/p>\n

\nReminder: You must<\/em> specify a Slurm Partition in your jobscript for all<\/em> job types<\/strong>: -p name<\/em><\/code><\/p>\n

You must also<\/em> specify a maximum wallclock time limit for your job (up to a permitted maximum time)<\/strong>: -t timelimit<\/em><\/code><\/p>\n

See table below.<\/strong>\n<\/div>\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
CSF3 Slurm Partitions and Compute Resources<\/caption>\n
Slurm Partition name<\/th>\nPartition Summary<\/th>\nCPU<\/th>\nMax Wallclock<\/acronym><\/th>\nCompute Resources<\/th>\nNotes<\/th>\n<\/tr>\n<\/thead>\n
AMD “Genoa” Nodes<\/th>\n<\/tr>\n
-p multicore<\/code><\/p>\n

(was -pe amd.pe<\/code> on SGE)<\/td>\n

Single-node parallel (2-168 cores) batch<\/strong> jobs. -n numcores<\/em><\/code><\/td>\nAMD Genoa<\/td>\n7 days -t 7-0<\/code><\/td>\n74x AMD nodes (168 cores per node). 8GB RAM per core.<\/td>\n12,432 cores available in total.<\/p>\n

This is the main parallel-job partition that most people will use.<\/strong><\/td>\n<\/tr>\n

-p interactive<\/code><\/td>\nSerial (1-core) and small parallel interactive<\/em><\/strong> jobs. Optional: -n numcores<\/em><\/code><\/p>\n

Serial (1-core) batch<\/em><\/strong> jobs.<\/td>\n

AMD Genoa<\/td>\n6 hours<\/strong><\/span> -t 0-6<\/code><\/td>\n2x AMD nodes (168 cores per node). 8GB RAM per core.<\/td>\n336 cores available in total. srun<\/code> (interactive), srun-x11<\/code> (interactive GUIs) and also sbatch<\/code> (jobscripts) accepted.<\/p>\n

For testing apps on AMD “Genoa”.<\/strong><\/td>\n<\/tr>\n

Intel Nodes<\/th>\n<\/tr>\n
-p multicore_small<\/code><\/p>\n

(was -pe smp.pe<\/code> on SGE)<\/td>\n

Single-node parallel (2-32 cores) batch<\/strong> jobs. -n numcores<\/em><\/code><\/td>\nIntel<\/td>\n7 days -t 7-0<\/code><\/td>\n21x Haswell nodes (24 cores, 5GB\/core) Optional: -C haswell<\/code><\/p>\n

28x Skylake nodes (32 cores, 6GB\/core) Optional: -C skylake<\/code><\/td>\n

1400 cores available in total (shared with the serial partition below)<\/td>\n<\/tr>\n
-p serial<\/code><\/td>\nSerial (1-core) batch<\/strong> jobs.<\/p>\n

Serial (1-core) interactive<\/em><\/strong> jobs.<\/td>\n

Intel<\/td>\n7 days -t 7-0<\/code><\/p>\n

6 hours<\/strong><\/span> -t 0-6<\/code><\/td>\n

21x Haswell nodes (24 cores, 5GB\/core) Optional: -C haswell<\/code><\/p>\n

28x Skylake nodes (32 cores, 6GB\/core) Optional: -C skylake<\/code><\/td>\n

1400 cores available in total (shared with the multicore_small partition above)<\/td>\n<\/tr>\n
High Memory Nodes<\/th>\n<\/tr>\n
-p himem<\/code><\/td>\nHigh memory jobs up to 2TB (1-32 cores). Optional: -n numcores<\/em><\/code><\/td>\nIntel<\/td>\n7 days -t 7-0<\/code><\/td>\n1x 512GB node (16 cores, 32GB per core) Optional: -C haswell<\/code><\/p>\n

4x 1.5TB node (32 cores, 40GB per core) Optional: -C cascadelake<\/code><\/p>\n

9x 2TB nodes (32 cores, 64GB per core) Optional: -C icelake<\/code> or -C ssd<\/code><\/td>\n

Job default is 32GB\/core unless you ask for more<\/em> – even if you add a -C<\/code> constraint! Use --mem=total<\/em>G<\/code> or --mem-per-cpu=num<\/em>G<\/code> to request more.
\nHigh Memory Docs<\/a><\/td>\n<\/tr>\n
-p vhimem<\/code><\/td>\nHigh memory jobs requiring up to 4TB. Optional: -n numcores<\/em><\/code><\/td>\nIntel<\/td>\n7 days -t 7-0<\/code><\/td>\n1x 4tb node (32 cores, 128GB per core) Optional: -C icelake<\/code> or -C ssd<\/code><\/td>\nRestricted access<\/strong> – request access<\/a><\/td>\n<\/tr>\n
GPU Nodes<\/th>\n<\/tr>\n
-p gpuV<\/code><\/td>\nNvidia V100 GPU batch and interactive jobs. Optional: -n numcores<\/em><\/code><\/td>\nIntel<\/td>\n4 days -t 4-0<\/code><\/td>\nNode spec: 4x Nvidia V100<\/p>\n

16GB GPU RAM<\/p>\n

32x Intel host cores<\/p>\n

192GB host RAM (5GB\/core)<\/p>\n

Required: -G NUM<\/em><\/code> or --gpus=NUM<\/em><\/code><\/td>\n

\n!!!NO LONGER AVAILABLE!!!
\nOctober 2025 – ALL v100s have been removed from service and replaced with L40S GPUs.<\/strong><\/td>\n<\/tr>\n
-p gpuA<\/code><\/td>\nNvidia A100 GPU batch and interactive jobs. Optional: -n numcores<\/em><\/code><\/td>\nAMD Milan<\/td>\n4 days -t 4-0<\/code><\/td>\nNode spec: 4x Nvidia A100s
\n80GB GPU RAM
\n48x AMD “Milan” host cores
\n512GB host RAM (10GB\/core) Required: -G NUM<\/em><\/code> or --gpus=NUM<\/em><\/code><\/td>\n		Max 12 CPU-cores per GPU<\/p>\n

19 nodes available = 76 GPUs<\/td>\n<\/tr>\n

-p gpuL<\/code><\/td>\nNvidia L40S GPU batch and interactive jobs. Optional: -n numcores<\/em><\/code><\/td>\nIntel<\/td>\n4 days -t 4-0<\/code><\/td>\nNode spec: 4x Nvidia L40S
\n48GB GPU RAM
\n2 x 24-core Intel “Sapphire Rapids” host cores
\n512GB host RAM (10GB\/core) Required: -G NUM<\/em><\/code> or --gpus=NUM<\/em><\/code><\/td>\n		Max 12 CPU-cores per GPU<\/p>\n

21 nodes available = 84 GPUs<\/td>\n<\/tr>\n

-p gpuA40GB<\/code><\/td>\nNvidia A100 (40GB) GPU batch and interactive jobs. Optional: -n numcores<\/em><\/code><\/td>\nAMD Milan<\/td>\n4 days -t 4-0<\/code><\/td>\nNode spec: 4x Nvidia A100s
\n40GB GPU RAM
\n48x AMD “Milan” host cores
\n512GB host RAM (10GB\/core) Required: -G NUM<\/em><\/code> or --gpus=NUM<\/em><\/code><\/td>\n		Very Restricted Access<\/strong>
\nMax 12 CPU-cores per GPU, 2 nodes available = 8 GPUs<\/td>\n<\/tr>\n
-p gpuH<\/code><\/td>\nNvidia H200 (141 GB) GPU batch jobs. Optional: -n numcores<\/em><\/code><\/td>\nIntel Xeon Emerald Rapids<\/td>\n4 days -t 4-0<\/code><\/td>\nNode spec: 8x Nvidia H200
\n141GB GPU RAM
\n64 “Emerald Rapids” host cores
\n1.5 TB host RAM (24GB\/core) Required: -G NUM<\/em><\/code> or --gpus=NUM<\/em><\/code><\/td>\n		Restricted Access<\/strong>
\nMax 8 CPU-cores per GPU, up to 3 nodes available = 24 GPUs.
\nH200 docs<\/a>.<\/td>\n<\/tr>\n
-p gpuH_short<\/code><\/td>\nNvidia H200 (141 GB) GPU batch and interactive jobs. Optional: -n numcores<\/em><\/code> or -c numcores<\/em><\/code><\/td>\nIntel Xeon Emerald Rapids<\/td>\n1 day -t 1-0<\/code><\/td>\nNode spec: 8x Nvidia H200
\n141GB GPU RAM
\n64 “Emerald Rapids” host cores
\n1.5 TB host RAM (24GB\/core) Required: -G NUM<\/em><\/code> or --gpus=NUM<\/em><\/code><\/td>\n		Restricted Access<\/strong>
\nMax 8 CPU-cores per GPU, up to 2 nodes available = 16 GPUs.
\nH200 docs<\/a>.<\/td>\n<\/tr>\n
HPC Pool<\/th>\n<\/tr>\n
-p hpcpool<\/code><\/td>\nMulti-node parallel (128-1024 cores) batch<\/strong> jobs. -n numcores<\/em><\/code><\/td>\nIntel Skylake<\/td>\n4 days -t 4-0<\/code><\/td>\n124x Skylake nodes (32 cores, 5GB\/core)<\/p>\n
-A hpc-proj-code<\/em>\r\n-N 4\r\n-n 128<\/pre>\n
 <\/td>\n
3968 cores available in total.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

*Information is correct as of 7th May 2025<\/em><\/p>\n

Jobscript Template<\/h2>\n

All jobs must specify the partition name (see above) and wallclock runtime limit (see the various time formats<\/a>) for the job.<\/p>\n

Note that the Slurm lines begin with #SBATCH<\/code> – not #$<\/code> as used on SGE. A common mistake is to use a $<\/code> symbol in the Slurm special line. It is NOT #$<\/strong>BATCH<\/code>.<\/p>\n

CPU Jobs<\/h3>\n
#!\/bin\/bash --login\r\n#SBATCH -p partitionname<\/em>    # Required (all jobs)<\/strong> - see table above\r\n#SBATCH -n numcores<\/em>         # Required (parallel jobs)<\/strong>  - defaults to 1 for serial\r\n#SBATCH -t timelimit<\/em>        # Required (all jobs)<\/strong> wallclock timelimit. Max permitted is 7 days (7-0)\r\n\r\n#SBATCH flag resources<\/em>      # See table above for additional high-memory, CPU architecture\r\n                            # and GPU resource flags.\r\n\r\n# We recommend purging your env\r\nmodule purge\r\nmodule load apps\/gcc<\/em>\/someapp<\/em>\/1.2.3<\/em>\r\n\r\n# $SLURM_NTASKS will be set to the number of cores requested above, if your app wants to know.\r\nsomeapp.exe<\/em>\r\n\r\n# Slurm knows to run $SLURM_NTASKS mpi processes if using mpirun\r\nmpirun some_mpi_app.exe<\/em>\r\n<\/pre>\n
For OpenMP parallel applications you can use:<\/p>\n
#!\/bin\/bash --login\r\n#SBATCH -n 1           # 1 task ($SLURM_NTASKS set to 1)\r\n#SBATCH -c numcores<\/em>    # cores-per-task ($SLURM_CPUS_PER_TASK set to this)\r\n#SBATCH -t timelimit<\/em>   # Max wallclock time\r\n\r\n# Inform OpenMP how many cores to use\r\nexport OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK\r\nsome_omp_app.exe<\/em>\r\n<\/pre>\n
Submit the job using<\/p>\n
sbatch jobscript<\/em>\r\n<\/pre>\n
Check on the status of the job using<\/p>\n
squeue\r\n<\/pre>\n
GPU Jobs<\/h3>\n
Please see the GPU Jobs<\/a> page.<\/p>\n
High-memory Jobs<\/h3>\n
Please see the High Memory Jobs<\/a> page.<\/p>\n
More jobscript options<\/h3>\n
For much more information on Slurm jobscript options, compared to SGE options, please see the SGE to Slurm<\/a> reference.<\/p>\n","protected":false},"excerpt":{"rendered":"
In Slurm, jobs are submitted to partitions – think of these as job queues for different types of jobs. In your jobscript, you must specify the partition, number of cores and max “wallclock” time your job is allowed to run for (up to a permitted maximum time.) There is now no default wallclock time – you must specify it. You may also need to specify other flags depending on which parition you submit to. For.. Read more »<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"parent":9105,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-9123","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/9123","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/comments?post=9123"}],"version-history":[{"count":24,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/9123\/revisions"}],"predecessor-version":[{"id":12193,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/9123\/revisions\/12193"}],"up":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/9105"}],"wp:attachment":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/media?parent=9123"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}