- Recent Posts & Updates

Slurm Partitions (Compute Resources)

In Slurm, jobs are submitted to partitions – think of these as job queues for different types of jobs. In your jobscript, you will need to specify the partition, number of cores and max “wallclock” time your job is allowed to run for (up to a permitted maximum time – there is now no default wallclock time – you must specify it.)

The table below provides an overview of the partitions (and the compute resources) currently available in the upgraded CSF3 cluster (Slurm).

The Slurm jobscript template (shown further down this page) gives you a starting point for your jobscript. Much more detail on Slurm jobscript options is available in the SGE to Slurm reference.

You will also need to use new commands on the login node: Slurm uses sbatch, squeue, scancel and srun commands, (not qsub, qstat, qdel and qrsh as used in SGE.)

Reminder: You must specify a Slurm Partition in your jobscript for all job types: -p name

You must also specify a maximum wallclock time limit for your job (up to a permitted maximum time): -t timelimit

See table below.

Additional hardware from the existing CSF3 cluster (SGE) will be migrated to the upgraded cluster (Slurm) in several phases – and the table will be updated accordingly.

CSF3 Slurm Partitions and Compute Resources
Slurm Partition name	Partition Summary	CPU	Max Wallclock	Compute Resources	Notes
AMD “Genoa” Nodes
`-p multicore`	Single-node parallel (2-168 cores) batch jobs. `-n numcores`	AMD Genoa	7 days `-t 7-0`	39x AMD (168 cores per node). 8GB RAM per core.	6,552 cores available in total. This is the main parallel-job partition that most people will use.
`-p interactive`	Small interactive jobs. Optional: `-n numcores` Serial (1-core) batch jobs.	AMD Genoa	1 hour `-t 0-1`	2x AMD (168 cores per node). 8GB RAM per core.	336 cores available in total. `srun` (interactive), `srun-x11` (interactive GUIs) and also `sbatch` (jobscripts) accepted. For testing apps on AMD “Genoa”.
Intel Nodes
`-p multicore_small`	Single-node parallel (2-32 cores) batch jobs. `-n numcores`	Intel	7 days `-t 7-0`	11x Haswell (24 cores, 5GB/core) Optional: `-C haswell` 15x Skylake (32 cores, 6GB/core) Optional: `-C skylake` Or use `-C scratch-old` to land on node with old CSF3 scratch mounted at `/scratch-old/username/`	744 cores available in total (shared with the serial partition below)
`-p serial`	Serial (1-core) batch jobs	Intel	7 days `-t 7-0`	11x Haswell (24 cores, 5GB/core) Optional: `-C haswell` 15x Skylake (32 cores, 6GB/core) Optional: `-C skylake` Or use `-C scratch-old` to land on node with old CSF3 scratch mounted at `/scratch-old/username/`	744 cores available in total (shared with the multicore_small partition above)
High Memory Nodes
`-p highmem`	High memory jobs up to 2TB (1-32 cores). Optional: `-n numcores`	Intel	7 days `-t 7-0`	1x 512GB node (16 cores, 32GB per core) Optional: `-C haswell` 2x 1.5TB node (32 cores, 40GB per core) Optional: `-C cascadelake` 1x 2TB nodes (32 cores, 64GB per core) Optional: `-C icelake`	Job default is 32GB/core unless you ask for more – even if you add a `-C` constraint! Use `--mem=totalG` or `--mem-per-cpu=numG` to request more. [LINK to HIMEM JOBS]
`-p vhighmem`	High memory jobs requiring up to 4TB. Optional: `-n numcores`	Intel	7 days `-t 7-0`	NONE AVAILABLE YET	Restricted access
GPU Nodes
`-p gpuV`	Nvidia V100 GPU batch and interactive jobs. Optional: `-n numcores`	Intel	4 days `-t 4-0`	4x Nvidia V100 16GB GPU RAM 32x Intel host cores 192GB host RAM (5GB/core) NONE AVAILABLE YET Required: `-G NUM` or `--gpus=NUM`	Restricted Access Max 8 CPU-cores per GPU
`-p gpuA`	Nvidia A100 GPU batch and interactive jobs. Optional: `-n numcores`	AMD Milan	4 days `-t 4-0`	1 node: 4x Nvidia A100s 80GB GPU RAM 48x AMD “Milan” host cores 512GB host RAM (10GB/core) NONE AVAILABLE YET Required: `-G NUM` or `--gpus=NUM`	Restricted Access Max 12 CPU-cores per GPU
`-p gpuL`	Nvidia L40s GPU batch and interactive jobs. Optional: `-n numcores`	Intel	4 days `-t 4-0`	1 node: 4x Nvidia L40s 48GB GPU RAM 2 x 24-core Intel “Sapphire Rapids” host cores 512GB host RAM (10GB/core) NONE AVAILABLE YET Required: `-G NUM` or `--gpus=NUM`	Restricted Access Max 12 CPU-cores per GPU
`-p gpuA40GB`	Nvidia A100 (40GB) GPU batch and interactive jobs. Optional: `-n numcores`	AMD Milan	4 days `-t 4-0`	4x Nvidia A100s 40GB GPU RAM 48x AMD “Milan” host cores 512GB host RAM (10GB/core) NONE AVAILABLE YET Required: `-G NUM` or `--gpus=NUM`	Very Restricted Access Max 12 CPU-cores per GPU
HPC Pool
`-p hpcpool`	Multi-node parallel (128-1024 cores) batch jobs. `-n numcores` Only 64 cores for testing: -n 64	Intel Skylake	4 days `-t 4-0` Currently 1 hour for testing: -t 0-1	128x Skylake (32 cores, 5GB/core) -A hpc-proj-code -N 2 -n 64 Only 2 nodes for testing: -N 2	4096 cores available in total.

*Information is correct as of 2nd April 2025

Jobscript Template

All jobs must specify the partition name (see above) and wallclock runtime limit (see the various time formats) for the job.

Note that the Slurm lines begin with #SBATCH – not #$ as used on SGE. A common mistake is to use a $ symbol in the Slurm special line. It is NOT #$BATCH.

#!/bin/bash --login
#SBATCH -p partitionname    # Required (all jobs) - see table above
#SBATCH -n numcores         # Required (parallel jobs)  - defaults to 1 for serial
#SBATCH -t timelimit        # Required (all jobs) wallclock timelimit. Max permitted is 7 days (7-0)

#SBATCH flag resources      # See table above for additional high-memory, CPU architecture
                            # and GPU resource flags.

# We recommend purging your env
module purge
module load apps/gcc/someapp/1.2.3

# $SLURM_NTASKS will be set to the number of cores requested above, if your app wants to know.
someapp.exe

# Slurm knows to run $SLURM_NTASKS mpi processes if using mpirun
mpirun some_mpi_app.exe

For OpenMP parallel applications you can use:

#!/bin/bash --login
#SBATCH -n 1           # 1 task ($SLURM_NTASKS set to 1)
#SBATCH -c numcores    # 84 cores-per-task ($SLURM_CPUS_PER_TASK set to 84)
#SBATCH -t timelimit   # Max wallclock time

# Inform OpenMP how many cores to use
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
some_omp_app.exe

Submit the job using

sbatch jobscript

Check on the status of the job using

squeue

For much more information on Slurm jobscript options, compared to SGE options, please see the SGE to Slurm reference.

Last modified on April 14, 2025 at 3:34 pm by George Leaver