- Recent Posts & Updates

Slurm Partitions (Compute Resources)

The table below provides an overview of the compute resources currently available in the upgraded CSF3 cluster (Slurm).

These are made available to your job via Slurm Partitions. Think of these as job queues for different types of jobs.

You must specify a Slurm Partition in your jobscript for all job types: -p name

You must also specify a maximum wallclock time limit for your job (up to a permitted maximum time): -t timelimit

See table below.

Additional hardware from the existing CSF3 cluster (SGE) will be migrated to the upgraded cluster (Slurm) in several phases – and the table will be updated accordingly.

CSF3 Slurm Partitions and Compute Resources
Slurm Partition name	Partition Summary	CPU	Max Wallclock	Compute Resources	Notes
AMD “Genoa” Nodes
`-p multicore`	Single-node parallel (2-168 cores) batch jobs. `-n numcores`	AMD Genoa	7 days `-t 7-0`	39x AMD (168 cores per node). 8GB RAM per core.	6,552 cores available in total. This is the main parallel-job partition that most people will use.
`-p interactive`	Small interactive jobs. Optional: `-n numcores` Serial (1-core) batch jobs.	AMD Genoa	1 hour `-t 0-1`	2x AMD (168 cores per node). 8GB RAM per core.	336 cores available in total. `srun` (interactive), `srun-x11` (interactive GUIs) and also `sbatch` (jobscripts) accepted. For testing apps on AMD “Genoa”.
Intel Nodes
`-p multicore_small`	Single-node parallel (2-32 cores) batch jobs. `-n numcores`	Intel	7 days `-t 7-0`	1x Haswell (24 cores, 5GB/core) Optional: `-C haswell` 1x Skylake (32 cores, 6GB/core) Optional: `-C skylake` Or use `-C scratch-old` to land on node with old CSF3 scratch mounted at `/scratch-old/username/`	58 cores available in total
`-p serial`	Serial (1-core) batch jobs	Intel	7 days `-t 7-0`	1x Haswell (24 cores, 5GB/core) Optional: `-C haswell` 1x Skylake (32 cores, 6GB/core) Optional: `-C skylake` Or use `-C scratch-old` to land on node with old CSF3 scratch mounted at `/scratch-old/username/`	58 cores available in total
High Memory Nodes
`-p highmem`	High memory jobs up to 2TB (1-32 cores). Optional: `-n numcores`	Intel	7 days `-t 7-0`	1x 512GB node (16 cores, 32GB per core) Optional: `-C haswell` 1x 1.5TB node (32 cores, 40GB per core) Optional: `-C cascadelake` 1x 2TB nodes (32 cores, 64GB per core) Optional: `-C icelake`	Job default is 32GB/core unless you ask for more – even if you add a `-C` constraint! Use `--mem=totalG` or `--mem-per-cpu=numG` to request more. [LINK to HIMEM JOBS]
`-p vhighmem`	High memory jobs requiring up to 4TB. Optional: `-n numcores`	Intel	7 days `-t 7-0`	NONE AVAILABLE YET	Restricted access
GPU Nodes
`-p gpuV`	Nvidia V100 GPU batch and interactive jobs. Optional: `-n numcores`	Intel	4 days `-t 4-0`	4x Nvidia V100 16GB GPU RAM 32x Intel host cores 192GB host RAM (5GB/core) Required: `-G NUM` or `--gpus=NUM`	Restricted Access Max 8 CPU-cores per GPU
`-p gpuA`	Nvidia A100 GPU batch and interactive jobs. Optional: `-n numcores`	AMD Milan	4 days `-t 4-0`	1 node: 4x Nvidia A100s 80GB GPU RAM 48x AMD “Milan” host cores 512GB host RAM (10GB/core) Required: `-G NUM` or `--gpus=NUM`	Restricted Access Max 12 CPU-cores per GPU
`-p gpuL`	Nvidia L40s GPU batch and interactive jobs. Optional: `-n numcores`	Intel	4 days `-t 4-0`	1 node: 4x Nvidia L40s 48GB GPU RAM 2 x 24-core Intel “Sapphire Rapids” host cores 512GB host RAM (10GB/core) Required: `-G NUM` or `--gpus=NUM`	Restricted Access Max 12 CPU-cores per GPU
`-p gpuA40GB`	Nvidia A100 (40GB) GPU batch and interactive jobs. Optional: `-n numcores`	AMD Milan	4 days `-t 4-0`	4x Nvidia A100s 40GB GPU RAM 48x AMD “Milan” host cores 512GB host RAM (10GB/core) NONE AVAILABLE YET Required: `-G NUM` or `--gpus=NUM`	Very Restricted Access Max 12 CPU-cores per GPU

*Information is correct as of 2nd April 2025

All jobs must specify the partition name (see above) and wallclock runtime limit (see the various time formats) for the job:

#!/bin/bash --login
#SBATCH -p partitionname    # Required (all jobs) - see table above
#SBATCH -n numcores         # Required (parallel jobs)  - defaults to 1 for serial
#SBATCH -t timelimit        # Required (all jobs) wallclock timelimit. Max permitted is 7 days (7-0)

#SBATCH flag resources      # See table above for additional high-memory, CPU architecture and GPU resource flags.

# We recommend purging your env
module purge
module load apps/gcc/someapp/1.2.3

# $SLURM_NTASKS will be set to the number of cores requested above, if your app wants to know.
someapp.exe

Submit the job using

sbatch jobscript

Check on the status of the job using

squeue

Last modified on April 3, 2025 at 2:47 pm by Chris Grave