{"id":9877,"date":"2025-05-14T13:15:39","date_gmt":"2025-05-14T12:15:39","guid":{"rendered":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/?page_id=9877"},"modified":"2026-02-23T15:49:50","modified_gmt":"2026-02-23T15:49:50","slug":"high-memory-jobs-slurm","status":"publish","type":"page","link":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/batch-slurm\/high-memory-jobs-slurm\/","title":{"rendered":"High Memory Jobs (Slurm)"},"content":{"rendered":"

Default Memory on the CSF<\/h2>\n

The AMD 168-core Genoa nodes (-p multicore<\/code>) have 8GB RAM per core. If your job needs more RAM on these nodes, request more cores.<\/p>\n

The standard Intel nodes (-p serial<\/code> and -p multicore_small<\/code>) have 4GB to 6GB of RAM per core. If your job needs more RAM on these nodes, request more cores.<\/p>\n

High-memory Intel nodes are also available, offering up to 2TB RAM in total , and if really needed, up to 4TB, and are described below<\/a>. First we describe how to check if your jobs are running out of memory (RAM).<\/p>\n

We only have a small number of high memory nodes. They should only<\/strong> be used for work that requires significant amounts of RAM. Incorrect use of these nodes may result in restrictions being placed on your account<\/strong>.<\/div>\n

How to check memory usage of your job<\/h2>\n

The batch system keeps track of the resources your jobs are using, and also records statistics about the job once it has finished.<\/p>\n

A completed (successful) job<\/h3>\n

You can see the peak<\/em> memory usage with the seff<\/code> command, passing in a JOBID:<\/p>\n

[mabcxyz1<\/em>@login1[csf3] ~]$ seff 12345<\/em>\r\nJob ID: 12345\r\nCluster: csf3.man.alces.network\r\nUser\/Group: username<\/em>\/xy<\/em>01\r\nState: COMPLETED (exit code 0)\r\nNodes: 1\r\nCores per node: 2\r\nCPU Utilized: 00:04:13\r\nCPU Efficiency: 49.41% of 00:08:32 core-walltime\r\nJob Wall-clock time: 00:04:16\r\nMemory Utilized: 21.45 GB<\/strong>                       # Peak memory usage<\/strong>\r\nMemory Efficiency:33.5%<\/strong> of 64.00 GB             # A low memory efficiency means this job did NOT need\r\n                                                # to use the himem partition. You should check this.\r\n<\/pre>\n
To check a specific jobarray task, use a JOBID of the form jobid_taskid<\/em>:<\/p>\n
seff 12345_501\r\n<\/pre>\n
Alternatively, use the sacct<\/code> command to obtain various stats about a job;<\/p>\n
sacct -j 12345<\/em>\r\n\r\n# Or to just get the memory usage\r\nsacct  -j 12345<\/em> -o maxrss\r\n<\/pre>\n
The sacct<\/code> command offers lots of options – use man sacct<\/code> to get more info.<\/p>\n
Depending on the software you are using you may also find memory usage reported in output files.<\/p>\n
A terminated (out of memory) job<\/h3>\n
If at any point during the running of the job, the job’s peak memory usage goes above<\/em> the limit that the job is permitted to use, then the job will be terminated by the batch system.<\/p>\n
The seff<\/code> command will show:<\/p>\n
[mabcxyz1<\/em>@login1[csf3] ~]$ seff 12345\r\nState: OUT_OF_MEMORY (exit code 0)\r\n<\/pre>\n
You may see the following in your slurm-12345.out<\/code> file:<\/p>\n
[mabcxyz1<\/em>@login1[csf3] ~]$  cat slurm-12345.out\r\n\r\n\/var\/spool\/slurmd\/job12345\/slurm_script: line 4: 1851022 Killed             .\/some-app.exe -in data.dat -out results.dat\r\nslurmstepd: error: Detected 1 oom_kill event in StepId=12345.batch. Some of the step tasks have been OOM Killed.\r\n                               #\r\n                               # OOM is \"out of memory\" - this means Slurm killed your job\r\n                               # because it tried to use more memory than allowed.\r\n<\/pre>\n
You will need to resubmit your job, either requesting more cores (if using the standard partitions) or use a high memory partition – see next.
\n<\/a><\/p>\n
Submitting High Memory Jobs<\/h2>\n
Memory is a “consumable” resource in the himem<\/code> and vhimem<\/code> partitions in the CSF (Slurm) cluster – you can specify how many cores AND<\/em> how much memory your job requires (in the multicore<\/code> partition, you can’t do this – instead you get a fixed amount of RAM per core.)<\/p>\n
For users that have previously used the CSF (SGE) cluster, there are no<\/em> mem1500<\/code>, mem2000<\/code> or mem4000<\/code> flags! There, memory was not<\/em> a consumable – you had to request a certain number of cores, and the -l memXXXX<\/code> flag would land the job on a node that gave a fixed amount of memory-per-core.<\/p>\n
On the CSF (Slurm) cluster, your jobs will land on any<\/em> of the high-memory nodes to meet your core and memory requirements. You can optionally<\/em> specify a CPU architecture if that is important to your job, but this will restrict the pool of nodes available for Slurm to choose from, and so might lead to longer queue-wait times.<\/p>\n
So, in summary, specify the amount<\/em> of memory your job requires. If you need more than one core, specify the number of cores. But the number of cores does not necessarily determine the amount of memory.<\/p>\n
High Memory Resources<\/h3>\n
The following resources are available to your job.<\/p>\n
The partition (and wallclock limit) are required. All other flags are optional.<\/p>\n
Note:<\/p>\n