{"id":9711,"date":"2025-05-06T17:07:32","date_gmt":"2025-05-06T16:07:32","guid":{"rendered":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/?page_id=9711"},"modified":"2026-02-24T09:50:08","modified_gmt":"2026-02-24T09:50:08","slug":"gpu-jobs-slurm","status":"publish","type":"page","link":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/batch-slurm\/gpu-jobs-slurm\/","title":{"rendered":"Nvidia GPU Jobs (Slurm)"},"content":{"rendered":"

Access<\/h2>\n

This page covers the Nvidia A100 (80GB), A100 (40GB) and L40S GPUs in Slurm.<\/p>\n

Notes for Slurm users – please read:<\/p>\n
    \n
  1. All users have access to the A100 (80GB) and L40S GPUs<\/strong> – you do not<\/em> need to submit a ticket to request access.<\/li>\n
  2. The A100 (40<\/strong>GB) GPUs are limited to a small number of users from a specific group.<\/li>\n
  3. The H200 GPUs have a specific access policy<\/a> (please check before requesting access.)<\/li>\n
  4. See the Slurm Partitions page<\/a> for the available hardware.<\/li>\n
  5. GPUs now run in “DEFAULT” compute mode, not “EXCLUSIVE_PROCESS”. Hence you can run multiple processes on the GPUs assigned to your job. Slurm will prevent other jobs from accessing the GPUs assigned to your job.<\/li>\n
  6. Feb 2026 update<\/strong>: a change has been made to the way in which the 4-GPU free-at-point-of-use GPU limits are applied – see below for details<\/a>.\n<\/ol>\n<\/div>\n

    Free at Point of Use access<\/h3>\n

    Please use the gpuA<\/code> and\/or gpuL<\/code> paritions.<\/p>\n

    20-Feb-2026: The current limits are<\/strong>: up to four A100(80GB) or L40S GPUs in use at any one time<\/strong>. These can now be all of the same GPU type. For example you can use 4 x A100(80GB) GPUs or 4 x L40S GPUs or a mixture of the two. You can submit as many jobs as you wish – the system will run them within the limits. <\/p>\n

    Users that have previous had temporary increased limits via the gpuax4<\/code> or gpulx4<\/code> QOSs should no longer use those QOSs. Simply submit your 4xGPU jobs and the new f@pou limits will allow the jobs to run.<\/p>\n

    PLEASE NOTE:<\/strong> Now that the free-at-point-of-use limits have been modified to allow 4 of the same type of GPUs to be used, we will NOT accept requests for access to more than 4 GPUs.<\/strong> <\/p>\n

    Contributor access<\/h3>\n

    Members of groups that have contributed GPUs will be informed of their limits when CSF accounts are created.<\/p>\n

    A100 (40GB) GPU access<\/h3>\n

    The A100-40G nodes have been funded by a specific research group and so access is very<\/em> restricted. You will be informed if you have access to these GPUs.<\/p>\n

    GPU batch job submission (Slurm)<\/h2>\n

    For jobs that require GPUs – running on one or more GPUs in a single compute node. A jobscript template is shown below. Please also consult the Partitions<\/a> page for details on available compute resources.<\/p>\n

    Please also consult the software page<\/a> for the code \/ application you are running for advice on running that application<\/strong><\/em>.<\/p>\n

    A GPU job script will run in the directory (folder) from which you submit the job. The jobscript takes the form:<\/p>\n

    #!\/bin\/bash --login\r\n### Choose ONE of the following partitions depending on your permitted access<\/strong>\r\n#SBATCH -p gpuA              # A100 (80GB) GPUs  [up to 12 CPU cores per GPU permitted]\r\n#SBATCH -p gpuA40GB          # A100 (40GB) GPUs  [up to 12 CPU cores per GPU permitted]\r\n#SBATCH -p gpuL              # L40S GPUs         [up to 12 CPU cores per GPU permitted]\r\n### Required flags\r\n#SBATCH -G N<\/em>                 # (or --gpus=N<\/em>) Number of GPUs \r\n#SBATCH -t 1-0               # Wallclock timelimit (1-0 is one day, 4-0 is max permitted)\r\n### Optional flags\r\n#SBATCH -n numcores<\/em>          # (or --ntasks=) Number of CPU (host) cores (default is 1)\r\n                             # See above for number of cores per GPU you can request.\r\n                             # Also affects host RAM allocated to job unless --mem=num<\/em> used.\r\n\r\nmodule purge\r\nmodule load libs\/cuda\/x.y.z<\/em>  # See below for specific versions\r\n\r\necho \"Job is using $SLURM_GPUS GPU(s) with ID(s) $CUDA_VISIBLE_DEVICES and $SLURM_NTASKS CPU core(s)\"\r\ngpuApp args ...<\/em>\r\n<\/pre>\n
    Note that the amount of host<\/em> RAM your job has access to is dependent on the number of CPU cores<\/em> you request, unless you request a specific amount of host memory for your job using the --mem=num<\/em>G<\/code> or --mem-per-gpu=num<\/em>G<\/code> flags. Note that the default units of memory are megabytes if no units are given (use G<\/code> for gigabytes.)<\/p>\n\n\n\n\n\n\n
    GPU<\/th>\nMax host cores per GPU<\/th>\nHost RAM per core (GB)<\/th>\nMax host RAM per GPU (GB)<\/th>\n<\/tr>\n
    A100(80GB)<\/td>\n12<\/td>\n10.4<\/td>\n125<\/td>\n<\/tr>\n
    A100(40GB)<\/td>\n12<\/td>\n10.4<\/td>\n125<\/td>\n<\/tr>\n
    L40S<\/td>\n12<\/td>\n10.4<\/td>\n125<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
    Available Hardware and Resources<\/h2>\n
    Please see the Partitions<\/a> page for details on available compute resources.<\/p>\n
    Software Applications<\/h2>\n
    A range of GPU capable software is available on the CSF.<\/p>\n
    List of installed GPU capable software.<\/a>
    \n    List of installed Machine Learning specific software.<\/a><\/p>\n
    <\/a><\/p>\n
    GPU Hardware and Driver<\/h2>\n
    The CSF (Slurm) will contain the following GPU nodes, which offer different types of Nvidia GPUs and host CPUs.<\/p>\n
    For the current<\/em> list of resources please see<\/strong> the Slurm Partitions page<\/a>.<\/p>\n
    The following information is for reference only – it does not mean that all of these nodes are currently available in Slurm!<\/strong><\/p>\n
    ***NO LONGER  IN SERVICE***<\/strong>17 GPU nodes each hosting 4 x Nvidia v100 GPUs (16GB GPU RAM) giving a total of 68 v100 GPUs. The node spec is:<\/p>\n