To monitor the resource usage of a running job, you’ll need to access the compute node where that job is running. This then allows you to run commands such as There are two ways to access the compute node where your job is running – see below.<\/p>\n Please note: if you don’t<\/em> have a job running on a compute node, you will not<\/em> be able to access that compute node.<\/p>\n It is now possible (and permitted) to access compute nodes using You can also use If you’ll be using a GUI tool to monitor your job, use:<\/p>\n To limit the amount of time your interactive session will run for, add the Run You can see the job stats – e.g., peak<\/em> memory usage – with the To check a specific jobarray task, use a JOBID of the form jobid_taskid<\/em>:<\/p>\n Alternatively, use the The Depending on the software you are using you may also find memory usage reported in output files.<\/p>\n If at any point during the running of the job, the job’s peak memory usage goes above<\/em> the limit that the job is permitted to use, then the job will be terminated by the batch system.<\/p>\n The You may see the following in your You will need to resubmit your job, either requesting more cores (if using the standard partitions) or use a high memory partition<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":" Monitoring Existing Jobs To monitor the resource usage of a running job, you’ll need to access the compute node where that job is running. This then allows you to run commands such as top or htop (for CPU\/host monitoring), or nvitop (for GPU monitoring). There are two ways to access the compute node where your job is running – see below. Please note: if you don’t have a job running on a compute node, you.. Read more »<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"parent":9105,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-9244","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/9244","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/comments?post=9244"}],"version-history":[{"count":20,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/9244\/revisions"}],"predecessor-version":[{"id":11963,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/9244\/revisions\/11963"}],"up":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/9105"}],"wp:attachment":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/media?parent=9244"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}top<\/code> or htop<\/code> (for CPU\/host monitoring), or nvitop<\/code> (for GPU monitoring).<\/p>\nUsing ssh<\/h3>\n
ssh<\/code> to monitor your jobs.<\/p>\n\r\n# On the login node, find out where your job is running\r\nsqueue\r\n JOBID PRIORITY PARTITION NAME USER ST ... NODELIST\r\n 123456 0.000054 gpuA myjob mabcxyz1 R ... node860<\/strong><\/em>\r\n\r\n# Now access the compute node\r\nssh node860<\/strong><\/em>\r\n\r\n# Run your monitoring command - for example:\r\ntop\r\nhtop\r\n# For Nvidia GPU jobs:\r\nmodule load tools\/bintools\/nvitop\r\nnvitop\r\n\r\n# To return to the login node:\r\nexit\r\n<\/pre>\n
Using srun<\/h3>\n
srun<\/code> to login to the node where the job is running, which gives you an interactive Slurm session on the node:<\/p>\n\r\nsrun --jobid JOBID<\/em> --pty bash\r\n<\/pre>\n
\r\nsrun-x11 --jobid JOBID<\/em> # NO \"--pty bash\" needed for srun-x11\r\n<\/pre>\n
-t timespec<\/em><\/code> flag to the srun<\/code> command. For example: -t 10<\/code> for 10 minutes.<\/p>\nEnding your monitoring session<\/h3>\n
exit<\/code> to end your interactive monitoring session. This will NOT terminate your batch job. You’ll return to the login node.<\/p>\nJob Statistics<\/h2>\n
A completed (successful) job<\/h3>\n
seff<\/code> command, passing in a JOBID:<\/p>\n[mabcxyz1<\/em>@login1[csf3] ~]$ seff 12345<\/em>\r\nJob ID: 12345\r\nCluster: csf3.man.alces.network\r\nUser\/Group: username<\/em>\/xy<\/em>01\r\nState: COMPLETED (exit code 0)\r\nNodes: 1\r\nCores per node: 2 # 2 CPUs requested in the jobscript\r\nCPU Utilized: 00:04:13\r\nCPU Efficiency: 49.41% of 00:08:32 core-walltime # <50% CPU usage suggests only 1 CPU was needed\r\nJob Wall-clock time: 00:04:16\r\nMemory Utilized: 21.45 GB<\/strong> # Peak memory usage<\/strong>\r\nMemory Efficiency:33.5%<\/strong> of 64.00 GB # A low memory efficiency means this job did NOT need\r\n # to use the himem partition. You should check this.\r\n<\/pre>\n
seff 12345_501\r\n<\/pre>\n
sacct<\/code> command to obtain various stats about a job;<\/p>\nsacct -j 12345<\/em>\r\n\r\n# Or to just get the memory usage\r\nsacct -j 12345<\/em> -o maxrss\r\n<\/pre>\n
sacct<\/code> command offers lots of options – use man sacct<\/code> to get more info.<\/p>\nA terminated (out of memory) job<\/h3>\n
seff<\/code> command will show:<\/p>\n[mabcxyz1<\/em>@login1[csf3] ~]$ seff 12345\r\nState: OUT_OF_MEMORY (exit code 0)\r\n<\/pre>\n
slurm-12345.out<\/code> file:<\/p>\n[mabcxyz1<\/em>@login1[csf3] ~]$ cat slurm-12345.out\r\n\r\n\/var\/spool\/slurmd\/job12345\/slurm_script: line 4: 1851022 Killed .\/some-app.exe -in data.dat -out results.dat\r\nslurmstepd: error: Detected 1 oom_kill event in StepId=12345.batch. Some of the step tasks have been OOM Killed.\r\n #\r\n # OOM is \"out of memory\" - this means Slurm killed your job\r\n # because it tried to use more memory than allowed.\r\n<\/pre>\n