Deleting or Modifying Jobs (Slurm)

Deleting jobs in Slurm

On occasion it may be necessary to remove a waiting job from the queue, or to stop a running job (and remove it from the queue). This is done using scancel . For example

scancel 18076
         #
         # replace 18076 with your job id

# To delete all of my jobs (running and pending) CAUTION!!!
scancel --me

You can identify the jobid typing the command squeue on its own.

Deleting Job Array Tasks

See the information on Job Arrays (Slurm) for how to delete all tasks in the job array or specific tasks.

Modifying jobs in slurm

Running jobs

Do not attempt to modify running jobs, if you have made a mistake in specifying job resources and the job is already running, “scancel” the job, clean up the outputs, correct your sbatch script and resubmit.

Pending jobs

scontrol can be used to modify pending jobs, use this with care as it is possible to put your job into a non-runnable state.

If you are in any doubt about how to edit jobs, use scancel, edit your sbatch script and resubmit. AI responses (search and chatbots) do not produce reliable Slurm commands if you do not know what you need to ask for.

Change time limit

Start by holding the job you want to change:

scontrol hold jobid=1234567

this will prevent the job from starting while you make changes. Then review the details of the job you want to change:

scontrol show jobid=1234567

Then change the time limit:

scontrol update jobid=1234567 TimeLimit=4-0

where the jobid is the id of your job and the TimeLimit can be set to any valid duration. If you get ‘permission denied’ then you have asked for a duration that is not permitted by the partition.

Check the changes you have made:

scontrol show jobid=1234567

Once you are happy, release the job:

scontrol release jobid=1234567

Change Partiton

For example, if you have a job of 32 cores or less that is experiencing a long wait (over 24 hours) for the multicore partition you could try the multicore_small instead.

Start by holding the job you want to change:

scontrol hold jobid=1234567

This will prevent the job from starting while you make changes. Then change the partition:

scontrol update jobid=1234567 Partition=multicore_small

The squeue command should show that it is now queuing for multicore_small.

Change CPU resources

Start by holding the job you want to change:

scontrol hold jobid=1234567

This will prevent the job from starting while you make changes. Then review the details of the job you want to change:

scontrol show jobid=1234567

The relevant parameters are:

NumNodes=1 NumCPUs=XX NumTasks=YY CPUs/Task=ZZ

whatever you change, NumCPUs must = NumTasks * CPUs/Task; However, CPUs/Task cannot be changed. For example, with:

NumNodes=1 NumCPUs=22 NumTasks=22 CPUs/Task=1

a valid change is:

scontrol update jobid=1234567 NumCPUs=20 NumTasks=20

but with:

NumNodes=1 NumCPUs=44 NumTasks=22 CPUs/Task=2

examples of valid changes are:

scontrol update jobid=1234567 NumCPUs=40 NumTasks=20         # Because CPUs/Task is fixed at 2
scontrol update jobid=1234567 NumCPUs=20 NumTasks=10         # Because CPUs/Task is fixed at 2

Check the changes you have made:

scontrol show jobid=1234567

Once you are happy, release the job:

scontrol release jobid=1234567