This page offers new CSF users a tutorial that covers usage of the Slurm batch system<\/em> to run a simple job on the CSF.<\/p>\n The tutorial also provides some information about the storage areas on the CSF, and also some common Linux commands used to manage your files.<\/p>\n After doing the tutorial you’ll be able to use the CSF. A further tutorial is also available for running more complicated parallel jobs.<\/p>\n If you are interested in attending the 1-day Intro to CSF training course which runs a couple of times each semester, please take a look at the course booking page<\/a> for details of the schedule and availability.<\/p>\n Before we begin the tutorial we’ll explain what the batch system<\/em> is and why we need to use it.<\/p>\n Click on each header below to expand the section:<\/p>\n Initially a batch system can be thought of as a job queue. You submit jobs to the queue and the system will pick them out of the queue to run them.<\/p>\n The jobs will do whatever commands you ask them to do (for example run an app such as a chemistry app, or a bioinformatics app or whatever application is appropriate to your work).<\/p>\n When the jobs finish, you should have some new files containing the results.<\/p>\n At this point you might be thinking you don’t like the idea of your work (jobs) waiting in a queue. How long will it queue for? Why can’t it just run immediately? Read on to find out more. The applications you’ll be running on the CSF usually need different amounts of memory, number of CPU cores or even GPUs. <\/p>\n You can request these specific resources for your job. Need a GPU? Simply request it in your job. Need a lot of memory to process a huge dataset? Simply ask for it.<\/p>\n The batch system<\/em> ensures your job only runs when all<\/em> of the required resources are available. It then allocates those resources to your job (so that it runs correctly) and makes sure no other jobs can grab your<\/em> resources.<\/p>\n But don’t worry if you’re not sure what resources you’ll need – there are sensible defaults. After trying the defaults, you might find your app needs more memory to process your data, or that it can use more CPU cores to make it run faster.<\/p>\n So you might find that your first few attempts at running jobs don’t actually complete successfully. Maybe you’ll need to run the jobs again but request more memory. Don’t worry – failed jobs don’t do any harm. You can simply delete the output files from these failed jobs (if there are any), modify your jobscript<\/em> to ask for more resources (more memory, CPUs, …) and then resubmit your jobs. The batch system also ensures fair usage for you and others – there are many<\/em> users and jobs on the system, all making different demands of the resources (memory, CPU cores, GPUs) and so allowing the batch system to choose exactly when to run your job is the only sensible way of running the system. <\/p>\n The fact that jobs are starting and finishing all the time means you rarely have to wait very long for your requested resource to become free so that your jobs can start.<\/p>\n There are other factors which control when jobs run (and how many of your jobs can run at the same time) but the use of a job queue should not<\/em> put you off using the system. An added bonus of a batch system is that once you’ve submitted<\/em> your jobs to the system, you don’t actually need to remain logged in. You can log off, go home or go to a meeting or do something else with your PC\/laptop.<\/p>\n Meanwhile the batch system will run your jobs. It can even email you when a job has finished.<\/p>\n Without<\/em> a batch system, you would need to remain logged in to the CSF until the job had finished<\/em>, which could be a problem for a simulation that takes several days to complete. Something to note about batch jobs is that you never see an application’s graphical user interface (GUI), if it has one. Batch jobs run without any interaction – all options \/ flags \/ input files etc will be specified on the command-line in a jobscript<\/em> (more on those later).<\/p>\n When the app is running, all output will be saved to files. This will be a new way of working if you are used to running an app in a desktop environment (e.g., on Windows). Running your code or application directly on the login nodes is not<\/strong><\/em> permitted.<\/p>\n The login nodes are for other tasks (transferring files on and off the system, editing jobscripts, submitting jobs to the system, checking results.) They don’t have a lot of memory, nor many cores, so trying to run your apps there is inefficient and may also adversely affect other users.<\/p>\n Applications found running on the login nodes may be killed by the sysadmins without warning.<\/strong><\/p>\n In this tutorial you can try out the sample job below – it shouldn’t take more than 10 minutes to work through the instructions on this page.\n<\/p><\/div>\n<\/details>\n The CSF3 now runs the Slurm batch system.<\/p>\n The three main Slurm commands you use are This tutorial assumes you are already<\/em> logged in to the CSF – please see the login instructions<\/a> for more information.<\/p>\n Here we describe in detail how to submit a simple, first job to the batch system. Please read all of the text, don’t just look for the commands to type, as it will explain why<\/em> you need to run the commands.<\/p>\n We will run a serial<\/em> job – i.e., it uses only one CPU core. We’ll see later that many of the real applications on the CSF can use more than one CPU core (a multi-core<\/em> job) to speed up their processing, giving you the results sooner.<\/p>\n You could also request more memory than the default 5-6GB of RAM. You could also request a GPU.<\/p>\n But initially a simple 1-core (serial<\/em>) job will help you become familiar with the principles of the batch system. These jobs are very common – you may well want to use this type of job in your real work after the tutorial.<\/p>\n Please remember: Do not<\/strong> simply run jobs on the login node – use the batch system as described below.<\/p>\n In the following steps we will be creating a jobscript<\/em> file. We will explain more about the file in the next step. The job will also create some files (any output generated by the job is saved to files).<\/p>\n Hence we first create a directory (folder) for the job to keep all of the files together in one place. This is important – you will likely run a lot of jobs on the CSF so it will be easier to manage all of your work if you keep your files tidy.<\/p>\n When you log in to the CSF you are placed in your home directory<\/em>. This area of storage is private to you and, importantly, is backed-up (not all storage areas on the CSF are backed-up). It is strongly recommended<\/strong> that you keep important files in your home directory<\/em> for safe keeping – and this includes your jobscripts.<\/p>\n Once you have logged in<\/a> you’ll be at the command-line prompt<\/em>:<\/p>\n Now create a directory (usually referred to as a folder<\/em> in Windows or MacOS) in your CSF home<\/em> storage area, for our first test job, by running the following commands at the prompt:<\/p>\n Notice that the prompt<\/em> has changed to indicate you have moved in to the The jobscript<\/em> file is the thing you submit to the batch system (i.e, the queue of jobs.) It is just a simple plain-text file. It serves two main purposes:<\/p>\n Hence jobscripts should be considered part of your work that needs to be kept securely in your home directory<\/em>. They are a record of how you ran a simulation or analysis, for example, or how you processed a particular dataset. Jobscripts are therefore part of your research methods.<\/p>\n We now use If you see an error similar to<\/p>\n Please ensure you have logged-in using the method that allows GUI apps to be used – this mostly affect Mac user, so please take a look at the Mac login instructions<\/a>.<\/p>\n However, you must run the following command<\/strong> on the login node to convert the file from Windows format to Linux format otherwise the job will report an error when you submit it to the batch system (this is only needed for jobscripts, not<\/strong> any other file)<\/p>\n But we recommend that Windows users install MobaXterm<\/a> to log in to the CSF. You can then run Here’s the jobscript content – put this in the text file you are creating<\/strong> either in gedit (run on the CSF login node) or notepad (run on your Windows PC): Note: lines must NOT be indented in your text file – there should NOT be any spaces at the start of the lines.<\/strong> Cut-n-paste from this web page will work correctly in most browsers in that it won’t copy any leading space.<\/p>\n This BASH script<\/em> has three parts:<\/p>\n Linux provides several scripting<\/em> languages but BASH is the one you use at the command-line once you’ve logged in. So we usually use it for jobscripts too. This means that any commands you would normally type at the command-line can also go into your jobscript to be run as part of a batch job.<\/li>\n In this simple jobscript the lines are:<\/p>\n In this example we have a trivial job which runs simple Linux commands to output the date and time, followed by the name of the compute node on which the job runs, then waits for two minutes and finally outputs the date and time again. In a real jobscript you would do something more interesting and useful – e.g., run MATLAB or Abaqus or a chemistry program.\n<\/li>\n<\/ol>\n We now copy the jobscript to your scratch<\/em> area.<\/p>\n PLEASE NOTE:<\/strong> the scratch area is a temporary<\/em> area – files unused in the last 3-months can be deleted by the system to free up space<\/strong>. You should always have a copy of important files in your home<\/em> area (or other research data storage visible on the CSF that your research group may have access to). Think of scratch<\/em> as fast, temporary<\/em> storage – if your job reads and writes large files it will be faster if run from scratch.<\/p>\n A good way of working is to create your important files in the home<\/em> area, then copy them to scratch when you need to use them in your jobs. That way you always have a safe copy in your home area.<\/p>\n So let’s copy<\/em> our jobscript to the scratch<\/em> area (we keep the original in our home<\/em> area for safe keeping):<\/p>\n We can now go into<\/em> the scratch area:<\/p>\n Our scratch directory is now our current working directory<\/em>. When we submit the job to the batch queue (see next step) it will run in the scratch area – remember, the job runs from which ever directory you are in<\/em> when you submit the job.<\/p>\n Any files that the job generates will also be written to the scratch area and if your job wants to read input data files (ours doesn’t in this example) then it would try to read them from the scratch area.<\/p>\n You will notice the prompt on the command-line will change to indicate where you are currently located:<\/p>\n Recap: So far we have created a directory for the jobscript in our home<\/em> area, written a jobscript text file there (where it is stored safely on backed-up storage), then copied it to the fast temporary scratch<\/em> storage and changed directory<\/em> to our scratch area where we’ll run the job from.<\/p>\n The next step is to actually submit the job to the batch system. Suppose, the above script is saved in a file called The job id To confirm that your job is queued, or perhaps already running, enter the command<\/p>\n If the job is still pending<\/strong> (waiting to run) the output from If your job is already running<\/strong>, the output will look like the following – notice the ST column shows “R” for running and the NODELIST column shows the name of the compute node<\/em> where your job is currently running. This shows that your job is not<\/em> using the login node, even though that’s where you are running commands, but is instead using one of the more powerful servers in the CSF.<\/p>\n If your jobs have finished, If something is wrong with your jobscript you’ll see F<\/strong> or some other code. There might also be a HINT: the most common error<\/strong> is making a typo in your jobscript. Errors don’t do any harm to the CSF, they just mean your job won’t do what you want it to do. Check the contents of your jobscript (see above for the original text.) <\/p>\n A typo on the first line of the jobscript is often the cause. Please type it carefully: If there is no output from the Each job will output least one file, containing any output that would normally have been printed to screen. This can including normal information from your app and also error message, if any occurred.<\/p>\n Let’s list<\/em> the files in the current directory using the Linux We can see our original jobscript To look at the contents of the output file:<\/p>\n In this example the output file contains:<\/p>\n shows the date, twice with a difference of 120 seconds (2 minutes), and the name of the compute node<\/em> on which the job ran, as expected (refer back to the commands we ran in our first jobscript<\/a>).<\/p>\n Note that the names of the output file is always, by default, This would generate an output file named You’ve now successfully run a job on the CSF. It was a simple 1-core<\/em> job (it used only one CPU core) to run some basic Linux commands. The output of the commands was captured in to the Earlier we said that the scratch<\/em> storage area is temporary (but fast). Hence if we want to keep the results from this job then we should copy them back to the home<\/em> storage area. Let’s assume we DO want to keep the output from this job. Apart from the usual Background: What is a batch system and why use it?<\/h2>\n
Join the queue…<\/summary>\n
\n<\/details>\nAsk for extra memory, cores or a GPU?<\/summary>\n
\n<\/details>\nFair usage<\/summary>\n
\n<\/details>\nLet the CSF get on with it<\/summary>\n
\n<\/details>\nNo GUI<\/summary>\n
\n<\/details>\nCan I just run my app on the Login Node?<\/summary>\n
Which batch system does the CSF use?<\/summary>\n
sbatch<\/code>, squeue<\/code> and possibly scancel<\/code>.<\/p>\n<\/details>\n10 Minute Tutorial: Submitting a First Job to the Batch System<\/h2>\n
What type of job will we run?<\/h3>\n
Step 0: Create a Folder for the Job Files<\/h3>\n
\r\n[mxyzabc1<\/em>@login1[csf3]<\/span> ~]$<\/strong> you will type your commands here, \"at the prompt\"<\/em>\r\n ^ ^ ^ ^\r\n | | | | \r\n | | | +--- The directory (folder) you are currently in<\/em>.\r\n | | | ~ means your home<\/em> folder, which is your private folder.\r\n | | |\r\n | | +--- Name of the system\r\n | |\r\n | +--- Name of the login node (some systems have more than one login node)\r\n |\r\n +--- Your username appears here\r\n<\/pre>\n\r\n# All of these commands are run on the CSF login node at the prompt<\/em>\r\nmkdir ~\/first-job # Make (create) the directory (folder)\r\ncd ~\/first-job # Change to (go into) the directory (folder)\r\n<\/pre>\n
first-job<\/code> folder:<\/p>\n\r\n[mxyzabc1<\/em>@login1[csf3]<\/span> first-job<\/strong>]$ \r\n ^\r\n |\r\n +--- The prompt shows we are now in the first-job folder\r\n<\/pre>\nStep 1: Create a “Jobscript” – a job description file<\/h3>\n
\n
gedit<\/code><\/a> or xnedit<\/code><\/a> or another editor, on the CSF login node (running text editors on the login node is<\/em> permitted) to create a file with exactly the following content (see below<\/a>). You can name the file anything you like, as long as there are no spaces in the name – in this example we use first-job.txt<\/code> but Linux doesn’t care what extension<\/em> you use – .txt<\/code> or .sbatch<\/code> or .jobscript<\/code> for example:<\/p>\n\r\n# Run this command on the CSF login node at the prompt<\/em>\r\ngedit first-job.txt\r\n #\r\n # If gedit opens, you can IGNORE any warnings \/ messages that appear in the terminal from gedit.\r\n # For example: (gedit:5246): dconf-WARNING **: .........\r\n #\r\n # Can't see gedit? Check your \"dock\" - it might have opened but the window is behind others on\r\n # your desktop.\r\n<\/pre>\n
\r\n(gedit:1639570): Gtk-WARNING **: 16:21:05.503: cannot open display<\/strong>:\r\n<\/pre>\n
\n
Notepad<\/code> and then transfer the file to CSF, although we don’t actually recommend this method. The file can have any name (we’re using first-job.txt<\/code> but anything will be OK – you’ll find that Notepad names files with .txt<\/code> at the end anyway).\n\r\n# Run this command on the CSF login node at the prompt<\/em> if jobscript was written in notepad\r\ndos2unix first-job.txt\r\n #\r\n # or whatever filename you used (we assume notepad adds .txt)\r\n<\/pre>\n
gedit<\/code> on the CSF login node and you’ll get a Linux text-editor very similar to Notepad. The file you write will be saved directly on the CSF and will not<\/em> need converting with dos2unix<\/code> because it is already in the correct format.\n<\/li>\n<\/ul>\n
\n<\/a><\/p>\n\r\n#!\/bin\/bash --login\r\n\r\n# Slurm options are those that begin with #SBATCH\r\n#SBATCH -p serial # Run in the \"serial\" partition (compute nodes dedicated to 1-core jobs)\r\n#SBATCH -t 5 # Allow a maximum wallclock time limit of 5 minutes\r\n # Our simple job actually only runs for about 2 minutes\r\n # but we always set the wallclock limit a little longer.\r\n # (Other time formats can be used for days and hours.)\r\n\r\n# Now the example commands to be executed (programs to be run) on a compute node:\r\n# In your real work, you'll run apps such as a chemistry app, or a bio-inf app.\r\n\/bin\/date\r\n\/bin\/hostname\r\n\/bin\/sleep 120\r\n\/bin\/date\r\n<\/pre>\n
\n
#!\/bin\/bash --login<\/code>, means that the file you create is treated as a BASH script.\n#SBATCH<\/code> are commands to the batch system – they provide information about your job.\n\n
#SBATCH -p serial<\/code> runs your job in the “serial” job area (partition). This is a dedicated set of compute nodes used to run serial (1-core) jobs.<\/li>\n#SBATCH -t 5<\/code> says that your job is allowed to run for no more<\/em> than 5 minutes, once it starts. It is perfectly fine if your job completes its work in less<\/em> time (and in fact our simple job will complete in about 2 minutes.) But if a job was still running once the wallclock time limit is reached, then the batch system will kill the job. So we always give a little extra time on the wallclock limit, just for safety.<\/li>\n<\/ul>\n<\/li>\nStep 2: Copy to scratch area<\/h3>\n
\r\ncp first-job.txt ~\/scratch\r\n<\/pre>\n
\r\ncd ~\/scratch\r\n<\/pre>\n
\r\n[mxyzabc1<\/em>@login2[csf3] scratch]$ \r\n #\r\n # The prompt shows your current directory\r\n<\/pre>\n
Step 3: Submit the Job to the Batch System<\/h3>\n
first-job.txt<\/code>. Then the following command will submit your job to the batch system:<\/p>\n\r\nsbatch first-job.txt\r\n\r\n# You'll see a message:\r\nSubmitted batch job 195501\r\n<\/pre>\n
195501<\/code> is a unique number identifying your job (obviously you will receive a different number). You may use this in other commands later.<\/p>\nStep 4: Check Job Status<\/h3>\n
\r\nsqueue\r\n<\/pre>\n
squeue<\/code> will look like the following – notice the ST column (short for “State”) – it shows “PD” for pending:<\/p>\n\r\n NODELIST\r\n JOBID PRIORITY PARTITION NAME USER ST SUBMIT_TIME START_TIME TIME NODES CPUS (REASON)\r\n195501 0.019104 serial first-jo mxyzabc1 PD<\/strong> 21\/05\/25 9:51 N\/A 0:00 1 1 (None)\r\n<\/pre>\n
\r\n NODELIST\r\n JOBID PRIORITY PARTITION NAME USER ST SUBMIT_TIME START_TIME TIME NODES CPUS (REASON)\r\n195501 0.019104 serial first-jo mxyzabc1 R<\/strong> 21\/05\/25 9:55 ... 9:55 0:05 1 1 node003\r\n<\/pre>\n
squeue<\/code> will show no output – meaning you have no jobs in the queue, either running or waiting.<\/p>\n\r\n[mxyzabc1<\/em>@login02 [CSF4] scratch]$ squeue\r\n JOBID PRIORITY PARTITION NAME USER ST SUBMIT_TIME START_TIME TIME NODES CPUS (REASON)\r\n #\r\n # No jobs listed means you have no jobs waiting or running (all jobs have finished)\r\n<\/pre>\n
REASON<\/code> to help diagnose the problem. Please contact us via the Connect Portal HPC Help form<\/a>, stating your job-ID and the system you are logged in to and we’ll let you know what has gone wrong.<\/p>\n#!\/bin\/bash --login<\/code> (no<\/strong> spaces at the start of the line.)<\/p>\nsqueue<\/code>command, your job has finished.<\/p>\nStep 5: Review Job Results\/Output<\/h3>\n
ls<\/code> command:<\/p>\n\r\nls\r\nfirst-job.txt slurm-195501<\/em>.out\r\n<\/pre>\n
first-job.txt<\/code> and a new file slurm-195501<\/em>.out<\/code> that has been generated by the job (remember, the job ID number 195501<\/em><\/code> will be different for your<\/em> job.)<\/p>\n\r\ncat slurm-195501<\/em>.out\r\n<\/pre>\n
\r\nWed May 21 09:55:49 BST 2025\r\nnode003\r\nWed May 21 09:57:49 BST 2025\r\n<\/pre>\n
slurm-JOBID<\/em>.out<\/code>. It might be easier to keep track of which job output which file if you make the output file use a similar name to that of your jobscript. You can change the start of the name of the output file by adding the following line to your jobscript (change myjobname<\/code> to something meaningful for your job)<\/p>\n\r\n#SBATCH -o %x.o%j # %x will be replaced by the jobscript name\r\n # %j will be replaced by the JOBID number\r\n<\/pre>\n
first-job.txt.o195501<\/code> (which will be familiar to users of the SGE batch system, which we used to use on the CSF.)<\/p>\nslurm-195501<\/em>.out<\/code> file. By changing the Linux commands to something more useful (e.g., to run your favourite chemistry application) you can get lots of real work done on the CSF.<\/p>\nStep 6: Copy Results back to “home”<\/h3>\n
slurm-195501<\/em>.out<\/code> file, it didn’t generate any other files. So we’ll just copy the .out<\/code> file back to