The CSF2 has been replaced by the CSF3 - please use that system! This documentation may be out of date. Please read the CSF3 documentation instead. To display this old CSF2 page click here. |
Batch Script and qsub Options
Batch job options can be specified in a qsub jobscript, for example using
#$ -cwd #$ -V
or on the qsub
command line, for example using
qsub -cwd -V ... filename [optional args]
Note: if filename is an executable (e.g., myapp.exe
) rather than a jobscript you must use the -b y
flag (please see below). We recommend using a jobscript however.
All of the possible flags are described in the manual page (man qsub
). The most commonly used options are briefly described below. Note that the order in which you specify options, either in the jobscript or on the command line does not matter.
SGE Switches
- -cwd
- Execute the job from the current (working) directory — the directory from which the
qsub
command is issued. If this option is not present, the job will be executed the user’s home directory. The.oNNNNN
and.eNNNNN
stdout and stderr files created by SGE for each job will also be written to the directory specified by this flag (or the home directory if not present) unless the-o
and-e
flags are used to override where these files are written. - -V
- (Uppercase V). This ensures that any environment settings you’ve made on the login node are inherited/passed to the compute node, including the settings applied by loading software
modulefiles
. A copy of your current environment is taken when you run theqsub
command (i.e., immediately, not when the job finally runs). Hence you can change your environment after running qsub, perhaps to set up for another job, or even log out and your job will still see the environment that was in place when you originally ran qsub. - -j y
- Merge the standard error stream into the standard output stream, i.e., job output and error messages are sent to the same
.o
file, rather than different files (usually.o
and.e
files). - -pe name.pe
- Specify the SGE parallel-environment to which a job is sent — see the section on running parallel jobs.
- -l resource
- Specify a resource to modify where in the system the job is placed. For example
-l highmem
to select a high-memory node. You may specify more than one resource flag, for example-l sandybridge -l 's_rt=00:10:00'
although not all combinations are supported. Resource flags exist for CPU architectures, job time limits, memory requirements, GPUs and interactivity so you should check those pages for details together with the parallel environment documentation to determine whether a resource and a PE are compatible (not all combinations are permitted). - -S /bin/bash
- (Uppercase S). Indicate your jobscript is written using
/bin/bash
shell syntax. See the introductory SGE information. - -N name
- (Uppercase N). Sets the job name, e.g.
-N my_job_name
to set job name tomy_job_name
. The.o
and.e
job output files will be named using this value — for examplemy_job_name.o12345
andmy_job_name.e12345
. If you don’t use the-N
option then the job output files will use the name of the jobscript (or executable) specified on theqsub
command-line. Do not use spaces in the name. - -o /path/to/dir
-e /path/to/dir
alternatively:
-o /path/to/dir/stdoutfile
-e /path/to/dir/stderrfile - Use either the directory form or the filename form. If a directory name is given, it specifies the path to a directory where the usual standard output stream (stdout) and standard error stream (stderr) files (
JobName.oNNNNN
andJobName.eNNNNN
respectively) will be written. The directories must already exist before the job runs – the batch system will not create them for you. If filenames are given, they specify the files to which stdout and stderr output will be written. No JobID number will be appended – your supplied filenames will be used as-is. If these flags are not used the standard output and error stream files will be written in the directory in which the job runs (see-cwd
). - -hold_jid jobid
- Specifies this job is conditional upon completion of a previous job or jobs, e.g.
-hold_jid jobID
to submit a job which will not start untiljobID
has completed.jobID
can be a job number (e.g., 89213) or job name (i.e., the earlier job was named using the-N
flag). Multiple jobIDs can be specified using a comma separated list of jobIDs. In that case the current job will not run until all specified jobIDs have finished. - -m bea
- Causes an email to be sent when the job begins, when it ends and/or if it is aborted. You can specify any or all of the
bea
letters. For example, most users only want to know when a job ends or aborts so use-m ea
.The email will be sent to your University email account.January 2016: Please note that you must put your email address in your jobscript or on the command line submission (see below for how) as it does not automatically detect you University email at the moment. We are looking into this.
- -M emailaddress
- (Uppercase M). Specify an email address to which
-m
status emails will be sent.By default your University email account will receive the email unless you use this option.January 2016: Please note that you must put your email address in your jobscript or on the command line submission as it does not automatically detect you University email at the moment. We are looking into this.
- -b y
- For use on the qsub command-line only. Indicates that the filename given on the
qsub
command line is an executable (binary) file, not a jobscript. This allows you to specify the executable directly on the command-line rather than in a job script. By default theqsub
command assumes the filename refers to a jobscript. For example, the following command line and jobscript (submitted withqsub myjobscript
) are equivalent:qsub -b y -cwd -V -l short /bin/hostname
and
#!/bin/bash #$ -cwd #$ -V #$ -l short /bin/hostname
It is up to the user which method they prefer. However we recommend writing a jobscript so that you can see how the job was submitted if referring back to an old job (perhaps submitted months ago) rather than trying to remember a command-line. It also allows the sysadmins to identify more easily any problems with jobs.
See the individual pages (in menu of left side of this page) for the PE names and resources available on this system.
SGE Environment Variables
The following environment variables are available for use in your jobscript when the job runs. They can be used to create unique names for output files, for example, by including the job id or name in the output filename.
- $NSLOTS
- The number of cores requested using the
-pe
flag or1
if running a serial job (no-pe
option specified). Use this variable if your application requires the number of cores to use on its command-line, rather than repeating the number in two places. This makes running jobs with different numbers of cores easier. For example:#$ -pe smp.pe 4 myapp -cores $NSLOTS -input sample.dat -output results.dat # # $NSLOTS will be automatically replaced with 4 in this example
You could also use this variable in the name of an output file if doing several runs with a different number of cores when timing your code. For example
#$ -pe smp.pe 4 myapp -cores $NSLOTS -input sample.dat -output results.${NSLOTS}cores.dat # # The output file will be named results.4cores.dat
- $JOB_ID
- The unique job id number assigned to the job at runtime by the batch system. You can use this to generate unique filenames that won’t be overwritten by other jobs. For example:
#$ -cwd #$ -V myapp -input sample.dat -output results.$JOB_ID.dat # # Output file will be named results.37823.dat where 37823 is my unique jobid.
- $JOB_NAME
- The value of the
-N
flag if present or the name of the jobscript if that flag is not used. Note that a unique jobid is always generated even if you use the-N
flag. For example:#$ -cwd #$ -N phase1 myapp -input sample.dat -output results.$JOB_NAME.$JOB_ID.dat # # Names the output file results.phase1.38795.dat (in this case)
- $SGE_TASK_ID
$SGE_TASK_FIRST
$SGE_TASK_LAST
$SGE_TASK_STEPSIZE - See the Job Arrays documentation for environment variables related to each task.
- $PE_HOSTFILE
- You will not normally need to use this variable in your jobscripts. However, some applications documented on the CSF software page process the names of the nodes on which your job will run in to their own format. This variable gives the name of a file containing the names of the nodes on which your job has been scheduled to run. Do not, however, change the value of this variable yourself.