{"id":30,"date":"2018-08-28T16:37:23","date_gmt":"2018-08-28T15:37:23","guid":{"rendered":"http:\/\/ri.itservices.manchester.ac.uk\/csf3\/?page_id=30"},"modified":"2025-05-22T15:55:58","modified_gmt":"2025-05-22T14:55:58","slug":"parallel-jobs","status":"publish","type":"page","link":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/batch\/parallel-jobs\/","title":{"rendered":"Parallel Jobs"},"content":{"rendered":"<p><script type=\"text\/javascript\">\n    function toggle() {\n        var x = document.getElementById(\"hidetext\");\n        if (x.style.display === \"none\") {x.style.display = \"block\";}\n        else {x.style.display = \"none\";}\n    }\n<\/script><\/p>\n<div class=\"warning\">The SGE batch system has been shutdown and the CSF upgraded to use the Slurm batch system. Please read the <a href=\"\/csf3\/batch-slurm\">CSF3 Slurm documentation<\/a> instead.<\/p>\n<p>To display this old SGE page, <a href=\"javascript:toggle()\">click here<\/a>\n<\/div>\n<div id=\"hidetext\" style=\"display: none\">\n<h2>Current Configuration and Parallel Environments<\/h2>\n<p>For jobs that require two or more CPU cores, the appropriate <acronym title=\"Son of Grid Engine\">SGE<\/acronym> <strong>p<\/strong>arallel <strong>e<\/strong>nvironment should be selected from the table below.<\/p>\n<p><em><strong>Please also consult the <a href=\"\/csf3\/software\">software page<\/a> specific for the code \/ application you are running for advice on the most suitable PE<\/strong><\/em>.<\/p>\n<p>A parallel job script takes the form:<\/p>\n<pre>#!\/bin\/bash --login\r\n#$ -cwd                       # Job will run in the current directory (where you ran qsub)\r\n<strong>#$ -pe <em>pename<\/em> <em>numcores<\/em><\/strong>        # Choose a PE name from the tables below and a number of cores\r\n\r\n# Load any required modulefiles\r\nmodule load <em>apps\/some\/example\/1.2.3<\/em>\r\n\r\n# Now the commands to be run in job. You MUST tell your app how many cores to use! There are usually\r\n# three way to do this. Note: $NSLOTS is automatically set to the number of cores given above.\r\n<\/pre>\n<p><a name=\"parapp\"><\/a><\/p>\n<ul>\n<li><a href=\"\/csf3\/software\/libraries\/openmp\/\">OpenMP<\/a> applications (multicore but all in a single compute node):\n<pre>export OMP_NUM_THREADS=$NSLOTS\r\n<em>the_openmp_app<\/em>\r\n<\/pre>\n<\/li>\n<li><a href=\"\/csf3\/software\/libraries\/openmpi\/\">MPI<\/a> applications (small jobs on a single node or larger jobs across multiple compute nodes)\n<pre>mpirun -n $NSLOTS <em>the_mpi_app<\/em> \r\n<\/pre>\n<\/li>\n<li>Other multicore apps that use their own command-line flags (you must check the app&#8217;s documentation for how to do this correctly). For example:\n<pre><em>the_bioinfo_app<\/em> <em>--numthreads<\/em> $NSLOTS         # This is an example - check <em>your<\/em> app's docs!\r\n<\/pre>\n<\/li>\n<\/ul>\n<p>The available <em>parallel environments<\/em> are now described. Use the name of a parallel environment in your jobscript.<\/p>\n<h2 id=\"petables\">AMD Parallel Environments<\/h2>\n<p><strong>New from September 2024<\/strong>. We are installing <a href=\"\/csf3\/batch\/new-amd-nodes-sept-2024\/\">new AMD compute nodes<\/a> in the CSF, and these will eventually be the majority of nodes.<\/p>\n<h3>Single Node Multi-core(SMP) and MPI Jobs<\/h3>\n<table width=\"100%\" class=\"striped\">\n<tbody>\n<tr>\n<td colspan=\"3\" valign=\"top\">PE name: <strong>amd.pe<\/strong> (NOTE: it is NOT smp.pe &#8211; that is for Intel CPUs &#8211; see below)<\/p>\n<ul>\n<li>For jobs of <strong>2 to 168 cores.<\/strong><\/li>\n<li>Jobs will use a <em>single<\/em> compute node. Use for OpenMP (or other multicore\/<acronym title=\"Symmetric Multi-Processing\">SMP<\/acronym> jobs) and <acronym title=\"Message Passing Interface\">MPI<\/acronym> jobs.<\/li>\n<li>8GB RAM per core.<\/li>\n<li>7 day runtime limit.<\/li>\n<li>Currently, jobs will run on AMD &#8220;Genoa&#8221; CPUs. All nodes provide <code>avx2<\/code> and <code>avx512<\/code> <em>vector extensions<\/em>.<\/li>\n<li>A large pool of nodes available &#8211; currently 10,248 cores.<\/li>\n<\/ul>\n<\/td>\n<\/tr>\n<tr>\n<th width=\"25%\"><a name=\"optres\"><\/a>Optional Resources<\/th>\n<th width=\"40%\">Max cores per job, RAM per core<\/th>\n<th>Additional usage guidance<\/th>\n<\/tr>\n<tr>\n<td><code>-l\u00a0<a href=\"\/csf3\/batch\/timelimits#shortjobs\">short<\/a><\/code><\/td>\n<td>Max 28 cores, 8GB\/core (Genoa nodes.) <strong>1 hour runtime<\/strong><\/td>\n<td><strong>Usually has shorter queue-wait times<\/strong>. Only 2 nodes available. This option is for test jobs and interactive only &#8211; DO NOT use it for production runs as it is unfair on those who need it for testing\/interactive.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h2>Intel Parallel Environments<\/h2>\n<h3>Single Node Multi-core(SMP) and small MPI Jobs<\/h3>\n<table width=\"100%\"  class=\"striped\">\n<tbody>\n<tr>\n<td colspan=\"3\" valign=\"top\">PE name: <strong>smp.pe<\/strong><\/p>\n<ul>\n<li>For jobs of <strong>2 to 32 cores.<\/strong><\/li>\n<li>Jobs will use a <em>single<\/em> compute node. Use for OpenMP (or other multicore\/<acronym title=\"Symmetric Multi-Processing\">SMP<\/acronym> jobs) and small <acronym title=\"Message Passing Interface\">MPI<\/acronym> jobs.<\/li>\n<li>4GB or 6GB RAM per core depending where the job runs.<\/li>\n<li>7 day runtime limit.<\/li>\n<li>Currently, jobs may be placed on either Broadwell (max 28 cores) or Skylake (max 32 cores) CPUs. See <a href=\"#optressmp\">optional resources<\/a> below to control this. The system will choose if not specified.<\/li>\n<li><strong>We recommend you do not specify a type of CPU<\/strong> unless absolutely necessary for your application\/situation as doing so reduces the pool of nodes available to you and can lead to an increased wait in the queue.<\/li>\n<li>Large pool of cores.<\/li>\n<li>The <em>optional<\/em> resource flags below can be used to modify your job. You should specify only <strong>one<\/strong> flag (if using any such flags) unless indicated in the table.<\/li>\n<li><strong>Note: Choosing a node type is not recommended as it can mean a much longer wait in the queue.<\/strong><\/li>\n<\/ul>\n<\/td>\n<\/tr>\n<tr>\n<th width=\"25%\"><a name=\"optressmp\"><\/a>Optional Resources<\/th>\n<th width=\"52%\">Max cores per job, RAM per core<\/th>\n<th>Additional usage guidance<\/th>\n<\/tr>\n<tr>\n<td><code>-l mem512<\/code><br \/>\n<code>-l mem1500<\/code><br \/>\n<code>-l mem2000<\/code><br \/>\n<code>-l mem4000<\/code> (restricted)<\/td>\n<td>Please see the <a href=\"\/csf3\/batch\/high-memory-jobs\/\">High Memory Jobs<\/a> page.<\/td>\n<td>High memory nodes. Jobs must genuinely need extra memory.<\/td>\n<\/tr>\n<tr>\n<td><code>-l\u00a0<a href=\"\/csf3\/batch\/timelimits#shortjobs\">short<\/a><\/code><\/td>\n<td>Max 24 cores, 4GB\/core (haswell nodes.) <strong>1 hour runtime<\/strong><\/td>\n<td><strong>Usually has shorter queue-wait times<\/strong>. Only 2 nodes available. This option is for test jobs and interactive only &#8211; DO NOT use it for production runs as it is unfair on those who need it for testing\/interactive.<\/td>\n<\/tr>\n<tr>\n<td colspan=\"3\">Most users will not need to use the following flags. They will restrict the pool of nodes available to your jobs, which will result in longer queue-wait times.<\/td>\n<\/tr>\n<tr>\n<td><code>-l\u00a0broadwell<\/code><\/td>\n<td>Max 28 cores, 5GB\/core<\/td>\n<td>Use only Broadwell cores.<\/td>\n<\/tr>\n<tr>\n<td><code>-l\u00a0skylake<\/code><\/td>\n<td>Max 32 cores, 6GB\/core<\/td>\n<td>Use only Skylake cores.<\/td>\n<\/tr>\n<tr>\n<td><code>-l\u00a0<a href=\"\/csf3\/batch\/intel-cores\/\">avx<\/a><\/code><\/td>\n<td>Limits will depend on what node type the system chooses and whether you include any memory options.<\/td>\n<td>System will choose one of Broadwell, Skylake CPUs<\/td>\n<\/tr>\n<tr>\n<td><code>-l\u00a0<a href=\"\/csf3\/batch\/intel-cores\/\">avx2<\/a><\/code><\/td>\n<td>Limits will depend on what node type the system chooses and whether you include any memory options.<\/td>\n<td>System will choose one of Broadwell, Skylake CPUs<\/td>\n<\/tr>\n<tr>\n<td><code>-l\u00a0<a href=\"\/csf3\/batch\/intel-cores\/\">avx512<\/a><\/code><\/td>\n<td>Max 32 cores, 6GB\/core<\/td>\n<td>Use only Skylake CPUs<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h3>Multi-node large MPI Jobs<\/h3>\n<table width=\"100%\">\n<tbody>\n<tr>\n<td colspan=\"3\" valign=\"top\">PE name: <strong>mpi-24-ib.pe<\/strong><\/p>\n<div class=\"warning\">\nThis PE HAS NOW BEEN RETIRED!! Use the AMD PE above, for larger parallel jobs (up to 168 cores).<\/p>\n<p>For mutli-node jobs larger than 168 cores, please see the <a href=\"\/csf3\/batch\/parallel-jobs\/\">HPC Pool<\/a>.\n<\/div>\n<\/td>\n<\/tr>\n<tr>\n<th width=\"25%\"><a name=\"optres\"><\/a>Optional Resources<\/th>\n<th width=\"40%\">Max cores per job, RAM per core<\/th>\n<th>Additional usage guidance<\/th>\n<\/tr>\n<tr>\n<td>NONE<\/td>\n<td>NONE<\/td>\n<td>NONE<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>The SGE batch system has been shutdown and the CSF upgraded to use the Slurm batch system. Please read the CSF3 Slurm documentation instead. To display this old SGE page, click here Current Configuration and Parallel Environments For jobs that require two or more CPU cores, the appropriate SGE parallel environment should be selected from the table below. Please also consult the software page specific for the code \/ application you are running for advice.. <a href=\"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/batch\/parallel-jobs\/\">Read more &raquo;<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"parent":22,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-30","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/30","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/comments?post=30"}],"version-history":[{"count":21,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/30\/revisions"}],"predecessor-version":[{"id":10079,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/30\/revisions\/10079"}],"up":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/22"}],"wp:attachment":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/media?parent=30"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}