When you run MATLAB at the University a network license token is checked out to you for the duration of your MATLAB session. We have 616 licenses for MATLAB itself and lesser amounts of licenses for the various toolboxes. If a license is not available on the network then MATLAB will fail with a licensing error.<\/p>\n
If you were to use MATLAB ‘normally’ on the CSF and ran N jobs simultaneously then you would be using (up to) N MATLAB licenses. On a large cluster such as the CSF it is possible for a single user to check out all available MATLAB licenses for hours at a time resulting in what would effectively be a denial of service attack for MATLAB. This is why we do not allow MATLAB to be executed directly on worker nodes.<\/p>\n
In order to get around this licensing problem, we make use of the MATLAB Compiler to produce standalone executables that can be run on worker nodes without the need for any network licenses. You should compile your code before submitting your jobs. If you try to compile code at the start of your jobscript then you could run out of compiler licenses (if your jobs require different matlab code – e.g., different parameters inside the matlab code, then you should pass in those parameters on the matlab command-line or via a .mat file).<\/p>\n
It is not possible to finely control the number of cores used by MATLAB jobs.\u00a0 There are only two options available for MATLAB code:<\/p>\n
Hence you can compile your code to be single-core (serial) code or multi-core, in which case it will use all cores on the compute node when the job runs.<\/p>\n
Some documents on the web refer to the\u00a0maxNumCompThreads<\/a>\u00a0function but this is unreliable, will soon be removed from MATLAB and should not be used.<\/p>\n Many MATLAB functions are multi-threaded by default and an extensive list of these can be found at\u00a0http:\/\/www.walkingrandomly.com\/?p=1894<\/a><\/p>\n In addition to the implicit, multi-threaded computation offered by these functions, you may wish to investigate use of the parallel computing toolbox. \u00a0This allows for explicit multi-core computation as well as GPU computing.<\/p>\n We will compile and submit the following four-line .m file. Create a file named From the bash prompt on the CSF login node, issue the following commands (lines starting with # are comments)<\/p>\n The Note: If you omit this option, you must<\/strong> run your compiled code as a parallel batch job<\/em> which requests all<\/strong> cores on a single compute node from the batch system (see below<\/a> for how to compile and run parallel code).<\/li>\n If you see an error similar to:<\/p>\n it means we need to increase the amount of memory you are permitted to use on the login node. Java often requires more than the default user limit. Please email us at its-ri-team@manchester.ac.uk<\/a> indicating you are trying to compile matlab code.<\/p>\n Note: do not compile code from within a batch job. If you have some matlab code that uses lots of different parameters it might be tempting to write a We now describe the files that the After running the compiler additional files will have been created in your directory. For the purposes of submitting jobs to the CSF, the only files we are interested in are:<\/p>\n Once you have mastered the basics of compilation further advice on compilation<\/a> is available. This deals with compiling multiple .m files, including directories of .m files, speeding up compilation time, use of toolboxes and other good advice.<\/p>\n If you have compiled using the option<\/p>\n then you submit your job for serial execution (so do not specify a PE) using a jobscript.<\/p>\n For example, we create a jobscript named Submit with the command: <\/p>\n where Two new files will appear – one for the standard output, and one for the standard error. See the SGE tutorial<\/a> for further details. Essential information about parallel matlab jobs<\/strong><\/p>\n Reminder: If your code makes no use, or very little use, of MATLAB parallel functions, you should compile serial MATLAB code as described above (add You compile multi-threaded MATLAB code using<\/p>\n and then you must<\/em> request all<\/em> of the cores (12, 16, 24) of a compute node in your jobscript using<\/p>\n This may<\/em> allow your MATLAB code to run faster but you may<\/em> also wait longer in the queue for an entire compute node to become free.<\/p>\n Please read on for the explanation:<\/p>\n Parallelisation in MATLAB is relatively complicated since there are many mechanisms by which it can be achieved. For example, many in-built MATLAB functions are automatically distributed across several processor cores if given large enough input (click here for a list<\/a> of such functions), so-called implicit parallelism.<\/p>\n If your code uses some of these functions then you may be tempted to omit the -R -singleCompThread<\/strong> option and submit to a parallel environment. However, this may not maximise your throughput<\/em> (the amount of work you get done on the CSF). For example, if only a short amount of time is spent in parallelized functions and the rest of your code is single threaded, you may spend longer in the CSF queue waiting for a 16-core node to become free (assuming you request 16 cores in your jobscript) than you gain from the parallel MATLAB sections of code. This becomes even more important if you have 100s of jobs to do – waiting for 16-core nodes to become free 100s of times could take a long time!). In this case you may be better off sticking with the serial jobs and keeping the -R -singleCompThread<\/strong> flag when compiling MATLAB. You’ll usually find that many single-core jobs spend less time in the CSF queues and so ultimately finish sooner, even if the actual runtime of the code is slightly longer than a parallel MATLAB job.<\/p>\n If however, you know that your code makes significant use<\/em> of implicitly parallel functions then you should omit -R -singleCompThread<\/strong> at the compilation stage and submit to the smp.pe environment. Note that it is impossible<\/em> to specify exactly<\/em> how many cores you want to use when working with implicitly parallel functions in MATLAB. Unfortunately MATLAB does not<\/strong> use familiar environment variables such as Instead, the parallel functions will automatically use as many cores as they can on the host machine. So, if you tell SGE to only use 2 cores (on the PE line your jobscript) MATLAB may still try to use all 12 (or 16, 24) cores in the node on which the job runs. This will trample on other jobs that have reserved those other cores. If you submit several 2-core jobs you could end up running up to 72 threads on a single compute node (six 2-core jobs on a 12-core node but MATLAB starts 12 threads for each of those six jobs) which would reduce your own jobs’ performance.<\/p>\n So it is essential<\/em> that you always request a whole compute node when working with parallel MATLAB jobs. This ensures only one job is running on the node and MATLAB is permitted to use all of the cores on that node. The CSF is a complex mixture of compute nodes because it has nodes containing 12, 16, 24 cores – please carefully follow the examples below to avoid problems – they look very similar, but each has a differences which are important and highlighted in bold in the scripts.<\/p>\n 12 core job – westmere compute node<\/strong><\/p>\n Please make sure you have read and understood the essential information section above.<\/p>\n Here is an example SGE script for a 12 core parallel MATLAB job called Submit with the command: 12 core job – sandybridge compute node<\/strong><\/p>\n Please make sure you have read and understood the essential information section above.<\/p>\n Here is an example SGE script for a 12 core parallel MATLAB job called Submit with the command: 16 core job – ivybridge compute node<\/strong><\/p>\n Please make sure you have read and understood the essential information section above.<\/p>\n Here is an example SGE script for a 16 core parallel MATLAB job called Submit with the command: 24 core job – haswell compute node<\/strong><\/p>\n Please make sure you have read and understood the essential information section above.<\/p>\n Here is an example SGE script for a 24 core parallel MATLAB job called Submit with the command: 16 core job – haswell compute node<\/strong><\/p>\n Please make sure you have read and understood the essential information section above.<\/p>\n Here is an example SGE script for a 16 core parallel MATLAB job called Submit with the command: MATLAB cannot be run on the AMD nodes. All of the above examples are for all of the possible Intel node types and which of these you use makes very little difference for MATLAB. The batch system spreads works as evenly as possible across all the different types of node so no one set of nodes is likely to be busier or quieter than another. Sandybridge, Ivybridge and Haswell may be slightly faster in terms of run time.<\/p>\n No, MATLAB cannot run on more than one CSF node. This is because the University does not have licenses for the distributed computing server product. The maximum job size is therefore all cores on one single compute node and jobs must use You may wish to pass command-line parameters to your compiled MATLAB code. For example suppose you wish to pass in a couple of numbers representing settings to be used by your code. The jobscript will look something like:<\/p>\n You will need to modify your Compile the code as before, for example:<\/p>\n In the above sample note that<\/p>\n The above method could be used within an SGE job array<\/a>. This is a single job that runs the same MATLAB code multiple times as individual tasks, but with different input parameters. More than one task can run at the same time. Each task uses a special SGE variable ($SGE_TASK_ID) to pass a different parameter to your MATLAB code. This is a great alternative to using a for loop, because rather than running each loop sequentially, the job array tasks can run at the same time i.e. in parallel.<\/p>\n In the example jobscript below the The MATLAB code will then read the first arg as shown earlier:<\/p>\n As before, compile the code.<\/p>\n Our batch system documentation contains lots of examples of how to use job array<\/a>.<\/p>\n If using job arrays<\/a> to run multiple instances of MATLAB (similar to condor), you may receive an error message about accessing a lock file:<\/p>\n This occurs when many job array tasks run concurrently and all try to access the same temporary directory used to store the lock file. The solution is to force MATLAB to create the temporary lock files on the nodes where the tasks are running rather than in your home or scratch space. Add the following lines to your jobscript before the Introduction When you run MATLAB at the University a network license token is checked out to you for the duration of your MATLAB session. We have 616 licenses for MATLAB itself and lesser amounts of licenses for the various toolboxes. If a license is not available on the network then MATLAB will fail with a licensing error. If you were to use MATLAB ‘normally’ on the CSF and ran N jobs simultaneously then you would.. Read more »<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"parent":233,"menu_order":0,"comment_status":"open","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-236","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/pages\/236","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/comments?post=236"}],"version-history":[{"count":24,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/pages\/236\/revisions"}],"predecessor-version":[{"id":4914,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/pages\/236\/revisions\/4914"}],"up":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/pages\/233"}],"wp:attachment":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/media?parent=236"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Basic Serial Compilation<\/h2>\n
example.m<\/code> using your preferred CSF text editor<\/a> (download example.m<\/a>)<\/p>\n\r\nmyVar = ones(5);\r\nsave('exampleoutput.mat', 'myVar')\r\ndisp('Hello from the CSF!')\r\ndisp('Be sure to save any variables you need to a mat file.')\r\n<\/pre>\n# This line loads the MATLAB environment\r\nmodule load apps\/binapps\/matlab\/R2017a\r\n\r\n# Perform the compilation, producing a serial (runs on one core) executable.\r\nmcc -R -singleCompThread -m example.m\r\n<\/pre>\n
mcc<\/code> command is the MATLAB compiler. The flags added are as follows:<\/p>\n\n
-R -singleCompThread<\/code> This option tells the compiler to produce a single threaded application and, unless you know that your application is going to take advantage of multiple cores, you should use it.\n-m example.m<\/code> This is your MATLAB source code – the .m<\/code> file – to be compiled.<\/li>\n<\/ul>\n\r\n[thread 140138232624896 also had an error]\r\nA fatal error has been detected by the Java Runtime Environment:\r\n\r\njava.lang.OutOfMemoryError: pthread_getattr_np\r\n\r\nInternal Error (os_linux_x86.cpp:681), pid=32693, tid=140138087524096\r\nError: pthread_getattr_np\r\n<\/pre>\n
.m<\/code> file for each set of parameters<\/code> and to compile a new version of your code for each run. But you could run out of compiler licenses if many jobs run at the same time. Instead, write your code to read parameters<\/em> from a .mat<\/code> file or from the command-line and then you only need to compile one version of your code. This should be done on the login node.<\/p>\nmcc<\/code> compiler will generate.<\/p>\nOutput files<\/h3>\n
\n
example<\/code> – This is the binary executable generated by the MATLAB compiler. It cannot be run directly.<\/li>\nrun_example.sh<\/code> – This is the wrapper script for the above executable and is the application<\/em> that you will actually run from within your jobscript<\/em>. So this is not<\/strong> the jobscript – you must still write a jobscript that runs this run_example.sh<\/code> script. It is a common mistake to think that run_example.sh<\/code> is the jobscript. It is not. It has been automatically generated by the MATLAB compiler.<\/li>\nexample.m<\/code> – This is your original .m<\/code> file that the compiler read to generate the above two file. You must not<\/strong> use this for anything else apart from compiling as above.<\/li>\n<\/ul>\nSubmitting serial jobs<\/h2>\n
-R -singleCompThread<\/pre>\n
example.qsub<\/code> as follows:<\/p>\n\r\n#!\/bin\/bash\r\n# ---- SGE options (lines start with #$): -------------------------------\r\n#$ -cwd\t\t# Run the job in the current directory\r\n#$ -V\t\t# Inherit environment settings (e.g. from loaded modulefiles)\r\n\r\n# ---- Commands to be executed (programs to be run) on a compute node ----\r\n.\/run_example.sh $MATLAB_HOME\r\n<\/pre>\n
qsub scriptname<\/em><\/pre>\n
scriptname<\/em><\/code> is replaced with the filename of your submission script (example.qsub<\/code> in this example).<\/p>\nOutput files<\/h3>\n
\n<\/a><\/p>\nBasic Parallel Compilation<\/h2>\n
-R -singleCompThread<\/code>). We now describe how to compile parallel (multi-core) matlab code:<\/p>\nmcc -m mypara.m\r\n<\/pre>\n
#$ -pe smp.pe 12 # or 16 or 24 depending on other flags (see below)\r\n<\/pre>\n
OMP_NUM_THREADS<\/code>.<\/p>\nSubmitting Parallel Jobs<\/h2>\n
run_myparallel.sh<\/code><\/p>\n#!\/bin\/bash\r\n#$ -pe smp.pe 12\r\n#$ -l westmere<\/strong>\r\n#$ -N MyParallel\r\n#$ -cwd\r\n#$ -V\r\n.\/run_myparallel.sh $MATLAB_HOME<\/pre>\n
qsub scriptname<\/code> where scriptname<\/code> is replaced with the filename of your submission script.<\/p>\nrun_myparallel.sh<\/code><\/p>\n#!\/bin\/bash\r\n#$ -pe smp.pe 12\r\n#$ -l sandybridge<\/strong>\r\n#$ -N MyParallel\r\n#$ -cwd\r\n#$ -V\r\n.\/run_myparallel.sh $MATLAB_HOME<\/pre>\n
qsub scriptname<\/code> where scriptname<\/code> is replaced with the filename of your submission script.<\/p>\nrun_myparallel.sh<\/code><\/p>\n\r\n#!\/bin\/bash\r\n#$ -pe smp.pe 16\r\n#$ -l ivybridge<\/strong>\r\n#$ -N MyParallel\r\n#$ -cwd\r\n#$ -V\r\n.\/run_myparallel.sh $MATLAB_HOME<\/pre>\n
qsub scriptname<\/code> where scriptname<\/code> is replaced with the filename of your submission script.<\/p>\nrun_myparallel.sh<\/code><\/p>\n#!\/bin\/bash\r\n#$ -pe smp.pe 24\r\n#$ -l haswell<\/strong>\r\n#$ -N MyParallel\r\n#$ -cwd\r\n#$ -V\r\n.\/run_myparallel.sh $MATLAB_HOME<\/pre>\n
qsub scriptname<\/code> where scriptname<\/code> is replaced with the filename of your submission script.<\/p>\nrun_myparallel.sh<\/code><\/p>\n#!\/bin\/bash\r\n#$ -pe smp.pe 16\r\n#$ -l haswell<\/strong>\r\n#$ -N MyParallel\r\n#$ -cwd\r\n#$ -V\r\n.\/run_myparallel.sh $MATLAB_HOME<\/pre>\n
qsub scriptname<\/code> where scriptname<\/code> is replaced with the filename of your submission script.<\/p>\nFrequently Asked Questions<\/h2>\n
Which compute node type should I run on?<\/h3>\n
Can I run a job on more than one compute node?<\/h3>\n
smp.pe<\/code> or they will fail. Please get in touch if you would like help assessing the requirements of your job.<\/p>\nHints, Tips and Code Samples<\/h2>\n
Passing Command-line Args to Compiled Code<\/h3>\n
#!\/bin\/bash\r\n#$ -cwd\r\n#$ -V\r\n#### This is a serial job. If you need to run the same code a lot with \r\n#### different parameters see the next tip about job arrays.\r\n\r\n# Pass in two numbers used by my MATLAB code: 500 and 10000 (for example) \r\n.\/run_myinput.sh $MATLAB_HOME 500 10000\r\n<\/pre>\n
myinput.m<\/code> file to read these args. You must make the entire code be run from a top-level function:<\/p>\n\r\n% myinput.m source code\r\nfunction exitcode =<\/strong> myinput(xparamarg, iters)<\/strong>\r\n % Args come in as strings. Convert to numbers:\r\n xparam=str2num<\/strong>(xparamarg);\r\n num_iters=str2num<\/strong>(iters);\r\n fprintf( 'Executing my code with xparam = %d, and %d iterations\\n', xparam, num_iters );\r\n\r\n % ...your code...\r\n\r\n % Set dummy function return val\r\n exitcode = 1;<\/strong>\r\nend<\/pre>\n
mcc -R -singleCompThread -m myinput.m\r\n<\/pre>\n
\n
str2num()<\/code>).<\/li>\n<\/ul>\nUsing the Job Array task ID as a Command-line Arg<\/h3>\n
$SGE_TASK_ID<\/code> value is passed to the MATLAB code. This can then do something unique with that value. For example:<\/p>\n#!\/bin\/bash\r\n#$ -cwd\r\n#$ -V\r\n### Jobarray with 1000 tasks numbered 1,2,...,1000\r\n#$ -t 1-1000\r\n\r\n# Include the next two lines to avoid a common problem with MATLAB job arrays (described in the next tip)\r\nexport MCR_CACHE_ROOT=\/tmp\/mcr_cache_root_$USER\r\nmkdir -p $MCR_CACHE_ROOT\r\n\r\n# Run our matlab code giving it the current task id on the command-line\r\n.\/run_myinput.sh $MATLAB_HOME $SGE_TASK_ID<\/pre>\n
% myinput.m source code\r\nfunction exitcode = myinput(task_id_arg<\/strong>)\r\n % Args come in as strings. Convert to numbers:\r\n sge_task_id=str2num<\/strong>(task_id_arg);\r\n fprintf( 'Executing my code with SGE_TASK_ID = %d\\n', sge_task_id);\r\n\r\n % ...your code...\r\n\r\n % Set dummy function return val\r\n exitcode = 1;<\/strong>\r\nend<\/pre>\n
mcc -R -singleCompThread -m myinput.m\r\n<\/pre>\n
MATLAB Job Array Error<\/h3>\n
terminate called after throwing an instance of 'dsFileBasedLockError'\r\nwhat(): \\\r\n Tried to obtain a lock on a directory without write permission: \\\r\n\/mnt\/iusers01\/xy01\/mabcxyz12\/.mcrCache7.17\/.deploy_lock.27<\/pre>\n
.\/run_xxxx.sh<\/code> line:<\/p>\nexport MCR_CACHE_ROOT=\/tmp\/mcr_cache_root_$USER\r\nmkdir -p $MCR_CACHE_ROOT<\/pre>\n
Further information<\/h2>\n
\n
\n