Compiling MATLAB code

Introduction

When you run MATLAB at the University a network license token is checked out to you for the duration of your MATLAB session. We have 616 licenses for MATLAB itself and lesser amounts of licenses for the various toolboxes. If a license is not available on the network then MATLAB will fail with a licensing error.

If you were to use MATLAB ‘normally’ on the CSF and ran N jobs simultaneously then you would be using (up to) N MATLAB licenses. On a large cluster such as the CSF it is possible for a single user to check out all available MATLAB licenses for hours at a time resulting in what would effectively be a denial of service attack for MATLAB. This is why we do not allow MATLAB to be executed directly on worker nodes.

In order to get around this licensing problem, we make use of the MATLAB Compiler to produce standalone executables that can be run on worker nodes without the need for any network licenses. You should compile your code before submitting your jobs. If you try to compile code at the start of your jobscript then you could run out of compiler licenses (if your jobs require different matlab code – e.g., different parameters inside the matlab code, then you should pass in those parameters on the matlab command-line or via a .mat file).

Controlling the number of cores used by MATLAB jobs

It is not possible to finely control the number of cores used by MATLAB jobs.  There are only two options available for MATLAB code:

• Use a single core, or
• Use all cores available on the compute node where the job runs

Hence you can compile your code to be single-core (serial) code or multi-core, in which case it will use all cores on the compute node when the job runs.

Some documents on the web refer to the maxNumCompThreads function but this is unreliable, will soon be removed from MATLAB and should not be used.

Many MATLAB functions are multi-threaded by default and an extensive list of these can be found at http://www.walkingrandomly.com/?p=1894

In addition to the implicit, multi-threaded computation offered by these functions, you may wish to investigate use of the parallel computing toolbox.  This allows for explicit multi-core computation as well as GPU computing.

Basic Serial Compilation

We will compile and submit the following four-line .m file. Create a file named example.m using your preferred CSF text editor (download example.m)

myVar = ones(5);
save('exampleoutput.mat', 'myVar')
disp('Hello from the CSF!')
disp('Be sure to save any variables you need to a mat file.')

From the bash prompt on the CSF login node, issue the following commands (lines starting with # are comments)

# This line loads the MATLAB environment
module load apps/binapps/matlab/R2017a

# Perform the compilation, producing a serial (runs on one core) executable.
mcc -R -singleCompThread -m example.m

The mcc command is the MATLAB compiler. The flags added are as follows:

• -R -singleCompThread This option tells the compiler to produce a single threaded application and, unless you know that your application is going to take advantage of multiple cores, you should use it.

  Note: If you omit this option, you must run your compiled code as a parallel batch job which requests all cores on a single compute node from the batch system (see below for how to compile and run parallel code).

• -m example.m This is your MATLAB source code – the .m file – to be compiled.

If you see an error similar to:

[thread 140138232624896 also had an error]
A fatal error has been detected by the Java Runtime Environment:

java.lang.OutOfMemoryError: pthread_getattr_np

Internal Error (os_linux_x86.cpp:681), pid=32693, tid=140138087524096
Error: pthread_getattr_np

it means we need to increase the amount of memory you are permitted to use on the login node. Java often requires more than the default user limit. Please email us at its-ri-team@manchester.ac.uk indicating you are trying to compile matlab code.

Note: do not compile code from within a batch job. If you have some matlab code that uses lots of different parameters it might be tempting to write a .m file for each set of parameters and to compile a new version of your code for each run. But you could run out of compiler licenses if many jobs run at the same time. Instead, write your code to read parameters from a .mat file or from the command-line and then you only need to compile one version of your code. This should be done on the login node.

We now describe the files that the mcc compiler will generate.

Output files

After running the compiler additional files will have been created in your directory. For the purposes of submitting jobs to the CSF, the only files we are interested in are:

• example – This is the binary executable generated by the MATLAB compiler. It cannot be run directly.
• run_example.sh – This is the wrapper script for the above executable and is the application that you will actually run from within your jobscript. So this is not the jobscript – you must still write a jobscript that runs this run_example.sh script. It is a common mistake to think that run_example.sh is the jobscript. It is not. It has been automatically generated by the MATLAB compiler.
• your jobscript.qsub – this does not yet exist. You will write the jobscript next…
• example.m – This is your original .m file that the compiler read to generate the above two file. You must not use this for anything else apart from compiling as above.

Once you have mastered the basics of compilation further advice on compilation is available. This deals with compiling multiple .m files, including directories of .m files, speeding up compilation time, use of toolboxes and other good advice.

Submitting serial jobs

If you have compiled using the option

-R -singleCompThread

then you submit your job for serial execution (so do not specify a PE) using a jobscript.

For example, we create a jobscript named example.qsub as follows:

#!/bin/bash
# ---- SGE options (lines start with #$): -------------------------------
#$ -cwd		# Run the job in the current directory
#$ -V		# Inherit environment settings (e.g. from loaded modulefiles)

# ---- Commands to be executed (programs to be run) on a compute node ----
./run_example.sh $MATLAB_HOME

Submit with the command:

qsub scriptname

where scriptname is replaced with the filename of your submission script (example.qsub in this example).

Output files

Two new files will appear – one for the standard output, and one for the standard error. See the SGE tutorial for further details.

Basic Parallel Compilation

Essential information about parallel matlab jobs

Reminder: If your code makes no use, or very little use, of MATLAB parallel functions, you should compile serial MATLAB code as described above (add -R -singleCompThread). We now describe how to compile parallel (multi-core) matlab code:

You compile multi-threaded MATLAB code using

mcc -m mypara.m

and then you must request all of the cores (12, 16, 24) of a compute node in your jobscript using

#$ -pe smp.pe 12    # or 16 or 24 depending on other flags (see below)

This may allow your MATLAB code to run faster but you may also wait longer in the queue for an entire compute node to become free.

Please read on for the explanation:

Parallelisation in MATLAB is relatively complicated since there are many mechanisms by which it can be achieved. For example, many in-built MATLAB functions are automatically distributed across several processor cores if given large enough input (click here for a list of such functions), so-called implicit parallelism.

If your code uses some of these functions then you may be tempted to omit the -R -singleCompThread option and submit to a parallel environment. However, this may not maximise your throughput (the amount of work you get done on the CSF). For example, if only a short amount of time is spent in parallelized functions and the rest of your code is single threaded, you may spend longer in the CSF queue waiting for a 16-core node to become free (assuming you request 16 cores in your jobscript) than you gain from the parallel MATLAB sections of code. This becomes even more important if you have 100s of jobs to do – waiting for 16-core nodes to become free 100s of times could take a long time!). In this case you may be better off sticking with the serial jobs and keeping the -R -singleCompThread flag when compiling MATLAB. You’ll usually find that many single-core jobs spend less time in the CSF queues and so ultimately finish sooner, even if the actual runtime of the code is slightly longer than a parallel MATLAB job.

If however, you know that your code makes significant use of implicitly parallel functions then you should omit -R -singleCompThread at the compilation stage and submit to the smp.pe environment. Note that it is impossible to specify exactly how many cores you want to use when working with implicitly parallel functions in MATLAB. Unfortunately MATLAB does not use familiar environment variables such as OMP_NUM_THREADS.

Instead, the parallel functions will automatically use as many cores as they can on the host machine. So, if you tell SGE to only use 2 cores (on the PE line your jobscript) MATLAB may still try to use all 12 (or 16, 24) cores in the node on which the job runs. This will trample on other jobs that have reserved those other cores. If you submit several 2-core jobs you could end up running up to 72 threads on a single compute node (six 2-core jobs on a 12-core node but MATLAB starts 12 threads for each of those six jobs) which would reduce your own jobs’ performance.

So it is essential that you always request a whole compute node when working with parallel MATLAB jobs. This ensures only one job is running on the node and MATLAB is permitted to use all of the cores on that node. The CSF is a complex mixture of compute nodes because it has nodes containing 12, 16, 24 cores – please carefully follow the examples below to avoid problems – they look very similar, but each has a differences which are important and highlighted in bold in the scripts.

Submitting Parallel Jobs

12 core job – westmere compute node

Please make sure you have read and understood the essential information section above.

Here is an example SGE script for a 12 core parallel MATLAB job called run_myparallel.sh

#!/bin/bash
#$ -pe smp.pe 12
#$ -l westmere
#$ -N MyParallel
#$ -cwd
#$ -V
./run_myparallel.sh $MATLAB_HOME

Submit with the command: qsub scriptname where scriptname is replaced with the filename of your submission script.

12 core job – sandybridge compute node

Please make sure you have read and understood the essential information section above.

Here is an example SGE script for a 12 core parallel MATLAB job called run_myparallel.sh

#!/bin/bash
#$ -pe smp.pe 12
#$ -l sandybridge
#$ -N MyParallel
#$ -cwd
#$ -V
./run_myparallel.sh $MATLAB_HOME

Submit with the command: qsub scriptname where scriptname is replaced with the filename of your submission script.

16 core job – ivybridge compute node

Please make sure you have read and understood the essential information section above.

Here is an example SGE script for a 16 core parallel MATLAB job called run_myparallel.sh

#!/bin/bash
#$ -pe smp.pe 16
#$ -l ivybridge
#$ -N MyParallel
#$ -cwd
#$ -V
./run_myparallel.sh $MATLAB_HOME

Submit with the command: qsub scriptname where scriptname is replaced with the filename of your submission script.

24 core job – haswell compute node

Please make sure you have read and understood the essential information section above.

Here is an example SGE script for a 24 core parallel MATLAB job called run_myparallel.sh

#!/bin/bash
#$ -pe smp.pe 24
#$ -l haswell
#$ -N MyParallel
#$ -cwd
#$ -V
./run_myparallel.sh $MATLAB_HOME

Submit with the command: qsub scriptname where scriptname is replaced with the filename of your submission script.

16 core job – haswell compute node

Please make sure you have read and understood the essential information section above.

Here is an example SGE script for a 16 core parallel MATLAB job called run_myparallel.sh

#!/bin/bash
#$ -pe smp.pe 16
#$ -l haswell
#$ -N MyParallel
#$ -cwd
#$ -V
./run_myparallel.sh $MATLAB_HOME

Submit with the command: qsub scriptname where scriptname is replaced with the filename of your submission script.

Frequently Asked Questions

Which compute node type should I run on?

MATLAB cannot be run on the AMD nodes. All of the above examples are for all of the possible Intel node types and which of these you use makes very little difference for MATLAB. The batch system spreads works as evenly as possible across all the different types of node so no one set of nodes is likely to be busier or quieter than another. Sandybridge, Ivybridge and Haswell may be slightly faster in terms of run time.

Can I run a job on more than one compute node?

No, MATLAB cannot run on more than one CSF node. This is because the University does not have licenses for the distributed computing server product. The maximum job size is therefore all cores on one single compute node and jobs must use smp.pe or they will fail. Please get in touch if you would like help assessing the requirements of your job.

Hints, Tips and Code Samples

Passing Command-line Args to Compiled Code

You may wish to pass command-line parameters to your compiled MATLAB code. For example suppose you wish to pass in a couple of numbers representing settings to be used by your code. The jobscript will look something like:

#!/bin/bash
#$ -cwd
#$ -V
#### This is a serial job. If you need to run the same code a lot with 
#### different parameters see the next tip about job arrays.

# Pass in two numbers used by my MATLAB code: 500 and 10000 (for example) 
./run_myinput.sh $MATLAB_HOME 500 10000

You will need to modify your myinput.m file to read these args. You must make the entire code be run from a top-level function:

% myinput.m source code
function exitcode = myinput(xparamarg, iters)
  % Args come in as strings. Convert to numbers:
  xparam=str2num(xparamarg);
  num_iters=str2num(iters);
  fprintf( 'Executing my code with xparam = %d, and %d iterations\n', xparam, num_iters );

  % ...your code...

  % Set dummy function return val
  exitcode = 1;
end

Compile the code as before, for example:

mcc -R -singleCompThread -m myinput.m

In the above sample note that

• The code is wrapped in a function and so will need an exit value – we set a dummy value at the end.
• The args passed in are presented as strings to your code. If they are to be used as numbers you must convert them to numbers (e.g., using str2num()).

Using the Job Array task ID as a Command-line Arg

The above method could be used within an SGE job array. This is a single job that runs the same MATLAB code multiple times as individual tasks, but with different input parameters. More than one task can run at the same time. Each task uses a special SGE variable ($SGE_TASK_ID) to pass a different parameter to your MATLAB code. This is a great alternative to using a for loop, because rather than running each loop sequentially, the job array tasks can run at the same time i.e. in parallel.

In the example jobscript below the $SGE_TASK_ID value is passed to the MATLAB code. This can then do something unique with that value. For example:

#!/bin/bash
#$ -cwd
#$ -V
### Jobarray with 1000 tasks numbered 1,2,...,1000
#$ -t 1-1000

# Include the next two lines to avoid a common problem with MATLAB job arrays (described in the next tip)
export MCR_CACHE_ROOT=/tmp/mcr_cache_root_$USER
mkdir -p $MCR_CACHE_ROOT

# Run our matlab code giving it the current task id on the command-line
./run_myinput.sh $MATLAB_HOME $SGE_TASK_ID

The MATLAB code will then read the first arg as shown earlier:

% myinput.m source code
function exitcode = myinput(task_id_arg)
  % Args come in as strings. Convert to numbers:
  sge_task_id=str2num(task_id_arg);
  fprintf( 'Executing my code with SGE_TASK_ID = %d\n', sge_task_id);

  % ...your code...

  % Set dummy function return val
  exitcode = 1;
end

As before, compile the code.

mcc -R -singleCompThread -m myinput.m

Our batch system documentation contains lots of examples of how to use job array.

MATLAB Job Array Error

If using job arrays to run multiple instances of MATLAB (similar to condor), you may receive an error message about accessing a lock file:

terminate called after throwing an instance of 'dsFileBasedLockError'
what(): \
 Tried to obtain a lock on a directory without write permission: \
/mnt/iusers01/xy01/mabcxyz12/.mcrCache7.17/.deploy_lock.27

This occurs when many job array tasks run concurrently and all try to access the same temporary directory used to store the lock file. The solution is to force MATLAB to create the temporary lock files on the nodes where the tasks are running rather than in your home or scratch space. Add the following lines to your jobscript before the ./run_xxxx.sh line:

export MCR_CACHE_ROOT=/tmp/mcr_cache_root_$USER
mkdir -p $MCR_CACHE_ROOT

Further information

• Some functions cannot be compiled (Everything from the symbolic toolbox for example). A full list of restrictions and exclusions can be found at
  • http://www.mathworks.com/help/toolbox/compiler/br2cqa0-1.html
• It is often possible to speed-up MATLAB code significantly using techniques such as vectorisation, mex files, parallelisation and more. If you would like advice on how to optimise your MATLAB application then email applicationsupport-eps@manchester.ac.uk