{"id":227,"date":"2018-09-04T18:53:32","date_gmt":"2018-09-04T17:53:32","guid":{"rendered":"http:\/\/ri.itservices.manchester.ac.uk\/csf3\/?page_id=227"},"modified":"2026-02-25T16:41:52","modified_gmt":"2026-02-25T16:41:52","slug":"modules","status":"publish","type":"page","link":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/software\/modules\/","title":{"rendered":"Modules"},"content":{"rendered":"<h2>Overview<\/h2>\n<p><em>Modules<\/em> is the system by which most software is made available. Modules allow, via a simple interface, the update of various paths (and other <em>environment settings<\/em>) that allow access to software.<\/p>\n<p>The use of modules also provides a convenient way to switch between different versions of the same software (such as compilers or applications such as matlab which are updated frequently).<\/p>\n<div class=\"hint\">\nNote: We now recommend loading modulefiles within your jobscript so that you have a full record of how the job was run. Alternatively, you may load modulefiles on the login node and let the job inherit these settings &#8211; jobs will do this by default. See <a href=\"#inherit\">below<\/a> for more information about these two methods.<\/td>\n<\/div>\n<h2>Module Commands<\/h2>\n<p>Your <em>environment<\/em> is controlled using the <code>module<\/code> command with various arguments. The most commonly used command is:<\/p>\n<pre>module load <em>name<\/em>\/<em>of<\/em>\/<em>module<\/em>\/<em>1.2.3<\/em>\r\n<\/pre>\n<p>These commands can be used directly on the login node or in your jobscripts.<\/p>\n<p>You may wish to <em>search<\/em> for modules on the login node while writing your jobscript AND do the actual loading of the modulefile inside the jobscript so that your jobs always use the same version of the application.<\/p>\n<p>The full list of module commands is given below.<\/p>\n<table class=\"striped\">\n<thead>\n<tr>\n<th width=\"30%\">Command<\/th>\n<th>Description<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td><code>module search <em>pattern<\/em><\/code><\/td>\n<td>Lists all modules that have <em>pattern<\/em> in their name. For example: <code>module search bio<\/code><br \/>\nThe <code>module search<\/code> command only works on the login node.<\/td>\n<\/tr>\n<tr>\n<td><code>module avail<\/code><\/td>\n<td>Lists all available modules for use on the CSF (this shows what software has been installed).<\/td>\n<\/tr>\n<tr>\n<td><code>module avail <em>category<\/em><\/code><\/td>\n<td>Lists all available modules whose names begin with <code>category<\/code> (change this to <code>compilers<\/code>, <code>apps<\/code>, <code>libs<\/code>, <code>tools<\/code>, <code>mpi<\/code>). Can be useful to narrow down the list of things you want to look for. See below.<\/td>\n<\/tr>\n<tr>\n<td><code>module list<\/code><\/td>\n<td>Lists modules that you have currently loaded in your environment, if any.<\/td>\n<\/tr>\n<tr>\n<td><code>module load <em>modulename<\/em><\/code><\/td>\n<td>Loads module <em>modulename<\/em><\/td>\n<\/tr>\n<tr>\n<td><code>module unload <em>modulename<\/em><\/code><\/td>\n<td>Unloads module <em>modulename<\/em> and clears any settings made by the module from your environment<\/td>\n<\/tr>\n<tr>\n<td><code>module switch <em>oldmodulename<\/em> <em>newmodulename<\/em><\/code><\/td>\n<td>Switches between two modules, unloading one and loading the other<\/td>\n<\/tr>\n<tr>\n<td><code>module purge<\/code><\/td>\n<td>Unload all modules from your environment. Very useful for starting with a clean environment (alternatively just log out and back in again).<\/td>\n<\/tr>\n<tr>\n<td><code>module initadd <em>modulename<\/em><\/code><\/td>\n<td>Run this command <em>once<\/em> to automatically ensure that a module is loaded whenever you log in to the CSF. It creates a <code>.modules<\/code> file in your home directory which acts as your personal configuration. We do <em>not<\/em> recommend doing this &#8211; you may forget which modulefiles you load upon login and other module loads may clash.<\/td>\n<\/tr>\n<tr>\n<td><code>module help <em>modulename<\/em><\/code><\/td>\n<td>May show longer description of the module if present in the modulefile<\/td>\n<\/tr>\n<tr>\n<td><code>module help<\/code><br \/>\n<code>module -H<\/code><\/td>\n<td>Show help about the <code>module<\/code> command<\/td>\n<\/tr>\n<tr>\n<td><code>module show <em>modulename<\/em><\/code><\/td>\n<td>Shows all of the settings that would be made by the module if you were to load it. Very useful to see what the module does.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<h2>Categories<\/h2>\n<p>The software available of the CSF has been placed in categories:<\/p>\n<ul>\n<li><code>apps<\/code> &#8211; for main applications which has subsections for the compiler used to build an application (where the Research Infrastructure team has compiled the application, for example, <code>intel-17.0<\/code>), or, where no compilation was required and the application could be simply unpacked and installed: <code>binapps<\/code>.<\/li>\n<li><code>compilers<\/code> &#8211; GCC and Intel compilers<\/li>\n<li><code>libs<\/code> &#8211; for example, NAG, FFTW, Intel&#8217;s Math Kernel Library (MKL)<\/li>\n<li><code>mpi<\/code> &#8211; for running large parallel applications<\/li>\n<li><code>tools<\/code> &#8211; for small useful utilities<\/li>\n<\/ul>\n<p>The <code>module avail<\/code> command will list everything available. You can view a single category at a time with:<\/p>\n<pre>module avail <em>category<\/em>\r\n                #\r\n                # replacing the optional <em>category<\/em> with one of the above\r\n<\/pre>\n<p>To use a module\/piece of software:<\/p>\n<pre>module load <em>name\/of\/module\/1.2.3<\/em>\r\n              #\r\n              # replacing <em>name\/of\/module\/1.2.3<\/em> with the module you want\r\n<\/pre>\n<p>For example:<\/p>\n<pre>module load compilers\/intel\/17.0.7\r\n<\/pre>\n<p>It is possible, and often necessary, to have multiple modules loaded at once. For example the following commands are equivalent:<\/p>\n<pre>module load compilers\/intel\/17.07\r\nmodule load tools\/gcc\/cmake\/3.13.2\r\n<\/pre>\n<p>or<\/p>\n<pre>module load compilers\/intel\/17.0.7 tools\/gcc\/cmake\/3.13.2<\/pre>\n<p>Some modulefiles have restricted access (you need to be in a certain unix group to load the modulefile). If you cannot see a module file in the list of available modules that you expect to see, please <a href=\"\/csf3\/overview\/help\/\">contact us<\/a> giving the name of the modulefile.<\/p>\n<h2>Bash Completion<\/h2>\n<p>The module paths on the CSF are lengthy, but you can use bash completion to help you. Press <code>TAB<\/code> part-way through typing a module name and the name will be completed for you or a list of possible completions will be given. Type a few more characters and hit <code>TAB<\/code> again to narrow the list down until you get the full module name.<\/p>\n<p><a name=\"inherit\"><\/a><\/p>\n<h2>Module environment settings in batch<\/h2>\n<p>Modulefiles can be loaded:<\/p>\n<ul>\n<li>on the login node <em>before<\/em> you submit a batch job<\/li>\n<li>or from the jobscript when the job actually <em>runs<\/em><\/li>\n<\/ul>\n<p>There are pros and cons to each method but which method you choose is up to you &#8211; both are valid.<\/p>\n<h3>Loading a modulefile on the login node<\/h3>\n<p>The advantages of loading a modulefile on the login node <em>before<\/em> you submit the job are that you can check the name of the modulefile is correct (any spelling mistakes and the modulefile won&#8217;t be loaded). You could also check the settings the modulefile makes (e.g., an environment variable) and then use that in your jobscript. The disadvantage is that the particular modulefile name used is not recorded in your jobscript. If you wish to run the job again (perhaps in a few months time) you may have forgotten which version of a modulefile you used.<\/p>\n<h3>Loading a modulefile in the jobscript<\/h3>\n<p>The key advantage of loading a modulefile in the jobscript is that the names of any modulefiles used by your job are recorded &#8211; this helps with reproducibility (you might want to run the job again in 6 months time so having the modulefiles written in the jobscript shows exactly which apps were used by the job.) However, if you spell a modulefile name incorrectly the mistake will only be detected when the job <em>runs<\/em>. Your job will probably fail and you&#8217;ll have to resubmit the job, waiting in the queue again.<\/p>\n<h3>Example of each method<\/h3>\n<p>You should write your jobscripts for the modulefile method you prefer:<\/p>\n<table class=\"jobscriptcompare\">\n<thead>\n<tr>\n<th width=\"35%\">Jobscript that loads a modulefile<\/th>\n<th>Jobscript that inherits a modulefile&#8217;s settings from the login node<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>\n<pre class='slurm'>\r\n#!\/bin\/bash <strong>--login<\/strong>\r\n#SBATCH -p <em>partition<\/em>\r\n#SBATCH -t <em>Wallclock<\/em>\r\n\r\nmodule load name\/of\/app\/1.2.3\r\n\r\napp.exe input.dat ....\r\n<\/pre>\n<\/td>\n<td>\n<pre class='slurm'>\r\n#!\/bin\/bash\r\n#SBATCH -p <em>partition<\/em>\r\n#SBATCH -t <em>Wallclock<\/em>\r\n\r\n# Settings are inherited from login node so do:\r\n# <em>module load name\/of\/app\/1.2.3<\/em> on the login node\r\n\r\napp.exe input.dat ....\r\n<\/pre>\n<\/td>\n<\/tr>\n<tr>\n<td>The <code>--login<\/code> flag on the first line is needed to make <code>module<\/code> commands work inside jobscripts.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p>Note: If loading the modulefile on the login node and inheriting the settings in the jobscript, the batch system takes a copy of your environment <em>at the time at which you run<\/em> the <code>sbatch<\/code> command, not when your job eventually runs. This means that after submitting your job with <code>sbatch<\/code>, you are free to change your environment or even log out and your job will still see the correct environment when it runs. For example:<\/p>\n<pre class='slurm'>\r\n# From the CSF3 login node, submit a job to run appA \r\nmodule load name\/of\/appA\/1.2.3\r\nsbatch job_to_run_appA\r\n  #\r\n  # This takes a copy <em>right now<\/em> of all of the environment\r\n  # settings made by appA's modulefile\r\n\r\n# Unload all modules, clear the settings\r\nmodule purge\r\n  #\r\n  # This does NOT affect the previous job. It has already\r\n  # copied all of the required settings. When the job eventually\r\n  # runs it will have all of the settings from the appA modulefile.\r\n\r\n# Set up for a different app\r\nmodule load name\/of\/appB\/4.5.6\r\nsbatch job_to_run_appB\r\n  #\r\n  # This take a copy <em>right now<\/em> of all of the environment\r\n  # settings made by appB's modulefile. \r\n\r\n# Even if we log out before the jobs have run, they have a copy\r\n# of all of the settings needed for each job.\r\nlogout\r\n<\/pre>\n<h2>Automatically loading modulefiles on log in<\/h2>\n<p>Note: We recommend that you only load modulefiles as and when you require them, not automatically on login. This way you avoid unexpected software clashes, interactions and environment problems that can cause job failure.<\/p>\n<p>If you do wish to always have certain modulefiles loaded when you log into the CSF then best practice is to specify them in your <code>.modules<\/code> environment file, not the <code>.bashrc<\/code>, or <code>.bash_profile<\/code>. Add the <code>module load<\/code> commands to the end of the <code>.modules<\/code> file and leave any other content in that file untouched. However, bear in mind that the warning above still applies.<\/p>\n<h2 id=\"privatemodules\">Private modules &#8211; software you have installed<\/h2>\n<p>If you have installed a piece of software in your home directory then you can create your own modulefile for it which saves you having to set variables on the command line, in your <code>.bashrc<\/code>, <code>.bash_profile<\/code> or in batch scripts.<\/p>\n<ul>\n<li>In your home dir create a directory called &#8216;privatemodules&#8217; (the name is key)\n<pre>mkdir ~\/privatemodules\r\n<\/pre>\n<\/li>\n<li>Within that dir create the required folders and modulefiles (see template below) so you get something like:\n<pre>cd ~\/privatemodules\r\npwd\r\n\/mnt\/iusers01\/xy01\/mxyzabc2\/privatemodules\r\n\r\n# Create a directory named after my own application\r\nmkdir myapp\r\n\r\n# Go in to the dir and create a text file named using the application version number (e.g., 2.0.3):\r\ncd myapp\r\ngedit 2.0.3\r\n       #\r\n       # Write your modulefile in this plain text file which is named '2.0.3'\r\n       # It is an unusual name for a text file but Linux allows any name to be used.\r\n       # It will make the 'module load' command used later look like the others on the CSF.\r\n       # See below for a template modulefile\r\n\r\n# You should now have a file in at the following location:\r\n~\/privatemodules\/myapp\/2.0.3   \r\n<\/pre>\n<\/li>\n<li>To load your new modulefile first load the &#8216;use.own&#8217; modulefile (only need to do this once).\n<pre>module load use.own\r\nmodule load myapp\/2.0.3\r\n<\/pre>\n<\/li>\n<\/ul>\n<h3>A template for a modulefile<\/h3>\n<pre>#%Module1.0\r\n\r\n## (The above line is required and <strong>must<\/strong> be the very first line)\r\n##\r\n## APP Modulefile\r\n##    -- Replace APP with the name of your app\r\n##    -- In this example we use a made-up app called 'paracode'\r\n##    -- It requires a few additions to our environment.\r\n\r\n## -- The 'module load' line below shoud be where you specify other modulefiles that may\r\n## -- be required. Remove the 'module load' line if you do not need it.\r\n## -- For example, compliers and the MPI library you used to build the software.\r\nmodule load compilers\/gcc\/x.x.x mpi\/gcc\/openmpi\/1.x.x\r\n\r\n## -- Get my home directory path in case we need it\r\nset     USER_HOME   $env(HOME)\r\n\r\n## -- Now, specify the path to the top level of your software install area.\r\n## -- Using 'set' just sets a local variable for use inside the modulefile.\r\nset     instdir     $USER_HOME\/mysoftware\/build\/paracode\r\n\r\n## -- Now add required environment variables as detailed in the software manual.\r\n## -- Using 'setenv' makes the variable 'visible' in your environment after loading the module.\r\nsetenv  PARACODE_DIR     $instdir\r\nsetenv  PARACODE_DATA    $instdir\/sample_data_dir\r\n\r\n## -- Can also add to environment variables that may already exist. We don't\r\n## -- want to overwrite these variables but instead add to them.\r\nprepend-path    PATH               $instdir\/bin\r\nprepend-path\tLD_LIBRARY_PATH    $instdir\/lib\r\n<\/pre>\n<h2>Loading modules from python scripts<\/h2>\n<p>It is possible to load modulefiles directly from within python scripts, or other scripts such as perl or ruby. This is not something most people will need to do, but if you are developing a pipeline in python, say, then you might prefer to do everything in python (the usual method is to load the modules in the jobscript before running the python command &#8211; your python script will then have all environment settings made by the modulefiles available to it.)<\/p>\n<p>To load modules directly in python:<\/p>\n<pre>\r\n# First take a local copy of a system file and modify it (correcting a misnamed directory path.)\r\n# We also rename it to give it a more meaningful name:\r\ncp \/opt\/clusterware\/opt\/modules\/init\/python.py modulefiles.py\r\nsed -i 's\/Modules\/modules\/' modulefiles.py\r\n<\/pre>\n<p>Now write your python code &#8211; e.g., <code>myapp.py<\/code>, containing:<\/p>\n<pre>\r\nimport sys\r\nimport os\r\n# Load the modulefiles.py to make the 'module' command available\r\nfrom modulefiles import module\r\n# Can now run module commands \r\nmodule('load', 'tool\/env\/proxy')\r\nmodule('load', 'tools\/gcc\/git\/2.24.0')\r\nmodule('list')\r\nos.system('git clone https:\/\/github.com\/someproject')\r\n<\/pre>\n<p>Can now run your python app using the <code>python<\/code> command (the default system-wide python is v2.7)<\/p>\n<pre>\r\npython myapp.py\r\n<\/pre>\n<p>If you want to use the <code>modulefile.py<\/code> file with python3, you&#8217;ll need to edit the file to modify some python2 syntax. The following commands will modify the file for you:<\/p>\n<pre>\r\nsed -i -e \"s\/if not os.environ.has_key('\\([A-Z]*\\)'):\/if '\\1' not in os.environ:\/\" modulefiles.py\r\nsed -i -e \"s\/exec output\/exec(output)\/\" modulefiles.py\r\n<\/pre>\n<p>You can then run your python app using:<\/p>\n<pre>\r\npython3 myapp.py\r\n<\/pre>\n<h2>Further Information<\/h2>\n<ul>\n<li>man module<\/li>\n<li><a href=\"http:\/\/modules.sourceforge.net\/\">Sourceforge Modules Website<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Overview Modules is the system by which most software is made available. Modules allow, via a simple interface, the update of various paths (and other environment settings) that allow access to software. The use of modules also provides a convenient way to switch between different versions of the same software (such as compilers or applications such as matlab which are updated frequently). Note: We now recommend loading modulefiles within your jobscript so that you have.. <a href=\"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/software\/modules\/\">Read more &raquo;<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"parent":33,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-227","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/227","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/comments?post=227"}],"version-history":[{"count":20,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/227\/revisions"}],"predecessor-version":[{"id":11971,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/227\/revisions\/11971"}],"up":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/33"}],"wp:attachment":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/media?parent=227"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}