Software

General Software Information

Many widely used pieces of software are installed centrally on the CSF by the Research Infrastructure team, so that you don’t need to use your own CSF storage to install applications. This also reduces duplication and allows you to spend your time on your research.

Software on the CSF is accessed through use of user environment modules which set up your environment to run a specific application – each application has its own modulefile which you load to give you access to that application. Applications should then be run via the batch system (SGE). Each application’s web-page, listed in the appropriate menu on the left, shows all the commands needed to load the modulefile and run the application in a batch job.

To run GUI/interactive software that can’t be run as a batch job, use qrsh to run interactively on a compute node, not directly on the login node.

Finding out what is available

New apps are (frequently) being compiled on the CSF3. To see what is available on the CSF, run the following commands on the login node:

module search keyword       # search for an app name, e.g., module search tensor   (try it)
module avail                # list all modulefiles!

A lot of software (mostly, but not only, bio-informatics apps) from the DPSF has also been transferred to the CSF3. To access this suite of applications please do

module load apps/dpsf       # Can also use apps/bioinf

Now running

module search keyword       # search for an app name
module avail                # list all modulefiles!

will show you a much larger list of available applications. These are untested on CSF3 – please alert us to any issues using them.

Note that Module names for some applications may be different to those on CSF2/DPSF. Some older versions of software may not be available on the CSF3.

Rule of thumb – don’t mix and match things! Finding out more about a piece of software

Most people don’t need to know the details of how a piece of software was compiled, or what environment it is setting. We will try and give useful details on this in our user docs if it is appropriate for a particular piece of software.

You may want to check that multiple pieces of software you are loading are/will be compatible. If you load something that has intel-17.0 in the name and something that has intel-18 in the name then this may cause an issue as you are loading software compiled with different versions of the intel compiler and thus different underlying libraries. However, something with gcc in the name and something with intel-1x.x in the name may be fine, it can be hard to tell. Note though, that a lot of our modulefiles load other modulefiles for you on which they are dependent and we will have ensured that these are compatible with one another.

If you would like to see what a specific modulefile loads or what other settings/variables are the default that you need to be careful not to override by any settings you make use the command:

module show modulename

where modulename is the modulefile of the software you are interested in.

Common examples we see:

Loading a version of openmpi compiled with intel-1x.x and loading a different version of the intel compiler. The openmpi modulefile will load the correct intel compiler version for you.
Loading an application compiled with a specific version of python and then loading a different version of python.

The above does mean that if you have a pipeline that uses several pieces of software then trying to handle it all within one batch job may be tricky. Consider using job dependencies to run each element or piece of software.

Requesting Additional Software

If you wish to use software we do not currently have on the system, or you require a newer version of an installed app, please contact us via its-ri-team@manchester.ac.uk. We will discuss the requirement with you and do our best to help.

Usually it is possible to add most freely available software and commercial packages, where licenses exist on campus. Commercial software which does not have a campus license can be more problematic to handle so please contact us before assuming that a purchase you wish to make will work on the CSF.

For software that only one user requires it might be deemed most suitable for you to install it in your home directory. Although we may be able to install software we cannot guarantee to provide in-depth support for every package – it is expected all users of the CSF take the time to learn how to use their required software packages effectively.

Please note that Linux has a lot of packages which we cannot list here, but which are available by default on the command line.

Installing your own Software

You may wish to download open-source applications and install them in your home directory, or additional Research Data Storage that your research group might have access to on the CSF. This is certainly possible – some hints and tips are given below.

We recommend creating a directory (folder) for all of your local software applications. Within there create a directory for the final installed version of an application (you may want to install several different versions over time) and also a directory for the source code to allow you to compile the application. For example:

mkdir p ~/software/app-name/
cd ~/software/app-name/
mkdir 1.2.3 build
        #     #
        #     # Repo or source code downloaded in to ~/software/app-name/build/
        #
        # Final install in ~/software/app-name/1.2.3/

All downloads from the web (including git repository clones) must be done via the University web-proxy. The CSF cannot access the outside world directly, for security. The proxy is an intermediate server than can access the outside world. Hence you will usually need to load the following modulefile before doing any downloads:

module load tools/env/proxy

To clone a git repository:

# From the login node, schedule an interactive session
qrsh -l short

# You will now be on a backend node
module load tools/gcc/git/2.24.0
cd ~/software/app-name/build/
git clone https://github.com/project-name/repo-name.git

# You will now usually have a directory named after the repo
cd repo-name

# If you wish to switch to a known tagged version 
git checkout tags/1.2.3

Further information about using git on the CSF.

The next step is often to configure the software so that it can be compiled correctly on the CSF. Typically GNU autoconf tools or cmake is used. Check the software’s installation notes. By default most software builds will assume you want to install the software in a central location. You don’t have the necessary permissions to do this on the CSF so you should specify a local installation directory. At this point, if you wish to use the Intel compilers you should load their modulefile.

# If using the intel compiler, load the modulefile (GNU compilers are default if no modulefile loaded)
module load compilers/intel/18.0.3

cd ~/software/app-name/build/repo-name

# GNU autoconf tools
./configure --prefix=~/software/app-name/1.2.3

# OR cmake
cmake -DCMAKE_INSTALL_PREFIX=~/software/app-name/1.2.3

It is then usually a case of running the make command to run the compilation.

# Compile the software in the 'build' area
make
# Copy the final version of the sofware to your 'install' area
make install

# Have a look in your 'install' area
ls ~/software/app-name/1.2.3