The CSF2 has been replaced by the CSF3 - please use that system! This documentation may be out of date. Please read the CSF3 documentation instead. To display this old CSF2 page click here. |
GROMACS v4.6.7
Overview
GROMACS is a package for computing molecular dynamics, simulating Newtonian equations of motion for systems with hundreds to millions of particles. GROMACS is designed for biochemical molecules with complicated bonded interactions (e.g. proteins, lipids, nucleic acids) but can also be used for non-biological systems (e.g. polymers).
Please do not add the -v flag to your mdrun command.It will write to a log file every second for the duration of your job and can lead to severe overloading of the file servers. |
This version is v4.6.7. The following flavours are available:
4.6.7 for all intel node types
- Single and double precision multi-threaded (OpenMP) versions:
mdrun
andmdrun_d
- Single and double precision MPI (not threaded) versions:
mdrun_mpi
andmdrun_d_mpi
- Compiled with Intel 14.0.3 compiler with the associated Intel MKL providing the FFT functions.
ngmx
has been included.
4.6.7 for Sandybridge and Ivybridge nodes only
- Single and double precision multi-threaded (OpenMP) versions:
mdrun
andmdrun_d
- Compiled with Intel 14.0.3 compiler with the associated Intel MKL providing the FFT functions and with
AVX_256
(an instruction set specific to these nodes) so WILL NOT work on Westmere nodes and NONE of the commands can be run on the login nodes. - We have no Sandybridge or Ivybridge nodes connected by Infiniband which means ONLY
smp.pe
job for this install. - There are no mpi versions of 4.6.7 for Sandybridge and Ivybridge nodes available on the CSF.
- This version will not run on highmem, twoday or short nodes (they are all Westmere).
ngmx
has been included.
Bugfix for g_hbond
Version 4.6.7 has the g_hbond fix included by default and so no separate build has been made for this version. See the GROMACS v4.5.4 CSF documentation for a description of that issue.
Restrictions on use
GROMACS is free software, available under the GNU General Public License.
Set up procedure
You must load the appropriate modulefile:
module load modulefile
replacing modulefile with one of the modules listed in the table below.
Version | Modulefile | Notes | Typical Executable name |
---|---|---|---|
Single precision multi-threaded | apps/intel-14.0/gromacs/4.6.7/single | non-MPI | mdrun |
Double precision multi-threaded | apps/intel-14.0/gromacs/4.6.7/double | non-MPI | mdrun_d |
Single precision MPI | apps/intel-14.0/gromacs/4.6.7/single-mpi | For MPI on Intel nodes using gigabit ethernet | mdrun_mpi |
Single precision MPI – Infiniband | apps/intel-14.0/gromacs/4.6.7/single-mpi-ib | For MPI on Intel or AMD nodes using infiniband | mdrun_mpi |
Double precision MPI | apps/intel-14.0/gromacs/4.6.7/double-mpi | For MPI on Intel nodes using gigabit ethernet | mdrun_mpi_d |
Double precision MPI – Infiniband | apps/intel-14.0/gromacs/4.6.7/double-mpi-ib | For MPI on Intel or AMD nodes using Infiniband | mdrun_mpi_d |
Single precision multi-threaded for AVX | apps/intel-14.0/gromacs/4.6.7/single-avx | non-MPI, Sandybridge and Ivybridge only | mdrun |
Double precision multi-threaded for AVX | apps/intel-14.0/gromacs/4.6.7/double-avx | non-MPI, Sandybridge and Ivybridge only | mdrun_d |
Interactive/Non-batch work/Job preparation
In order to prepare your jobs or post process them you may need to make use of commands such as grompp
. These will not work on the CSF login node because the software was compiled with AVX_256
which is not compatible with the login nodes. We have therefore allocated ONE sandybridge node to allow you to run these commands via qrsh. To do so type:
qrsh -l inter -l short -l sandybridge
which will give access to the sandybridge compute node. Then run your commands. When you have finished close the connection to the compute node with exit
(failure to do this may result in the compute node being unavailable to other users who need it). Then submit your computation/simulation to batch as per the above examples.
DO NOT run mdrun on this compute node – all computational work MUST be submitted to batch.
Running the application in batch
First load the required module (see above) and create a directory containing the required input data files.
You MUST ensure that as well as requesting a number of cores from a suitable parallel environment you also tell gromacs how many cores it may use. These two numbers must be the same, which can be ensured through correct use of some variables and/or flags depending on the version of gromacs being used. Failure to set this information causes gromacs to run incorrectly, overload compute nodes and potentially trample on jobs belonging to other users. All of the examples below ensure that jobs use the cores requested.
If you are running a different version to those used in the examples below ensure you replace the executable, e.g. mdrun
with the executable appropriate to the version based on the table above.
Please do not add the -v flag to your mdrun command.It will write to a log file every second for the duration of your job and can lead to severe overloading of the file servers. |
Multi-threaded single-precision on intel nodes, 2 to 16 cores
Note that GROMACS v4.6.7 (unlike v4.5.4) does not support the -nt
flag to set the number of threads when using the multithreaded OpenMP (non-MPI) verison. Instead set the OMP_NUM_THREADS
environment variable as shown below.
An example batch submission script to run the single-precision mdrun executable with 12 threads:
#!/bin/bash #$ -S /bin/bash #$ -cwd #$ -V #$ -pe smp.pe 12 export OMP_NUM_THREADS=$NSLOTS mdrun
Submit with the command: qsub scriptname
The system will run your job on a Westmere, a Sandybridge or an Ivybridge node depending on what is available. This option goes to the biggest pool of nodes. To get a more optimised run on Sandybridge or Ivybridge you should be using a modulefile with ‘avx’ in the name and using the instructions below.
Multi-threaded single-precision AVX on Sandybridge nodes 2 to 12 cores
An example batch submission script to run the double-precision mdrun_d
executable with 8 threads:
#!/bin/bash #$ -S /bin/bash #$ -cwd #$ -V #$ -pe smp.pe 8 #$ -l sandybridge export OMP_NUM_THREADS=$NSLOTS mdrun
Submit with the command: qsub scriptname
Multi-threaded single-precision AVX on Ivybridge nodes, 2 to 16 cores
Note that GROMACS v4.6.7 (unlike v4.5.4) does not support the -nt
flag to set the number of threads when using the multithreaded OpenMP (non-MPI) verison. Instead set the OMP_NUM_THREADS
environment variable as shown below.
An example batch submission script to run the single-precision mdrun executable with 12 threads:
#!/bin/bash #$ -S /bin/bash #$ -cwd #$ -V #$ -pe smp.pe 12 #$ -l ivybridge export OMP_NUM_THREADS=$NSLOTS mdrun
Submit with the command: qsub scriptname
Single-precision MPI with Infiniband, 48 cores or more in multiples of 24
An example batch submission script to run the single precision mdrun_mpi
executable with 24 MPI processes on 24 cores with the orte-24-ib.pe parallel environment (Intel nodes using infiniband):
#!/bin/bash #$ -S /bin/bash #$ -cwd #$ -V #$ -pe orte-24-ib.pe 48 mpiexec -n $NSLOTS mdrun_mpi
Submit with the command: qsub scriptname
Illegal instruction
If during job setup/post-processing you try to run gromacs commands on the login nodes and get the following error:
Illegal instruction
this is because you have an AVX only version of the modulefile loaded which is not compatible with the login node. You should load the non-AVX version and then unload it and reload the AVX version at job submission time.
If you get it in a batch job then it is because you did not specify whether to run on a sandybridge or ivybridge node in your job script.
Further info
- You can see a list of all the installed GROMACS utilities with the command:
ls $GMXDIR/bin
- GROMACS web page
- GROMACS manuals
- GROMACS user mailing list
Updates
Dec 2014 – 4.6.7 installed with AVX support (specific user request for this) and documentation written.
Nov 2013 – Documentation for 4.5.4 and 4.6.1 split in to two pages.
May 2013 – Gromacs 4.6.1 and Plumed 1.3 installed.