The CSF2 has been replaced by the CSF3 - please use that system! This documentation may be out of date. Please read the CSF3 documentation instead. To display this old CSF2 page click here. |
GROMACS v 2016.3
Overview
GROMACS is a package for computing molecular dynamics, simulating Newtonian equations of motion for systems with hundreds to millions of particles. GROMACS is designed for biochemical molecules with complicated bonded interactions (e.g. proteins, lipids, nucleic acids) but can also be used for non-biological systems (e.g. polymers).
Please do not add the -v flag to your mdrun command.It will write to a log file every second for the duration of your job and can lead to severe overloading of the file servers. |
Significant Change in this Version
Within Gromacs 2016, the different gromacs commands (e.g., mdrun
, grompp
, g_hbond
) should now be run using the command:
gmx command
where command
is the name of the command you wish to run (without any g_
prefix), for example:
gmx mdrun
The gmx
command changes its name to reflect the gromacs flavour being used but the command
does not change. For example, if using the mdrun
command:
# New 2016.3 method # Previous 5.0.4 method gmx mdrun mdrun gmx_d mdrun mdrun_d mpirun -n $NSLOTS gmx_mpi mdrun mpirun -n $NSLOTS mdrun_mpi mpirun -n $NSLOTS gmx_mpi_d mdrun mpirun -n $NSLOTS mdrun_mpi_d
The complete list of command
names can be found by running the following on the login node:
module load apps/intel-17.0/gromacs/2016.3/single gmx help commands # The following commands are available: anadock dyecoupl mdmat sans anaeig dyndom mdrun sasa analyze editconf mindist saxs angle eneconv mk_angndx select bar enemat morph sham bundle energy msd sigeps check filter nmeig solvate chi freevolume nmens sorient cluster gangle nmtraj spatial clustsize genconf order spol confrms genion pairdist tcaf convert-tpr genrestr pdb2gmx traj covar grompp pme_error trjcat current gyrate polystat trjconv density h2order potential trjorder densmap hbond principal tune_pme densorder helix rama vanhove dielectric helixorient rdf velacc dipoles help rms view disre hydorder rmsdist wham distance insert-molecules rmsf wheel do_dssp lie rotacf x2top dos make_edi rotmat xpm2ps dump make_ndx saltbr
Notice that the command names do NOT start with g_
and do NOT reference the flavour being run (e.g., _mpi_d
). Only the main gmx
command changes its name to reflect the flavour (see below for list of modulefiles for the full list of flavours available).
To obtain more help about a particular command run:
gmx help command
For example
gmx help mdrun
Helper scripts
To assist with moving to the new command calling method, we have recreated some of the individual commands that you may have used in your jobscript. For example, you can continue to use mdrun
(or mdrun_d
) instead of the new gmx mdrun
(or gmx_d mdrun
) in this release. These extra commands are automatically included in your environment when you load the gromacs modulefiles. This old method uses the flavour of gromacs in the command name (see above for comparison of new and old commands).
However, please note that the following commands are new to 2016.3 and so can only be run using the new method (gmx command
):
# New commands that can only be run using: gmx command check help convert-tpr insert-molecules distance pairdist dump sasa freevolume solvate gangle view
Available Flavours
This version is 2016.3. The following flavours are available:
2016.3 for all Intel node types
Note: ability to run on all Intel nodes implies lower optimization.
- Single and double precision multi-threaded (OpenMP) versions:
mdrun
andmdrun_d
- Single and double precision MPI (not threaded) versions:
mdrun_mpi
andmdrun_d_mpi
- Compiled with Intel 17.0.2 compiler with the associated Intel MKL providing the FFT functions.
ngmx
has been included.
2016.3 for Sandybridge and Ivybridge (and Haswell, Broadwell nodes) only
Note: ability to run on only Sandybridge, Ivybridge (and Haswell) nodes implies higher optimization. Note that an even higher level of optimization, and an MPI version, is available for Haswell nodes (see below).
- Single and double precision multi-threaded (OpenMP) versions:
mdrun
andmdrun_d
- Compiled with Intel 17.0.2 compiler with the associated Intel MKL providing the FFT functions and with AVX_256 (an instruction set specific to these nodes) so WILL NOT work on Westmere nodes and NONE of the commands can be run on the login nodes.
- We have no Sandybridge or Ivybridge nodes connected by Infiniband which means ONLY
smp.pe
(single-node, multicore) jobs for these types nodes. - There are no MPI versions of 2016.3 for Sandybridge and Ivybridge nodes available on the CSF.
- This version will not run on highmem, twoday or short nodes (they are all Westmere).
ngmx
has been included.
2016.3 for Haswell and Broadwell nodes only
Note: ability to run on only Haswell and Broadwell nodes implies higher optimization.
- Single and double precision single-node, multi-threaded (OpenMP) versions:
mdrun
andmdrun_d
- Single and double precision multi-node (MPI) versions:
mdrun_mpi
andmdrun_mpi_d
- Compiled with Intel 17.0.2 compiler with the associated Intel MKL providing the FFT functions and with AVX2_256 (an instruction set specific to these nodes which provides further optimization than AVX_256) so WILL NOT work on Westmere, Sandybridge and Ivybridge nodes.
- Single-node multi-core
smp.pe
jobs can use these nodes. - Multi-core MPI
orte-24-ib.pe
jobs can use these nodes – the Haswell and Broadwell nodes have InfiniBand networking. - This version will not run on highmem, twoday or short nodes (they are all Westmere).
ngmx
has been included.
Bugfix for g_hbond
Version 2016.3 has the g_hbond fix included by default and so no separate build has been made for this version. See the GROMACS v4.5.4 CSF documentation for a description of that issue.
Restrictions on use
GROMACS is free software, available under the GNU General Public License.
Set up procedure
You must load the appropriate modulefile:
module load modulefile
replacing modulefile with one of the modules listed in the table below.
Version | Modulefile | Notes | Typical Executable name |
---|---|---|---|
Single precision multi-threaded (single-node) | apps/intel-17.0/gromacs/2016.3/single | non-MPI | mdrun or gmx mdrun |
Double precision multi-threaded (single-node) | apps/intel-17.0/gromacs/2016.3/double | non-MPI | mdrun_d or gmx_d mdrun |
Single precision MPI (single-node) | apps/intel-17.0/gromacs/2016.3/single-mpi | For MPI on Intel nodes using gigabit ethernet | mdrun_mpi or gmx_mpi mdrun |
Single precision MPI (multi-node, Infiniband) | apps/intel-17.0/gromacs/2016.3/single-mpi-ib | For MPI on Intel or AMD nodes using infiniband | mdrun_mpi or gmx_mpi mdrun |
Double precision MPI (single-node) | apps/intel-17.0/gromacs/2016.3/double-mpi | For MPI on Intel nodes using gigabit ethernet | mdrun_mpi_d or gmx_mpi_d mdrun |
Double precision MPI (multi-node, Infiniband) | apps/intel-17.0/gromacs/2016.3/double-mpi-ib | For MPI on Intel or AMD nodes using Infiniband | mdrun_mpi_d or gmx_mpi_d mdrun |
AVX optimized builds for Sandybridge and Ivybridge nodes | |||
Single precision multi-threaded for AVX (single-node) | apps/intel-17.0/gromacs/2016.3/single-avx | non-MPI, Sandybridge and Ivybridge only | mdrun or gmx mdrun |
Double precision multi-threaded for AVX (single-node) | apps/intel-17.0/gromacs/2016.3/double-avx | non-MPI, Sandybridge and Ivybridge only | mdrun_d or gmx_d mdrun |
AVX2 optimized builds for Haswell and Broadwell 24-core nodes (new April 2016) | |||
Single precision multi-threaded for AVX2 (single-node) | apps/intel-17.0/gromacs/2016.3/single-avx2 | non-MPI, Haswell only | mdrun or gmx mdrun |
Double precision multi-threaded for AVX2 (single-node) | apps/intel-17.0/gromacs/2016.3/double-avx2 | non-MPI, Haswell or Broadwell only | mdrun_d or gmx_d mdrun |
Single precision MPI (single/multi-node, Infiniband) for AVX2 | apps/intel-17.0/gromacs/2016.3/single-avx2-mpi-ib | For MPI on Intel Haswell nodes using infiniband | mdrun_mpi or gmx_mpi mdrun |
Double precision MPI (single/multi-node, Infiniband) for AVX2 | apps/intel-17.0/gromacs/2016.3/double-avx2-mpi-ib | For MPI on Intel Haswell or Broadwell nodes using infiniband | mdrun_mpi_d or gmx_mpi_d mdrun |
Interactive/Non-batch work/Job preparation
In order to prepare your jobs or post process them you may need to make use of commands such as grompp
. These will not work on the CSF login node because the software was compiled with AVX_256 which is not compatible with the login nodes. We have therefore allocated ONE sandybridge node to allow you to run these commands via qrsh. To do so type:
qrsh -l inter -l short -l sandybridge
which will give access to the sandybridge compute node. Then run your commands. When you have finished close the connection to the compute node with exit
(failure to do this may result in the compute node being unavailable to other users who need it). Then submit your computation/simulation to batch as per the above examples.
DO NOT run mdrun on this compute node – all computational work MUST be submitted to batch.
Running the application in batch
First load the required module (see above) and create a directory containing the required input data files.
Please NOTE the following which important for running jobs correctly and efficiently:
Ensure you inform gromacs how many cores it can use. This is done using the $NSLOTS
variable which is automatically set for you in the jobscript to be the number of cores you request in the jobscript header (see later for complete examples). You can use either of the following methods depending whether you want a multi-core job (running on a single compute node) or a larger job running across multiple compute nodes:
# Multi-core (single-node) or Multi-node MPI jobs mpirun -n $NSLOTS mdrun_mpi # Old method (v5.0.4 and earlier) mpirun -n $NSLOTS mdrun_mpi_d # Old method (v5.0.4 and earlier) mpirun -n $NSLOTS gmx_mpi mdrun # New method (v5.1.4 and later) mpirun -n $NSLOTS gmx_mpi_d mdrun # New method (v5.1.4 and later)
or
# Single-node multi-threaded job export OMP_NUM_THREADS=$NSLOTS # Do this for all versions mdrun # Old method (v5.0.4 and earlier) mdrun_d # Old method (v5.0.4 and earlier) export OMP_NUM_THREADS=$NSLOTS # Do this for all versions gmx mdrun # New method (v5.1.4 and later) gmx_d mdrun # New method (v5.1.4 and later)
The examples below can be used for single precision or double precision gromacs. Simply run mdrun
(single precision) or mdrun_d
(double precision).
Please do not add the -v flag to your mdrun command.It will write to a log file every second for the duration of your job and can lead to severe overloading of the file servers. |
Multi-threaded single-precision on Intel nodes, 2 to 24 cores
Note that GROMACS 2016.3 (unlike v4.5.4) does not support the -nt
flag to set the number of threads when using the multithreaded OpenMP (non-MPI) version. Instead set the OMP_NUM_THREADS
environment variable as shown below.
An example batch submission script to run the single-precision mdrun executable with 12 threads:
#!/bin/bash #$ -S /bin/bash #$ -cwd #$ -V #$ -pe smp.pe 12 # Can specify 2 to 24 cores in smp.pe # 2-12 includes Westmere, Sandybridge, Ivybridge, Haswell, Broadwell # 13-16 forces use of Ivybridge # 17-24 forces use of Haswell or Broadwell # Can force use of a particular architecture (see below) export OMP_NUM_THREADS=$NSLOTS mdrun # # This is the old naming convention (it will still work in this release) # The new gromacs convention is to run: gmx mdrun
Submit with the command: qsub scriptname
The system will run your job on a Westmere, a Sandybridge or an Ivybridge node depending on what is available. This option goes to the biggest pool of nodes. To get a more optimised run on Sandybridge or Ivybridge you should be using a modulefile with ‘avx’ in the name and using the instructions below.
Multi-threaded double-precision, AVX on Sandybridge nodes, 2 to 12 cores
An example batch submission script to run the double-precision mdrun_d
executable with 8 threads:
#!/bin/bash #$ -S /bin/bash #$ -cwd #$ -V #$ -pe smp.pe 8 #$ -l sandybridge # Force use of sandybridge nodes export OMP_NUM_THREADS=$NSLOTS mdrun_d # # This is the old naming convention (it will still work in this release) # The new gromacs convention is to run: gmx_d mdrun
Submit with the command: qsub scriptname
Multi-threaded single-precision, AVX on Ivybridge nodes, 2 to 16 cores
Note that GROMACS 2016.3 (unlike v4.5.4) does not support the -nt
flag to set the number of threads when using the multithreaded OpenMP (non-MPI) version. Instead set the OMP_NUM_THREADS
environment variable as shown below.
An example batch submission script to run the single-precision mdrun executable with 16 threads:
#!/bin/bash #$ -S /bin/bash #$ -cwd #$ -V #$ -pe smp.pe 16 #$ -l ivybridge # Force use of Ivybridge nodes export OMP_NUM_THREADS=$NSLOTS mdrun
Submit with the command: qsub scriptname
Multi-threaded single-precision, AVX2 on Haswell nodes, 2 to 24 cores
Note that GROMACS 2016.3 (unlike v4.5.4) does not support the -nt
flag to set the number of threads when using the multithreaded OpenMP (non-MPI) version. Instead set the OMP_NUM_THREADS
environment variable as shown below.
An example batch submission script to run the single-precision mdrun executable with 16 threads:
#!/bin/bash #$ -S /bin/bash #$ -cwd #$ -V #$ -pe smp.pe 24 #$ -l haswell # Force use of Haswell nodes export OMP_NUM_THREADS=$NSLOTS mdrun
Submit with the command: qsub scriptname
Single precision MPI (single-node), 2 to 24 cores
An example batch submission script to run the double-precision mdrun_mpi
executable on 8 cores using mpi:
#!/bin/bash #$ -S /bin/bash #$ -cwd #$ -V #$ -pe smp.pe 8 mpirun -n $NSLOTS mdrun_mpi
Submit with the command: qsub scriptname
Double precision MPI (single-node), 2 to 24 cores
An example batch submission script to run the double-precision mdrun_mpi_d
executable on 8 cores using mpi:
#!/bin/bash #$ -S /bin/bash #$ -cwd #$ -V #$ -pe smp.pe 4 mpirun -n $NSLOTS mdrun_mpi_d # # This is the old naming convention (it will still work in this release) # The new gromacs convention is to run: mpirun -n $NSLOTS gmx_mpi_d mdrun
Submit with the command: qsub scriptname
Single-precision AVX2, MPI with Infiniband, 48 cores or more in multiples of 24
An example batch submission script to run the single precision mdrun_mpi
executable with 48 MPI processes (48 cores on two 24-core nodes) with the orte-24-ib.pe
parallel environment (Intel Haswell nodes using infiniband):
#!/bin/bash #$ -S /bin/bash #$ -cwd #$ -V #$ -pe orte-24-ib.pe 48 # EG: Two 24-core Intel Haswell nodes mpirun -n $NSLOTS mdrun_mpi # # This is the old naming convention (it will still work in this release) # The new gromacs convention is to run: mpirun -n $NSLOTS gmx_mpi mdrun
Submit with the command: qsub scriptname
Illegal instruction
If during a batch job you try to run gromacs and get the following error:
Illegal instruction
This is because you have an AVX or AVX2 only version of the modulefile loaded which is not compatible with the compute nodes on which your job is running. Ensure your jobscript requests the correct type of compute node.
Error about OpenMP and cut-off scheme
If you encounter the following error:
OpenMP threads have been requested with cut-off scheme Group, but these are only supported with cut-off scheme Verlet
then please try using the mpi version of the software. Note that is is possible to run mpi versions on a single node (example above).
Further info
- You can see a list of all the installed GROMACS utilities with the command:
ls $GMXDIR/bin
- GROMACS web page
- GROMACS manuals
- GROMACS user mailing list
Updates
Aug 2018 – 2016.3 installed with AVX support (GPU support with Intel compiler not possible)
Sep 2017 – 5.1.4 installed with AVX support (GPU support with Intel compiler not possible)
Apr 2015 – 5.0.4 installed with AVX support (GPU support with Intel compiler not possible)
Dec 2014 – 4.6.7 installed with AVX support (specific user request for this) and documentation written.
Nov 2013 – Documentation for 4.5.4 and 4.6.1 split in to two pages.
May 2013 – Gromacs 4.6.1 and Plumed 1.3 installed.