{"id":4682,"date":"2020-07-22T17:08:47","date_gmt":"2020-07-22T16:08:47","guid":{"rendered":"http:\/\/ri.itservices.manchester.ac.uk\/csf3\/?page_id=4682"},"modified":"2024-01-03T15:12:24","modified_gmt":"2024-01-03T15:12:24","slug":"gromacs-2020-3-with-gpu-builds","status":"publish","type":"page","link":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/gromacs-2020-3-with-gpu-builds\/","title":{"rendered":"Gromacs 2020.3 (CPU & GPU versions)"},"content":{"rendered":"

Overview<\/h2>\n

GROMACS is a package for computing molecular dynamics, simulating Newtonian equations of motion for systems with hundreds to millions of particles. GROMACS is designed for biochemical molecules with complicated bonded interactions (e.g. proteins, lipids, nucleic acids) but can also be used for non-biological systems (e.g. polymers).<\/p>\n

\nPlease do not<\/strong> add the -v<\/code> flag to your mdrun<\/code> command.<\/em><\/p>\n

It will write to a log file every second for the duration of your job and can lead to severe overloading of the file servers.<\/td>\n<\/div>\n

Significant Change in this Version<\/h2>\n

Within Gromacs 2020, the different gromacs commands (e.g., mdrun<\/code>, grompp<\/code>, g_hbond<\/code>) should now be run using the command:<\/p>\n

gmx command<\/em>\r\n<\/pre>\n
where command<\/em><\/code> is the name of the command you wish to run (without any g_<\/code> prefix), for example:<\/p>\n
gmx mdrun\r\n<\/pre>\n
The gmx<\/code> command changes its name to reflect the gromacs flavour being used but the command<\/em><\/code> does not change. For example, if using the mdrun<\/code> command:<\/p>\n
# New 2020.3 method                  # Previous 5.0.4 method\r\n# =================                  # =====================\r\ngmx   mdrun                          mdrun\r\ngmx_d mdrun                          mdrun_d\r\nmpirun -n $NSLOTS gmx_mpi   mdrun    mpirun -n $NSLOTS mdrun_mpi\r\nmpirun -n $NSLOTS gmx_mpi_d mdrun    mpirun -n $NSLOTS mdrun_mpi_d\r\n<\/pre>\n
The complete list of command<\/em><\/code> names can be found by running the following on the login node:<\/p>\n
gmx help commands<\/pre>\n
# The following commands are available:\r\nanadock\t\t\tgangle\t\t\trdf\r\nanaeig\t\t\tgenconf\t\t\trms\r\nanalyze\t\t\tgenion\t\t\trmsdist\r\nangle\t\t\tgenrestr\t\trmsf\r\nawh\t\t\tgrompp\t\t\trotacf\r\nbar\t\t\tgyrate\t\t\trotmat\r\nbundle\t\t\th2order\t\t\tsaltbr\r\ncheck\t\t\thbond\t\t\tsans\r\nchi\t\t\thelix\t\t\tsasa\r\ncluster\t\t\thelixorient\t\tsaxs\r\nclustsize\t\thelp\t\t\tselect\r\nconfrms\t\t\thydorder\t\tsham\r\nconvert-tpr\t\tinsert-molecules\tsigeps\r\ncovar\t\t\tlie\t\t\tsolvate\r\ncurrent\t\t\tmake_edi\t\tsorient\r\ndensity\t\t\tmake_ndx\t\tspatial\r\ndensmap\t\t\tmdmat\t\t\tspol\r\ndensorder\t\tmdrun\t\t\ttcaf\r\ndielectric\t\tmindist\t\t\ttraj\r\ndipoles\t\t\tmk_angndx\t\ttrajectory\r\ndisre\t\t\tmorph\t\t\ttrjcat\r\ndistance\t\tmsd\t\t\ttrjconv\r\ndo_dssp\t\t\tnmeig\t\t\ttrjorder\r\ndos\t\t\tnmens\t\t\ttune_pme\r\ndump\t\t\tnmtraj\t\t\tvanhove\r\ndyecoupl\t\torder\t\t\tvelacc\r\ndyndom\t\t\tpairdist\t\tview\r\neditconf\t\tpdb2gmx\t\t\twham\r\neneconv\t\t\tpme_error\t\twheel\r\nenemat\t\t\tpolystat\t\tx2top\r\nenergy\t\t\tpotential\t\txpm2ps\r\nfilter\t\t\tprincipal\r\nfreevolume\t\trama\r\n<\/pre>\n
Notice that the command names do NOT start with g_<\/code> and do NOT reference the flavour being run (e.g., _mpi_d<\/code>). Only the main gmx<\/code> command changes its name to reflect the flavour (see below for list of modulefiles for the full list of flavours available).<\/p>\n
To obtain more help about a particular command run:<\/p>\n
gmx help command<\/em>\r\n<\/pre>\n
For example<\/p>\n
gmx help mdrun\r\n<\/pre>\n
Helper scripts<\/h3>\n
To assist with moving to the new command calling method, we have recreated some of the individual commands that you may have used in your jobscript. For example, you can continue to use mdrun<\/code> (or mdrun_d<\/code>) instead of the new gmx mdrun<\/code> (or gmx_d mdrun<\/code>) in this release. These extra commands are automatically included in your environment when you load the gromacs modulefiles. This old method uses the flavour of gromacs in the command name (see above for comparison of new and old commands).<\/p>\n
However, please note that some commands are new (since 2020.1) and so can only be run using the new method (gmx command<\/em><\/code>):<\/p>\n
Available Flavours<\/h2>\n
For version 2020.3 we have compiled multiple versions of Gromacs, each of which is optimised for a particular CPU architecture. We have also built versions with GPU support (note, GPU versions of Gromacs only support single precision). The module file has been written to detect which CPU the compute node is using and to automatically select the correct Gromacs executable. If you want to ensure you get a particular level of opimisation specify an architecture<\/a> in the jobscript e.g -l skylake<\/code>.<\/p>\n
2020.3 for Ivybridge (and Haswell, Broadwell and Skylake nodes) only<\/h3>\n
With AVX optimisation.<\/p>\n
2020.3 for Haswell and Broadwell (and Skylake) nodes only<\/h3>\n
With AVX2 optimisation.<\/p>\n
2020.3 for Skylake nodes only<\/h3>\n
With AVX-512 optimisation.<\/p>\n
2020.3 for Skylake nodes with GPU acceleration.<\/h3>\n
With AVX-512 optimisation and with GPU acceleration turned on. Note only single precision versions are available with GPU acceleration.<\/p>\n
Restrictions on use<\/h2>\n
GROMACS is free software, available under the GNU General Public License.<\/p>\n
Set up procedure<\/h2>\n
You must load the appropriate modulefile:<\/p>\n
module load modulefile<\/em>\r\n<\/pre>\n
replacing modulefile<\/em> with one of the modules listed in the table below. The module file will auto-detect and pick a version of Gromacs with AVX optimisations to match the CPU of the compute node(s) you are assigned.<\/p>\n
 <\/p>\n\n\n\n\n\n\n\n
Version<\/th>\nModulefile<\/th>\nNotes<\/th>\nTypical Executable name<\/th>\n<\/tr>\n
Single precision multi-threaded (single-node)<\/td>\napps\/intel-18.0\/gromacs\/2020.3\/single<\/td>\nnon-MPI with GPU acceleration available<\/td>\nmdrun<\/code> or gmx\u00a0mdrun<\/code><\/td>\n<\/tr>\n
Double precision multi-threaded (single-node)<\/td>\napps\/intel-18.0\/gromacs\/2020.3\/double<\/td>\nnon-MPI<\/td>\nmdrun_d<\/code> or gmx_d\u00a0mdrun<\/code><\/td>\n<\/tr>\n
Single precision MPI<\/td>\napps\/intel-18.0\/gromacs\/2020.3\/single_mpi<\/td>\nFor MPI<\/td>\nmdrun_mpi<\/code> or gmx_mpi\u00a0mdrun<\/code><\/td>\n<\/tr>\n
Double precision MPI<\/td>\napps\/intel-18.0\/gromacs\/2020.3\/double_mpi<\/td>\nFor MPI<\/td>\nmdrun_mpi_d<\/code> or gmx_mpi_d\u00a0mdrun<\/code><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
Running the application<\/h2>\n
Please do not<\/strong> run GROMACS on the login node.<\/p>\n
Important notes regarding running jobs in batch<\/h3>\n
We now recommend that the relevant module file (single\/double non-MPI\/MPI) is loaded as part of your batch script. The module file itself will select the most suitable build of Gromacs for the processor architecture you end up running on.<\/p>\n
Please NOTE the following which is important for running jobs correctly and efficiently:<\/p>\n
Ensure you inform gromacs how many cores it can use. This is done using the $NSLOTS<\/code> variable which is automatically set for you in the jobscript to be the number of cores you request in the jobscript header (see later for complete examples). You can use either of the following methods depending whether you want a multi-core job (running on a single compute node) or a larger job running across multiple compute nodes:<\/p>\n
# Multi-core (single-node) or Multi-node MPI jobs\r\n\r\nmpirun -n $NSLOTS mdrun_mpi         # Old method (v5.0.4 and earlier)\r\nmpirun -n $NSLOTS mdrun_mpi_d       # Old method (v5.0.4 and earlier)\r\n\r\nmpirun -n $NSLOTS gmx_mpi mdrun     # New method (v5.1.4 and later)\r\nmpirun -n $NSLOTS gmx_mpi_d mdrun   # New method (v5.1.4 and later)<\/pre>\n
or<\/p>\n
# Single-node multi-threaded job\r\n\r\nexport OMP_NUM_THREADS=$NSLOTS      # Do this for all versions\r\nmdrun                               # Old method (v5.0.4 and earlier)\r\nmdrun_d                             # Old method (v5.0.4 and earlier)\r\n\r\nexport OMP_NUM_THREADS=$NSLOTS      # Do this for all versions\r\ngmx mdrun                           # New method (v5.1.4 and later)\r\ngmx_d mdrun                         # New method (v5.1.4 and later)\r\n\r\n# Single-node multi-threaded job with GPU acceleration.\r\n\r\n<\/pre>\n
The examples below can be used for single precision or double precision gromacs. Simply run mdrun<\/code> (single precision) or mdrun_d<\/code> (double precision).<\/p>\n
\nPlease do not<\/strong> add the -v<\/code> flag to your mdrun<\/code> command.<\/em><\/p>\n
It will write to a log file every second for the duration of your job and can lead to severe overloading of the file servers.\n<\/p><\/div>\n
Multi-threaded single-precision on Intel nodes, 2 to 32 cores<\/h3>\n
Note that GROMACS 2020.3 (unlike v4.5.4) does not<\/strong> support the -nt<\/code> flag to set the number of threads when using the multithreaded OpenMP (non-MPI) version. Instead set the OMP_NUM_THREADS<\/code> environment variable as shown below.<\/p>\n
An example batch submission script to run the single-precision<\/strong> mdrun executable with 12 threads:<\/p>\n
#!\/bin\/bash --login\r\n#$ -cwd\r\n#$ -pe smp.pe 12            # Can specify 2 to 32 cores in smp.pe\r\n                           \r\nmodule load apps\/intel-18.0\/gromacs\/2020.3\/single\r\nexport OMP_NUM_THREADS=$NSLOTS\r\nmdrun\r\n  #\r\n  # This is the old naming convention (it will still work in this release)\r\n  # The new gromacs convention is to run: gmx mdrun\r\n<\/pre>\n
Submit with the command: qsub scriptname<\/code><\/p>\n
The system will run your job on a Ivybridge, Haswell, Broadwell or Skylake node depending on the number of cores requested and what is available. Not specifying (recommende) an architecture means that your job will start as soon as any type of compute node that can accommodate it becomes available (it gives the job the biggest pool of nodes to target). To get a more optimised run on Haswell\/Broadwell you should specify the architecture you require, but note that it may take longer for your job to start as specifying an architecture reduces the size of the pool that the system can target.<\/p>\n
Multi-threaded double-precision on Intel nodes, 2 to 32 cores<\/h3>\n
An example batch submission script to run the double-precision<\/strong> mdrun_d<\/code> executable with 8 threads:<\/p>\n
#!\/bin\/bash --login\r\n#$ -cwd\r\n#$ -pe smp.pe 24\r\nmodule load apps\/intel-18.0\/gromacs\/2020.3\/double\r\nexport OMP_NUM_THREADS=$NSLOTS\r\nmdrun_d\r\n  #\r\n  # This is the old naming convention (it will still work in this release)\r\n  # The new gromacs convention is to run: gmx_d mdrun<\/pre>\n
Submit with the command: qsub scriptname<\/code><\/p>\n
Single precision MPI (single-node), 2 to 32 cores<\/h3>\n
An example batch submission script to run the double-precision mdrun_mpi<\/code> executable on 8 cores using mpi:<\/p>\n
#!\/bin\/bash --login\r\n#$ -cwd\r\n#$ -pe smp.pe 8            \r\nmodule load apps\/intel-18.0\/gromacs\/2020.3\/single_mpi                                          \r\nmpirun -n $NSLOTS mdrun_mpi\r\n<\/pre>\n
Submit with the command: qsub scriptname<\/code><\/p>\n
Double precision MPI (single-node), 2 to 32 cores<\/h3>\n
An example batch submission script to run the double-precision<\/strong> mdrun_mpi_d<\/code> executable on 8 cores using mpi:<\/p>\n
#!\/bin\/bash --login\r\n#$ -cwd\r\n#$ -V\r\n#$ -pe smp.pe 8\r\nmodule load apps\/intel-18.0\/gromacs\/2020.3\/double_mpi                                           \r\nmpirun -n $NSLOTS mdrun_mpi_d\r\n  #\r\n  # This is the old naming convention (it will still work in this release)\r\n  # The new gromacs convention is to run: mpirun -n $NSLOTS gmx_mpi_d mdrun\r\n<\/pre>\n
Submit with the command: qsub scriptname<\/code><\/p>\n
Single-precision, MPI, 48 cores or more in multiples of 24<\/h3>\n
An example batch submission script to run the single precision<\/strong> mdrun_mpi<\/code> executable with 48 MPI processes (48 cores on two 24-core nodes) with the mpi-24-ib.pe<\/code> parallel environment (Intel Haswell nodes using infiniband):<\/p>\n
#!\/bin\/bash --login\r\n#$ -cwd\r\n#$ -pe mpi-24-ib.pe 48           # EG: Two 24-core Intel Haswell nodes\r\nmodule load apps\/intel-18.0\/gromacs\/2020.3\/single_mpi\r\nmpirun -n $NSLOTS gmx_mpi mdrun\r\n  #\r\n  # This is the old naming convention (it will still work in this release)\r\n  # The new gromacs convention is to run: mpirun -n $NSLOTS gmx_mpi mdrun<\/pre>\n
Submit with the command: qsub scriptname<\/code><\/p>\n
Double-precision, MPI, 48 cores or more in multiples of 24<\/code><\/h3>\n
An example batch submission script to run the single precision<\/strong> mdrun_mpi<\/code> executable with 48 MPI processes (48 cores on two 24-core nodes) with the mpi-24-ib.pe<\/code> parallel environment (Intel Haswell nodes using infiniband):<\/code><\/p>\n
#!\/bin\/bash --login\r\n#$ -cwd\r\n#$ -pe mpi-24-ib.pe 48           # EG: Two 24-core Intel Haswell nodes\r\nmodule load apps\/intel-18.0\/gromacs\/2020.3\/double_mpi\r\nmpirun -n $NSLOTS gmx_mpi_d mdrun\r\n<\/pre>\n
Submit with the command: qsub scriptname<\/code><\/code><\/p>\n
Multi-threaded single-precision on a single node with one GPU.<\/h3>\n
You need to request being added to the relevant group to access GPUs<\/a> before you can run GROAMACS on them.<\/strong><\/p>\n
Please note that if you have ‘free at the point of use’ access to the GPUs then the maximum number of GPUs you can request is 2<\/p>\n
The maximum number of CPU cores that anyone can request is 8 per GPU.<\/p>\n
#!\/bin\/bash --login\r\n#$ -cwd\r\n#$ -pe smp.pe 8             #Specify the number of CPUs, maximum of 8 per GPU.\r\n#$ -l v100               #This requests a single GPU.\r\n\r\nmodule load apps\/intel-18.0\/gromacs\/2020.3\/single\r\ngmx mdrun -ntmpi 1 -ntomp ${NSLOTS} ...\r\n<\/pre>\n
Submit with the command: qsub scriptname<\/code><\/p>\n
This requests 1 thread mpi rank for the GPU and $NSLOTS (8 in this case) OpenMP threads per rank.<\/p>\n
Multi-threaded single-precision on a single node with multiple GPUs<\/h3>\n
You need to request being added to the relevant group to access GPUs<\/a> before you can run GROAMACS on them.<\/strong><\/p>\n
Please note that if you have ‘free at the point of use’ access to the GPUs then the maximum number of GPUs you can request is 2 (please therefore follow the previous example).<\/p>\n
The maximum number of CPU cores that anyone can request is 8 per GPU requested e.g. 1 GPU and 8 cores, 2 GPUs and 16 cores.<\/p>\n
#!\/bin\/bash --login\r\n#$ -cwd\r\n#$ -pe smp.pe 16            #Specify the number of CPUs, maximum of 8 per GPU.\r\n#$ -l v100=2             #Specify we want a GPU (nvidia_v100) node with two GPUs, maximum is 4.\r\n\r\nmodule load apps\/intel-18.0\/gromacs\/2020.3\/single\r\ngmx mdrun -ntmpi 2 -ntomp 8 ...\r\n<\/pre>\n
Here ntmpi<\/code> is the number of (thread)mpi ranks and ntomp<\/code> is the number of OpenMP threads per rank.<\/p>\n
For the example above where we have requested 2 GPUs (and therefore have a maximum of 16 cores to use) sensible combinations are<\/p>\n
-ntmpi 2 -ntomp 8   # 1 Rank per GPU, 8 threads per rank\r\n-ntmpi 4 -ntomp 4   # 2 Ranks per GPU, 4 threads per rank\r\n-ntmpi 8 -ntomp 2   # 4 Ranks per GPU, 2 threads per rank\r\n<\/pre>\n
If you have time to experiment you can try each combination to see which gives the best performance, if not, use the following<\/p>\n
export OMP_NUM_THREADS=$((NSLOTS\/NGPUS))\r\ngmx mdrun -ntmpi ${NGPUS} -ntomp ${OMP_NUM_THREADS}\r\n<\/pre>\n
Submit with the command: qsub scriptname<\/code><\/p>\n
Error about OpenMP and cut-off scheme<\/h2>\n
If you encounter the following error:<\/p>\n
OpenMP threads have been requested with cut-off scheme Group, but these \r\nare only supported with cut-off scheme Verlet\r\n<\/pre>\n
then please try using the mpi version of the software. Note that is is possible to run mpi versions on a single node (example above).<\/p>\n
Further info<\/h2>\n