HOOMD-blue
Overview
HOOMD-Blue is a general purpose particle simulation toolkit. It performs hard particle Monte Carlo simulations of a variety of shape classes, and molecular dynamics simulations of particles with a range of pair, bond, angle, and other potentials. The software is accessed as a python module.
Version 2.5.1 is installed on the CSF. This supports parallel CPU processing (MPI, TBB) and GPU processing (CUDA).
You need to request being added to the relevant group to access GPUs before you can run HOOMD-blue on them.
Restrictions on use
There are no restrictions on accessing this software on the CSF. It is released under the BSD 3-Clause license and all use must adhere to that license.
Please ensure you cite your use of HOOMD-blue. The references are reported by the software at the start of a simulation or can be found on the citing HOOMD-blue page.
Set up procedure
We now recommend loading modulefiles within your jobscript so that you have a full record of how the job was run. See the example jobscript below for how to do this. Alternatively, you may load modulefiles on the login node and let the job inherit these settings.
Load one of the following modulefiles:
module load apps/gcc/hoomd-blue/2.5.1
This will load all necessary MPI, CUDA and Python modulefiles for you.
Running the application
Please do not run HOOMD-blue on the login node. Jobs should be submitted to the compute nodes via batch.
The hoomd python module accepts various command-line flags. To see what these are, use the following command. See also our jobscript examples below for various GPU related flags:
python -c "import hoomd; hoomd.context.initialize()" --help
In the following examples we use the simple example code available in the HOOMD-blue hoomd python package documentation. For example, create a python script named simple.py as follows:
import hoomd
from hoomd import md
hoomd.context.initialize()
# create a 10x10x10 square lattice of particles with name A
hoomd.init.create_lattice(unitcell=hoomd.lattice.sc(a=2.0, type_name='A'), n=10)
# specify Lennard-Jones interactions between particle pairs
nl = md.nlist.cell()
lj = md.pair.lj(r_cut=3.0, nlist=nl)
lj.pair_coeff.set('A', 'A', epsilon=1.0, sigma=1.0)
# integrate at constant temperature
all = hoomd.group.all();
md.integrate.mode_standard(dt=0.005)
hoomd.md.integrate.langevin(group=all, kT=1.2, seed=4)
# run 10,000 time steps
# hoomd.run(10e3)
# For CSF3 tests we run a little longer (1,000,000 steps)
hoomd.run(10e5)
Note: HOOMD-blue is typically used on GPUs hence we give GPU job examples first.
Single GPU batch job submission
You need to request being added to the relevant group to access GPUs before you can run HOOMD-blue on them.
The following jobscript will run HOOMD-blue on a single GPU:
#!/bin/bash --login #$ -cwd #$ -l v100=1 # A single GPU # Load the modulefile in the jobscript (use the version required for your work) module load apps/gcc/hoomd-blue/2.5.1 # Run on the single GPU assigned to your job python simple.py --gpu=0 # # Always use id 0 to mean a single GPU. # Your job may have been assigned to a GPU with a different # physical id but 0 here means the single GPU your job has # been given.
Submit the jobscript using:
qsub scriptname
where scriptname is the name of your jobscript.
Note that in the output from the job you will see something similar to:
HOOMD-blue is running on the following GPU(s): [0] Tesla V100-SXM2-16GB 80 SM_7.0 @ 1.53 GHz, 16160 MiB DRAM, MNG
The [0] is not necessarily the GPU physical id assigned to your job. This just means you have one GPU assigned to your job (if you run a multi-GPU job you will see them all listed here). DO NOT specify GPU ids on the python command line.
Multi GPU batch job using NVLink submission
You need to request being added to the relevant group to access GPUs before you can run HOOMD-blue on them.
The GPU nodes in the CSF contain NVLink hardware linking the GPUs within a single compute node together. This speeds up communication between GPUs in a multi-GPU job. To run HOOMD-blue using multiple GPUs in a single compute node using the NVLink hardware, use the following jobscript:
#!/bin/bash --login #$ -cwd #$ -l v100=2 # A multi-GPU job (can be 2--4) # We only need a single CPU hence no -pe line # Load the modulefile in the jobscript (use the version required for your work) module load apps/gcc/hoomd-blue/2.5.1 # Run on the first and second GPUs assigned to our job. We must always use 0,1 here. python simple.py --gpu=0,1 # # Always use ids 0,1 to mean the two GPUs. # Your job may have been assigned to GPUs with different # physical ids but 0,1 here means the two GPUs your job has # been given.
Submit the jobscript using:
qsub scriptname
where scriptname is the name of your jobscript.
As before, the GPU ids must be 0,1 for a two-GPU job. The physical ids may be different but the CUDA library will map 0,1 to the physical ids.
Multi GPU batch job using MPI submission
You need to request being added to the relevant group to access GPUs before you can run HOOMD-blue on them.
This method uses multiple MPI processes with each one accessing a single GPU. This is not so useful in the CSF at moment because (a) we run the GPUs in compute-exclusive mode, so only a single MPI process can access a GPU, and (b) we do not support multi-node GPU jobs currently. In future this may be more useful.
To use the MPI method of running multi-GPU jobs use the following jobscript:
#!/bin/bash --login #$ -cwd #$ -l v100=2 # A multi-GPU job (can be 2--4) #$ -pe smp.pe 2 # We need an MPI process for each GPU # Load the modulefile in the jobscript (use the version required for your work) module load apps/gcc/hoomd-blue/2.5.1 # Run python via mpirun to start multiple copies. Each uses one GPU. # $NSLOTS is set automatically to the number of CPU cores given above. # DO NOT specify any GPU ids. They will be detected automatically. mpirun -n $NSLOTS python simple.py # # DO NOT specify GPU ids when using the MPI method. # hoomd will assign a valid GPU to each MPI process.
Submit the jobscript using:
qsub scriptname
where scriptname is the name of your jobscript.
In future we may be able to combine the NVLink method with the MPI method to support multi-node GPU jobs.
CPU Serial batch job submission
It would be unusual to run HOOMD-blue in a serial CPU job but it is possible. Create a batch submission script (which will load the modulefile in the jobscript), for example:
#!/bin/bash --login #$ -cwd # Job will run from the current directory # NO -V line - we load modulefiles in the jobscript # Load the modulefile in the jobscript (use the version required for your work) module load apps/gcc/hoomd-blue/2.5.1 # The job will run on a CPU-only mode and so hoomd will run in CPU mode # $NSLOTS will be set automatically to one core in a single core job. python simple.py --mode=cpu --nthreads=$NSLOTS # # This flag is optional. If no GPUs are detected then # CPU mode will be used by default.
Submit the jobscript using:
qsub scriptname
where scriptname is the name of your jobscript.
Multi-core Parallel batch job submission
Some HOOMD-blue classes can use multiple CPU cores within a single compute node. This is done using the Intel Thread Building Blocks library and support for this has been compiled in to HOOMD-blue on the CSF. The TBB modulefile will be loaded automatically when you load the hoomd modulefile.
Use a jobscript similar to the following:
#!/bin/bash --login #$ -cwd # Job will run from the current directory #$ -pe smp.pe 16 # Number of CPU cores (can be 2--32) # NO -V line - we load modulefiles in the jobscript # Load the modulefile in the jobscript (use the version required for your work) module load apps/gcc/hoomd-blue/2.5.1 # The job will run on a CPU-only mode and so hoomd will run in CPU mode. # $NSLOTS is set automatically to the number of cores requested above python simple.py --mode=cpu --nthreads=$NSLOTS # # This flag is optional. If no GPUs are detected then # CPU mode will be used by default.
Submit the jobscript using:
qsub scriptname
where scriptname is the name of your jobscript.
Note that it is also possible to use the OMP_NUM_THREADS=$NSLOTS method to set the number of cores to use if you are more familiar with OpenMP.
Multi-node (MPI) Parallel batch job submission
If your domain is large enough then you can also run multi-node CPU-only parallel jobs. Use a jobscript similar to:
#!/bin/bash --login #$ -cwd #$ -pe orte-24-ib.pe 48 # Multi-node job (two x 24-core compute nodes) # Load the modulefile in the jobscript (use the version required for your work) module load apps/gcc/hoomd/2.5.1 # $NSLOTS is set automatically to the number of cores requested above mpirun -n $NSLOTS python simple.py
Submit the jobscript using:
qsub scriptname
where scriptname is the name of your jobscript.
Further info
Updates
None.