DL_MESO

Overview

DL_MESO is a general purpose mesoscale simulation package which supports both Lattice Boltzmann Equation (LBE) and Dissipative Particle Dynamics (DPD) methods.

Versions 2.5 and 2.6 are installed. v2.5 was compiled with intel compilers 12.0.5 and openmpi 1.6. v2.6 was compiled with intel compilers 14.0.4 and openmpi 1.6.

Multiple versions of 2.6 are installed with bugfixes (see modulefile info below). Details of the bugfixes can be found on the DL_MESO INFOMAIL website.

Restrictions on use

Whilst the software is free for academic usage there are limitations within the DL_MESO license agreement which must be strictly adhered to by users. All users who wish to use the software must request access to the ‘dlmeso’ unix group. A copy of the full license is also available on the CSF in $dlmeso_home/$dlmeso_ver/LICENCE. Important points to note are:

• No industrially-funded work must be undertaken using the software. See clauses 2.1.3 and 2.2 of the license.
• The software is only available to Staff and Students of the University of Manchester. Users are reminded that they must not share their password with anyone, or allow anyone else to utlise their account.
• Citation of the software must appear in any published work. See clause 4.2 for the required text.

There is no access to the source code on the CSF.

Set up procedure

Once you have been added to the unix group please load the modulefile:

DL_MESO v2.6 with bugfixes to 14 April 2017 Modulefiles

In April 2017 we added a new installation which contains all the bugfixes released up to 14th April 2017 (up to INFOMAIL 42).

module load apps/intel-14.0/dl_meso/2.6-bf-20170414

This will allow you to run v2.6 as a serial (1-core app) or a multi-threaded (i.e., single compute node) app.

If you wish to run the software in parallel using MPI you will also require one of the following mpi modulefiles:

module load mpi/intel-14.0/openmpi/1.6

for jobs of 2 cores to 24 cores.

OR this one for jobs of 48 cores or more which are a multiple of 24 (targets the Infiniband connected nodes)

module load mpi/intel-14.0/openmpi/1.6-ib

DL_MESO v2.6 with bugfixes to 14 April 2017 Modulefiles compiled with fftw3

These modulefiles contain ONLY executables for DPD. You do not need to load an mpi or fftw3 modulefile as this is automatically done for you.

For FFTW3 floating precision on 2-24 cores please use:

module load apps/intel-15.0/dl_meso/2.6-bf-20170414-fftw3-float-mpi

For FFTW3 floating precision on 48 cores or more which are a multiple of 24 (targets the Infiniband connected nodes) cores please use:

module load apps/intel-15.0/dl_meso/2.6-bf-20170414-fftw3-float-mpi-ib

For FFTW3 double precision on 2-24 cores please use:

module load apps/intel-15.0/dl_meso/2.6-bf-20170414-fftw3-double-mpi

For FFTW3 double precision on 48 cores or more which are a multiple of 24 (targets the Infiniband connected nodes) cores please use:

module load apps/intel-15.0/dl_meso/2.6-bf-20170414-fftw3-double-mpi-ib

DL_MESO v2.6 with bugfixes to 28 Nov 2016 Modulefiles

In January 2017 we added a new installation which contains all the bugfixes released up to 28th November 2016 (up to INFOMAIL 37).

module load apps/intel-14.0/dl_meso/2.6-bf-20161128

This will allow you to run v2.6 as a serial (1-core app) or a multi-threaded (i.e., single compute node) app.

If you wish to run the software in parallel using MPI you will also require one of the following mpi modulefiles:

module load mpi/intel-14.0/openmpi/1.6

for jobs of 2 cores to 24 cores.

OR this one for jobs of 48 cores or more which are a multiple of 24 (targets the Infiniband connected nodes)

module load mpi/intel-14.0/openmpi/1.6-ib

DL_MESO v2.6 Modulefiles

module load apps/intel-14.0/dl_meso/2.6

This will allow you to run v2.6 as a serial (1-core app) or a multi-threaded (i.e., single compute node) app.

If you wish to run the software in parallel using MPI you will also require one of the following mpi modulefiles:

module load mpi/intel-14.0/openmpi/1.6

for jobs of 2 cores to 24 cores.

OR this one for jobs of 48 cores or more which are a multiple of 24 (targets the Infiniband connected nodes)

module load mpi/intel-14.0/openmpi/1.6-ib

DL_MESO v2.5 Modulefiles

module load apps/intel-12.0/dl_meso/2.5

If you wish to run the software in parallel you will also require one of the following mpi modulefiles:

module load mpi/intel-12.0/openmpi/1.6

for jobs of 2 cores or more, but fewer than 12 (targets ethernet connected nodes).

OR this one for jobs of 48 cores or more which are a multiple of 24 (targets the Infiniband connected nodes)

module load mpi/intel-12.0/openmpi/1.6-ib

Running the application

You will notice that there are some differences between the User Manual and the CSF installation. The java interface is not available. Serial and parallel executables are available on the system. For both versions they are named as follows:

DL_MESO v2.6 executables:

Executable	Simulation
slbe.exe	Serial LBE
plbe.exe	Parallel LBE (uses MPI – single and multi-node jobs)
plbe-omp.exe	Parallel LBE (uses OpenMP – single-node multi-threaded jobs)
sdpd.exe	Serial DPD
pdpd.exe	Parallel DPD (uses MPI – single and multi-node jobs)
pdpd-omp.exe	Parallel DPD (uses OpenMP – single-node multi-threaded jobs)
pdpd-fftw3-double.exe	Parallel DPD (uses MPI and FFTW double precision – single and multi-node jobs)
pdpd-fftw3-float.exe	Parallel DPD (uses MPI and FFTW float precision – single and multi-node jobs)

DL_MESO v2.5 executables:

Executable	Simulation
slbe.exe	Serial LBE
plbe.exe	Parallel LBE
sdpd.exe	Serial DPD
pdpd.exe	Parallel DPD

Serial Batch job examples

Serial LBE batch job submission

• Make sure you have the dl_meso modulefile loaded.
• Set up a directory from which your job will run, with all the required input files in it.
• Write a job submission script, for example, in a file called jobscript:

     
#!/bin/bash
### SGE Job Stuff
#$ -cwd
#$ -V
#$ -S /bin/bash

slbe.exe

• Submit: qsub jobscript

Serial DPD batch job submission

• Make sure you have the dl_meso modulefile loaded.
• Set up a directory from which your job will run, with all the required input files in it.
• Write a job submission script, for example, in a file called jobscript:

     
#!/bin/bash
### SGE Job Stuff
#$ -cwd
#$ -V
#$ -S /bin/bash

sdpd.exe

• Submit: qsub jobscript

Parallel (multi-core) Batch job examples

It is highly recommended that you run scaling tests on 2,4,6,8,10,12,16,18,20,22,24 cores before moving on to running larger jobs to see how well your job performs as the number of cores increases.

Parallel LBE batch job submission – 2 to 24 cores using MPI

• Make sure you have the dl_meso and non-ib mpi modulefile loaded.
• Set up a directory from which your job will run, with all the required input files in it.
• Write a job submission script, for example, in a file called jobscript and asking for 6 cores:

     
#!/bin/bash
### SGE Job Stuff
#$ -cwd
#$ -V
#$ -S /bin/bash
#$ -pe smp.pe 6

mpirun -n $NSLOTS plbe.exe

• Submit: qsub jobscript

Parallel LBE batch job submission – 2 to 24 cores using OpenMP

• Make sure you have the dl_meso modulefile loaded.
• Set up a directory from which your job will run, with all the required input files in it.
• Write a job submission script, for example, in a file called jobscript and asking for 6 cores:

    
#!/bin/bash
### SGE Job Stuff
#$ -cwd
#$ -V
#$ -S /bin/bash
#$ -pe smp.pe 6

export OMP_NUM_THREADS=$NSLOTS
plbe-omp.exe

• Submit: qsub jobscript

Parallel DPD batch job submission – 2 to 24 cores using MPI

• Make sure you have the dl_meso and non-ib mpi modulefile loaded.
• Set up a directory from which your job will run, with all the required input files in it.
• Write a job submission script, for example, in a file called jobscript and asking for 12 cores:

     
#!/bin/bash
### SGE Job Stuff
#$ -cwd
#$ -V
#$ -S /bin/bash
#$ -pe smp.pe 12

mpirun -n $NSLOTS pdpd.exe

• Submit: qsub jobscript

Parallel DPD batch job submission – 2 to 24 cores using OpenMP

• Make sure you have the dl_meso modulefile loaded.
• Set up a directory from which your job will run, with all the required input files in it.
• Write a job submission script, for example, in a file called jobscript and asking for 12 cores:

     
#!/bin/bash
### SGE Job Stuff
#$ -cwd
#$ -V
#$ -S /bin/bash
#$ -pe smp.pe 12

export OMP_NUM_THREADS=$NSLOTS
pdpd-omp.exe

• Submit: qsub jobscript

Parallel batch job submissions for more jobs of 24 cores or more and a multiple of 12

• As above, but use the inifiniband mpi modulefile and replace smp.pe with orte-24-ib.pe and an integer equal to or greater than 48 which is a multiple of 24.
• Please ensure you have done some scaling first on 2, 4, 6 etc cores to ensure that you are seeing a benefit from increasing the number of cores.

Further info

• DL_MESO Homepage
• DL_MESO User Manual
• Example data and cases can be found in /opt/gridware/apps/intel-14.0/dl_meso/2.6/DEMO – please see the User Manual for further details.

Updates

Version 2.6 installed 19 Jan 2016.
Version 2.5 installed 22 Jan 2014.