MUMPS

Overview

MUltifrontal Massively Parallel sparse direct Solver. See http://graal.ens-lyon.fr/MUMPS/. MUMPS is a distributed multifrontal solver (F90, MPI based) for the solution of large sparse systems of linear equations A x = b using Gaussian elimination. The parallel version of MUMPS is installed.

Compiled for AMD Bulldozer nodes only. This software will not work on Intel nodes (will return Illegal instruction error).

MUMPS was compiled against the following libraries, which are all available in the installation tree:

• ScaLAPACK, which also provides BLACS (v2.0.2 – compiled from source)
• PT-SCOTCH (v5.1.12_esmumps – compiled from source)
• ParMETIS (v4.0.2 – compiled from source)

The following library is external to the MUMPS installation

• ACML, which provides BLAS and LAPACK (v5.1.0 – provided by AMD as an optimized library)

All compilation was performed with the AMD Open64 compiler using -O3 -march=bdver1 optimization and architecture flags and the OpenMPI 1.6 InfiniBand MPI layer.

PT-SCOTCH and ParMETIS both use 64bit integers for indices. The following config options were used:

• PT-Scotch: IDXSIZE64
• ParMetis: IDXTYPEWIDTH 64, REALTYPEWIDTH 32

Restrictions on use

MUMPS is free software under the CeCILL-C licence (essentially GNU LGPL); see the Conditions of Use for full details. However, versions linked with the proprietary ParMETIS and ACML libraries can’t be distributed; see also the ParMETIS licence.

Set up procedure

You must be on a Bulldozer node to compile and run any programs which use the MUMPS libraries. See the OpenMPI on AMD Bulldozer Compute Nodes page for full details, but essentially you should do:

qrsh -l bulldozer -l short
module load libs/open64-4.5.2/mumps/4.10.0

This will also load the Open64 compiler, ACML and OpenMPI IB modules: compilers/amd/4.5.2, libs/amd/acml/5.1.0 and mpi/open64-4.5.2/openmpi/1.6-ib-amd-bd.

Running the application

The MUMPS header files and libraries are located in the following directories, with $MUMPSROOT and other variables being set by the modulefile.

• $MUMPSROOT/include
• $MUMPSROOT/lib

Compilation requires the use of mpicc and mpif90 when linking against MUMPS libraries. The easiest method of compiling is to write a Makefile which includes the MUMPS compilation settings file

• $MUMPSROOT/Makefile.inc

For example, create a Makefile that looks similar to the one below (download Makefile.mk – you may want to rename it from Makefile.mk to Makefile)

# Sample MUMPS Makefile
# Note 1: Requires the MUMPS modulefile to be loaded
# Note 2: Indented lines MUST begin with a TAB, not 8 spaces (cut-n-paste warning!)

# Makefile.inc (below) requires that topdir be set.
topdir=$(MUMPSROOT)
libdir=$(topdir)/lib

# Compilation settings. In particular CC=mpicc, FC=mpif90, FL=mpif90.
include $(topdir)/Makefile.inc
LIBMUMPS_COMMON = $(libdir)/libmumps_common$(PLAT)$(LIBEXT)

# My executables (Fortran and C examples) - MODIFY as reqd.
all: myFmumps.exe myCmumps.exe

# FORTRAN example using single precision library. Assumes source file is named myFmumps.F - MODIFY as reqd.
LIBSMUMPS = $(libdir)/libsmumps$(PLAT)$(LIBEXT) $(LIBMUMPS_COMMON)
myFmumps.exe: myFmumps.o
        $(FL) -o $@ $(OPTL) myFmumps.o $(LIBSMUMPS) $(LORDERINGS) $(LIBS) $(LIBBLAS) $(LIBOTHERS)


# C example using double precision library. Assumes source file is named myCmumps.c - MODIFY as reqd.
# NOTE: We DO use FL (mpif90) to link a C code against the MUMPS libraries.
LIBDMUMPS = $(libdir)/libdmumps$(PLAT)$(LIBEXT) $(LIBMUMPS_COMMON)
myCmumps.exe: myCmumps.o
        $(FL) -o $@ $(OPTL) myCmumps.o  $(LIBDMUMPS) $(LORDERINGS) $(LIBS) $(LIBBLAS) $(LIBOTHERS)

clean:
        rm -f *.exe *.o

# How .F and .c source files are compiled
.SUFFIXES: .c .F .o
.F.o:
        $(FC) $(OPTF) $(INCS) -I. -I$(topdir)/include -c $*.F $(OUTF)$*.o
.c.o:
        $(CC) $(OPTC) $(INCS) -I. -I$(topdir)/include -c $*.c $(OUTC)$*.o

Simply run make to compile with the above Makefile.

Example Code

Simple example programs are available in $MUMPSROOT/examples/. To use the sample code as a basis for your own projects, copy the examples, modify and recompile, as shown:

qrsh -l bulldozer -l short
module load libs/open64-4.5.2/mumps/4.10.0     # Will load compiler, ACML and openmpi BD modules
mkdir -p ~/scratch/mumps/examples              # Or what ever you want to call the directory
cd ~/scratch/mumps/examples
cp $MUMPSROOT/examples/* .                     # Notice the . at the end of the command
# Modify the .F or .c source files if using the examples as a starting point.
# Now recompile.
make clean
make

Various settings used for compilation are given in the include file

$MUMPSROOT/Makefile.inc

The external libraries against which the MUMPS libraries were compiled are available in

$MUMPSBASE/external/

Serial batch job submission

Only the parallel version of MUMPS has been compiled.

Parallel batch job submission

You must be on an AMD Bulldozer node to run any code dependent on MUMPS. These nodes have a requirement that jobs be a multiple of 64 processes. The following jobscript will run one of the MUMPS example programs (name the file mumps.qsub for example):

#!/bin/bash
#$ -S /bin/bash
#$ -cwd
#$ -V
#$ -pe orte-64bd-ib.pe 64     # Must be a multiple of 64

module load libs/open64-4.5.2/mumps/4.10.0

# See the examples/ directory copied earlier
mpirun -n $NSLOTS ./dsimpletest < input_simpletest_real

Submit the job to the batch system from the login node using

# You should be on the login node, not a bulldozer, node at this point
module load libs/open64-4.5.2/mumps/4.10.0
qsub -V mumps.qsub
   #
   # or what ever you have called your jobscript
   #

Further info

The MUMPS user guide is available at $MUMPSROOT/doc/userguide_4.10.0.pdf or see the online documentation at the following pages:

• MUMPS: http://graal.ens-lyon.fr/MUMPS/
• PT-SCOTCH: http://www.labri.fr/perso/pelegrin/scotch/
• ParMETIS: http://glaros.dtc.umn.edu/gkhome/metis/parmetis/overview
• ScaLAPACK: http://www.netlib.org/scalapack

Updates