HYPRE

Overview

HYPRE is a library of high performance preconditioners and solvers featuring multigrid methods for the solution of large, sparse linear systems of equations on massively parallel computers.

Restrictions on use

There are no restrictions on access to the HYPRE libraries on CSF. The software is released under a MIT and APACHE licenses. All usage must adhere to one of these licenses.

Set up procedure

We now recommend loading modulefiles within your jobscript so that you have a full record of how the job was run. See the example jobscript below for how to do this. Alternatively, you may load modulefiles on the login node and let the job inherit these settings.

Load one of the following modulefiles to set up your environment:

# All versions have been compiled with the system default GCC 9.3.0 compiler.

module load libs/gcc/hypre/2.25.0-serial      # No parallel features enabled
module load libs/gcc/hypre/2.25.0-omp         # Multi-threaded parallelism (OpenMP)
module load libs/gcc/hypre/2.25.0-mpi         # Multi-node parallelism (OpenMPI 4.0.1)
module load libs/gcc/hypre/2.25.0-cuda        # GPU (CUDA 10.1.168), serial CPU

# All versions have been compiled with the system default GCC 4.8.5 compiler.
# MPI and CUDA modulefiles will be loaded automatically where needed.

module load libs/gcc/hypre/2.18.2-serial      # No parallel features enabled
module load libs/gcc/hypre/2.18.2-omp         # Multi-threaded parallelism (OpenMP)
module load libs/gcc/hypre/2.18.2-mpi         # Multi-node parallelism (OpenMPI 4.0.1)
module load libs/gcc/hypre/2.18.2-cuda        # GPU (CUDA 10.1.168), serial CPU
module load libs/gcc/hypre/2.18.2-mpi-cuda    # GPU (CUDA 10.1.168), MPI (CUDA-aware) CPU

Compiling an HYPRE-capable application

You will use the HYPRE installation on CSF3 when compiling your own software or as a dependency when building other downloaded applications. This allows you to add HYPRE functionality to your own apps. There are no executable programs in the installation – you must compile something to generate a HYPRE application.

The modulefiles will set an environment variable named ${HYPREDIR} which can then be used in your compilation process (e.g., in a Makefile or directly on the command-line) to access the header and library files:

• To inform the compiler of the header file directory use:

  gcc -I${HYPREDIR}/include ....

• To inform the compiler of the library file use:

  gcc ... -L${HYPREDIR}/lib -lHYPRE

  The library is a shared library named libHYPRE.so.

• In a Makefile ensure you use ${HYPREDIR} rather than $HYPREDIR.

An example compilation command could be

gcc -I${HYPREDIR}/include example_hypre.c -o example_hypre -L${HYPREDIR}/lib -lHYPRE

Running an HYPRE-capable application

You must load the HYPRE modulefile before running your HYPRE-capable application.

Please do not run HYPRE-capable applications on the login node. Jobs should be submitted to the compute nodes via batch.

Serial batch job submission

Create a batch submission script (which will load the modulefile in the jobscript), for example:

#!/bin/bash --login
#$ -cwd             # Job will run from the current directory
                    # NO -V line - we load modulefiles in the jobscript

# Load the serial (non-MPI) modulefile
module load libs/gcc/hypre/2.18.2-serial

# Run my application I compiled earlier
./myhypreapp arg1 ...

Submit the jobscript using:

qsub scriptname

where scriptname is the name of your jobscript.

Parallel OpenMP batch job submission

Create a batch submission script (which will load the modulefile in the jobscript), for example:

#!/bin/bash --login
#$ -cwd             # Job will run from the current directory
#$ -pe smp.pe 8     # Number of cores, can be 2--32
                    # NO -V line - we load modulefiles in the jobscript

# Load the serial (non-MPI) modulefile
module load libs/gcc/hypre/2.18.2-omp

# Inform the ap how many cores it can use. $NSLOTS is set to the number of cores from above.
export OMP_NUM_THREADS=$NSLOTS

# Run my application I compiled earlier
./myhypreapp arg1 ...

Submit the jobscript using:

qsub scriptname

where scriptname is the name of your jobscript.

Parallel MPI batch job submission

Create a batch submission script (which will load the modulefile in the jobscript), for example:

#!/bin/bash --login
#$ -cwd             # Job will run from the current directory
#$ -pe mpi-24-ib 48 # Number of cores, must be 48 or more in multiples of 24
                    # NO -V line - we load modulefiles in the jobscript

# Load the serial (non-MPI) modulefile
module load libs/gcc/hypre/2.18.2-mpi

# Run my application I compiled earlier
mpirun -n $NSLOTS ./myhypreapp arg1 ...

Submit the jobscript using:

qsub scriptname

where scriptname is the name of your jobscript.

Further info

See the HYPRE website for full documentation.

Updates

None.