OpenFOAM

Overview

OpenFOAM (Open source Field Operation And Manipulation) is a C++ toolbox for the development of customized numerical solvers, and pre-/post-processing utilities for the solution of continuum mechanics problems, including computational fluid dynamics (CFD).

Due to a split in the OpenFoam organisation in 2016, there are two main versions; openfoam.org and openfoam.com. We have installations of both versions, see modulefile information below for further notes. To complicate matters further there are several minor forks of OpenFoam, if you make a request, please be precise in which version.

Restrictions on use

OpenFOAM is distributed by the OpenFOAM Foundation and is freely available and open source, licensed under open source licenses. All CSF users may use this software.

Set up procedure

This slightly different to most other CSF applications.

OpenFOAM expects the variable FOAM_RUN to be set for your job and to contain the relevant files and directories. It is recommended to use scratch and then copy back any needed results to your home directory

In your jobscript you can use one of the following module load commands for the different versions on CSF3.

OpenFOAM.org versions

module load apps/gcc/openfoam/14             # Includes OpenFOAM.org 14
module load apps/gcc/openfoam/13             # Includes OpenFOAM.org 13
module load apps/gcc/openfoam/12             # Includes OpenFOAM.org 12
module load apps/gcc/openfoam/11             # Includes OpenFOAM.org 11
module load apps/gcc/openfoam/10             # Includes OpenFOAM.org 10
module load apps/gcc/openfoam/10-89f925d     # Includes OpenFOAM.org 10 in the designated commit
module load apps/gcc/openfoam/9              # Includes OpenFOAM.org 9, rheotools
module load apps/gcc/openfoam/8              # Includes OpenFOAM.org 8, swak4foam
module load apps/gcc/openfoam/7-addons       # Includes OpenFOAM.org 7, swak4foam and rheotools
module load apps/gcc/openfoam/7              # Previously accidentally labelled as v1906

OpenFOAM.com versions

This version contains, in addition to the main OpenFOAM tools, “customer sponsored developments and contributions from the community, including the OpenFOAM Foundation. This Official OpenFOAM release contains several man years of client-sponsored developments of which much has been transferred to, but not released in the OpenFOAM Foundation branch”.

module load apps/gcc/openfoam/v2506         # Includes METIS decomp library
module load apps/gcc/openfoam/v2312         # Includes METIS decomp library
module load apps/gcc/openfoam/v2306         # Includes METIS decomp library
module load apps/gcc/openfoam/v2212         # Includes METIS decomp library and swak4foam
module load apps/gcc/openfoam/v2106
module load apps/gcc/openfoam/v2012         # Includes METIS decomp library and swak4foam
module load apps/gcc/openfoam/v2006         # Includes METIS decomp library and swak4foam
module load apps/gcc/openfoam/v1912         # Includes METIS decomp library and swak4foam
module load apps/gcc/openfoam/v1906
module load apps/gcc/openfoam/v1812

Running the application

User are requested not to output data every timestep of their simulation if not needed. This can create a huge number of files and directories in your scratch area (we have seen millions of files generated). Please ensure you modify your controlDict file to turn off writing at every timestep. For example, set purgeWrite 5 to keep just the most recent 5 timesteps worth and set a suitable writeInterval. Please check the controlDict online documentation for more keywords and options.

To check your scratch usage (space consumed and number of files) run the following command on the login node: scrusage

Serial batch job submission

1. Ensure you have followed the Set up Procedure (loading the modulefile and sourcing the dot file.)
2. Now in the top directory ($FOAM_RUN) where you have set up your job/case (the one containing 0, constant and system) create a batch submission script, called for example openfoam.slurm containing:

   #!/bin/bash --login
   #SBATCH -p serial        # (or --partition=) Submit to the serial nodes
   #SBATCH -t 1-0           # Wallclock time limit, 1-0 is 1 day (max permitted is 7-0, 7 days)
   
   module purge
   module load apps/gcc/openfoam/6
   
   mkdir -p /scratch/$USER/OpenFoam
   export FOAM_RUN=/scratch/$USER/OpenFoam
   source $foamDotFile
   cd $FOAM_RUN
   interFoam
   

   Submit the job using sbatch jobscript.

   In the jobscripts, replace interFoam with the OpenFOAM executable appropriate to your job/case.

Parallel batch job submission – single node multi-core

1. Ensure you have followed the Set up Procedure (loading the modulefile and sourcing the dot file.)
2. You will need to decompose your case before you can run it. Ensure that you have a file called decomposeParDict in your job/case system directory, specifying the number of cores you wish to use with numberOfSubdomains and a suitable decompositon method, e.g. simple, and related settings (link to relevant documentation).
3. Now run this command from the top level directory of your job/case (the directory containing 0, constant and system directories):

   # All OpenFOAM versions have this serial executable (DO NOT RUN THIS ON THE LOGIN NODES!)
   decomposePar
   

   Note that the decomposePar app can use a lot of memory for dense or complicated meshes. If your job doesn’t complete successfully, you can achieve the same result with a parallel job using:

   # Only "OpenFOAM.com" versions (v1812, ..., v2012 and so on) have this executable
   mpirun -n $NSLOTS redistributePar -decompose -parallel
   

4. Next, still in the top directory, create a batch submission script, called for example openfoam.par.slurm containing:

   #!/bin/bash --login
   #SBATCH -p multicore       # (or --partition=) Submit to the AMD Genoa nodes
   #SBATCH -n 4               # (or --ntasks=) Number of cores
   #SBATCH -t 1-0             # Wallclock time limit, 1-0 is 1 day (max permitted is 7-0, 7 days)
   
   module purge
   module load apps/gcc/openfoam/6
   
   mkdir -p /scratch/$USER/OpenFoam
   export FOAM_RUN=/scratch/$USER/OpenFoam
   source $foamDotFile
   cd $FOAM_RUN
   # mpirun uses $SLURM_NTASKS cores by default (set by the -n flag above)
   mpirun interFoam -parallel
   

   Submit the job using sbatch jobscript.

   replacing interFoam with the OpenFOAM executable appropriate to your job/case.

   The number of cores requested must match the numberOfSubdomains setting you made earlier – if it doesn’t your job will fail.

5. After the simulation is finished you may need to recombine the distributed mesh (and results) in to one mesh. The following can be run in a serial (1-core) job to do that:

   # All OpenFOAM versions have this serial executable (DO NOT RUN THIS ON THE LOGIN NODE!)
   reconstructPar
   

   Alternatively, to perform this step in a parallel job (e.g., at the end of your main simulation job) you can use:

   # Only "OpenFOAM.com" versions (v1812, ..., v2012 and so on) have this executable
   mpirun redistributePar -recompose -parallel
   

   This will generate a directory for each time-step in you simulation or just the last time-step if that is the only one you keep.

Parallel batch job submission – multi node

Follow the same steps as with the parallel – single node multi-core job submission above.
The only difference is in the batch submission script, where you must specify the number of Nodes you need and choose a partition that supports multi-node jobs.
Jobs in the multinode partition must use all cores on each node. You can find the number of cores per node in our Partitions table.
It should contain:

#!/bin/bash --login
#SBATCH -p multinode       # Partition is required
#SBATCH -N 7               # (or --nodes=) Number of nodes 
#SBATCH -n 280             # (or --ntasks=) Total number of cores. One core per MPI process
                           # where n = N x number_of_cores_per_node 
#SBATCH -t 2-0             # Wallclock limit (days-hours). Required!

module purge
module load apps/gcc/openfoam/6

mkdir -p /scratch/$USER/OpenFoam
export FOAM_RUN=/scratch/$USER/OpenFoam
source $foamDotFile
cd $FOAM_RUN
# mpirun uses $SLURM_NTASKS cores by default (set by the -n flag above)
mpirun interFoam -parallel

For efficiency, to avoid having cores sitting idle, please adjust the number of cores requested to an exact multiple of the cores that each node has (currently 40 for multinode). If this is not possible, then you can force the mpirun command to use a specific number of cores which is slightly less than the total requested from Slurm:

mpirun -np 256 interFoam -parallel

You also need to ensure that the numberOfSubdomains value in your decomposeParDict file matches exactly the value you set in mpirun’s -np flag.

Node and Core limits

There are limits of minimum and maximum number of cores and number of nodes configured. The limits depend on the partition requested but also on the type of your account, for example free-at-the-point-of-use (fatpou01) accounts have a lower core-count limit. Requesting a value above or below the configured limits will result in a job submission error.

Please consult our Partition Table which is kept up to date according to our current configuration.

Reducing disk space

OpenFOAM can generate a lot of output files – especially if the results of every time-step are written to disk (we strongly discourage this!) Once you’ve post-processed your time-step files, do you need to keep them? If not you could simply delete the files:

# Caution - this will delete a lot of files - scratch is NOT backed up!
cd ~/scratch/my-openfoam-sim
rm -rf processor*

If you ran the reconstructPar app to recombine the results from each CPU, you will still have a file for every time-step in the postProcessing directory. Do you need this – for example, if you have generated a movie file of the results, you might not want the individual time-step files:

# Caution - this will delete a lot of files - scratch is NOT backed up!
cd ~/scratch/my-openfoam-sim
rm -rf postProcessing

If you do want to keep the files, archiving them in to a single compressed file will save a lot of space. While time-step files might be individually small, the fact that the Lustre filesystem has a minimum block size means very small files actually consume more space than is really needed for them.

The following job-script will archive all of your step files in to a single compressed tar archive:

#!/bin/bash
#SBATCH -p multicore_small      # (or --partition=) Submit to the 32-core Intel nodes
#SBATCH -n 4                    # (or --ntasks=) Number of cores
#SBATCH -t 0-1                  # Wallclock time limit (0-1 is 1 hour, max permitted is 7-0, 7 days)

module purge
module load tools/gcc/pigz/2.4
tar cf - processor* postProcessing |  pigz -p $SLURM_NTASKS > my-openfoam-sim.tar.gz

Submit the job from the directory where your OpenFOAM files are located, using sbatch jobscript where jobscript is the name of your file.

Once the job has finished you can remove the individual directories as shown above.

You should also copy the my-openfoam-sim.tar.gz file to your home directory:

cp my-openfoam-sim.tar.gz ~

or to your Research Data Storage area.

If you ever need to extract the files from the archive, simply run:

tar xzf my-openfoam-sim.tar.gz

It will recreate the above directories and files in your current directory.

Additional advice

• When changing the number of cores you will need to adjust your input files appropriately and ensure decomposePar is re-run.
• If the decomposePar command takes more than a few minutes to run or uses significant resource on the login node then please include it in your job submission script instead by including it on the line before the mpirun so that it executes as part of the batch job on the compute node.

Further info

• The OpenFOAM tutorials are very good and ideal for testing and getting used to setting up a job on the CSF before you proceed to production runs of your own work. A tutorial on how to run OpenFOAM v9 on CSF3 is available in:

  /opt/apps/apps/gcc/openfoam/9/openfoam9_training_csf3_rans.tar.gz

• OpenFOAM documentation.
• swak4foam website.
• Useful swak4foam tutorials on the NTNU HPC Wiki.