{"id":8321,"date":"2024-10-04T18:13:07","date_gmt":"2024-10-04T17:13:07","guid":{"rendered":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/?page_id=8321"},"modified":"2025-08-26T17:10:42","modified_gmt":"2025-08-26T16:10:42","slug":"gaussian16-linda","status":"publish","type":"page","link":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/software\/applications\/gaussian16-linda\/","title":{"rendered":"Gaussian16 Linda (multinode)"},"content":{"rendered":"

Gaussian<\/a> is a general purpose suite of electronic structure programs. The “Linda” version adds multi-node (larger parallel jobs) capability to Gaussian.<\/p>\n

Versions g16c01 Linda is installed on the CSF. It is available as binaries only. The source code is not<\/em> available on the CSF.<\/p>\n

Gaussian 16 (non-Linda) is also available on the CSF<\/a>.
\nGaussian 09 (non-Linda) is also available on the CSF<\/a>.<\/p>\n

Restrictions on use<\/h2>\n
Gaussian Linda is only available to members for Prof. Kaltsoyannis’s group. All requests to access this version must be approved by Prof. Kaltsoyannis.<\/div>\n

Please contact us via our help form<\/a> to request access to Gaussian 16 Linda.<\/p>\n

Set up procedure<\/h2>\n

There is only one version available on the CSF – optimized for Intel “Haswell” architectures or equivalently any CPU that provides AVX2 vector extensions. This includes all Intel and AMD compute nodes in the CSF.<\/p>\n

The detectcpu<\/code> modulefile must be loaded inside your jobscript, not<\/em> on the login node.<\/p>\n

After being added to the relevant unix group, you will be able to access the executables by loading the modulefile:<\/p>\n

module load apps\/binapps\/gaussian\/g16c01_em64t_detectcpu_linda\r\n<\/pre>\n
Gaussian MUST ONLY<\/strong> be run in batch. Please DO NOT<\/strong> run g16<\/code> on the login nodes. Computational work found to be running on the login nodes will be killed WITHOUT WARNING<\/strong>.<\/p>\n
Gaussian Scratch<\/h2>\n
Gaussian uses an environment variable $GAUSS_SCRDIR<\/code> to specify a directory path for where to write scratch<\/em> (temporary) files (two-electron integral files, integral derivative files and a read-write<\/em> file for temporary workings).<\/p>\n
It is set to your scratch<\/em> directory (~\/scratch<\/code>) when you load the modulefile. This is a Lustre filesystem which provides good I\/O performance. Do not be tempted to use your home directory for Gaussian scratch files – the files can be huge making the home area at risk of going over quota. So this gives you a sensible default location for temp files if you wish.<\/p>\n
Using a Scratch Directory per Job<\/h3>\n
We recommend using a different scratch directory for each job<\/strong>. This improves file access times if you run many jobs – writing 1000s of scratch files to a single directory can slow down your jobs. It is much better to create a directory for each<\/em> job within your scratch<\/code> area. It is also then easy to delete the entire directory if Gaussian has left unwanted scratch files behind.<\/p>\n
\r\n# In the jobscript:\r\nexport GAUSS_SCRDIR=\/scratch\/$USER\/gau_temp_$SLURM_JOB_ID\r\nmkdir -p $GAUSS_SCRDIR\r\n<\/pre>\n
See below for example jobscripts<\/a> that do this.<\/p>\n
Using a node-local directory<\/h3>\n
A faster, but smaller, local $TMPDIR<\/code> on each compute node is available should users prefer to use that. It can be more efficient if you have a need to create lots of small files, but space is limited. The AMD 168-core Genoa nodes in the multicore<\/code> partition all have a 1.6TB local disk. Use this with caution – if you fill the local disk, your job will crash!<\/strong>.<\/p>\n
\r\n# Slurm creates the temp $TMPDIR. FOR SMALL FILES ONLY.<\/strong>\r\n# It also deletes the folder (and all files) at the end of the job.\r\n# You should take a copy of any files from this area within your jobscript<\/em>.\r\n# You will NOT have access to this temp area after the job has finished.\r\nexport GAUSS_SCRDIR=$TMPDIR\r\n<\/pre>\n
If your job writes a huge .rwf<\/code>, say, then it will likely run out of space in this area and your job will crash. <\/p>\n
Cleaning up scratch files<\/h3>\n
Gaussian should<\/em> delete scratch files automatically when a job completes successfully or dies cleanly. However, it often fails to do this. Scratch files are also not<\/em> deleted when a job is killed externally or terminates abnormally so that you can use the scratch files to restart the job (if possible). Consequently, leftover files may accumulate in the scratch directory, and it is your responsibility to delete these files. Please check periodically<\/strong> whether you have a lot of temporary Gaussian files that can be deleted.<\/p>\n
Very large Gaussian scratch files<\/h3>\n
Occasionally some jobs create .rwf<\/code> files which are very large (several TB). The batch system will not permit a job to create files bigger than 4TB. If your gaussian job fails and the .rwf<\/code> file is 4TB then it may be that this limit has prevented your job from completing. You should re-run the job and in your input file request that the .rwf<\/code> file be split into multiple files. For example to split the file into two 3TB files:<\/p>\n
%rwf=\/scratch\/$USER\/myjob\/one.rwf,3000GB,\/scratch\/$USER\/myjob\/two.rwf,3000GB\r\n<\/pre>\n
Number of Cores<\/h2>\n
\n
If your Gaussian input file contains the number of cores – e.g.:<\/p>\n
%NProcsShared\r\n%nprocs\r\n<\/pre>\n
then you must remove this line from your input file<\/strong>.<\/p>\n
The Linda modulefile will work out the correct number of cores to use, based on your jobscript settings.<\/p>\n<\/div>\n
Parallel batch job<\/h2>\n
You must<\/em> submit your job to the hpcpool<\/code> partition on the CSF – this is the only multi-node capable partition. Our installation will use the fast InfiniBand network for Linda communication. Submitting to the multicore<\/code>partition (to run on a single compute node) will not work<\/strong> (you can simply use non-Linda Gaussian for those jobs.)<\/p>\n
\r\n#!\/bin\/bash --login\r\n#SBATCH -p hpcpool          # (or --partition=) Use the HPC Pool nodes\r\n#SBATCH -N 4                # (or --nodes=) Number of compute nodes (each has 32 cores.)\r\n#SBATCH --ntasks-per-node=1 # Number of Linda workers to run on each compute node (1 is recommended!)\r\n#SBATCH -c 32               # (or --cpus-per-task=) Number of cores each Linda worker will use on its assigned compute node.\r\n#SBATCH -t 1-0              # Maximum wallclock time (1-0 is 1 day, max permitted is 4-0.)\r\n##### You must have been given permission to run in an HPC Pool project<\/strong>\r\n#SBATCH -A hpc-projectcode<\/em>  # The HPC Account code is required<\/strong>\r\n\r\n## Set up scratch dir (please do this!)\r\nexport GAUSS_SCRDIR=\/scratch\/$USER\/gau_temp_$SLURM_JOB_ID\r\nmkdir -p $GAUSS_SCRDIR\r\n\r\n# Load g16 linda\r\nmodule purge\r\nmodule load apps\/binapps\/gaussian\/g16c01_em64t_detectcpu_linda\r\n\r\n$g16root\/g16\/g16 < file.inp > file.out\r\n<\/pre>\n
Submit the job using sbatch jobscript<\/em><\/code> where jobscript<\/em><\/code> is the name of your jobscript.<\/p>\n
Automatic Settings<\/h3>\n
Note that the Gaussian Linda module will make automatically a lot of the environment settings required by Gaussian. These include:<\/p>\n
# The Linda modulefile will set these variables automatically. In general, DO NOT SET THESE YOURSELF<\/strong>!\r\nGAUSS_PDEF or GAUSS_CDEF   # to specify the number of cores each Linda worker will use (DO NOT SET YOURSELF)\r\nGAUSS_MDEF                 # to specify the memory to use (default is 5GB per core on HPC Pool nodes)\r\nGAUSS_WDEF                 # the names of the nodes to use for Linda (DO NOT SET YOURSELF)\r\nGAUSS_SDEF                 # the remote process launcher - Slurm's \"srun\" (DO NOT SET YOURSELF)\r\nGAUSS_LFLAGS2              # the linda stacksize (default is \"--LindaOptions -s 20000000\")\r\nGAUSS_LFLAGS               # optional flags, e.g., \"-vv\" for debugging (default is unset<\/em>)\r\n<\/pre>\n
If you want to see what the above variables have been set to, add the following to your jobscript after<\/em> the module load<\/code> line:<\/p>\n
\r\nenv | grep GAUSS_\r\n<\/pre>\n
Optional Settings<\/h3>\n
NOTE: We DO NOT recommend that you change the defaults. But we document them here for completeness.<\/p>\n
It is possible to override the default amount of memory per core used by Gaussian, by setting the following before<\/strong> you load the modulefile:<\/p>\n
\r\nexport GB_PER_CORE=4      # Some number of GB per core. Using too much will crash your job. Default is 5.\r\nmodule load ...\r\n<\/pre>\n
It is also possible to increase the number of Linda workers per node, which defaults to one. Each Linda worker will use all of the cores available to it on the compute nodes. The default of 1 is strongly<\/em> recommended, and increasing this will likely reduce performance. But you can set this before<\/strong> loading the modulefile if you want to experiment:<\/p>\n
\r\n#SBATCH --ntasks-per-node=2   # Some number of Linda workers per node. The default of 1 is usually best.\r\nmodule load ...\r\n<\/pre>\n
Further info<\/h2>\n