{"id":1341,"date":"2024-10-04T18:39:46","date_gmt":"2024-10-04T17:39:46","guid":{"rendered":"https:\/\/ri.itservices.manchester.ac.uk\/csf4\/?page_id=1341"},"modified":"2025-01-21T08:31:46","modified_gmt":"2025-01-21T08:31:46","slug":"gaussian16-linda","status":"publish","type":"page","link":"https:\/\/ri.itservices.manchester.ac.uk\/csf4\/software\/applications\/gaussian16-linda\/","title":{"rendered":"Gaussian16 Linda (multinode)"},"content":{"rendered":"

Gaussian<\/a> is a general purpose suite of electronic structure programs. The “Linda” version adds multi-node (larger parallel jobs) capability to Gaussian.<\/p>\n

Versions g16c01 Linda is installed on the CSF. It is available as binaries only. The source code is not available on the CSF.<\/p>\n

Gaussian 16 (non-Linda) is also available on the CSF<\/a>.<\/p>\n

Restrictions on use<\/h2>\n
Gaussian Linda is only available to members for Prof. Kaltsoyannis’s group. All requests to access this version must be approved by Prof. Kaltsoyannis.<\/div>\n

Please contact us via our help form<\/a> to request access to Gaussian 16 Linda.<\/p>\n

Set up procedure<\/h2>\n

There is only one version available on the CSF – optimized for Intel “Haswell” architectures or equivalently any CPU that provides AVX2 vector extensions. This includes all nodes in the CSF.<\/p>\n

The detectcpu<\/code> modulefile must be loaded inside your jobscript, not<\/em> on the login node.<\/p>\n

After being added to the relevant unix group, you will be able to access the executables by loading the modulefile:<\/p>\n

module load gaussian\/g16c01_em64t_detectcpu_linda\r\n<\/pre>\n
Gaussian MUST ONLY<\/strong> be run in batch. Please DO NOT<\/strong> run g16<\/code> on the login nodes. Computational work found to be running on the login nodes will be killed WITHOUT WARNING<\/strong>.<\/p>\n
Gaussian Scratch<\/h2>\n
Gaussian uses an environment variable $GAUSS_SCRDIR<\/code> to specify a directory path for where to write scratch<\/em> (temporary) files (two-electron integral files, integral derivative files and a read-write<\/em> file for temporary workings). It is set to your scratch<\/em> directory (~\/scratch<\/code>) when you load the modulefile. This is a Lustre filesystem which provides good I\/O performance.<\/p>\n
Do not<\/strong> use your home directory for Gaussian scratch files – the files can be huge making the home area at risk of going over quota. We also recommend using a directory-per-job in your scratch area. See below for how to do this.<\/p>\n
A faster, but smaller, local \/tmp<\/code> on each compute node is available should users prefer to use that. It can be more efficient if you have a need to create lots of small files, but space is limited. The minimum \/tmp<\/code> on intel compute nodes is 800GB, the largest is 3.5TB.<\/p>\n
Gaussian should<\/em> delete scratch files automatically when a job completes successfully or dies cleanly. However, it often fails to do this. Scratch files are also not<\/em> deleted when a job is killed externally or terminates abnormally so that you can use the scratch files to restart the job (if possible). Consequently, leftover files may accumulate in the scratch directory, and it is your responsibility to delete these files. Please check periodically<\/strong> whether you have a lot of temporary Gaussian files that can be deleted.<\/p>\n
Using a Scratch Directory per Job<\/h3>\n
We now recommend using a different scratch directory for each job. This improves file access times if you run many jobs – writing 1000s of scratch files to a single directory can slow down your jobs. It is much better to create a directory for each<\/em> job within your scratch<\/code> area. It is also then easy to delete the entire directory if Gaussian has left unwanted scratch files behind.<\/p>\n
The example jobscripts below show how to use this method (it is simple – just two extra lines in your jobscript).<\/p>\n
Very large Gaussian scratch files<\/h3>\n
Occasionally some jobs create .rwf<\/code> files which are very large (several TB). The batch system will not permit a job to create files bigger than 4TB. If your gaussian job fails and the .rwf<\/code> file is 4TB then it may be that this limit has prevented your job from completing. You should re-run the job and in your input file request that the .rwf<\/code> file be split into multiple files. For example to split the file into two 3TB files:<\/p>\n
%rwf=\/scratch\/$USER\/myjob\/one.rwf,3000GB,\/scratch\/$USER\/myjob\/two.rwf,3000GB\r\n<\/pre>\n
Number of Cores<\/h2>\n
\n
If your Gaussian input file contains the number of cores – e.g.:<\/p>\n
%NProcsShared\r\n%nprocs\r\n<\/pre>\n
then you must remove this line from your input file<\/strong>.<\/p>\n
The Linda modulefile will work out the correct number of cores to use, based on your jobscript settings.<\/p>\n<\/div>\n
Parallel batch job<\/h2>\n
You must<\/em> submit your job to the multinode<\/code> partition on the CSF so that it can run as a multi-node job. Our installation will use the fast InfiniBand network for Linda communication. Submitting to the multicore<\/code> partition (to run on a single compute node) will not work<\/strong>.<\/p>\n
#!\/bin\/bash --login\r\n#SBATCH -p multinode             # (or --partition) A multinode job\r\n#SBATCH --nodes=3                # (or -N 3) Number of compute nodes\r\n#SBATCH --ntasks-per-node=1      # Number of Linda workers per compute node\r\n#SBATCH --cpus-per-task=40       # (or -c 40) Number of cores each<\/em> Linda worker uses\r\n\r\n## Set up scratch dir (please do this!)\r\nexport GAUSS_SCRDIR=\/scratch\/$USER\/gau_temp_$JOB_ID\r\nmkdir -p $GAUSS_SCRDIR\r\n\r\n# Load g16 linda \r\nmodule load gaussian\/g16c01_em64t_detectcpu_linda\r\n\r\n$g16root\/g16\/g16 < file.inp > file.out\r\n<\/pre>\n
Submit the job using sbatch jobscript<\/em><\/code> where jobscript<\/em><\/code> is the name of your jobscript.<\/p>\n
Automatic Settings<\/h3>\n
The Gaussian Linda module will make automatically a lot of the environment settings required by Gaussian. These include:<\/p>\n
# The Linda modulefile will set these variables automatically. In general, DO NOT SET THESE YOURSLEF!\r\nGAUSS_PDEF or GAUSS_CDEF   # to specify the number of cores (DO NOT SET YOURSELF)\r\nGAUSS_MDEF                 # to specify the memory to use (default is 3.5GB per core)\r\nGAUSS_WDEF                 # the names of the nodes to use for Linda (DO NOT SET YOURSELF)\r\nGAUSS_SDEF                 # the remote process launcher (DO NOT SET YOURSELF)\r\nGAUSS_LFLAGS2              # the linda stacksize (default is \"--LindaOptions -s 20000000\")\r\nGAUSS_LFLAGS               # optional flags, e.g., -vv for debugging (default is unset<\/em>)\r\n<\/pre>\n
Optional Settings<\/h3>\n
NOTE: We DO NOT recommend that you change the defaults. But we document them here for completeness.<\/p>\n
It is possible to override the default amount of memory per core used by Gaussian, by setting the following before<\/strong> you load the modulefile:<\/p>\n
export GB_PER_CORE=4      # Some number of GB per core. Using too much will crash your job.\r\nmodule load ...\r\n<\/pre>\n
It is also possible to increase the number of Linda workers per node, which defaults to one. Each Linda worker will use all of the cores available to it on the compute nodes. The default of 1 is strongly<\/em> recommended, and increasing this will likely reduce performance. But you can change this before<\/strong> in your jobscript options:<\/p>\n
#SBATCH --ntasks-per-node=2    # Number of Linda workers per node (1 is recommended)\r\n#SBATCH --cpus-per-task=20     # Adjust so that product is 40 (2x20=40)\r\nmodule load ...\r\n<\/pre>\n
Further info<\/h2>\n