ABAQUS

New CSF users: Please see the Restrictions on Use section below – you need to complete some paperwork to access Abaqus on the CSF.

Overview

ABAQUS is a suite of general-purpose, nonlinear finite-element analysis programs. Available versions:

• 2022
• 2021
• 2020
• 2019
• 2018
• 2017
• 6.13-3 This version is not intended for long term use. We will be removing it at the end of July 2020. Please use or upgrade to 2018 ASAP.

You must not run ABAQUS/standard or ABAQUS/explicit or ABAQUS/CAE on the login node.

Downloading .odb files – advice

If you wish to download your results to look at in the Abaqus GUI on your desktop (for example the .odb file) please DO NOT immediately delete the file from your scratch area after downloading it. You may leave the file in scratch until you are happy with the results. A number of users have downloaded, then immediately deleted the file from the CSF. While we do ask that users keep their scratch directories tidy, you should always check that the download actually worked correctly and that the Abaqus GUI can read your downloaded file. Note that files deleted from scratch cannot be recovered – you would have to rerun your simulation.

You should also copy your important results to your CSF home directory once you are happy with those results. The CSF scratch is not to be used for long term storage and is not backed up. Your home directory is.

License tokens

The maximum number of Abaqus license tokens available, in total, to all CSF jobs is now restricted to 450. This is to ensure fair usage of license tokens on all systems across campus.

If the maximum number of tokens is reached on the license server, your CSF job may fail or Abaqus may appear to be running but will in fact be queued in the license server waiting for licenses to become free. This will be reported in the job’s stderr file (jobscript.eNNNNNN). In this case, your job is using up its CSF job runtime (e.g., 7 days) and so it is recommended that you add the following to your jobscript before the line that runs Abaqus (see jobscripts below for more examples):

. $ABAQUS_HOME/liccheck.sh

Note that the above line starts with a ‘dot’, then a space. This script will requeue your job on the CSF if no Abaqus licenses are available.

To ensure license tokens are not exhausted by CSF jobs:

• Do not submit large jobarrays that run Abaqus.
• Do a few scaling runs with different numbers of cores to ensure you use the optimum number for your jobs. Using more cores uses more licenses but does not necessarily make your job go any faster.

Restrictions on use

The following are conditions of our site licence with ABAQUS, Inc:

ABAQUS may be used, on campus only, by any employee or student of the University of Manchester, or other person authorised by the University (additional checks required), for the purpose of the normal business of the University. The software may not be used for research, consultancy or services leading to commercial exploitation of the software, or ANY purpose which leads to it being commercially exploited by the employer of students on industrial placement or part-time courses. You are not permitted to copy or transfer the software or documentation.

To use ABAQUS on the CSF you must be a registered postgraduate or staff user of the CSF and you must complete the Application for Authority to use ABAQUS on the CSF (PDF), make sure both you and your supervisor sign it (in ink), then scan the form and email it to its-ri-team@manchester.ac.uk (or attach it with your response if you’re doing it as a reply to an open related IT ticket).

Once your form has been received you will be added to the abaqus unix group and sent a confirmation email.

Set up procedure

To access abaqus please load the following modulefile by entering

module load apps/binapps/abaqus/2022
         ## Replace abq2018 in the examples on this page with abq2022.

module load apps/binapps/abaqus/2021
         ## Replace abq2018 in the examples on this page with abq2021

module load apps/binapps/abaqus/2020
         ## Replace abq2018 in the examples on this page with abq2020

module load apps/binapps/abaqus/2019
         ## Replace abq2018 in the examples on this page with abq2019

module load apps/binapps/abaqus/2018

module load apps/binapps/abaqus/2017
         ## Replace abq2018 in the examples on this page with abq2017

module load apps/binapps/abaqus/6.13
         ## This version is not intended for long term use. 
         ## We will be removing it at the end of 2019. 
         ## Please use or upgrade to 2018 ASAP
         ## Replace abq2018 in the examples on this page with abq6133

Unlike CSF2 we do not have a generic abq modulefile – this is because of differences in the install structure of 6.13-3 and newer versions.

Each version of Abaqus has a system-wide ABAQUS environment file which provides default settings for things like memory usage, and in most cases these default settings are sufficient.

If you need your own settings then create an environment file in your home or working directory called abaqus_v6.env and ensure it contains only your own customizations, e.g memory. For complex configurations please consult the Abaqus manuals.

The configuration for 2019 is not controlled by a single environment file but instead three files – abaqus_v6.env, custom_v6.env and lnx86_64.env are all involved and should be sufficient. Consult the abaqus manuals for further details.

Abaqus example problems

Abaqus comes with many example problems which can be used to become familiar with Abaqus and with job submission on the CSF. These example problems are described in the Abaqus documentation, and can be obtained using the Abaqus fetch command. For example, after loading the Abaqus module enter the following at the command line to extract the input file for test problem s4d:

abq2018 fetch job=s4d

This will extract the input file s4d.inp to your current directory and this can be used in your first test jobscripts. To run the computation defined by this input file replace input=myabqjob with input=s4d in the commands and script examples given below.

Memory requirements

You must explicitly specify memory requirements, using memory= on the ABAQUS command-line. Abaqus uses this memory setting to control memory usage for:

• all input file processing during Abaqus/Standard and Abaqus/Explicit jobs
• and the analysis phase of Abaqus/Standard jobs

See Section 3.4.1 Managing memory and disk use in Abaqus in the ABAQUS manual for more information. Note: there is no memory management parameter for Abaqus/Explicit analysis phase but it is used during input file processing. Hence all Abaqus jobs on the CSF must specify the memory usage.

Therefore you should estimate how much memory your job is likely to need before you submit your job to the CSF. For example use the ABAQUS datacheck parameter with the command to ABAQUS as described in the manual, e.g. for input file myabqjob.inp use the serial jobscript

#!/bin/bash --login
#$ -cwd

module load apps/binapps/abaqus/2018

abq2018 datacheck job=jobname input=myabqjob interactive
                        #             #
                        #             # Use: input=s4d if using the sample s4d files
                        #
                        # Use: job=s4d if using the sample s4d files

Submit the job using qsub myjobscript.

The output files contain an estimate of minimum and optimal (to minimise i/o) memory requirements. For example, have a look at the jobname.dat (for example the s4d.dat file). It will contain a section such as:

                   M E M O R Y   E S T I M A T E

 PROCESS      FLOATING PT       MINIMUM MEMORY        MEMORY TO
              OPERATIONS           REQUIRED          MINIMIZE I/O
             PER ITERATION         (MBYTES)           (MBYTES)

     1          5.88E+11              732               3161

Here we are told the minimum memory we should run with is 732MB and the optimum amount to run with (reduce disk I/O) is 3161MB (i.e., approx 3GB). Given that the Intel cores each have 4GB on the CSF, specifying 4GB would be suitable.

Once you have an estimate of memory, choose an appropriate parallel environment and number of cores to give the desired amount of memory, as described below. Note: it is recommended you chose the number of cores such that you have more than the optimal memory size reported by the datacheck procedure.

Note: when you run your full ABAQUS analysis, the output will also include memory estimates. If your job has a problem look to see if these estimates are higher than the memory available to your job. You may need to adjust your jobscript to run on more cores to give Abaqus more memory.

If unsure how to proceed regarding memory, please email the helpdesk for advice.

Disk space requirements – use scratch (not home)

Abaqus can produce large files, often 10s of GBs in size. Please DO NOT run abaqus jobs from your home directory, there is not enough disk space. Please use the scratch filesystem instead.

Scratch is a separate filesystem designed especially for running jobs in and has a lot more space available. Every user has a symlink from their home directory to their scratch directory – $HOME/scratch. You can access the directory simply by typing:

cd scratch            # If not currently in your 'home' folder, use cd ~/scratch

and then creating directories for your jobs, putting all your related files in that directory and then submitting you job from there. For example:

1. cd scratch
2. mkdir my_abaqus_job
3. cd my_abaqus_job
4. gedit jobscript.txt
5. Upload your files to this directory or create files in it as appropriate.
6. qsub jobscript.txt

Please see the documentation below for detailed jobscripts and submission examples.

To work out how much space you are currently using (on any filesystem):

du -sh

which will tell you the total size of the directory you are in and all the directories below it.

To see a file size:

ls -lh filename

where filename is the name of the file you are interested in.

Running the application

You must not run ABAQUS/standard or ABAQUS/explicit or ABAQUS/CAE on the login node.

Executable name

All of the notes and examples below use version 2018 and hence run the Abaqus executable abq2018. If you are using a newer version (e.g., 2021) please change the executable name in your jobscript appropriately (e.g., abq2021).

Double precision

If you wish to run using double precision you will need to add double=both to your jobscript command line. Note that both can be replaced with constraint or explicit or off. Care may be needed when doing this in conjunction with user sub-routines. Please see the official abaqus documentation for full details.

Where to run your job

The choice of parallel environment (PE) in your jobscript will determine where it runs. This is the line in the jobscript:

#$ -pe name.pe N

where name.pe is one of the parallel environment names set up on the CSF and N is the number of cores to use for your job.

See the examples below for the most common uses of parallel environments for Abaqus.

Serial batch job submission

• Create a batch job submission script/text file using gedit, for example abqtest1.txt as follows:

#!/bin/bash --login
#$ -cwd

# Load the software
module load apps/binapps/abaqus/2018

# Ensure licenses are available. If not, job will automatically requeue.
. $ABAQUS_HOME/liccheck.sh

abq2018 job=myabqjob input=myabqjob scratch=$HOME/scratch memory="4 gb" interactive
                                                                   #
# 1 core on an Intel node has access to 4GB. Hence memory="4 gb"   #

Note: The above goes IN a file, do not simply type it into the CSF command line as it will start a job on the login node and that can cause serious system issues.

Submit the above jobscript using

qsub jobscript

where jobscript is the name of your text file.

• Note the request for 4GB of memory. Jobs on the CSF run on CPU cores that share memory, therefore it is important you don’t ask for more memory per core than is available for your job.
  • By default serial jobs run on the Intel compute nodes with 12 cores sharing 48GB, i.e. 4GB of RAM per core. Since a serial job runs on 1 core, you should normally ask for 4GB memory.
  • If you need more memory per core, try options such as -l mem256 (described here) and adjust the memory request.
  • If you need more memory, also consider running on more than one core (see the sections on Parallel batch job submission below).
• You must include the word interactive at the end of your abaqus line – this ensures that it runs in SGE, without it your job will fail.
• To submit the job: qsub jobscript (replacing ‘jobscript’ with the name of your script)

Parallel batch job submission

• Please be sure that your job will run on/is designed for more than one core/CPU. If you are unsure then we recommend running serial jobs.
• Note that Abaqus parallel memory limits are per compute node not per job. Hence you only need to specify the maximum memory to be used by one compute node even if you are going to be using several nodes to provide you with the total number of cores requested. See below for an example.

Parallel batch job submission example (single node, maximum 32 cores)

If the number of cores fits on a single compute node, create a batch job submission script, using gedit, requesting the shared memory smp parallel environment. For example:

#!/bin/bash --login
#$ -cwd               # Run from current directory
#$ -pe smp.pe 4       # Use 4 cores on a single node (max 32 in smp.pe)

# Load the software
module load apps/binapps/abaqus/2018

# Ensure licenses are available. If not, job will automatically requeue.
. $ABAQUS_HOME/liccheck.sh

abq2018 job=myabqjob input=myabqjob cpus=$NSLOTS scratch=$HOME/scratch memory="16 gb" interactive
                                                                                 #
#   4 cores on an Intel node. Each core has access to 4GB hence 4 x 4GB = 16GB.  #

Note: The above goes IN a file, do not simply type it into the CSF command line as it will start a job on the login node and that can cause serious system issues.

• You must include the word interactive at the end of your abaqus line – this ensures that it runs in SGE, without it your job will fail.
• Note the memory request, this job runs on 4 cores of an Intel node with 4GB memory per core, i.e. 4 x 4GB = 16GB memory in total. Change this to match the number of cores you request.
• You must include cpus=$NSLOTS to ensure the job uses the number of cores requested on the pe line.
• To submit the job: qsub jobscript (replacing jobscript with the name of your script)

It is highly recommended that you plot the time taken for jobs to complete versus number of cores for your typical Abaqus jobs, to identify the optimal number of cores to use for your work.

Parallel batch job submission example (multi-node, 48 cores or more in multiples of 24 up to 120 max)

The following example uses two nodes to provide 48 cores.

Please note – this is important for your job!

• The memory request is the maximum to be used by a single node individually, not the total of the two nodes.

The memory request is per-node, not per-job. In this example the 24-core nodes requested in the #$ -pe line have 128GB RAM each but assuming 120GB usable (5GB per core) is sensible. The job will effectively use 256GB (or more sensibly 240GB RAM) in total across the two nodes but we do not specify that total on the abaqus command line. We only specify 128 gb (or more sensibly 120GB RAM) because each single node will use that amount.

#!/bin/bash --login
#$ -cwd                  # Run from current directory
#$ -pe mpi-24-ib.pe 48   # 48 cores = two Intel 24-core nodes (InfiniBand network)

# Load the software
module load apps/binapps/abaqus/2018

# Ensure licenses are available. If not, job will automatically requeue.
. $ABAQUS_HOME/liccheck.sh

abq2018 job=myabqjob input=myabqjob cpus=$NSLOTS scratch=$HOME/scratch memory="128 gb" interactive
                                                                                 #
#  48 cores = two 24-core nodes. Each node has 128GB ram. Hence memory="128 gb"  #
#  It will be more sensible to specify 120GB so as to not run out of memory.     #

User Defined Sub-Routines

Three methods for using user sub-routines are given below. Please note that you will need to load a compiler, the recommended ones are also below.

Recommended compilers

For Abaqus 2018 the Intel Fortran Composer XE 16 Update 1 is recommended. We have update 7 on the CSF. Please try:

module load apps/binapps/abaqus/2018 compilers/intel/17.0.7

Method 1: Compile and run in batch

You need to load a compiler (see above) and add the user option with the name of your code (in this case subroutine.f) to your abq6133 line.

Example job submission script:

#!/bin/bash --login
#$ -cwd               # Run from current directory

# Load the software and Intel compiler suite
module load apps/binapps/abaqus/2018 compilers/intel/17.0.7

# Ensure licenses are available. If not, job will automatically requeue.
. $ABAQUS_HOME/liccheck.sh

abq2018 job=myabqjob input=myabqjob user=subroutine.f scratch=$HOME/scratch interactive

Note: The above goes IN a file, do not simply type it into the CSF command line as it will start a job on the login node and that can cause serious system issues.

Compile using abaqus and then run in batch

Load a compiler (see above). If the subroutines are in a file called subroutine.f, use the following command at the prompt on the login node:

abq2018 make library=subroutines.f

This will create a shared library called libstandardU.so and an object file called subroutine.o . You can then submit your job (ensuring that both of new files are in the job directory). Example submision script:

#!/bin/bash --login
#$ -cwd               # Run from current directory

# Load the software and Intel compiler suite
module load apps/binapps/abaqus/2018 compilers/intel/17.0.7

# Ensure licenses are available. If not, job will automatically requeue.
. $ABAQUS_HOME/liccheck.sh

abq2018 job=myabqjob input=myabqjob user=subroutine.o  scratch=$HOME/scratch interactive

Note: On the abaqus command line we only reference subroutine.o. Abaqus will automatically find the libstandardU.so

Precompiled user sub-routine

Load the appropriate compiler, ensure you have the libstandardU.so and subroutine.o in your job directory and then submit your job. Example submision script:

#!/bin/bash --login
#$ -cwd               # Run from current directory

# Load the software and Intel compiler suite
module load apps/binapps/abaqus/2018 compilers/intel/17.0.7

# Ensure licenses are available. If not, job will automatically requeue.
. $ABAQUS_HOME/liccheck.sh

abq2018 job=myabqjob input=myabqjob user=subroutine.o scratch=$HOME/scratch interactive

Note: On the abaqus command line we only reference subroutine.o. Abaqus will automatically find the libstandardU.so

Abaqus Restart – running jobs for more than 7 days

Nov 2018 – These are provisional instructions and may need some correcting

A job can run for up to 7 days. It is possible to do checkpointing in abaqus whereby you can restart jobs that run out of time. The relevant section of the abaqus analysis manual section 9.1.1.

The following examples gives some idea of how it works:

Ensure that your input file (e.g., mysim.inp) has requested restart information be written:

• For Abaqus/Standard:

  *RESTART, WRITE, OVERLAY
  

• For Abaqus/Explicit:

  *RESTART, WRITE, OVERLAY, NUMBER INTERVAL=n
  

  where n is the numerber of time intervals a step is divided into (default is 1 if this keyword is not given).
  The OVERLAY keyword means that previous steps’ restart information is overwritten by the current step’s restart information each time Abaqus saves data to the restart file. This significantly reduces the size of a restart file. Without the OVERLAY keyword the restart files can be enormous.

Other keywords can be used to reduce the frequency of writes (the default is to write restart information each time step). Writing restart files will slow down your simulation. Please read the Abaqus online manual (section 9.1.1 – see link above) for more information on the RESTART keyword options.

To restart a simulation:

For Abaqus/Standard: If the last complete information for an increment in the result file is from Step2, increment 4. The following two line input file (e.g., mynewsim.inp) could be used to restart Abaqus job and continue writing the restart file (.res):

*HEADING
*RESTART, READ, STEP=2, INC=4, WRITE

You would then add the following flags to your abaqus command-line in your jobscript (see earlier for how to write a correct jobscript):

abq2018 job=mynewsim oldjob=mysim cpus=$NSLOTS scratch=$HOME/scratch memory="48 gb" interactive
    #                                                                        #  
    #                                                                        # Change to whatever
    #                                                                        # amount you used
    #                                                                        # in your original job.
    # Change to whatever version you use

For Abaqus/Explicit: Suppose a one step job stopped prior to completion because a CPU time limit was exceeded and the you have decide that a second step should be added with new boundary Condition definitions. The following input file could be used to Restart this job, complete the remaining part of Step 1, and complete Step 2

*HEADING
*RESTART, READ, STEP=1
**
** This define Step 2
**
*STEP
*DYNAMIC, EXPLICIT
  , .003
*BOUNDARY, OP=NEW

*END STEP

Please note that Restart files can be very large for very large models or for jobs involving many Restart increments. The best solution is to use a command called OVERLAY as an extra parameter on the *RESTART line. In this command the previous Restart file is NOT copied to the beginning of a new Restart file when the job restarted (only the data at a restart increments requested in the current run are saved to the new Restart file (please see Section 9.1.1).

Abaqus error: number of cpus exceed the number of cpus available

If your job output contains the following:

Abaqus Error: The number of cpus (72) exceeds the number of cpus available (12).
Abaqus/Analysis exited with error(s).

it may indicate that you are trying to run a procedure that is not supported across multiple cores or multiple nodes. In this instance please try running on a single core (no #$ -pe line in your job script) or in smp.pe on a maximum of 32 cores.

Section 3.5.2 of the Abaqus manual for Abaqus/Standard Parallel execution says

“The direct sparse solver cannot be used on multiple compute nodes of a computer cluster if:

• the analysis also includes an eigenvalue extraction procedure, or
• the analysis requires features for which MPI-based parallel execution of element operations is not supported.

In addition, the direct sparse solver cannot be used on multiple nodes of a computer cluster for analyses that include any of the following:

• multiple load cases with changing boundary conditions (“Multiple load case analysis,” Section 6.1.4), and
• the quasi-Newton nonlinear solution technique (“Convergence criteria for nonlinear problems,” Section 7.2.3).”

ABAQUS CAE (the Abaqus GUI)

This method of using abaqus is not well suited to running jobs. It is best for pre-processing (e.g., setting up your mesh/model and .inp file for use in a batch job) and post-processing work (e.g., visualizing the result of a simulation). Please submit all long or resource intensive simulation jobs to the batch system as detailed above.

To display the Abaqus GUI on your desktop you’ll need an XServer to be running on your local PC/laptop:

• if you use windows we recommend you log in to the CSF using MobaXterm as this has an XServer built in
• if you use a mac/MacOSX install XQuartz on MacOSX (XQuartz is a free XServer for Mac) and then enable X with a -X on your ssh command line
• if you use linux this has a built in X server which you can enable on your ssh command with -X

If you need to use Abaqus CAE on the CSF a lot (not recommended as CSF is predominantly for batch work, input files can be created on your local install and uploaded) an alternative is to use the Virtual Desktop Service to access the CSF via a remote Linux desktop which runs our central SSH/X2GO gateways. This method is recommended if working at home – it will give you better performance, making Abaqus feel more responsive than if you access the CSF directly with GlobalProtect running.

You must use the qrsh facility to run Abaqus CAE (the Abaqus GUI) on a compute node. This will log you in to a back-end compute node, which have more CPUs and memory than the login nodes. However, please note: the maximum time allocated for interactive use is 1 hour.

An example of connecting interactively to a compute node, load abaqus and run CAE is given below:

# Run the qrsh command at the CSF login node prompt:
[mabcxyz1@hlogin1 [csf3] ~]$ qrsh -l short

    # Wait until you are logged in to a compute node (e.g., node407).

# Now on the compute node, run:
module load apps/binapps/abaqus/2019
abq2019 cae -mesa

# When finished with the Abaqus GUI, quit Abaqus then log out of the compute node with the command:
exit

[mabcxyz1@hlogin1 [csf3] ~]$ you are now back at the login node

This will perform software rendering which is very slow but will at least allow Abaqus to display the GUI correctly. You will get better performance if using the Virtual Desktop Service to access the CSF when working from home.

If you get a message saying that your job could not be scheduled then the short nodes may be in full use (there are only two of them). You will need to try again later.

If the Abaqus GUI appears washed out, as is known to occur in Ubuntu 12.04, enter the following

export XLIB_SKIP_ARGB_VISUALS=1

in the CSF terminal window before using qrsh.

Further info

• IT Services Abaqus Information
• The Abaqus online documentation is provided online and in order to use it you need to first create an account using using your @manchester.ac.uk email address.
• The University of Manchester finite-elements mailing list has been created to communicate information relating to finite elements, including Abaqus. To subscribe send an email from your University of Manchester email account to listserv@listserv.manchester.ac.uk containing nothing except subscribe finite-elements in the main body of the email.