Abaqus

Overview

ABAQUS is a suite of general-purpose, nonlinear finite-element analysis programs. Available versions:

• 2022
• 2021
• 2020
• 2018

You must not run ABAQUS/standard or ABAQUS/explicit or ABAQUS/CAE on the login node.

Restrictions on use

The following are conditions of our site licence with ABAQUS, Inc:

ABAQUS may be used, on campus only, by any employee or student of the University of Manchester, or other person authorised by the University (additional checks required), for the purpose of the normal business of the University. The software may not be used for research, consultancy or services leading to commercial exploitation of the software, or ANY purpose which leads to it being commercially exploited by the employer of students on industrial placement or part-time courses. You are not permitted to copy or transfer the software or documentation.

To use ABAQUS on the CSF you must be a registered postgraduate or staff user of the CSF and you must complete the Application for Authority to use ABAQUS on the CSF (PDF), make sure both you and your supervisor sign it (in ink), then scan the form and email it to its-ri-team@manchester.ac.uk (or attach it with your response if replying to an open helpdesk ticket).

Once your form has been received you will be added to the abaqus unix group and sent a confirmation email.

General Advice for Abaqus Users

Downloading .odb files – advice

If you wish to download your results to look at in the Abaqus GUI on your desktop (for example the .odb file) please DO NOT immediately delete the file from your scratch area after downloading it. You may leave the file in scratch until your are happy with the results. A number of users have downloaded, then immediately deleted the file from the CSF, before they had checked the download completed successfully. While we do ask that users keep their scratch directories tidy, you should always check that the download actually worked correctly and that the Abaqus GUI can read your downloaded file. Note that files deleted from scratch cannot be recovered – you would have to rerun your simulation.

Keeping your important results safe

You should also copy your important results to your CSF home directory, or additional Research Data Storage visible on the CSF, once you are happy with those results. The CSF scratch is not to be used for long term storage and is not backed up. Your home or RDS directory is.

License tokens

The maximum number of Abaqus license tokens available, in total, to all CSF jobs is now restricted to 450. This is to ensure fair usage of license tokens on all systems across campus.

If the maximum number of tokens is reached on the license server, your CSF job may fail or Abaqus may appear to be running but will in fact be queued in the license server waiting for licenses to become free. This will be reported in the job’s stderr file (slurm-NNNNN.out or jobscript.eNNNNNN). In this case, your job is using up its CSF job runtime (e.g., 7 days) and so it is recommended that you add the following to your jobscript before the line that runs Abaqus (see jobscripts below for more examples):

. $ABAQUS_HOME/liccheck.sh

Note that the above line starts with a ‘dot’, then a space. This script will requeue your job on the CSF if no Abaqus licenses are available.

To ensure license tokens are not exhausted by CSF jobs:

• Submit only a few jobs, wait for them to finish then submit further jobs.
• Do not submit jobarrays that run Abaqus.
• Do a few scaling runs with different numbers of cores to ensure you use the optimum number for your jobs. Using more cores uses more licenses but does not necessarily make your job go any faster.

Set up procedure

To access abaqus, please load one of the following modulefiles in your jobscript, using:

module load abaqus/2022
    ## Replace abq2020 in the jobscript examples on this page with abq2022
module load abaqus/2021
    ## Replace abq2020 in the jobscript examples on this page with abq2021
module load abaqus/2020
    ## Jobscript examples on this page all use this version. Change as required.
module load abaqus/2018
    ## Replace abq2020 in the jobscript examples on this page with abq2018

Abaqus example problems

Abaqus comes with many example problems which can be used to become familiar with Abaqus and with job submission on the CSF. These example problems are described in the Abaqus documentation, and can be obtained using the Abaqus fetch command. For example, after loading the Abaqus module enter the following at the command line to extract the input file for test problem s4d:

abq2020 fetch job=s4d

This will extract the input file s4d.inp to your current directory and this can be used in your first test jobscripts. To run the computation defined by this input file replace input=myabqjob with input=s4d in the commands and script examples given below.

Memory requirements

You must explicitly specify memory requirements, using memory= on the ABAQUS command-line. Abaqus uses this memory setting to control memory usage for:

• all input file processing during Abaqus/Standard and Abaqus/Explicit jobs
• and the analysis phase of Abaqus/Standard jobs

See Section 3.4.1 Managing memory and disk use in Abaqus in the ABAQUS manual for more information. Note: there is no memory management parameter for Abaqus/Explicit analysis phase but it is used during input file processing. Hence all Abaqus jobs on the CSF must specify the memory usage.

Therefore you should estimate how much memory your job is likely to need before you submit your job to the CSF. For example use the ABAQUS datacheck parameter with the command to ABAQUS as described in the manual, e.g. for input file myabqjob.inp use the serial jobscript

#!/bin/bash --login
#SBATCH -n 1          # (--ntasks=) 1 core

module load abaqus/2020

abq2020 datacheck job=jobname input=myabqjob interactive
                        #             #
                        #             # Use: input=s4d if using the sample s4d files
                        #
                        # Use: job=s4d if using the sample s4d files

Submit the job using sbatch myjobscript.

The output files contain an estimate of minimum and optimal (to minimise i/o) memory requirements. For example, have a look at the jobname.dat (for example the s4d.dat file). It will contain a section such as:

                   M E M O R Y   E S T I M A T E

 PROCESS      FLOATING PT       MINIMUM MEMORY        MEMORY TO
              OPERATIONS           REQUIRED          MINIMIZE I/O
             PER ITERATION         (MBYTES)           (MBYTES)

     1          5.88E+11              732               3161

Here we are told the minimum memory we should run with is 732MB and the optimum amount to run with (reduce disk I/O) is 3161MB (i.e., approx 3GB). Given that the Intel cores each have 4GB on the CSF, specifying 4GB would be suitable.

Once you have an estimate of memory, choose an appropriate parallel environment and number of cores to give the desired amount of memory, as described below. Note: it is recommended you chose the number of cores such that you have more than the optimal memory size reported by the datacheck procedure.

Note: when you run your full ABAQUS analysis, the output will also include memory estimates. If your job has a problem look to see if these estimates are higher than the memory available to your job. You may need to adjust your jobscript to run on more cores to give Abaqus more memory.

If unsure how to proceed regarding memory, please email the helpdesk for advice.

Disk space requirements – use scratch (not home)

Abaqus can produce large files, often 10s of GBs in size. Please DO NOT run abaqus jobs from your home directory, there is not enough disk space. Please use the scratch filesystem instead.

Scratch is a separate filesystem designed especially for running jobs in and has a lot more space available. Every user has a symlink from their home directory to their scratch directory – $HOME/scratch. You can access the directory simply by typing:

cd scratch            # If not currently in your 'home' folder, use cd ~/scratch

and then creating directories for your jobs, putting all your related files in that directory and then submitting you job from there. For example:

1. cd scratch
2. mkdir my_abaqus_job
3. cd my_abaqus_job
4. gedit jobscript.txt
5. Upload your files to this directory or create files in it as appropriate.
6. sbatch jobscript.txt

Please see the documentation below for detailed jobscripts and submission examples.

To work out how much space you are currently using (on any filesystem):

du -sh

which will tell you the total size of the directory you are in and all the directories below it.

To see a file size:

ls -lh filename

where filename is the name of the file you are interested in.

Running the application

You must not run ABAQUS/standard or ABAQUS/explicit or ABAQUS/CAE on the login node.

Executable name

All of the notes and examples below use version 2020 and hence run the Abaqus executable abq2020.

Double precision

If you wish to run using double precision you will need to add double=both to your jobscript command line. Note that both can be replaced with constraint or explicit or off. Care may be needed when doing this in conjunction with user sub-routines. Please see the official abaqus documentation for full details.

Serial batch job submission

• Create a batch job submission script/text file using gedit, for example abqtest1.txt as follows:

#!/bin/bash --login
#SBATCH -n 1          # (--ntasks=) 1 core

# Load the software
module load abaqus/2020

# Ensure licenses are available. If not, job will automatically requeue.
. $ABAQUS_HOME/liccheck.sh

abq2020 job=myabqjob input=myabqjob scratch=$HOME/scratch memory="4 gb" interactive
                                                                   #
# 1 core on an Intel node has access to 4GB. Hence memory="4 gb"   #

Note: The above goes IN a file, do not simply type it into the CSF command line as it will start a job on the login node and that can cause serious system issues.

Submit the above jobscript using

sbatch jobscript

where jobscript is the name of your text file.

• Note the request for 4GB of memory. Jobs on the CSF run on CPU cores that share memory, therefore it is important you don’t ask for more memory per core than is available for your job.
• If you need more memory, also consider running on more than one core (see the sections on Parallel batch job submission below).
• You must include the word interactive at the end of your abaqus line – this ensures that it runs in SLURM, without it your job will fail.
• To submit the job: sbatch jobscript (replacing ‘jobscript’ with the name of your script)

Parallel batch job submission

• Please be sure that your job will run on/is designed for more than one core/CPU. If you are unsure then we recommend running serial jobs.
• Note that Abaqus parallel memory limits are per compute node not per job. Hence you only need to specify the maximum memory to be used by one compute node even if you are going to be using several nodes to provide you with the total number of cores requested. See below for an example.

Parallel batch job submission example (single node, maximum 40 cores)

If the number of cores fits on a single compute node, create a batch job submission script, using gedit, requesting the shared memory smp parallel environment. For example:

#!/bin/bash --login
#SBATCH -p multicore # (or --partition=) One compute node will be used
#SBATCH -n 4         # (or --ntasks=) Use 4 cores on a single node (can be 2 to 40) 

# Load the software 
module load abaqus/2020 

# Ensure licenses are available. If not, job will automatically requeue. 

. $ABAQUS_HOME/liccheck.sh abq2020 job=myabqjob input=myabqjob cpus=$SLURM_NTASKS scratch=$HOME/scratch memory="16 gb" interactive
                                                                                                    #
                                        # 4 cores on an Intel node. Each core has access to 4GB hence 4 x 4GB = 16GB. #

Note: The above goes IN a file, do not simply type it into the CSF command line as it will start a job on the login node and that can cause serious system issues.

• You must include the word interactive at the end of your abaqus line – this ensures that it runs in SLURM, without it your job will fail.
• Note the memory request, this job runs on 4 cores of an Intel node with 4GB memory per core, i.e. 4 x 4GB = 16GB memory in total. Change this to match the number of cores you request.
• You must include cpus=$SLURM_NTASKS to ensure the job uses the number of cores requested on the ntasks line.
• To submit the job: sbatch jobscript (replacing jobscript with the name of your script)

It is highly recommended that you plot the time taken for jobs to complete versus number of cores for your typical Abaqus jobs, to identify the optimal number of cores to use for your work.

Parallel batch job submission example (multi-node, 80 cores or more in multiples of 40 up to 200 max)

The following example uses two nodes to provide 48 cores.

Please note – this is important for your job!

• The memory request is the maximum to be used by a single node individually, not the total of the two nodes.

The memory request is per-node, not per-job. In this example the 40-core nodes each have 160GB RAM available (for jobs). The job will effectively use 2 x 160GB = 320GB in total across the two nodes but we do not specify that total on the abaqus command line. We only specify 160 gb because each single node will use that amount.

#!/bin/bash --login
#SBATCH -p multinode # (or --partition=) Two or more compute nodes will be used
#SBATCH -n 80        # (or --ntasks=) 80 cores 

# Load the software
module load abaqus/2020

# Ensure licenses are available. If not, job will automatically requeue.
. $ABAQUS_HOME/liccheck.sh

# Dynamically create and populate abaqus_v6.env environmnet file with list of hosts. Abaqus needs to know this in order to run across multiple nodes. 
create_abaqus_host_list.sh

#  Clear SLURM_GTIDS variable, failure to do so will cause Abaqus to get stuck due to the MPI that Abaqus ships with not supporting the SLURM scheduler
unset SLURM_GTIDS

abq2020 job=myabqjob input=myabqjob cpus=$SLURM_NTASKS scratch=$HOME/scratch memory="160 gb" interactive
                                                                                       #
      #  80 cores = 2 x 40-core nodes. Each node has 160GB RAM. Hence memory="160 gb"  #

User Defined Sub-Routines

Three methods for using user sub-routines are given below. Please note that you will need to load a compiler, the recommended ones are also below.

Recommended compilers

For Abaqus 2020 the Intel Fortran Composer XE 16 Update 1 is recommended. We have a newer version on the CSF but it should be fine to use this version: Please try:

module load abaqus/2020 iccifort/2020.2.254

Method 1: Compile and run in batch

You need to load a compiler (see above) and add the user option with the name of your code (in this case subroutine.f) to your abq6133 line.

Example job submission script:

#!/bin/bash --login
#SBATCH -n 1          # (--ntasks=) 1 core

# Load the software and Intel compiler suite
module load abaqus/2020 iccifort/2020.2.254

# Ensure licenses are available. If not, job will automatically requeue.
. $ABAQUS_HOME/liccheck.sh

abq2020 job=myabqjob input=myabqjob user=subroutine.f scratch=$HOME/scratch interactive

Note: The above goes IN a file, do not simply type it into the CSF command line as it will start a job on the login node and that can cause serious system issues.

Compile using abaqus and then run in batch

Load a compiler (see above). If the subroutines are in a file called subroutine.f, use the following command at the prompt on the login node:

abq2020 make library=subroutines.f

This will create a shared library called libstandardU.so and an object file called subroutine.o . You can then submit your job (ensuring that both of new files are in the job directory). Example submision script:

#!/bin/bash --login
#SBATCH -n 1          # (--ntasks=) 1 core

# Load the software and Intel compiler suite
module load abaqus/2020 iccifort/2020.2.254

# Ensure licenses are available. If not, job will automatically requeue.
. $ABAQUS_HOME/liccheck.sh

abq2020 job=myabqjob input=myabqjob user=subroutine.o  scratch=$HOME/scratch interactive

Note: On the abaqus command line we only reference subroutine.o. Abaqus will automatically find the libstandardU.so

Precompiled user sub-routine

Load the appropriate compiler, ensure you have the libstandardU.so and subroutine.o in your job directory and then submit your job. Example submision script:

#!/bin/bash --login
#SBATCH -n 1          # (--ntasks=) 1 core

# Load the software and Intel compiler suite
module load abaqus/2020 iccifort/2020.2.254

# Ensure licenses are available. If not, job will automatically requeue.
. $ABAQUS_HOME/liccheck.sh

abq2020 job=myabqjob input=myabqjob user=subroutine.o scratch=$HOME/scratch interactive

Note: On the abaqus command line we only reference subroutine.o. Abaqus will automatically find the libstandardU.so

Abaqus Restart – running jobs for more than 7 days

These are provisional instructions and make need some correcting

The maximum length of time a job can run for on the CSF is 7 days. It is possible to do checkpointing in abaqus whereby you can restart jobs that run out of time. The relevant section of the abaqus analysis manual section 9.1.1.

The following examples gives some idea of how it works:

Ensure that your input file (e.g., mysim.inp) has requested restart information be written:

• For Abaqus/Standard:

  *RESTART, WRITE, OVERLAY
  

• For Abaqus/Explicit:

  *RESTART, WRITE, OVERLAY, NUMBER INTERVAL=n
  

  where n is the numerber of time intervals a step is divided into (default is 1 if this keyword is not given).
  The OVERLAY keyword means that previous steps’ restart information is overwritten by the current step’s restart information each time Abaqus saves data to the restart file. This significantly reduces the size of a restart file. Without the OVERLAY keyword the restart files can be enormous.

Other keywords can be used to reduce the frequency of writes (the default is to write restart information each time step). Writing restart files will slow down your simulation. Please read the Abaqus online manual (section 9.1.1 – see link above) for more information on the RESTART keyword options.

To restart a simulation:

For Abaqus/Standard: If the last complete information for an increment in the result file is from Step2, increment 4. The following two line input file (e.g., mynewsim.inp) could be used to restart Abaqus job and continue writing the restart file (.res):

*HEADING
*RESTART, READ, STEP=2, INC=4, WRITE

You would then add the following flags to your abaqus command-line in your jobscript (see earlier for how to write a correct jobscript):

abq2020 job=mynewsim oldjob=mysim cpus=$SLURM_NTASKS scratch=$HOME/scratch memory="48 gb" interactive
    #                                                                              #  
    #                                                                              # Change to whatever
    #                                                                              # amount you used
    #                                                                              # in your original job.
    # Change to whatever version you use

For Abaqus/Explicit: Suppose a one step job stopped prior to completion because a CPU time limit was exceeded and the you have decide that a second step should be added with new boundary Condition definitions. The following input file could be used to Restart this job, complete the remaining part of Step 1, and complete Step 2

*HEADING
*RESTART, READ, STEP=1
**
** This define Step 2
**
*STEP
*DYNAMIC, EXPLICIT
  , .003
*BOUNDARY, OP=NEW

*END STEP

Please note that Restart files can be very large for very large models or for jobs involving many Restart increments. The best solution is to use a command called OVERLAY as an extra parameter on the *RESTART line. In this command the previous Restart file is NOT copied to the beginning of a new Restart file when the job restarted (only the data at a restart increments requested in the current run are saved to the new Restart file (please see Section 9.1.1).

Abaqus error: number of cpus exceed the number of cpus available

If your job output contains the following:

Abaqus Error: The number of cpus (72) exceeds the number of cpus available (12).
Abaqus/Analysis exited with error(s).

it may indicate that you are trying to run a procedure that is not supported across multiple cores or multiple nodes. In this instance please try running on a single core (use #SBATCH -n 1 in your job script) or in #SBATCH -n 40 (2 to 40 cores).

Section 3.5.2 of the Abaqus manual for Abaqus/Standard Parallel execution says

“The direct sparse solver cannot be used on multiple compute nodes of a computer cluster if:

• the analysis also includes an eigenvalue extraction procedure, or
• the analysis requires features for which MPI-based parallel execution of element operations is not supported.

In addition, the direct sparse solver cannot be used on multiple nodes of a computer cluster for analyses that include any of the following:

• multiple load cases with changing boundary conditions (“Multiple load case analysis,” Section 6.1.4), and
• the quasi-Newton nonlinear solution technique (“Convergence criteria for nonlinear problems,” Section 7.2.3).”

ABAQUS CAE (the Abaqus GUI)

This method of using abaqus is not well suited to running jobs. It is best for pre-processing (e.g., setting up your mesh/model and .inp file for use in a batch job) and post-processing work (e.g., visualizing the result of a simulation). Please submit all long or resource intensive simulation jobs to the batch system as detailed above.

To display the Abaqus GUI on your desktop you’ll need an XServer to be running – so log in using MobaXterm on Windows or install XQuartz on MacOSX. Alternatively use the Virtual Desktop Service to access the CSF via a remote Linux desktop which runs our central SSH/X2GO gateways. This method is recommended if working at home – it will give you better performance, making Abaqus feel more responsive than if you access the CSF directly with GlobalProtect running.

You must use the srun facility to run Abaqus CAE (the Abaqus GUI) on a compute node. This will log you in to a back-end compute node, which have more CPUs and memory than the login nodes. However, please note: the maximum time allocated for interactive use is 1 hour.

An example of connecting interactively to a compute node, load abaqus and run CAE is given below:

# Run the srun command at the CSF login node prompt:
[mabcxyz1@login02 [CSF4] ~]$ srun --tty bash

    # Wait until you are logged in to a compute node (e.g., node407).

# Now on the compute node, run:
module load /abaqus/2020
abq2020 cae -mesa

# When finished with the Abaqus GUI, quit Abaqus then log out of the compute node with the command:
exit

[mabcxyz1@login01 [CSF4] ~]$ you are now back at the login node

This will perform software rendering which is very slow but will at least allow Abaqus to display the GUI correctly. You will get better performance if using the Virtual Desktop Service to access the CSF when working from home.

If you get a message saying that your job could not be scheduled then the short nodes may be in full use (there are only two of them). You will need to try again later.

If the Abaqus GUI appears washed out, as is known to occur in Ubuntu 12.04, enter the following

export XLIB_SKIP_ARGB_VISUALS=1

in the CSF terminal window before using srun.

Further info

• IT Services Abaqus Information
• The Abaqus documentation is provided online and in order to use it you need to first create an account at https://iam.3ds.com/service/social/ using your @manchester.ac.uk email address. The URL for the documentation itself is http://50.16.225.63/
• Compiling, Linking and Debugging ABAQUS user subroutines
• Dassault Systèmes have an online version of their 5 day Introduction to Abaqus course notes available to all 3DS Academy members.
• The University of Manchester finite-elements mailing list has been created to communicate information relating to finite elements, including Abaqus. To subscribe send an email from your University of Manchester email account to listserv@listserv.manchester.ac.uk containing nothing except subscribe finite-elements in the main body of the email.