{"id":650,"date":"2018-10-29T15:18:43","date_gmt":"2018-10-29T15:18:43","guid":{"rendered":"http:\/\/ri.itservices.manchester.ac.uk\/csf3\/?page_id=650"},"modified":"2026-01-16T15:36:26","modified_gmt":"2026-01-16T15:36:26","slug":"singularity","status":"publish","type":"page","link":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/software\/applications\/singularity\/","title":{"rendered":"Singularity \/ Apptainer"},"content":{"rendered":"

Overview<\/h2>\n

Singularity<\/a> provides a mechanism to run containers where containers can be used to package entire scientific workflows, software and libraries, and even data.<\/p>\n

Version 1.4.2-1.el9 is installed on the CSF3.<\/p>\n

Please note: Docker is not<\/strong> available on the CSF for security reasons. Instead you should use Singularity or Apptainer. You can convert your Docker images to Singularity \/ Apptainer images.<\/p>\n

Please see below<\/a> for info on using your own containers.\n<\/div>\n

Restrictions on use<\/h2>\n

The software is licensed under the BSD 3-clause “New” or “Revised” License<\/a>.<\/p>\n

Set up procedure<\/h2>\n

Please note:<\/strong> you no longer need to use a modulefile<\/strong>.<\/p>\n

We have installed Apptainer, which can be run using the command apptainer<\/code> or singularity<\/code>, as a system-wide command. So you can simply run the commands you wish to run without loading any modulefiles<\/strong>.<\/p>\n

For example:<\/p>\n

# Ensure you have NO singularity modulefiles loaded ('module purge' will unload all modulefiles)\r\n\r\n[username<\/em>@login2[csf3] ~]$ singularity --version\r\napptainer version 1.4.2-1.el9\r\n  #\r\n  # It is apptainer that is installed on the CSF. Hence the\r\n  # 'singularity' command is an alias for apptainer.\r\n\r\n[username<\/em>@login2[csf3] ~]$ apptainer --version\r\napptainer version 1.4.2-1.el9\r\n<\/pre>\n
Singularity version 3<\/h3>\n
If you need singularity v3, e.g., for Nextflow, please first try using the system-wide version – i.e., do not<\/em> load any singularity modulefiles.<\/p>\n
The system-wide singularity is provided by Apptainer v1.4.2, and this includes many of the features that you may be wanting from singularity v3.<\/p>\n
In short, you will likely be able to do everything you need to do simply by NOT loading any singularity or apptainer modulefiles.<\/p>\n
Running the application<\/h2>\n
Please do not run Apptainer \/ Singularity containers on the login node. Jobs should be submitted to the compute nodes via batch. You may run the command on its own to obtain a list of flags:<\/p>\n
\r\napptainer\r\nUsage:\r\n  apptainer [global options...] \r\n\r\nAvailable Commands:\r\n  build       Build an Apptainer image\r\n  cache       Manage the local cache\r\n  capability  Manage Linux capabilities for users and groups\r\n  checkpoint  Manage container checkpoint state (experimental)\r\n  completion  Generate the autocompletion script for the specified shell\r\n  config      Manage various apptainer configuration (root user only)\r\n  delete      Deletes requested image from the library\r\n  exec        Run a command within a container\r\n  inspect     Show metadata for an image\r\n  instance    Manage containers running as services\r\n  key         Manage OpenPGP keys\r\n  keyserver   Manage apptainer keyservers\r\n  oci         Manage OCI containers\r\n  overlay     Manage an EXT3 writable overlay image\r\n  plugin      Manage Apptainer plugins\r\n  pull        Pull an image from a URI\r\n  push        Upload image to the provided URI\r\n  registry    Manage authentication to OCI\/Docker registries\r\n  remote      Manage apptainer remote endpoints\r\n  run         Run the user-defined default command within a container\r\n  run-help    Show the user-defined help for an image\r\n  search      Search a Container Library for images\r\n  shell       Run a shell within a container\r\n  sif         Manipulate Singularity Image Format (SIF) images\r\n  sign        Add digital signature(s) to an image\r\n  test        Run the user-defined tests within a container\r\n  verify      Verify digital signature(s) within an image\r\n  version     Show the version for Apptainer\r\n\r\nRun 'apptainer --help' for more detailed usage information.\r\n<\/pre>\n
Serial batch job submission<\/h3>\n
Create a batch submission script, for example:<\/p>\n
\r\n#!\/bin\/bash --login \r\n#SBATCH -p serial       # (or --partition=) Run on the nodes dedicated to 1-core jobs\r\n#SBATCH -t 2-0          # Wallclock time limit (2-0 is 2 days, max permitted is 7-0)\r\n\r\n# We'll use the system-wide command, hence no apptainer modulefiles to load\r\nmodule purge\r\n\r\napptainer run mystack.simg\r\n<\/pre>\n
Submit the jobscript using:<\/p>\n
sbatch scriptname<\/em><\/pre>\n
where scriptname<\/em> is the name of your jobscript.<\/p>\n
Parallel batch job submission<\/h3>\n
You should check the Apptainer Documentation<\/a> for how to ensure your Apptainer \/ Singularity container can access the cores available to your job. For example, MPI applications inside a container usually require the apptainer command itself to be run via mpirun<\/code> in the jobscript.<\/p>\n
Create a jobscript similar to the following:<\/p>\n
#!\/bin\/bash --login\r\n#SBATCH -p multicore    # (or --partition=) Run on the AMD 168-core Genoa nodes\r\n#SBATCH -n 8            # (or --ntasks=) Number of cores\r\n#SBATCH -t 2-0          # Wallclock time limit (2-0 is 2 days, max permitted is 7-0)\r\n\r\n# We'll use the system-wide command, hence no apptainer modulefiles to load\r\nmodule purge\r\n\r\n# mpirun knows to run $SLURM_NTASKS processes (which is the-n number above)\r\nmpirun apptainer exec name_of_container<\/em> name_of_app_inside_container<\/em>\r\n<\/pre>\n
Submit the jobscript using:<\/p>\n
sbatch scriptname<\/em><\/pre>\n
where scriptname<\/em> is the name of your jobscript.<\/p>\n
Using your own containers<\/h2>\n
You may want to use your own containers on the CSF3 – that’s fine. You will need to have a \/scratch<\/code> directory (stub) within<\/em> the container to bind to the \/scratch<\/code> directory on the CSF.<\/p>\n
Rebuilding with a scratch mount point<\/h3>\n
If building from an image definition .def<\/code> file, please include the line <\/p>\n
\r\n%post\r\n...\r\nmkdir \/scratch\r\n<\/pre>\n
in the %post<\/code> section.<\/p>\n
If using a prebuilt .sif<\/code> (or .simg<\/code>) container image and you don’t have a .def file available, then follow the steps below to rebuild with a \/scratch<\/code> directory within. It is suggested to do the below steps in scratch, as it is faster than you home directory:<\/p>\n
\r\n# From the image .sif create a sandbox directory which you can edit  \r\napptainer build --sandbox mysandbox myimage.sif\r\n# add an empty \/scratch dir in the sandbox\r\nmkdir mysandbox\/scratch\r\n# build a new image based on the sandbox directory\r\napptainer build myimage-csf3ready.sif mysandbox\r\n<\/pre>\n
Adding scratch and home at runtime<\/h3>\n
If you don’t want to rebuild your image, you can make scratch, home directories and any additional Research Data Storage you may have access to, using the following command when you run apptainer:<\/p>\n
\r\napptainer exec -B \/scratch,\/mnt myimage.sif<\/em>\r\n<\/pre>\n
Converting from a Docker container – example 1<\/h3>\n
Many Docker images exist that can be converted to apptainer \/ singularity images. We user an apptainer recipe (.def<\/code> file) to pull down the Docker container and build an Apptainer image from it.<\/p>\n
The example below uses https:\/\/hub.docker.com\/r\/cp2k\/cp2k\/<\/a>.
\nThere are 2 methods to build the container in a way that is CSF3 compatible (i.e it includes a \/scratch directory).
\nThe end result will be exactly the same with both methods.<\/p>\n
1) Build using a .def file<\/h4>\n
First create a .def file named cp2k-csf.def with the contents below:<\/p>\n
\r\nBootStrap: docker\r\nFrom: cp2k\/cp2k\r\n\r\n%post\r\n   mkdir \/scratch \r\n<\/pre>\n
Then build the container with the command:<\/p>\n
apptainer build cp2k-csf.sif cp2k-csf.def<\/pre>\n
2) Convert to .sif then add \/scratch in 2 steps<\/h4>\n
\r\napptainer build cp2k.sif docker:\/\/cp2k\/cp2k\r\napptainer build --sandbox cp2k-sandbox cp2k.sif \r\nmkdir cp2k-sandbox\/scratch\r\napptainer build cp2k-csf.sif cp2k-sandbox\r\n<\/pre>\n
Converting from a Docker container – example 2<\/h3>\n
The following example shows how a container can be used to make available old versions of software that might be difficult to obtain or install on the current system.<\/p>\n
We were asked to install Tensorflow 2.15 with TensorFlow Probability 0.23. The following method can be used to create an Apptainer image for this:<\/p>\n
First create a text file named tensorflow-gpu-2.150.def<\/code> containing:<\/p>\n
\r\nBootstrap: docker\r\n\r\nFrom: tensorflow\/tensorflow:2.15.0-gpu\r\n\r\n%post\r\n  pip install tensorflow-probability==0.23.0\r\n<\/pre>\n
On the CSF login node, build the image:<\/p>\n
\r\nmodule purge\r\napptainer build tensorflow-gpu-2.15.0.sif tensorflow-gpu-2.15.0.def\r\n<\/pre>\n
Now test in an interactive session on the CSF (1 x L40S GPU, 4 CPU cores, 30 minute session):<\/p>\n
\r\nsrun -p gpuL -G 1 -n 4 -t 30 --pty bash\r\nmodule purge\r\napptainer exec -B \/scratch,\/mnt --nv tensorflow-gpu-2.15.0.sif python3 my_python_code<\/em>.py\r\n\r\n# When finished test, return to the login node\r\nexit\r\n<\/pre>\n
You should see the Tensorflow results displayed to your terminal.<\/p>\n
Running a container<\/h3>\n
When running your container, please remember to bind scratch (and also \/mnt which will make your home directory available) and run your jobs from there:<\/p>\n
\r\napptainer run -B \/scratch,\/mnt my_container.sif arg1 arg2 ...<\/em>\r\n<\/pre>\n
Alternatively, you can set the following environment variable:<\/p>\n
\r\nexport APPTAINER_BINDPATH=\"\/scratch,\/mnt\"\r\napptainer run my_container.sif arg1 arg2 ...<\/em>\r\n<\/pre>\n
Running GPU containers<\/h3>\n
If your app will be using a GPU, you’ll need to submit the job to GPU nodes<\/a> as usual. Your jobscript should load a CUDA modulefile<\/a>.<\/p>\n
The simplest method of running a GPU container (from a jobscript or srun interactive session) is simply to do:<\/p>\n
\r\napptains exec -B \/scratch,\/mnt --nv myimage.sif<\/em> args...<\/em>\r\n<\/pre>\n
This will make available to the container the GPUs that have been assigned to your job.<\/p>\n
The following instructions are from the pre-Slurm CSF can be ignored (TBC).<\/p>\n
We like to use the following code in a script or jobscript to run containers – it will automatically pass the required GPU flags and settings to singularity if needed:<\/p>\n
\r\n# Note: If running a GPU-enabled container your jobscript must load a 'libs\/cuda'\r\n# modulefile before you use the code below.\r\n\r\n# These env vars (without the APPTAINER_) will be visible inside the image at runtime\r\nexport APPTAINER_HOME=\"$HOME\"\r\nexport APPTAINER_LANG=\"$LANG\"\r\n\r\n# Bind the CSF's real \/scratch and \/mnt dirs to empty dirs inside the image\r\nexport APPTAINER_BINDPATH=\"\/scratch,\/mnt\"\r\n\r\n# A GPU job on the CSF will have set $CUDA_VISIBLE_DEVICES, so test\r\n# whether it is set or not (-n means \"non-zero\")\r\nif [ -n \"$CUDA_VISIBLE_DEVICES\" ]; then\r\n   # We are a GPU job. Set the special SINGULARITYENV_CUDA_VISIBLE_DEVICES to limit\r\n   # which GPUs the container can see.\r\n   export APPTAINERENV_CUDA_VISIBLE_DEVICES=\"$CUDA_VISIBLE_DEVICES\"\r\n   # This is the nvidia flag for the apptainer command line\r\n   NVIDIAFLAG=--nv\r\nfi\r\n\r\n# We use the 'sg' command to ensure the container is run with your own group id.\r\nsg $GROUP -c \"apptainer run $NVIDIAFLAG my_container.sif arg1 arg2 ...<\/em>\"\r\n<\/pre>\n
Building your own Singularity image<\/h3>\n
You can build your own sifs for use on the CSF3 via the online resource: https:\/\/cloud.sylabs.io\/builder<\/a><\/p>\n
Please remember to include <\/p>\n
mkdir \/scratch<\/pre>\n
 in the definition instructions. Be aware also that this resource is not affiliated with The University of Manchester.<\/p>\n
Further info<\/h2>\n