Conda – Miniforge3

Overview

Conda is package and environment manager. It allows you to create separate environments, each containing their own files, packages (typically python packages), and package dependencies. The contents of each environment do not interact with each other.

Hence it allows you to work on multiple python projects without package installs in one project breaking another project.

New versions of Conda are regularly released, but environments created with older versions are generally usable with newer versions.

On the CSF we recommend the Minforge3 implementation of Conda because it is aligned with the open conda-forge community and repository.

The Bioconda channel is already configured within our Miniforge modules.

Restrictions on use

Conda is open source but packages and repositories may not be entirely open or available under the same open source license.

Miniforge installer code uses BSD-3-Clause license.

Set up procedure

We recommend loading modulefiles within your jobscripts so that you have a full record of how the job was run. See the example jobscript below for how to do this. Alternatively, you may load modulefiles on the login node and let the job inherit these settings.

Load one of the following modulefiles:

module load apps/binapps/conda/miniforge3/25.9.1
module load apps/binapps/conda/miniforge3/25.3.0

Loading the module will set all parameters to allow Conda to function, but will not activate the conda base environment. To use a Conda environment, you first need to create it.

The information provided here is for quick reference, for more complete documentation please see the official Conda getting started guide.

Create a Conda environment

Create an environment called “myenv” with a specific Python version and the pandas package:

conda create -n myenv python==3.11 pandas

Conda will look up the requested packages and if they are available, will present you with package plan. You do not have to install python, Conda is used to manage a wide range of different software.

Type ‘Y’ to accept the plan and allow Conda to install the software.

When the install is complete you will be prompted to activate the environment:

conda activate myenv

Once your environment is active your shell will look something like:

(myenv) [a12345bc@login2[csf3] ~]$

To deactivate an environment:

conda deactivate

You can install/uninstall any other available Conda packages with:

conda install PACKAGE
conda remove PACKAGE

Mamba

If your packages are particularly complicated and have many dependencies, using mamba to install into an empty Conda environment may be faster. This tends to be the case with Bioinformatics pipeline environments. To use mamba:

conda create -n myenv
conda activate myenv
mamba install PACKAGE1 PACKAGE2

Running jobs within a Conda environment

While you can set up your environment on the login nodes, jobs must be submitted to the compute nodes via batch.

Serial batch job submission

Create a batch submission script which loads the modulefile, checking that you are loading the version you want, for example:

#!/bin/bash --login
#SBATCH -p serial     # (or --partition=) Run on the nodes dedicated to 1-core jobs
#SBATCH -t 4-0        # Wallclock time limit. 4-0 is 4 days. Max permitted is 7-0.

# Start with a clean environment - modules are inherited from the login node by default.
module purge
module load apps/binapps/conda/miniforge3/VERSION

conda activate myenv

python3 myscript.py

Submit the jobscript using:

sbatch scriptname

where scriptname is the name of your jobscript.

Parallel batch job submission

If the app is multicore capable, the following parallel jobscript includes an example of how you might set the number of threads/processes automatically from SLURM variables, whether this is necessary will depend on your code.

#!/bin/bash --login
#SBATCH -p multicore  # (or --partition=) Run on the AMD 168-core nodes
#SBATCH -n 16         # (or --ntasks=) Number of cores to use.
#SBATCH -t 4-0        # Wallclock time limit. 4-0 is 4 days. Max permitted is 7-0.

## Start with a clean environment - modules are inherited from the login node by default.
module purge
module load apps/binapps/conda/miniforge3/VERSION

conda activate myenv

## set any multi-tread parameters you may need from the slurm parameter
## THREADS will be 16 in this example
THREADS=$SLURM_NTASKS_PER_NODE

## run code, assuming that the -t option is number of threads
python3 myscript.py -t $THREADS

Submit the jobscript using:

sbatch scriptname

where scriptname is the name of your jobscript.

Hints and tips

Conda environments and packages can be large, which can lead to your home directory or other spaces getting full, the following actions can help you clean up:

Review your environments and remove any you no longer need:

conda env list

conda env remove -n NAME

Clean up cached downloads (will not delete environments):

conda clean -a

You can specify a different location when you create a new environment provided you have write access there:

conda create -p /some/path/you/have/write/access/myenv

and activate with the full path:

conda activate /some/path/you/have/write/access/myenv

Reproducible environments

It is good practice to keep a record of exactly what packages and versions you used to produce your work. Conda supports this by allowing you to export the complete list of packages in an environment to a YAML file:

conda export -n NAME --no-builds --file environment.yaml 

We suggest putting this environment.yaml into your source control alongside your code or scripts.

You can subsequently re-create a deleted environment using the file:

conda env create --file=environment.yaml

Share environment details with a collaborator via a specification file

With the environment active:

conda list --explicit > specification.txt

To make an environment from a specification:

conda create --prefix INSERT-FULL-PATH/env-name --file specification.txt

Further details can be found in the conda export documentation.

Migrating from Anaconda Python – check your ~/.bashrc file

If you’ve previously used Anaconda Python on the CSF to create conda environments, you may wish to check that your login environment is not automatically setting things up for Anaconda, as follows:

If when using Anaconda python, you may have run conda init bash at some point. This will have added something to your ~/.bashrc file, which will cause Anaconda settings to be applied every time you login to the CSF. You can now remove the following lines from your ~/.bashrc file (which is just a text file):

gedit ~/.bashrc
  #
  # Remove everything bewteen the lines show below, including the two line!
  # This occurs at the bottom of your ~/.bashrc file.

# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!

... remove all of the script code and these surrounding lines ...

# <<< conda initialize <<<

Moving a conda environment

You may need to move a conda environment:

• To free up space in your home directory
• Because your home directory will change path due to an account status change on CSF3. You must act before your home directory changes

Where are Conda environments stored?

Every conda environment is stored in its own directory, the default location is a hidden directory in your home:

~/.conda/envs

If you specified the -p or --prefix option when you created the environment, it will be in whatever path you provided.

You can view where your conda environments are located with the command:

conda env list

Considerations before moving

• If you have your own installation of miniforge3/miniconda3/Anaconda3 in your home directory and your home directory will change, moving is slightly different. The conda base environment cannot be moved as it is tied to the conda software installation.
• If you have installed all your tools into a single conda base environment located in your home directory, then you will need to extract this into a new location before switching to the minforge3 module.
• You may also need to clean up your ~/.bashrc file if you have ever run conda init or hard-coded any conda paths.

If all your environments apart from base are in a non-home location (created with -p or --prefix) and you don’t use base, you are very unlikely to have to move them for changes to your home location.

Moving with clone

The simplest way to move an environment within the same system is with the clone tool, which makes a copy of the whole environment in a new location.

If you have your own conda installation in your home, have been adding packages to the base environment and your home will move, you must clone your base environment into a new location with a new name (not base), using your original installation of conda with:

conda create --prefix /INSERT-FULL-PATH/envs/my-env --clone base

To clone a normal (non-base) environment, for example ‘my-env’:

conda create --prefix /INSERT-FULL-PATH/envs/my-env --clone my-env

In both cases the new environment can be activated (from a clean session) with:

module load apps/binapps/conda/miniforge3

conda activate /INSERT-FULL-PATH/envs/my-env

Tidying moved environments

When an environment is stored in a --prefix location, by default the whole path will be shown on the command line when active, to just show the environment name:

conda config --set env_prompt '({name})'

To tell conda where to search and not have to provide the full path to a non default location every time add the location to your .condarc file without any environment name:

envs_dirs:
  - /INSERT-FULL-PATH/envs
pkgs_dirs:
  - /INSERT-FULL-PATH/conda_pkgs

If you are moving conda environments and are uncertain about the best approach, please get in touch.

Example App Installs

Here we provide some example application installs we’ve used (or suggested) in response to application install requests. Please note that you should check each app’s own documentation and install notes for full details.

FBPIC on GPUs

See the FBPIC Github page.

module load apps/binapps/conda/miniforge3/25.9.1
conda create -n FBPIC_GPU_rit
conda activate FBPIC_GPU_rit
conda install -c conda-forge numba scipy pyfftw mpi4py openmpi cupy cuda-version=13.0 cuda-cudart cuda-nvcc cuda-nvrtc
pip install --no-cache-dir fbpic

Jobscripts should contain the commands:

module purge
module load apps/binapps/conda/miniforge3/25.9.1
conda activate FBPIC_GPU_rit
python ionization_script.py

SCENIC+

See the SCENIC+ Github page.

Whilst there are instructions on that page showing how to install from source inside a conda environment, we’ve done an installation from the BioConda channel because the source installation failed on the CSF:

module load apps/binapps/conda/miniforge3/25.9.1
conda create -n scenicplus -y
conda activate scenicplus
conda install -c bioconda scenicplus -y

To use the installation, add to your jobscripts:

module load apps/binapps/conda/miniforge3/25.9.1
conda activate scenicplus

Further information

Conda Documentation