Conda – Miniforge3
Overview
Conda is package and environment manager. It allows you to create separate environments, each containing their own files, packages (typically python packages), and package dependencies. The contents of each environment do not interact with each other.
Hence it allows you to work on multiple python projects without package installs in one project breaking another project.
New versions of Conda are regularly released, but environments created with older versions are generally usable with newer versions.
On the CSF we recommend the Minforge3 implementation of Conda because it is aligned with the open conda-forge community and repository.
The Bioconda channel is already configured within our Miniforge modules.
Restrictions on use
Conda is open source but packages and repositories may not be entirely open or available under the same open source license.
Miniforge installer code uses BSD-3-Clause license.
Set up procedure
We recommend loading modulefiles within your jobscripts so that you have a full record of how the job was run. See the example jobscript below for how to do this. Alternatively, you may load modulefiles on the login node and let the job inherit these settings.
Load one of the following modulefiles:
module load apps/binapps/conda/miniforge3/26.3.2 # Provides python 3.13.13 unless specified module load apps/binapps/conda/miniforge3/25.9.1 # Provides python 3.12.12 unless specified module load apps/binapps/conda/miniforge3/25.3.0 # Provides python 3.12.9 unless specified
Loading the module will set all parameters to allow Conda to function, but will not activate the conda base environment. To use a Conda environment, you first need to create it.
The information provided here is for quick reference, for more complete documentation please see the official Conda getting started guide.
Create a Conda environment
Create an environment called “myenv” with a specific Python version and the pandas package:
conda create -n myenv python==3.11 pandas
# #
# # Packages you wish to install inside your conda
# # env. This is optional, you can also do it later.
#
# Installing a version of python is optional
# (see above for default python versions)
Conda will look up the requested packages and if they are available, will present you with package plan. You do not have to install python, Conda is used to manage a wide range of different software.
Type ‘Y’ to accept the plan and allow Conda to install the software.
When the install is complete you will be prompted to activate the environment:
conda activate myenv
Once your environment is active your shell will look something like:
(myenv) [a12345bc@login2[csf3] ~]$
#
# Name of active conda env you are now working in.
# For example, python package install will go into this env.
You can install/uninstall any other available Conda packages with the following commands. The packages will be installed into the currently active conda env:
conda install PACKAGE conda remove PACKAGE
To deactivate an environment:
conda deactivate
# Your prompt will return to the default, showing no active env
[a12345bc@login2[csf3] ~]$
Remember, when you want to use a package that you’ve installed in a conda env, you must always first activate the conda env.
Mamba
If your packages are particularly complicated and have many dependencies, using mamba to install into an empty Conda environment may be faster. This tends to be the case with Bioinformatics pipeline environments. To use mamba:
conda create -n myenv conda activate myenv mamba install PACKAGE1 PACKAGE2
Running jobs within a Conda environment
While you can set up your environment on the login nodes, jobs must be submitted to the compute nodes via batch.
Serial batch job submission
Create a batch submission script which loads the modulefile, checking that you are loading the version you want, for example:
#!/bin/bash --login #SBATCH -p serial # (or --partition=) Run on the nodes dedicated to 1-core jobs #SBATCH -t 4-0 # Wallclock time limit. 4-0 is 4 days. Max permitted is 7-0. # Start with a clean environment - modules are inherited from the login node by default. module purge module load apps/binapps/conda/miniforge3/VERSION conda activate myenv python3 myscript.py
Submit the jobscript using:
sbatch scriptname
where scriptname is the name of your jobscript.
Parallel batch job submission
If the app is multicore capable, the following parallel jobscript includes an example of how you might set the number of threads/processes automatically from SLURM variables, whether this is necessary will depend on your code.
#!/bin/bash --login #SBATCH -p multicore # (or --partition=) Run on the AMD 168-core nodes #SBATCH -n 16 # (or --ntasks=) Number of cores to use. #SBATCH -t 4-0 # Wallclock time limit. 4-0 is 4 days. Max permitted is 7-0. ## Start with a clean environment - modules are inherited from the login node by default. module purge module load apps/binapps/conda/miniforge3/VERSION conda activate myenv ## set any multi-tread parameters you may need from the slurm parameter ## THREADS will be 16 in this example THREADS=$SLURM_NTASKS_PER_NODE ## run code, assuming that the -t option is number of threads python3 myscript.py -t $THREADS
Submit the jobscript using:
sbatch scriptname
where scriptname is the name of your jobscript.
Hints and tips
Conda environments and packages can be large, which can lead to your home directory or other spaces getting full, the following actions can help you clean up:
Review your environments and remove any you no longer need:
conda env list conda env remove -n NAME
Clean up cached downloads (will not delete environments):
conda clean -a
You can specify a different location when you create a new environment provided you have write access there:
conda create -p /some/path/you/have/write/access/myenv
and activate with the full path:
conda activate /some/path/you/have/write/access/myenv
Reproducible environments
It is good practice to keep a record of exactly what packages and versions you used to produce your work. Conda supports this by allowing you to export the complete list of packages in an environment to a YAML file:
conda export -n NAME --no-builds --file environment.yaml
We suggest putting this environment.yaml into your source control alongside your code or scripts.
You can subsequently re-create a deleted environment using the file:
conda env create --file=environment.yaml
With the environment active:
conda list --explicit > specification.txt
To make an environment from a specification:
conda create --prefix INSERT-FULL-PATH/env-name --file specification.txt
Further details can be found in the conda export documentation.
Migrating from Anaconda Python – check your ~/.bashrc file
If you’ve previously used Anaconda Python on the CSF to create conda environments, you may wish to check that your login environment is not automatically setting things up for Anaconda, as follows:
If when using Anaconda python, you may have run conda init bash at some point. This will have added something to your ~/.bashrc file, which will cause Anaconda settings to be applied every time you login to the CSF. You can now remove the following lines from your ~/.bashrc file (which is just a text file):
gedit ~/.bashrc # # Remove everything bewteen the lines show below, including the two line! # This occurs at the bottom of your ~/.bashrc file. # >>> conda initialize >>> # !! Contents within this block are managed by 'conda init' !! ... remove all of the script code and these surrounding lines ... # <<< conda initialize <<<
Moving a conda environment
You may need to move a conda environment:
- To free up space in your home directory
- Because your home directory will change path due to an account status change on CSF3. You must act before your home directory changes
Where are Conda environments stored?
Every conda environment is stored in its own directory, the default location is a hidden directory in your home:
~/.conda/envs
If you specified the -p or --prefix option when you created the environment, it will be in whatever path you provided.
You can view where your conda environments are located with the command:
conda env list
Considerations before moving
- If you have your own installation of miniforge3/miniconda3/Anaconda3 in your home directory and your home directory will change, moving is slightly different. The conda
baseenvironment cannot be moved as it is tied to the conda software installation. - If you have installed all your tools into a single conda
baseenvironment located in your home directory, then you will need to extract this into a new location before switching to the minforge3 module. - You may also need to clean up your
~/.bashrcfile if you have ever runconda initor hard-coded any conda paths.
If all your environments apart from base are in a non-home location (created with -p or --prefix) and you don’t use base, you are very unlikely to have to move them for changes to your home location.
Moving with clone
The simplest way to move an environment within the same system is with the clone tool, which makes a copy of the whole environment in a new location.
If you have your own conda installation in your home, have been adding packages to the base environment and your home will move, you must clone your base environment into a new location with a new name (not base), using your original installation of conda with:
conda create --prefix /INSERT-FULL-PATH/envs/my-env --clone base
To clone a normal (non-base) environment, for example ‘my-env’:
conda create --prefix /INSERT-FULL-PATH/envs/my-env --clone my-env
In both cases the new environment can be activated (from a clean session) with:
module load apps/binapps/conda/miniforge3 conda activate /INSERT-FULL-PATH/envs/my-env
Moving with conda-pack
This is yet another simple method. With conda-pack you can pack your existing environments in tarball archives before your account is moved. Then after the account has been moved, you can restore your environments using the tarball. You can use the tarball to restore the virtual environment even in any other computer. The biggest advantage is that it will rewrite all hard-coded paths in the environments and fix the activation scripts.
First of all, you will need to install the conda-pack package itself. You don’t need to install it inside any virtual environment, you install it within your home directory:
module purge module load apps/binapps/conda/miniforge3/25.9.1 pip install conda-pack --user
Next look up all your conda virtual environments using:
conda env list
This will show the list of the conda virtual environment that you have. Output will look like:
# conda environments: # # * -> active # + -> frozen MyPytorchTensorboard /mnt/iusers01/somegroup01/youruserid/.conda/envs/MyPytorchTensorboard mypandas /mnt/iusers01/somegroup01/youruserid/.conda/envs/mypandas
Pack your virtual environment into tarball:
conda pack -n MyPytorchTensorboard -o MyPytorchTensorboard.tar.gz conda pack -n mypandas -o mypandas.tar.gz # TIP: You can create/store the tarball in your scratch directory # if home directory space is limited
The conda-pack additionally packs a conda-unpack utility inside the tarball it creates.
The conda-unpack utility does everything needed for restoring the conda environment in a new environment.
These tarballs can be used to restore the virtual environment after account has been moved to different group/contrbution in CSF3.
They can be also used to restore the virtual environment in any other server/computer as well.
Restoring your virtual environment using conda-unpack from the tarball that was created created using conda-pack:
After the account move, login to your CSF3 account and do this:
# Remove the old virtual environment directory with all their contents if still present:
rm -rf ~/.conda/envs/MyPytorchTensorboard
rm -rf ~/.conda/envs/mypandas
# Create the virtual environment directory:
mkdir ~/.conda/envs/MyPytorchTensorboard
mkdir ~/.conda/envs/mypandas
# Extract the tarballs to their respective directories under ~/.conda/envs:
tar -xzf MyPytorchTensorboard.tar.gz -C ~/.conda/envs/MyPytorchTensorboard/
tar -xzf mypandas.tar.gz -C ~/.conda/envs/mypandas/
# Run the conda-unpack utility
~/.conda/envs/MyPytorchTensorboard/bin/conda-unpack
~/.conda/envs/mypandas/bin/conda-unpack
# Running conda-unpack utility rewrites all hard-coded paths in the environment
# and fixes the activation scripts
# Check if the environment is detected
conda env list
# The conda environment should get detected, if not, activate by using the path once:
conda activate ~/.conda/envs/MyPytorchTensorboard
# Check again if the environment is detected
conda env list
Moving with environment.yaml
This is yet another method. You can export a environment.yaml file which can be used to create virtual environment again.
This the environment.yaml file will contain list of packages that are present in the virtual environment, not the packages themselves.
One caveat in this method is that if your existing virtual environment is very old, there are chances that some of the dependent packages are no longer available in their public repositories from where they can be downloaded and installed again later when you use the environment.yaml file to re-create the virtual environment. It is however better to create and save an environment.yaml file any way to fall back to something if some problem happens. The environment.yaml file can be tweaked to make it work in such circumstances.
Export the environment to environment.yaml file:
module purge module load apps/binapps/conda/miniforge3/25.9.1 conda env list conda export --name myenv --format=environment-yaml > environment.yaml
Use the generated environment.yaml file to create a virtual environment:
module purge module load apps/binapps/conda/miniforge3/25.9.1 # Remove the virtual environment directory first if present rm -rf ~/.conda/envs/myenv # Create the virtual environment conda create --name myenv --file environment.yaml
Tidying moved environments
When an environment is stored in a --prefix location, by default the whole path will be shown on the command line when active, to just show the environment name:
conda config --set env_prompt '({name})'
To tell conda where to search and not have to provide the full path to a non default location every time add the location to your .condarc file without any environment name:
envs_dirs: - /INSERT-FULL-PATH/envs pkgs_dirs: - /INSERT-FULL-PATH/conda_pkgs
If you are moving conda environments and are uncertain about the best approach, please get in touch.
Example App Installs
Here we provide some example application installs we’ve used (or suggested) in response to application install requests. Please note that you should check each app’s own documentation and install notes for full details.
FBPIC on GPUs
See the FBPIC Github page.
module load apps/binapps/conda/miniforge3/25.9.1 conda create -n FBPIC_GPU_rit conda activate FBPIC_GPU_rit conda install -c conda-forge numba scipy pyfftw mpi4py openmpi cupy cuda-version=13.0 cuda-cudart cuda-nvcc cuda-nvrtc pip install --no-cache-dir fbpic
Jobscripts should contain the commands:
module purge module load apps/binapps/conda/miniforge3/25.9.1 conda activate FBPIC_GPU_rit python ionization_script.py
SCENIC+
See the SCENIC+ Github page.
Whilst there are instructions on that page showing how to install from source inside a conda environment, we’ve done an installation from the BioConda channel because the source installation failed on the CSF:
module load apps/binapps/conda/miniforge3/25.9.1 conda create -n scenicplus -y conda activate scenicplus conda install -c bioconda scenicplus -y
To use the installation, add to your jobscripts:
module load apps/binapps/conda/miniforge3/25.9.1 conda activate scenicplus