Conda – Miniforge3
Overview
Conda is package and environment manager.
New versions of Conda are regularly released but environments created with older versions are generally usable with newer versions.
On the CSF we recommend the Minforge3 implementation of Conda because it is aligned with the open conda-forge community and repository.
The Bioconda channel is already configured within our Miniforge modules.
Restrictions on use
Conda is open source but packages and repositories may not be entirely open or available under the same open source license.
Miniforge installer code uses BSD-3-Clause license.
Set up procedure
We recommend loading modulefiles within your jobscripts so that you have a full record of how the job was run. See the example jobscript below for how to do this. Alternatively, you may load modulefiles on the login node and let the job inherit these settings.
Load one of the following modulefiles:
module load apps/binapps/conda/miniforge3/25.9.1 module load apps/binapps/conda/miniforge3/25.3.0
Loading the module will set all parameters to allow Conda to function, but will not activate the conda base environment. To use a Conda environment, you first need to create it.
The information provided here is for quick reference, for more complete documentation please see the official Conda getting started guide.
Create a Conda environment
Create an environment called “myenv” with a specific Python version and the pandas package:
conda create -n myenv python==3.11 pandas
Conda will look up the requested packages and if they are available, will present you with package plan. You do not have to install python, Conda is used to manage a wide range of different software.
Type ‘Y’ to accept the plan and allow Conda to install the software.
When the install is complete you will be prompted to activate the environment:
conda activate myenv
Once your environment is active your shell will look something like:
(myenv) [a12345bc@login2[csf3] ~]$
To deactivate an environment:
conda deactivate
You can install/uninstall any other available Conda packages with:
conda install PACKAGE conda remove PACKAGE
Mamba
If your packages are particularly complicated and have many dependencies, using mamba to install into an empty Conda environment may be faster. This tends to be the case with Bioinformatics pipeline environments. To use mamba:
conda create -n myenv conda activate myenv mamba install PACKAGE1 PACKAGE2
Running jobs within a Conda environment
While you can set up your environment on the login nodes, jobs must be submitted to the compute nodes via batch.
Serial batch job submission
Create a batch submission script which loads the modulefile, checking that you are loading the version you want, for example:
#!/bin/bash --login #SBATCH -p serial # (or --partition=) Run on the nodes dedicated to 1-core jobs #SBATCH -t 4-0 # Wallclock time limit. 4-0 is 4 days. Max permitted is 7-0. # Start with a clean environment - modules are inherited from the login node by default. module purge module load apps/binapps/conda/miniforge3/VERSION conda activate myenv python3 myscript.py
Submit the jobscript using:
sbatch scriptname
where scriptname is the name of your jobscript.
Parallel batch job submission
If the app is multicore capable, the following parallel jobscript includes an example of how you might set the number of threads/processes automatically from SLURM variables, whether this is necessary will depend on your code.
#!/bin/bash --login #SBATCH -p multicore # (or --partition=) Run on the AMD 168-core nodes #SBATCH -n 16 # (or --ntasks=) Number of cores to use. #SBATCH -t 4-0 # Wallclock time limit. 4-0 is 4 days. Max permitted is 7-0. ## Start with a clean environment - modules are inherited from the login node by default. module purge module load apps/binapps/conda/miniforge3/VERSION conda activate myenv ## set any multi-tread parameters you may need from the slurm parameter ## THREADS will be 16 in this example THREADS=$SLURM_NTASKS_PER_NODE ## run code, assuming that the -t option is number of threads python3 myscript.py -t $THREADS
Submit the jobscript using:
sbatch scriptname
where scriptname is the name of your jobscript.
Hints and tips
By default conda environments are created in a hidden subdirectory in your home directory:
~/.conda/
This can lead to your home directory getting full, the following actions can help you clean up:
Review your environments and remove any you no longer need:
conda env list conda env remove -n NAME
Clean up cached downloads (will not delete environments):
conda clean -a
You can specify a different location when you create a new environment provided you have write access there:
conda create -p /some/path/you/have/write/access/myenv
and activate with the full path:
conda activate /some/path/you/have/write/access/myenv
Reproducible environments
It is good practice to keep a record of exactly what packages and versions you used to produce your work. Conda supports this by allowing you to export the complete list of packages in an environment to a YAML file:
conda export -n NAME --no-builds --file environment.yaml
We suggest putting this environment.yaml into your source control alongside your code or scripts.
You can subsequently re-create a deleted environment using the file:
conda env create --file=environment.yaml
Further details can be found in the conda export documentation.
Moving from Anaconda Python – check your ~/.bashrc file
If you’ve previously used Anaconda Python on the CSF to create conda environments, you may wish to check that your login environment is not automatically setting things up for Anaconda, as follows:
If when using Anaconda python, you may have run conda init bash at some point. This will have added something to your ~/.bashrc file, which will cause Anaconda settings to be applied every time you login to the CSF. You can now remove the following lines from your ~/.bashrc file (which is just a text file):
gedit ~/.bashrc # # Remove everything bewteen the lines show below, including the two line! # This occurs at the bottom of your ~/.bashrc file. # >>> conda initialize >>> # !! Contents within this block are managed by 'conda init' !! ... remove all of the script code and these surrounding lines ... # <<< conda initialize <<<
Example App Installs
Here we provide some example application installs we’ve used (or suggested) in response to application install requests. Please note that you should check each app’s own documentation and install notes for full details.
FBPIC on GPUs
See FBPIC Github page.
module load apps/binapps/conda/miniforge3/25.9.1 conda create -n FBPIC_GPU_rit conda activate FBPIC_GPU_rit conda install -c conda-forge numba scipy pyfftw mpi4py openmpi cupy cuda-version=13.0 cuda-cudart cuda-nvcc cuda-nvrtc pip install --no-cache-dir fbpic
Jobscripts should contain the commands:
module purge module load apps/binapps/conda/miniforge3/25.9.1 conda activate FBPIC_GPU_rit python ionization_script.py