Conda<\/a> is package and environment manager. <\/p>\n New versions of Conda are regularly released but environments created with older versions are generally usable with newer versions.<\/p>\n On the CSF we recommend the Minforge3 implementation of Conda because it is aligned with the open conda-forge community and repository<\/a>.<\/p>\n The Bioconda channel<\/a> is already configured within our Miniforge modules.<\/p>\n Conda is open source but packages and repositories may not be entirely open or available under the same open source license.<\/p>\n Miniforge installer code uses BSD-3-Clause license.<\/p>\n We recommend loading modulefiles within your jobscripts so that you have a full record of how the job was run. See the example jobscript below for how to do this. Alternatively, you may load modulefiles on the login node and let the job inherit these settings.<\/p>\n Load one of the following modulefiles:<\/p>\n Loading the module will set all parameters to allow Conda to function, but will not activate the conda base environment. To use a Conda environment, you first need to create it.<\/p>\n The information provided here is for quick reference, for more complete documentation please see the official Conda getting started guide.<\/a><\/p>\n Create an environment called “myenv” with a specific Python version and the pandas package:<\/p>\n Conda will look up the requested packages and if they are available, will present you with package plan. You do not have to install python, Conda is used to manage a wide range of different software.<\/p>\n Type ‘Y’ to accept the plan and allow Conda to install the software.<\/p>\n When the install is complete you will be prompted to activate the environment:<\/p>\n Once your environment is active your shell will look something like:<\/p>\n To deactivate an environment:<\/p>\n You can install\/uninstall any other available Conda packages with:<\/p>\n If your packages are particularly complicated and have many dependencies, using While you can set up your environment on the login nodes, jobs must<\/strong> be submitted to the compute nodes via batch.<\/p>\n Create a batch submission script which loads the modulefile, checking that you are loading the version you want, for example:<\/p>\n Submit the jobscript using: <\/p>\n where scriptname<\/em> is the name of your jobscript.<\/p>\n If the app is multicore capable, the following parallel jobscript includes an example of how you might set the number of threads\/processes automatically from SLURM variables, whether this is necessary will depend on your code.<\/p>\n Submit the jobscript using: <\/p>\n where scriptname<\/em> is the name of your jobscript.<\/p>\n By default conda environments are created in a hidden subdirectory in your home directory:<\/p>\n This can lead to your home directory getting full, the following actions can help you clean up:<\/p>\n Review your environments and remove any you no longer need:<\/p>\n Clean up cached downloads (will not delete environments):<\/p>\n You can specify a different location when you create a new environment provided you have write access there:<\/p>\n and activate with the full path:<\/p>\n It is good practice to keep a record of exactly what packages and versions you used to produce your work. Conda supports this by allowing you to export the complete list of packages in an environment to a YAML file:<\/p>\n We suggest putting this environment.yaml into your source control alongside your code or scripts.<\/p>\n You can subsequently re-create a deleted environment using the file:<\/p>\n Further details can be found in the conda export documentation.<\/a><\/p>\n If you’ve previously used Anaconda Python on the CSF to create conda environments, you may wish to check that your login environment is not automatically setting things up for Anaconda, as follows:<\/p>\n If when using Anaconda python, you may have run Here we provide some example application installs we’ve used (or suggested) in response to application install requests. Please note that you should check each app’s own documentation and install notes for full details.<\/p>\n See the FBPIC Github<\/a> page.<\/p>\n Jobscripts should contain the commands:<\/p>\n See the SCENIC+ Github<\/a> page.<\/p>\n Whilst there are instructions on that page showing how to install from source inside a conda environment, we’ve done an installation from the BioConda<\/a> channel because the source installation failed on the CSF:<\/p>\n To use the installation, add to your jobscripts:<\/p>\n Conda Documentation<\/a><\/p>\n","protected":false},"excerpt":{"rendered":" Overview Conda is package and environment manager. New versions of Conda are regularly released but environments created with older versions are generally usable with newer versions. On the CSF we recommend the Minforge3 implementation of Conda because it is aligned with the open conda-forge community and repository. The Bioconda channel is already configured within our Miniforge modules. Restrictions on use Conda is open source but packages and repositories may not be entirely open or available.. Read more »<\/a><\/p>\n","protected":false},"author":25,"featured_media":0,"parent":86,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-11406","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/11406","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/users\/25"}],"replies":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/comments?post=11406"}],"version-history":[{"count":13,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/11406\/revisions"}],"predecessor-version":[{"id":12120,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/11406\/revisions\/12120"}],"up":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/86"}],"wp:attachment":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/media?parent=11406"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Restrictions on use<\/h2>\n
Set up procedure<\/h2>\n
\r\nmodule load apps\/binapps\/conda\/miniforge3\/25.9.1\r\nmodule load apps\/binapps\/conda\/miniforge3\/25.3.0\r\n<\/pre>\n
Create a Conda environment<\/h3>\n
\r\nconda create -n myenv python==3.11 pandas\r\n<\/pre>\n
\r\nconda activate myenv\r\n<\/pre>\n
\r\n(myenv) [a12345bc@login2[csf3] ~]$\r\n<\/pre>\n
\r\nconda deactivate\r\n<\/pre>\n
\r\nconda install PACKAGE\r\nconda remove PACKAGE\r\n<\/pre>\n
Mamba<\/h3>\n
mamba<\/code> to install into an empty Conda environment may be faster. This tends to be the case with Bioinformatics pipeline environments. To use mamba<\/code>:<\/p>\n\r\nconda create -n myenv\r\nconda activate myenv\r\nmamba install PACKAGE1 PACKAGE2\r\n<\/pre>\n
Running jobs within a Conda environment<\/h2>\n
Serial batch job submission<\/h3>\n
\r\n#!\/bin\/bash --login\r\n#SBATCH -p serial # (or --partition=) Run on the nodes dedicated to 1-core jobs\r\n#SBATCH -t 4-0 # Wallclock time limit. 4-0 is 4 days. Max permitted is 7-0.\r\n\r\n# Start with a clean environment - modules are inherited from the login node by default.\r\nmodule purge\r\nmodule load apps\/binapps\/conda\/miniforge3\/VERSION\r\n\r\nconda activate myenv\r\n\r\npython3 myscript.py\r\n\r\n<\/pre>\n
sbatch scriptname<\/em><\/pre>\n
Parallel batch job submission<\/h3>\n
\r\n#!\/bin\/bash --login\r\n#SBATCH -p multicore # (or --partition=) Run on the AMD 168-core nodes\r\n#SBATCH -n 16 # (or --ntasks=) Number of cores to use.\r\n#SBATCH -t 4-0 # Wallclock time limit. 4-0 is 4 days. Max permitted is 7-0.\r\n\r\n## Start with a clean environment - modules are inherited from the login node by default.\r\nmodule purge\r\nmodule load apps\/binapps\/conda\/miniforge3\/VERSION\r\n\r\nconda activate myenv\r\n\r\n## set any multi-tread parameters you may need from the slurm parameter\r\n## THREADS will be 16 in this example\r\nTHREADS=$SLURM_NTASKS_PER_NODE\r\n\r\n## run code, assuming that the -t option is number of threads\r\npython3 myscript.py -t $THREADS\r\n<\/pre>\n
sbatch scriptname<\/em><\/pre>\n
Hints and tips<\/h2>\n
\r\n~\/.conda\/\r\n<\/pre>\n
\r\nconda env list\r\n\r\nconda env remove -n NAME\r\n<\/pre>\n
\r\nconda clean -a\r\n<\/pre>\n
\r\nconda create -p \/some\/path\/you\/have\/write\/access\/myenv\r\n<\/pre>\n
\r\nconda activate \/some\/path\/you\/have\/write\/access\/myenv\r\n<\/pre>\n
Reproducible environments<\/h3>\n
\r\nconda export -n NAME --no-builds --file environment.yaml \r\n<\/pre>\n
\r\nconda env create --file=environment.yaml\r\n<\/pre>\n
Moving from Anaconda Python – check your ~\/.bashrc file<\/h3>\n
conda init bash<\/code> at some point. This will have added something to your ~\/.bashrc<\/code> file, which will cause Anaconda settings to be applied every time you login to the CSF. You can now remove the following lines from your ~\/.bashrc<\/code> file (which is just a text file):<\/p>\n\r\ngedit ~\/.bashrc\r\n #\r\n # Remove everything bewteen the lines show below, including the two line!\r\n # This occurs at the bottom of your ~\/.bashrc file.\r\n\r\n# >>> conda initialize >>>\r\n# !! Contents within this block are managed by 'conda init' !!\r\n\r\n... remove all of the script code and these surrounding lines<\/strong><\/em> ...\r\n\r\n# <<< conda initialize <<<\r\n<\/pre>\n
Example App Installs<\/h2>\n
FBPIC on GPUs<\/h3>\n
\r\nmodule load apps\/binapps\/conda\/miniforge3\/25.9.1\r\nconda create -n FBPIC_GPU_rit\r\nconda activate FBPIC_GPU_rit\r\nconda install -c conda-forge numba scipy pyfftw mpi4py openmpi cupy cuda-version=13.0 cuda-cudart cuda-nvcc cuda-nvrtc\r\npip install --no-cache-dir fbpic\r\n<\/pre>\n
\r\nmodule purge\r\nmodule load apps\/binapps\/conda\/miniforge3\/25.9.1\r\nconda activate FBPIC_GPU_rit\r\npython ionization_script<\/em>.py\r\n<\/pre>\n
SCENIC+<\/h3>\n
\r\nmodule load apps\/binapps\/conda\/miniforge3\/25.9.1\r\nconda create -n scenicplus -y\r\nconda activate scenicplus\r\nconda install -c bioconda scenicplus -y\r\n<\/pre>\n
\r\nmodule load apps\/binapps\/conda\/miniforge3\/25.9.1\r\nconda activate scenicplus\r\n<\/pre>\n
Further information<\/h2>\n