{"id":577,"date":"2021-03-02T16:10:16","date_gmt":"2021-03-02T16:10:16","guid":{"rendered":"http:\/\/ri.itservices.manchester.ac.uk\/csf4\/?page_id=577"},"modified":"2025-06-04T15:17:34","modified_gmt":"2025-06-04T14:17:34","slug":"anaconda-python","status":"publish","type":"page","link":"https:\/\/ri.itservices.manchester.ac.uk\/csf4\/software\/applications\/anaconda-python\/","title":{"rendered":"Anaconda Python"},"content":{"rendered":"\n\n\n
June 2023: The proxy is no longer available<\/strong>.
\nIt will not be possible to download packages from external sites (e.g., when creating a conda env<\/em>) on CSF4. Please consider using your CSF3 account for this task. See the CSF3 interactive notes<\/a> for more information.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Overview<\/h2>\n

Anaconda<\/a> is a completely free enterprise-ready Python distribution for large-scale data processing, predictive analytics, and scientific computing general-purpose statistical software package. It has 100+ of the most popular Python packages for science, math, engineering, data analysis.<\/p>\n

Versions available are listed below in ‘Set up Procedure’.<\/p>\n

Restrictions on use<\/h2>\n

There are no restrictions on access Anaconda Python on the CSF. All users should read the End User License Agreement<\/a> before using the software.<\/p>\n

Set up procedure<\/h2>\n

We now recommend loading modulefiles within your jobscript so that you have a full record of how the job was run. See the example jobscript below for how to do this. Alternatively, you may load modulefiles on the login node and let the job inherit the settings.<\/p>\n

Load one of the following modulefiles (most recent listed at the top, oldest at the bottom):<\/p>\n

# Python 3.x\r\nmodule load anaconda3\/2022.10          # Python 3.9.13\r\nmodule load anaconda3\/2020.07          # Python 3.8.3\r\nmodule load anaconda3\/5.3.0            # Python 3.7.0\r\nmodule load anaconda3\/5.2.0            # Python 3.6.5\r\n\r\n# Python 2.x\r\nmodule load anaconda2\/5.3.0            # Python 2.7.15\r\n<\/pre>\n
Additional Centrally Installed Packages<\/h2>\n
Please note that on CSF4 we will not<\/strong> be installing additional packages in to Anaconda Python. Instead all users are encourages to use conda environments<\/em> to install packages in self-contained environments within your home directory (see below<\/a> for details). This is because Anaconda python uses its own package manager which may conflict with the package manager we are using on CSF4 for app installs.<\/p>\n
We may be able to install additional python packages against the non-anaconda ordinary python installations. For example, the following modulefiles are available to provided non-anaconda python:<\/p>\n
# Some of the non-anaconda python installations\r\nmodule load python\/3.8.2-gcccore-9.3.0\r\nmodule load python\/3.7.4-gcccore-9.3.0\r\nmodule load python\/2.7.18-gcccore-9.3.0\r\n<\/pre>\n
A full list of python related modulefiles is available by running:<\/p>\n
module avail python\r\n<\/pre>\n
Running the application<\/h2>\n
Please do not run python on the login node. Jobs should be submitted to the compute nodes via batch. If you wish to do interactive development with python, please start an interactive session<\/a> first.<\/p>\n
Serial batch job submission<\/h3>\n
Before submitting the job we will write the following simple python script for use as an example:<\/p>\n
# fib-example.py\r\nparents, babies = (1, 1)\r\nwhile babies < 100:\r\n    print('This generation has %d babies' % babies)\r\n    parents, babies = (babies, parents + babies)\r\n<\/pre>\n
Now create a batch submission script that we will submit to the batch system:<\/p>\n
#!\/bin\/bash --login\r\n#SBATCH -p serial         # (--partition=serial) Option - serial is the default\r\n\r\n# Load the version you require\r\nmodule load anaconda3\/5.3.0\r\n\r\n# Execute our simple python script we created above:\r\npython fib-example.py<\/em>\r\n<\/pre>\n
Submit the jobscript using:<\/p>\n
sbatch scriptname<\/em><\/pre>\n
where scriptname<\/em> is the name of your jobscript.<\/p>\n
Parallel batch job submission<\/h3>\n
A simple parallel numpy<\/code> example script is:<\/p>\n
# eig-example.py\r\nimport numpy\r\n\r\ndef test_eigenvalue():\r\n  i=500\r\n  data=numpy.random.rand(i,i)\r\n  result=numpy.linalg.eig(data)\r\n  return result\r\n\r\nprint(test_eigenvalue())\r\n<\/pre>\n
and the corresponding jobscript is:<\/p>\n
#!\/bin\/bash --login\r\n#SBATCH -p multicore    # (--partition=multicore)\r\n#SBATCH -n 8            # (--ntasks=8) Number of cores (2-40)\r\n\r\n# Load the version you require\r\nmodule load anaconda3\/5.3.0\r\n\r\n# Inform numpy how many cores to use. $SLURM_NTASKS is automatically set to the number given above.\r\nexport OMP_NUM_THREADS=$SLURM_NTASKS\r\n\r\npython eig-example.py<\/em>\r\n<\/pre>\n
Adding packages<\/h2>\n
You will need to load the following modulefile before proceeding to allow access to the outside world, in addition to your prefer anaconda version’s modulefile:<\/p>\n
module load proxy\r\n<\/pre>\n
Anaconda packages<\/h3>\n
To see what is already installed in the central conda installation:<\/p>\n
conda list\r\n<\/pre>\n
To see if a package is available<\/p>\n
conda list package<\/em>\r\n<\/pre>\n
where package<\/code><\/em> is replaced with the name of the package you want.<\/p>\n
The next section provides details on conda environments<\/em> which allow you to install your own packages to extend the capabilities of the central install.<\/p>\n
<\/a><\/p>\n
Conda environments<\/h3>\n
A conda environment<\/em> provides a localised installation of packages managed by Anaconda python. Multiple environments can be set up for different groups of packages (e.g., your machine learning<\/em> packages and your chemistry<\/em> packages). The packages are installed in a directory in your home area. The following commands will install and activate \/ deactivate a particular environment.<\/p>\n
Run these commands on the login node or in an interactive session to set up a new conda environment:<\/p>\n
module load anaconda3\/5.3.0\r\nmodule load proxy\r\n<\/pre>\n
Now create the conda environment. You will then be able to add conda (python) packages to it. Packages will be stored in your home dir in ~\/.conda\/envs\/<\/code>. There are a couple of ways you can do this (we use the name test_env<\/code> in the examples below). When running the commands below, ignore any message about updating conda – you won’t be able to update the centrally installed version:<\/p>\n
There are two alternatives. If you want to fix exactly which version of python you are working with, use option 2:<\/p>\n
    \n
  1. Create an empty<\/em> environment. You can add packages to it later (use -n on 2019.07 version):\n
    conda create -n test_env<\/strong>\r\n<\/pre>\n<\/li>\n
  2. Alternatively, create a new environment that contains various standard packages<\/em> such as pip, setuptools etc (it will tell you what is being installed). You can specify a particular version of python (to match the centrally installed version) or conda will download the latest available and install that in your environment:\n
    # If you want to ensure you use the same version of python as in the central install:\r\npython --version\r\n    #\r\n    # Make a note of: 3.7.0<\/strong> (or whatever your version is)\r\n\r\n# Create the conda environment (remove the =3.7.0 to use the latest version of python)\r\nconda create -n test_env<\/strong> python=3.7.0<\/strong>\r\n[y]\r\n<\/pre>\n<\/li>\n<\/ol>\n
    In both of the above examples, you can add the name of a python package to be installed while creating your environment. For example:<\/p>\n
    conda create -n test_env packagename<\/em>\r\n<\/pre>\n
    The following commands can be used in a jobscript or interactive session every time you want to use your conda environment:<\/p>\n
    # Activate virtual env - these commands can be run inside a jobscript or an interactive session\r\n\r\n[username<\/em>@login02 [CSF4] ~]$ source activate test_env\r\n                                #\r\n                                # Can also use 'conda activate test_env' but only on the\r\n                                # login node. In jobscripts you should use 'source'.\r\n\r\n(test_env) [username<\/em>@login02 [CSF4] ~]$\r\n   #\r\n   # Notice that your prompt changes to indicate the\r\n   # name of the active conda environment.\r\n\r\n(test_env) [username<\/em>@login02 [CSF4] ~]$ python myscript.py<\/em>\r\n                                                    #\r\n                                                    # import xyz<\/em>\r\n                                                    # ...\r\n                                                    # exit()\r\n\r\n# If you wanted to install other packages in the current active env:\r\n(test_env) [username<\/em>@login02 [CSF4] ~]$ conda install packagename<\/em>\r\n\r\n# If you wanted to remove a package from the current active env:\r\n(test_env) [username<\/em>@login02 [CSF4] ~]$ conda remove packagename<\/em>\r\n\r\n# When you are finished, switch off the conda environment.\r\n# Your python code will only have access to the packages provided by the central install.\r\n(test_env) [username<\/em>@login02 [CSF4] ~]$  source deactivate\r\n\r\n[username<\/em>@login02 [CSF4] ~]$ \r\n   #\r\n   # Your prompt is now how it normally is, indicating\r\n   # that no conda environment is active.\r\n<\/pre>\n
    Some other actions you may wish to perform with your environments:<\/p>\n
    # Get a list of all your conda environments (active env shown with a *)\r\nconda env list\r\n   # conda environments:\r\n   #\r\n   gputest                  <HOME>\/.conda\/envs\/gputest\r\n   my_bioinf_env            <HOME>\/.conda\/envs\/my_bioinf_env\r\n   base                  *  \/opt\/software\/RI\/apps\/Anaconda3\/5.2.0\r\n     #\r\n     # Note that the 'base' env is the central install area and is shown as the\r\n     # default environment (with a *) when you have not activated any of your\r\n     # own environments. You will NOT be able to install packages into 'base'\r\n     # because of the file permissions. You must create an activate one of your\r\n     # own environments before installing any extra python packages.\r\n\r\n\r\n# Get a list of the packages installed in the currently active environment\r\nconda list\r\n\r\n# Install other packages in a named virtual env (for example we install a packed named pillow).\r\n# If you don't specify the name it will install in the currently active package.\r\nconda install -n test_env pillow\r\n\r\n# To remove the test_env virtual env files (deletes files from your ~\/.conda\/envs\/ directory)\r\nconda remove -n test_env --all\r\n<\/pre>\n
    It is possible to activate other channels<\/em> inside a conda environment so that you can install packages from different channels. For example, to use BioConda<\/a> in a conda env:<\/p>\n
    # This example uses the BioConda channel and installs a package named emirge from that channel.\r\n# Note that emirge requires python 2.7 so we use the older Anaconda v2 installation:\r\nmodule load anaconda2\/5.3.0         # Provides Python 2.7.15\r\nmodule load proxy\r\n\r\n# Create a new conda env\r\nconda create -n my_bioinf_env python=2.7.15\r\n   #\r\n   # (ignore warning about new version of conda)\r\n   # \r\n   # \r\n   # Proceed ([y]\/n)? y<\/strong>\r\n\r\n# Activate the environment\r\nsource activate my_bioinf_env             # Can also use 'conda activate my_bioinf_env'\r\n\r\n# Add the BioConda channels to the env - these are the places where packages\r\n# are downloaded from. The BioConda website says the order of these is important.\r\nconda config --env --add channels defaults\r\nconda config --env --add channels bioconda\r\nconda config --env --add channels conda-forge\r\n\r\n# Install the emirge package (use your own packages here)\r\nconda install emirge\r\n\r\n# We use emirge at the command-line but you may need to run python\r\n# and import a library, for example.\r\nemirge.py --help\r\n\r\n# Deactivate the env when done\r\nsource deactivate\r\n<\/pre>\n
    pip installation inside a conda env<\/h3>\n
    The above section showed how to create a conda env and then use the conda install<\/code> command to install packages in to that environment.<\/p>\n
    You may also want to perform pip<\/code> installations in to your conda env. This is possible but you need to perform a couple of extra steps:<\/p>\n
    # First move a config file you may have out of the way. If this is present it\r\n# will force the pip install to occur outside of the conda env.\r\nmv ~\/.pydistutils.cfg ~\/.pydistutils.cfg.ignore\r\n\r\n# Now load which ever version of python you need. For example:\r\nmodule load apps\/anaconda3\/5.3.0\r\n\r\n# Load the proxy\r\nmodule load proxy\r\n\r\n# Check which version of python we have. We'll force the conda env to use that version.\r\npython --version\r\n    # Make a note of: 3.7.0\r\n\r\n# Create the conda env containing some basic python packages, including 'pip'.\r\n# It is important to use the 'pip' package installed inside you conda env so\r\n# that it knows to install pypi packages inside the env.\r\nconda create -n myenv<\/strong> python=3.7.0\r\n[y]\r\n\r\n# Activate the env\r\nconda activate myenv<\/strong>\r\n\r\n# Now install your pip package(s)\r\npip install --log pip.log packagename<\/em>\r\n  #\r\n  # Note: If pip reports errors about the proxy, you are using a very new\r\n  # version of pip. Try using the other proxy modulefile that adds\r\n  # 'http:\/\/' to the proxy settings:\r\n  # module swap proxy proxy2\r\n\r\n# You should now be able to use your pypi package:\r\npython\r\nimport packagename<\/em>\r\n<\/pre>\n
    pip\/pypi installation (outside of a conda env)<\/h3>\n
    An alternative to using conda environments is to install a package in to your home area using pip<\/code>. This will make the package available every time you run python, but makes it difficult to separate packages and dependencies for different tasks. The above conda environments method is recommended.<\/p>\n
    You will need to load the following modulefile before proceeding to allow access to the outside world:<\/p>\n
    module load proxy\r\n<\/pre>\n
    To install a package in to your home directory storage area:<\/p>\n
    pip install --user package<\/em>\r\n<\/pre>\n
    where package<\/code> is replaced with the name of the package you want. This will install the package to a hidden directory called .local<\/code> in your home directory. It should be automatically picked up by python, you can test thus:<\/p>\n
    python\r\nimport package<\/em>\r\nhelp (package<\/em>)\r\n<\/pre>\n
    Hints and Tips<\/h2>\n
    Got a handy tip? Please send it in to its-ri-team@manchester.ac.uk<\/a> …<\/p>\n
    Plotting graphs with Pyplot<\/h3>\n
    If you want to plot a graph to a PNG file, say, in batch, try the following (see this stackoverflow question and answer<\/a>):<\/p>\n
    # Create a file named graph.py:\r\n\r\nimport matplotlib as mpl\r\n# Agg backend will render without X server on a compute node in batch\r\nmpl.use('Agg')\r\nimport matplotlib.pyplot as plt\r\n\r\nfig = plt.figure()\r\nax = fig.add_subplot(111)\r\nax.plot(range(10))\r\n# Save the graph to a .png file (you cannot use plt.show() in a batch job)\r\nfig.savefig('temp.png')\r\n<\/pre>\n
    Then, to quickly test from the CSF login node, load the modulefile and submit a batch job (without writing a jobscript):<\/p>\n
    module load apps\/anaconda3\/5.2.0\r\nqsub -b y -cwd -V -j y -l short python .\/graph.py\r\n<\/pre>\n
    When job completes, view the image file on the login node using the Linux eog<\/code> tool:<\/p>\n
    eog temp.png\r\n<\/pre>\n
    Further info<\/h2>\n