{"id":7560,"date":"2024-05-07T17:07:38","date_gmt":"2024-05-07T16:07:38","guid":{"rendered":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/?page_id=7560"},"modified":"2025-10-08T14:24:30","modified_gmt":"2025-10-08T13:24:30","slug":"pycharm","status":"publish","type":"page","link":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/software\/tools\/pycharm\/","title":{"rendered":"PyCharm"},"content":{"rendered":"

Overview<\/h2>\n

PyCharm<\/a> is an integrated development environment (IDE) for data science and & web development and is used for programming in Python. It provides code analysis, a graphical debugger, an integrated unit tester, integration with version control systems.There is a Full-fledged Professional version and Free Community version.<\/p>\n

PyCharm is written in Java and Python and comes bundled with Java.<\/p>\n

PyCharm is not installed on the CSF<\/strong>. Instead, you will connect PyCharm installed on your PC\/laptop to a Python Interpreter installed on the CSF.<\/p>\n

\nPlease note: this “remote access” feature is currently available only in the Professional Edition of PyCharm, not in the Free Community version. You will require a PyCharm license for your PC\/laptop to use that version.<\/p>\n

Research IT DO NOT<\/strong> provide licenses.\n<\/div>\n

This has been tested with PyCharm Professional version 2024.1.1 running on a PC\/laptop with Windows 10\/11, Ubuntu 22.04 and Xubuntu 22.04. It should work in other operating systems as well but we are unable to debug problems you your<\/em> PC\/latop – please contact the the main IT Helpdesk<\/a> for help with that.<\/p>\n

Restrictions on use<\/h2>\n

Please ensure you comply with the PyCharm licensing requirements on your<\/em> PC\/laptop. While the Community edition is free to use for personal and commercial development, licensed under Apache 2.0, the full featured Professional Edition needs commercial license to be purchased.<\/p>\n

This page discusses running PyCharm in your Laptop\/Mac\/PC and connecting it to CSF3.<\/p>\n

Set up overview<\/h2>\n
    \n
  1. PyCharm must be installed on your<\/em> PC\/laptop first, and a project<\/em> created. You do not install\/run PyCharm on the CSF.<\/li>\n
  2. You will then run Python on the CSF to act as the “SSH Remote Interpreter” to which PyCharm (running on your<\/em> PC\/laptop) will connect. If your wish to use specific Python virtual environment on the CSF3, it can be activated on the CSF so that PyCharm uses that virtual env.<\/li>\n
  3. The project<\/em> folder configured on your<\/em> PC\/laptop gets automatically synced to a respective project folder on the CSF, once the above setup is complete.<\/li>\n
  4. If a Python code is run in the PyCharm IDE at this stage, it will be executed on the login node of the CSF3, which is NOT permitted<\/strong>. Instead, the codes must be submitted as job to the CSF3 batch system, which will then run the code in the usual manner – on a suitable compute node. This is done by the Python_Submit.py<\/code> given below.You must<\/strong> copy the script to your project folder and open it in PyCharm. The Python_Submit.py<\/code> file must then be edited to describe your job (e.g., number of CPU cores, high memory, GPUs) and then run in PyCharm IDE. This will submit your project’s python code as a job. The qstat<\/code> command will also be run to show your job queue, all from within PyCharm.<\/li>\n
  5. You can then use the terminal provided in PyCharm IDE, or a separate terminal, to ssh to CSF3 as normal, and monitor the submitted job and analyse the output.<\/li>\n<\/ol>\n

    Connect PyCharm installed on PC\/laptop to a Python Interpreter on the CSF<\/h2>\n

    Start PyCharm Professional on your<\/em> Laptop\/Mac\/PC, then open your project (or create a new one.)<\/p>\n

    If you haven’t created a project yet, create one first.<\/p>\n

    Then, go to:
    \nFile > Settings<\/code>
    \nProject > Python Interpreter<\/code>
    \nAdd Interpreter > On SSH<\/code>
    \n(see images below)
    \n    \"screenshot\"<\/a><\/p>\n

    \"screenshot\"<\/a>
    \nSelect New<\/code> and type in:
    \nHost: csf3.itservices.manchester.ac.uk<\/code>
    \nPort: 22<\/code>
    \nUsername: your-csf3-username<\/em><\/code>
    \n(see image below)<\/p>\n

    \"screenshot\"<\/a>
    \nThen hit Next<\/code>:<\/p>\n

    Type your password in the Password field and hit Next<\/code> in the next step.<\/p>\n

    At the Duo two-factor login open your Duo app<\/em> and then in your University of Manchester<\/em> account click Show passcode<\/code> and hit Refresh passcode<\/code>.<\/p>\n

    DO NOT type 1 and hit enter to confirm from Duo. This method will not work. View and type the Passcode instead.<\/strong><\/p>\n

    Type the refreshed passcode in the Passcode field and hit Next<\/code>.
    \nEven if it fails for the first time, try again.
    \nYou might need to type in your password and passcode twice.<\/em><\/p>\n

    \"screenshot\"<\/a><\/p>\n

    After successful connection you may see something like this:<\/p>\n

    \"screenshot\"<\/a><\/p>\n

    If you do not see the above messages, don’t worry, the connection is most probably established when the Next<\/code> button turns to blue colour.<\/p>\n

    Hit Next<\/code>.<\/p>\n

    In the next stage (4\/4)<\/code>, if you have created a conda environment on the CSF that you wish to use, select Conda Environment<\/code> in the left panel.<\/p>\n

    Here you have the option for selecting other virtual environments<\/em> as well.
    \nGive it some time to search and then click on the small folder icon in the Conda Executable<\/code> field.<\/p>\n

    \"screenshot\"<\/a><\/p>\n

    Now browse and select the Conda Executable in CSF3.<\/p>\n

    \"screenshot\"<\/a><\/p>\n

    TIP:<\/strong>
    \nTo find the path of the Conda Executable<\/em> of your environment, do something like the following in a MobaXterm\/SSH terminal connected to the CSF:<\/p>\n

    module load apps\/binapps\/anaconda3\/2021.11\r\nsource activate my_conda_env<\/em>\r\nwhich conda\r\n<\/pre>\n
    This will show the path of the conda executable<\/em> which you need to browse to and select.
    \nFor example:<\/p>\n
    \/opt\/apps\/apps\/binapps\/anaconda3\/2021.11\/condabin\/conda\r\n<\/pre>\n
    It might be different for your conda version and environment.<\/p>\n
    Now hit the Load Environments<\/code> button on the right of Conda Executable<\/code> field.<\/p>\n
    You will be able to see all of the conda environments that you have created on the CSF under: Use existing environment<\/code>
    \nFrom the list, select the one that you need.<\/p>\n
    \"screenshot\"<\/a><\/p>\n
    Then click on the small folder icon at the end of Sync folders<\/code> field.<\/p>\n
    \"screenshot\"<\/a><\/p>\n
    In the new window that pops up, set the path to your CSF3 folder where you want to keep your project files on CSF3 synced. For example, you may wish to store your sync’d projects in ~\/pycharm_projects\/my_project_1<\/em>\/<\/code>.<\/p>\n
    \"screenshot\"<\/a>
    \nNext hit OK<\/code><\/p>\n
    Keep Automatically upload project files to the server<\/code> checked on, and hit Create<\/code>.<\/p>\n
    \"screenshot\"<\/a><\/p>\n
    Setup is now complete!<\/strong><\/p>\n
    At this stage whenever you create new files, it gets saved locally on your<\/em> PC\/laptop project folder and<\/em> also gets synced\/uploaded to the CSF project folder.<\/p>\n
    The Python_Submit.py code<\/h2>\n
    Any Python code that you have opened in PyCharm IDE at this stage, if run in PyCharm IDE, would start running on the CSF login node! This is NOT<\/strong> permitted.We DO NOT<\/strong> want this to happen – your must NOT<\/strong> run your PyCharm project code on the CSF login node! Login nodes are for logging into the CSF3 and submitting jobs to the batch system.<\/p>\n<\/div>\n
    Instead, we will use a helper<\/em> Python script to run your<\/em> python code via the batch system. The Python_Submit.py<\/code> script below will do this for you, but you will need to edit the file. You can use a new copy of this script in each PyCharm project. That way, projects can request different CSF resources (e.g., one project may require a GPU node, while another project may need a high-memory node.)<\/p>\n
    When you run the Python_Submit.py<\/code> script in the PyCharm IDE, it will create a jobscript (to run your python project code) and submit that jobscript to the CSF3 batch system.<\/p>\n
    It will then display the squeue<\/code> output to show your job queue.<\/p>\n
    Save a copy and Edit the Python_Submit.py script<\/h3>\n
    You should download Python_Submit.py from here<\/a> or cut-n-paste a copy of the code below in to a file named
    \nPython_Submit.py<\/code> in your project directory (remember that when you create this file on your PC\/laptop in your PyCharm project, PyCharm will upload it to the CSF folder that contains a sync’d copy of your project files.)<\/p>\n
    The editable parameters of the script must be modified by you to suit your<\/em> requirements -e.g., whether you need more cores, or a high memory node or a GPU node. The editable parameters have comments added for ease of understanding.<\/p>\n
    \r\n### OCT 2025: updated to be compatible with the upgraded SLURM batch system\r\n\r\nimport os\r\n\r\n## Do not remove any of the quotes \" \" (for single-line strings) or \"\"\" \"\"\" (for multi-line strings)\r\n\r\n# EDIT ME: Give the name for your jobscript file here\r\njobscript = \"MyPyCharmJob\"\r\n\r\n# EDIT ME: Give the maximum wallclock time that your job will be allowed to run for\r\n# Use the SLURM format d-hh:mm:ss (days-hours:minutes:seconds) OR d-hh OR mm\r\n# WITH Double Quotes in all cases\r\nwallclock = \"4-00:00:00\"\r\n\r\n# EDIT ME: Give the name of the SLURM partition where the script will run\r\n# (OCT2025) Available options including the double-quotes: \"serial\", \"multicore\",\r\n# \"multicore_small\", \"himem\", \"vhimem\", \"gpuA\", \"gpuL\" and \"gpuA40GB\"\r\npartition = \"multicore\"\r\n\r\n# EDIT ME: Specify no. of cores (WITHOUT double-quotes) required for your job\r\ncores = 2\r\n\r\n# EDIT ME: Only if  himem and vhimem partitions chosen\r\n# Write the amount of memory requested in Gigabytes (WITHOUT double-quotes)\r\n# Ignored if the partition is not a GPU partition\r\nmem = 1000\r\n\r\n# EDIT ME: GPU jobs only (gpuL or gpuA partition chosen)\r\n# Number of GPUs requested. Ignored if the partition is not a GPU partition\r\ngpus = 1\r\n\r\n# EDIT ME: Load necessary Anaconda module and any others you require\r\n# Do not remove any of the multi-line string quotes \"\"\" \"\"\".\r\nmodule_command = \"\"\"\r\nmodule purge\r\nmodule load apps\/binapps\/anaconda3\/2024.10\r\n\"\"\"\r\n\r\n# EDIT ME: Set to True if your python code uses OpenMP for multithreading\r\n# It will set the environment variable OMP_NUM_THREADS=$SLURM_NTASKS\r\n# Available options: True or False (without quotes, First Letter Capital)\r\nopenMP = True\r\n\r\n# EDIT ME: Activate your conda environment (set to \"\" to ignore conda environments)\r\nconda_command = \"source activate my_conda_env\"\r\n\r\n# EDIT ME: The command to run your Python code goes here:\r\n# (ensure you have saved your python code in the working\/project directory)\r\npython_command = \"python MyPythonCode.py\"\r\n\r\n# De-activate conda environment after python code has completed running (set to \"\" to ignore conda environments)\r\nconda_deactivate_command = \"source deactivate\"\r\n\r\n################################################################################\r\n# DO NOT EDIT BEYOND THIS LINE #################################################\r\n################################################################################\r\n\r\nprint('Submitting ' + jobscript + '...')\r\n\r\npwd = os.getcwd()\r\n\r\n# Jobscript file starts\r\njob_sh = '#!\/bin\/bash --login' + '\\n'\r\njob_sh += '#SBATCH -J ' + jobscript + '\\n'\r\njob_sh += '#SBATCH -t ' + wallclock + '\\n'\r\njob_sh += '#SBATCH -p ' + partition + '\\n'\r\n\r\nif partition != 'serial':\r\n    job_sh += '#SBATCH -n ' + str(cores) + '\\n'\r\n\r\nif (partition == 'himem') or (partition == 'vhimem') and (mem > 0):\r\n    job_sh += '#SBATCH --mem=' + str(mem) + 'G\\n'\r\nelif (partition == 'gpuA') or (partition == 'gpuL') or (partition == 'gpuA40GB') and (gpus > 0):\r\n    job_sh += '#SBATCH -G ' + str(gpus) + '\\n'\r\n\r\n# if present, add module load command\r\nif module_command != '':\r\n    job_sh += module_command + '\\n'\r\n\r\n# Help to ensure correct number of OpenMP cores used\r\nif openMP:\r\n    job_sh += '\\n' + 'export OMP_NUM_THREADS=$SLURM_NTASKS' + '\\n'\r\n\r\n# if present, activate requested conda environment\r\nif conda_command != '':\r\n    job_sh += conda_command + '\\n'\r\n\r\n# if present, add Python code execution command\r\nif python_command != '':\r\n    job_sh += '\\n' + python_command + '\\n'\r\n\r\n# if present, de-activate conda environment after execution of Python code\r\nif conda_deactivate_command != '':\r\n    job_sh += '\\n' + conda_deactivate_command + '\\n'\r\n\r\n# Print job related useful info to output file\r\njob_sh += \"\"\"printf \"$JOB_ID\\\\n\"\r\nhostname\r\ndate\r\n\"\"\"\r\n##################################################################\r\n# Save jobscript and submit to batch\r\n##################################################################\r\n# write jobscript to a file in the PWD\r\njob_filename = os.path.join(pwd, jobscript + '.sbatch')\r\nwith open(job_filename, 'w') as job_file:\r\n    job_file.write(job_sh)\r\n\r\njob_submitted = os.system('sbatch ' + job_filename)\r\nif job_submitted == 0:\r\n    print('Job submitted to the CSF3 SLURM batch system. Here is the summary of your submitted jobs:')\r\n    os.system('squeue --me')\r\nelse:\r\n    print('Failed to submit job!')\r\n\r\n<\/pre>\n
    Further info<\/h2>\n