Page Contents

PyCharm

Overview

PyCharm is an integrated development environment (IDE) for data science and & web development and is used for programming in Python. It provides code analysis, a graphical debugger, an integrated unit tester, integration with version control systems.There is a Full-fledged Professional version and Free Community version.

PyCharm is written in Java and Python and comes bundled with Java.

PyCharm is not installed on the CSF. The following instruction is for adding/connecting Python Interpreter in CSF3 to PyCharm installed in users laptop/Mac/PC. Please note that this feature is currently available only in the Professional Edition of PyCharm, not in the Free Community version.

This has been tested with PyCharm Professional version 2024.1.1 running in user devices with Windows 10/11, Ubuntu 22.04 and Xubuntu 22.04 operating systems and should work in other operating systems as well.

Restrictions on use

While the Community edition is free to use for personal and commercial development licensed under Apache 2.0, the full featured Professional Edition needs commercial license to be purchased.

PyCharm must not be run on login nodes of CSF3. However it can be run in an interactive session in CSF3.

This page discusses running PyCharm in your Laptop/Mac/PC and connecting it to CSF3.

Set up overview

PyCharm must be installed in the user device first and a project created. PyCharm is not needed to be installed/run on CSF3.

Next we will connect users account in CSF3, and use Python in CSF3 account as the SSH Remote Interpreter of the PyCharm project in the Laptop. If a specific virtual environment is used by the user in CSF3, it can be used as well.

The project folder configured in laptop gets automatically synced with respective project folder in CSF3 once the above setup is complete.

If a Python code is run in PyCharm IDE at this stage, it will be executed in the login node of the CSF3 which we don’t want. We want the codes to be submitted as job to the CSF3 batch system, which will then run the code in suitable compute nodes. This is done by the Python_Submit.py given below. Users must copy the code to their project folder and open it in PyCharm. The Python_Submit.py, which is self descriptive with sufficient comments, must be edited as per the requirement and then run in PyCharm IDE. This will submit the python code of the user as a job and will print the qstat, all from within PyCharm.

User can then use terminal provided in PyCharm IDE or a separate terminal to ssh to CSF3 and monitor the submitted job and analyse the output.

Steps for adding/connecting Python Interpreter in CSF3 to PyCharm installed in users laptop

Start PyCharm Professional in your Laptop/Mac/PC and open your project.
If you haven’t created a project yet, create one first.

Next go to:

File > Settings

Project > Python Interpreter

Add Interpreter > On SSH

Select New and type in:
Host: csf3.itservices.manchester.ac.uk
Port: 22
Username: [your-csf3-username]
Then hit Next

Type your password in the Password field and hit Next in the next step.

At the Duo two-factor login open Duo app and click Show passcode and hit Refresh passcode it.
*Do not type 1 and hit enter to confirm from Duo. This method will not work. View and type the Passcode instead.
Type the passcode in the Passcode field and hit Next.
Even if it fails for the first time, try again.
You might need to type in your password and passcode twice.

After successful connection you will see something like this:

Hit Next.

In the next stage (4/4) select Conda Environment in the left panel, if you have created and use any conda virtual environment for your work in CSF3.
Here you have the option for selecting other virtual environments as well.
Give it some time to search and then click on the small folder icon in the Conda Executable field.

Now browse and select Conda Executable in CSF3.

TIP:
To find the path of the Conda Executable of your environment, do something like the following in a MobaXterm/SSH terminal connected to CSF3:

module load apps/binapps/anaconda3/2021.11
source activate my_conda_env
which conda

This will show the path of the conda executable which you need to browse to and select.
For example:
/opt/apps/apps/binapps/anaconda3/2021.11/condabin/conda
It might be different for your conda environment

Now hit Load Environments button on the right of Conda Executable field.
You will be able to see all the conda environments that you have created in CSF3 under: Use existing environment
From the list, select the one that you need.

Then click on the small folder icon at the end of Sync folders field.

In the new window that pops up, set the path to your CSF3 folder where you want to keep your project files in CSF3 synced.

Keep Automatically upload project files to the server checked on and hit Create.

Next hit OK

Setup is complete.

At this stage whenever you create new files, it gets saved locally in the Laptop/Mac/PC project folder and also gets synced/uploaded to CSF3 project folder.

The Python_Submit.py code

Any Python code that you have opened in PyCharm IDE at this stage, if run in PyCharm IDE, will start running in the login node of CSF.

We do not want this to happen as this could overload the login nodes, which are not meant to run your codes/programs.
Login nodes are for logging into the CSF3 and submitting jobs to the Batch system of CSF3, which in turn runs your job in suitable compute nodes under it.

To reduce complexity you can use the following Python_Submit.py, which can submit your codes to the CSF3 batch system.

Python_Submit.py when run in PyCharm IDE, will create a submit script suitable for your python code and submit it to the CSF3 batch system.

It will then display the qstat.

The editable parameters of Python_Submit.py must be modified by you to suit your requirements.
The editable parameters have necessary comments added for ease of understanding.

Python_Submit.py

import os
import time
import shutil

pwd = os.getcwd()

## Do not remove any of the triple quotes """ """

# Give the name for your job here
jobscript = 'MyPyCharmJob'

# Request high memory for CPU jobs *only.
# Available options : "No" or "mem256" or "mem512" (*including the double quotes)
memory_request = "No"

# Specify no. of cores required for your job
cores = 2

# Set to True if you want to use GPU
use_gpu = False

# Load necessary Anaconda module
module_command = """module load apps/binapps/anaconda3/2021.11 """

# Activate your conda environment
conda_command = """source activate my_conda_env """

# Your command to run your Python code goes here:
# First save your python code in the working/project directory
python_command = """python MyPythonCode.py """

# De-activate conda environment after python code has completed running
conda_deactivate_command = """source deactivate """

################################################################################
# DO NOT EDIT BEYOND THIS LINE #################################################
################################################################################

print
'Submitting ' + jobscript + '...'

# Jobscript file starts
job_sh = """#!/bin/bash --login
#$ -cwd
#$ -N """ + jobscript + """
"""
if cores > 1:
    job_sh += """#$ -pe smp.pe """ + str(cores) + '\n'

# if additional memory requested
if memory_request == "mem256":
    job_sh += """#$ -l """ + memory_request + '\n'

if memory_request == "mem512":
    job_sh += """#$ -l """ + memory_request + '\n'

## Only the below "-l" line may be edited to select between v100 and A100 GPUs and their numbers
# if you need 2 GPUs change v100=1 to v100=2
# if you need A100 change to a100=1 or a100=2
if use_gpu:
    job_sh += """### GPU requested
#$ -l v100=1
"""

# Un-comment for multithreaded jobs
job_sh += """export OMP_NUM_THREADS=$NSLOTS """

# if present, add module load command
if module_command != '':
    job_sh += '\n' + module_command + '\n'
# if present, activate requested conda environment
if conda_command != '':
    job_sh += '\n' + conda_command + '\n'
# if present, add Python code execution command
if python_command != '':
    job_sh += '\n' + python_command + '\n'
# if present, de-activate conda environment after execution of Python code
if conda_deactivate_command != '':
    job_sh += '\n' + conda_deactivate_command + '\n'



# Print job related useful info to output file
job_sh +="""printf "$JOB_ID\\n"
hostname
date
printf "\\n"
"""

# write jobscript to a file in the PWD
job_filename = os.path.join(pwd, jobscript + '.sh')
with open(job_filename, 'w') as job_file:
    job_file.write(job_sh)

# submit the job and print qstat
print
''
job_submitted = os.system('qsub ' + job_filename)
if job_submitted == 0:
    print
    'Job submitted to the CFS3 SGE batch system. Here is the summary of your submitted job:'
    os.system('qstat')

Further info

Next time you start PyCharm after closing it or restarting laptop/Mac/PC, you will have to authenticate again using password and passcode
PyCharm Website/

Updates

None.

Last modified on May 13, 2024 at 4:02 pm by Abhijit Ghosh

Page Contents