Known Issues and Workarounds – July 2024 OS upgrade

July 2024 – the iCSF Operating System was upgraded from Scientific Linux 7.9 to Alma 9.4. As a result, there’s a possibility certain pieces of software will require re-installing, or perhaps additional modulefiles will need to be loaded in your environment.

We list below, the known problems and workarounds discovered so far.

Your own testing of your jobs will help us. If you discover any such things in your own jobs, please do let us know!

PLEASE NOTE: The incline256 nodes will soon be retired and replaced entirely with 2TB nodes. Therefore we recommend connecting to and using the newly upgraded incline2000 (incline2000.itservices.manchester.ac.uk) nodes.

ssh -X incline2000.itservices.manchester.ac.uk

How to report an issue

If you would like to report a problem with a piece of software, please do ONE of the following (no need to do both).

  1. Fill out the following request form and do include the following.
    • Request Type = General Enquiry about HPC/HTC
    • System in Use = iCSF
    • Software (name & version) = Name of Software inc. Version
    • Additional Information = In the body of text please outline the error, be sure to include any error messages, relevant locations/directories, etc.

    OR

  2. Send and  email its-ri-team@manchester.ac.uk. In the subject line please include the the following,
    • Subject – iCSF – Software May2024 upgrade – nameOfSoftware + version“.
    • Email body – In the body of text please outline the error, be sure to include any error messages, location of job scripts, relevant locations, etc.

Known Issues and Workarounds

R & Rstudio not working

R/Rstudio fails to open.

Users are encouraged to use the latest version of R and Rstudio. The latest version of R addresses a number of security concerns found in previous versions. Older version of R will not run on the incline2000 nodes.

Known applications affected: R & Rstudio (follow link for more info.)

Solution

Please ensure you are connecting to incline2000 and loading the latest version of R and Rstudio

# Connect to incline2000
ssh -X incline2000.itservices.manchester.ac.uk

# Load latest version of R
module load apps/gcc/R/4.4.1

# Load latest version of rstudio
module load apps/binapps/rstudio/2024.04.2-any-r

Please read the R & Rstudio page for further info.

Host Key Error on incline/icsf Nodes

We have now taken some old nodes out of service. These were accessed via the addresses incline2000.itservices.manchester.ac.uk and icsf.itservices.manchester.ac.uk and incline256.itservices.manchester.ac.uk. Those addresses are still valid but they will now log you into one of the new incline2000 nodes. This may cause you to see an error when logging in. See below on how to fix it:

When logging in to the incline2000.itservices.manchester.ac.uk nodes using:

ssh -X username@incline2000.itservices.manchester.ac.uk
     #
     # (optional - the error may occur below with or without -X)

you may receive one or both the following errors about host identification:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ED25519 key sent by the remote host is
SHA256:1xTLr+ipUFWUHBeSq3ZF5lnOMGy73OzuQOzJFA2oZm8.
Please contact your system administrator.
Add correct host key in /home/username/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /home/username/.ssh/known_hosts:3
  remove with:
  ssh-keygen -f "/home/username/.ssh/known_hosts" -R "incline2000"
Host key for incline2000 has changed and you have requested strict checking.
Host key verification failed.

Solution

The easiest way to fix this if logging in from Linux (e.g., our X2GO desktop), or from MacOSX or in MobaXterm is to run the following command(s) in your terminal window (note: you must not be logged in to the iCSF at this point):

# Do this on your laptop/PC in a terminal NOT logged in to the iCSF
ssh-keygen -f "/home/username/.ssh/known_hosts" -R 10.99.203.69
ssh-keygen -f "/home/username/.ssh/known_hosts" -R incline2000.itservices.manchester.ac.uk
                #
                # Use the correct path and username reported in your error!
                # It may be different to that shown here and where username is
                # whatever username you use on YOUR laptop/PC.

You should then log in using ssh:

ssh -X username@incline2000.itservices.manchester.ac.uk

you will be asked to accept the new key:

The authenticity of host 'incline2000.itservices.manchester.ac.uk (10.99.203.69)' can't be established.
ECDSA key fingerprint is xx:xx:xx:xx:xx......:xx.
Are you sure you want to continue connecting (yes/no)? yes
                                                        #
                                                        # Type yes and press Enter

It is safe to do so – answer yes at the prompt. You will then be logged in to one of the 2TB compute nodes.

Last modified on August 1, 2024 at 9:22 am by Chris Grave