Tutorials

 

File Transfer

Estimated Time to Complete: 20 minutes

Overview

Working on the various compute platforms such as the iCSF and CSF you will often find yourself having to transfer files between your local computer and the platform you are working on. There are various methods of transferring files to and from the various compute platforms Service. This Tutorial will show how to transfer files using the recommended method between your local computer and any of the compute platforms. All methods take advantage of the Secure copy Protocol (scp). The scp command allows you to copy files over a ssh connection. This is useful if you want to transport files between a local desktop and remote host e.g. CSF/iCSF. The scp command uses ssh therefore any files copied in this way are done so in a secure manner via. The scp command can be used in three ways:

  1. Copy from a (remote) server to your computer
  2. Copy from your computer to a (remote) server
  3. Copy from a (remote) server to another (remote) server.

Requirements

  • Windows – WinSCP installed
  • Linux/MacOS – Some basic command line knowledge
  • CSF/iCSF Account
  • Sample Data – SampleData.zip
  • If Off-Campus you will need to be connected to the UoM VPN (Global Protect) with 2FA

Further information and instructions on other methods can be found by following this link.

WinSCP

Video Demonstration

Video WinSCP File Transfer

How To

    1. If you haven’t already download and extract the SampleData.zip dataset to your local desktop.
    2. Open WinSCP and from the Login window create a new session.

Please note: If the Login window does not appear you can create a new session by doing the following:Session > New Session

    1. Under the Login Window enter the following in
      • File Protocol: Select SCP
      • Host name: Enter one of the following hosts depending on your destination.
System Hostname used for Access
iCSF/incline incline256.itservices.manchester.ac.uk
CSF3 csf3.itservices.manchester.ac.uk
Zrek zrek.itservices.manchester.ac.uk
RDS-SSH rds-ssh.itservices.manchester.ac.uk
Condor submitter.itservices.manchester.ac.uk

Please note:For the purposes of this tutorial we will connect to incline256.itservices.manchester.ac.uk

      • Port: Enter22
      • User Name: Enter your central username
      • Password: Leave blank
      • Select Advanced Under the Connection option enter the following
          • Server Response timeout: Enter 60
          • Select OK

        </ul

      • Select Save
      • Select Login

Please note: The first time you connect using WinSCP you will likely receive the following message  ‘Continue connecting to an unknown server and add its host key to cache…‘ This is a fundamental feature of SSH, helping to prevent your data being intercepted by a third party by providing a known “key”  Select Yes

    • When prompted for Password: Enter your central password
  1.  The main WinSCP commander window should open this is similar to a Windows explorer window. You should notice the window is split into a left pane and right pane.
    • Left Pane = Local Computer
    • Right Pane = Remote Directory e.g. incline.
  2.  Now lets transfer a file. If you haven’t already download and extract the SampleData.zip dataset to your local desktop.
      • Now in left pane of the WinSCP commander window navigate to where you extracted the SampleData.zip.

    Please note: It the file is not appearing it maybe necessary to click the refresh button

    • Now in the right pane lets create a new directory for the file. Select >Directory> and give it a name e.g. uploads
    • Navigate into to the newly created directory
    • In order to transfer the SampleData directory simple drag’n’drop from one pane to the other
  3.  Now lets connect to incline and confirm the transfer was successful
    • $ ssh -X incline256.itservices.manchester.ac.uk
    • In you incline home directory list contents by using the ls command – You should notice the uploads directory has been created.
    • Lets list the contents of the uploads directory using the following command ls uploads – You should see the SampleData has successfully been transferred.
  4. Imagine you performed some analysis on the data and produced a plot. You should notice a Rplot.pdf file in the SampleData/Outputs folder lets copy that from incline back to our Downloads folder simply by drag’n’droping  the file from the left hand pane to the right hand pane using WinSCP

SCP via Terminal

The SCP utility is available via the Terminal appication in both the Linux and MacOS operarting systems.

In the below example we use incline256.itservices.manchester.ac.uk as our remote destination/host. This can be changed depending on the system in use see below table a for list of the available various.

System Hostname used for Access
iCSF/incline incline256.itservices.manchester.ac.uk
CSF3 csf3.itservices.manchester.ac.uk
Zrek zrek.itservices.manchester.ac.uk
RDS-SSH rds-ssh.itservices.manchester.ac.uk
Condor submitter.itservices.manchester.ac.uk

Video Demonstration

Video Create a New Session

How to

  1. If you haven’t already download SampleData.zip
  2. Open the Terminal Application
  3. Change directory to the location downloaded the SampleData
  • cd ~/Downloads
  • unzip SampleData.zip
  • ls
  1. Now lets copy the SampleData folder over to Incline (or other compute platform)
  • scp -r SampleData mzysscg3@incline256.itservices.manchester.ac.uk:~/uploads/

    PLEASE NOTE: The syntax of the scp command is as follows scp [options][source][Destination]. The destination directory also needs to exist prior to copying any data. In the above example we included the -r option so it will copy the directory recursively i.e. everything within the directory. The source file is the SampleData folder and the Destination is mzysscg3@incline256.itservices.manchester.ac.uk:~/uploads anything after : denotes where the file/will be copied to. In this particular example we are coping to ~/uploads the ~ denotes our home directory . If you wanted you could specify a different directory e.g. :/some/other/directory/

  1. Now lets hop onto incline256 to check the folder has been copied successfully over
  • ssh -X username@incline256.itservices.manchester.ac.uk
  • cd ~ Moves us into our home directory
  • ls You should see the SampleData directory listed
  • Let check the contents of the SampleData directory ls SampleData you should see everything was successfully copied over.
  1. Now imagine you performed some analysis on the data and produced a plot. You should notice a .pdf file in the SampleData/Outputs folder lets copy that from incline to our desktop.
  • On our desktop/laptop lets first create a directory for the file to be copied into e.g.
  • mkdir -p ~/Downloads/fromIncline the -p creates any parent directories
  • scp -r mzysscg3@incline256.itservices.manchester.ac.uk:~/SampleData/Outputs/Rplots.pdf ~/Downloads/fromIncline

Next Steps

You should now be able to transfer files between your local computer and various computational platforms. Have a go a transferring some some more files and directories.

If you experienced any problems or require further support then please email its-ri-team@manchester.ac.uk

Further Reading

Virtual Desktop Service
WinSCP
Research IT Documentation on File Transfers.

Feedback