Copying Files between (CIFS and NFS) Shares

Introduction

Reminder: In the following notes, the term “share” means a “storage area” – i.e., Research Data Storage that has been allocated to you.

Research Data Storage is provided in two “flavours”: NFS shares are available only on Research IT run Linux systems, such as the compute platforms – the CSF and iCSF. CIFS(SMB) shares are not available on these systems but can be mapped as network drives on your desktop/laptop.

Some people have both types of storage. This page details how to transfer files between a CIFS(SMB) share and an NFS share.

The information here also applies if you have additional Research Data Storage NFS shares (e.g., extra storage on the HPC platforms) and wish to move a large number of files efficiently between that and your CSF/iCSF home directory. By using the “rds-ssh” server, you reduce the load on the compute system login nodes.

rds-ssh Overview

Both flavours of storage – CIFS(SMB) shares and Linux NFS shares – are available on a server named rds-ssh. This makes it easy to copy files from one flavour of storage to the other simply by logging in to the rds-ssh server and doing the file copy there.

It is not necessary to map either share onto your desktop/laptop to copy files from one share to another. In fact we discourage doing this. By doing the copy on the rds-ssh server (details below) you ensure the file transfer takes place entirely within the data-centre and is usually much faster than transfers over the campus network (or your home broadband) via your desktop/laptop.

How To Copy Files Efficiently between CIFS(SMB) and NFS Shares

This is a summary of the procedure to efficiently copy files from one storage share to another – by using the rds-ssh server:

  1. Obtain an account on the RDS-SSH service! (in short, email its-ri-team@manchester.ac.uk and tell us the names of your storage areas.)
  2. Log-in to the RDS-SSH service by using an SSH client (on MS Windows this could be Mobaxterm or PuTTY; on Linux/MacOS use the command-line ssh)
  3. On the rds-ssh server, run the following command (to ensure all of your required CIFS(SMB) shares are available): 
    mount-my-cifs

    If anything is missing contact its-ri-team@manchester.ac.uk for help. Note that your NFS storage will already be available, it is only the CIFS(SMB) storage that you have to mount because it requires your password to mount it – the sysadmins cannot bypass this security.

  4. On the rds-ssh server, copy files from one share to another — see the examples below.

See below for more details on how to copy files between your storage areas once you’ve done the above steps. You can do the copy at the Linux command-line or using a graphical file browser.

How NOT To Do it!

When copying files from one RDS share to another, it is important to ensure the copy is taking place on the RDS-SSH server and not via your local desktop/laptop.

At no point should you download the files to your desktop/laptop from one share then upload it to another! This can be terribly slow because you usually have only a slow network connection to your desktop/laptop (and you actually transfer the files twice over that slow network – download then upload).

If you have mapped-as-a-drive one or both shares to your desktop, it is still quicker to log in to the rds-ssh server and do the copy there. Doing the copy on your desktop/laptop means the files go over your local network (e.g., the campus network or your home broadband.) This could be slow. By logging in to the rds-ssh server, the files only ever go over the fast data-centre network.

How TO do it method 1 — Using the rds-ssh server’s command-line

Here we log in to the rds-ssh server (using any ssh tool) and do the file copying on the rds-ssh server’s command-line:

ssh username@rds-ssh.itservices.manchester.ac.uk
         #
         # Use your own central username (and password, then DUO, when prompted)

We are now at the rds-ssh home directory. If you have a CSF/iCSF account this will be the same home directory used on those machines.

Your other RDS shares will be available in the /mnt/ directory. The RI Team will have given you the exact path names.

Once you’ve logged in to rds-ssh, if your have CIFS(SMB) storage shares, run:

[username@gorg ~]$  mount-my-cifs
                         #
                         # If asked, enter your UoM password to make the CIFS(SMB) storage available.
                         # The Research IT sysadmins cannot do this for you.
  • Note that the rds-ssh server is actually made up of two servers named gorg and zola. So you may see zola mentioned in the prompt instead: 
    [username@zola ~]$  mount-my-cifs

    If you log in to rds-ssh.itservices.manchester.ac.uk you will land on one of the above servers. Running mount-my-cifs after logging in will ensure your storage has been mounted on the particular server you have landed on.

Copying files using the cp command

Here we show an example of copying from one of our CIFS shares to our CSF home directory:

# Ensure we are at the rds-ssh home directory (same as our CSF/iCSF home dir)
cd ~

# Copy all .dat files from our mounted CIFS share to a CSF directory (sampledata)
# Research IT will tell you the correct path for your storage share
cp /mnt/cifs-fac01/mxyzabc1-ProjectName/*.dat  ~/sampledata/

# Copy an entire sub-directory from our CIFS share to another of our NFS shares
# Research IT will tell you the correct path for your storage shares
cp -r /mnt/cifs-fac01/mxyzabc1-ProjectName/results/  /mnt/fac01-data01/mxyzabc1/

The main thing to notice here is that your CIFS(SMB) shares will be available via a directory name similar to /mnt/cifs-fac01/username-ProjectName (or something similar), your other NFS shares will be in the /mnt/ directory and your CSF/iCSF home directory (if you have an account on the central compute platforms) is also your home directory on rds-ssh.

The usual Linux command-line file management commands, e.g rsync, can all be used.

Copying files using the rsync command

Here we show an example of using rsync from one of our CIFS(SMB) shares to our CSF home directory:

# Ensure we are at the rds-ssh home directory (same as our CSF/iCSF home dir)
cd ~

# Copy a directory from our mounted CIFS share to a CSF directory (sampledata)
# Research IT will tell you the correct path for your storage share
rsync -av /mnt/cifs-fac01/mxyzabc1-ProjectName/mydatafiles  ~/sampledata
                                                          #
                                                          # NO / here. This is important
                                                          # Using a / here changes the
                                                          # behaviour of rsync!

Note that the rsync command is fussy about / characters at the end of directory names. If you put a / at the end of the source directory name, rsync will copy the files from inside the directory but won’t create an actual directory in your CSF home.

rsync -av dirA ~/targetDir
              #
              # No / here.
              # The result of this command is
              # ~/targetDir/dirA/all files from dirA

rsync -av dirA/ ~/targetDir
              #
              # / here.
              # The result of this command is
              # ~/targetDir/all files from dirA

How TO do it method 2 — Using the rds-ssh server’s GUI app

A graphical file manager named xfe is installed on the rds-ssh server if you prefer not to use command-line tools. You must have an X11 server running on your desktop machine to use this (on Windows we recommend using MobaXTerm to log in to the rds-ssh server because it provides X11 for you without any extra setup).

Log in to rds-ssh as follows (from MobaXTerm or linux/MacOS command-line):

ssh -X username@rds-ssh.itservices.manchester.ac.uk
     #   #
     #   # Use your own central username (and password when prompted)
     #
     # Mac users: Please try -Y instead if you have problems with the GUI.

Once you’ve logged in to rds-ssh, if your have CIFS(SMB) storage shares, run:

[username@gorg ~]$  mount-my-cifs
                         #
                         # If asked, enter your UoM password to make the CIFS(SMB) storage available.
                         # The Research IT sysadmins cannot do this for you.
  • Note that the rds-ssh server is actually made up of two servers named gorg and zola. So you may see zola mentioned in the prompt instead: 
    [username@zola ~]$  mount-my-cifs

    If you log in to rds-ssh.itservices.manchester.ac.uk you will land on one of the above servers. Running mount-my-cifs after logging in will ensure your storage has been mounted on the particular server you have landed on.

Now start the file manager:

xfe

When the GUI pops up, press CTRL+F3 to open two panels (both initially showing the same directory) to make dragging and dropping files between directories easier. You can also open two panels from the menus via View -> Two Panels.

Initially we are at the rds-ssh home directory. If you have a CSF/iCSF account this will be the same home directory used on those machines.

Your other RDS shares will be available in the /mnt/ directory. The RI Team will have given you the exact path names. Navigate to that directory in the GUI and find your shares – CIFS shares will usually be named similar to /mnt/cifs-fac01/username-ProjectName (or something similar), and your CSF/iCSF home directory is also your home directory on rds-ssh.

The image shows the xfe browser (opened on rds-ssh from a Windows desktop). Click on the image for a large version.
XFE file browser
You can now drag and drop files between the two storage areas.

Last modified on October 3, 2024 at 9:18 am by George Leaver