Copying Files between (CIFS and NFS) Shares

NFS shares are available only on ITServices-run Linux systems, such as the CSF and iCSF. CIFS shares are not available on these systems but are typically available as network-drives on your desktop. This page details how to move files between a CIFS share and an NFS share.

The information here also applies if you have an NFS share and wish to move large number of files or data efficiently between that and your CSF/iCSF home directory (using rds-ssh to do this reduces the load on the compute system login nodes).

Use the RDS-SSH service.

Overview

Both MS Windows-style CIFS shares and Linux-style NFS shares from the Research Data Storage Service are available via a host named rds-ssh. So it is easy to copy files from one to the other (should you have access to both types of share) simply by accessing the rds-ssh host and doing the file copy there.

It is not necessary to map either share on your laptop or desktop to copy files from one share to another; the method below ensures file transfer takes place entirely within the data centre and is usually much faster than transfer over the campus network via your desktop/laptop.

How To Copy Files Efficiently between CIFS and NFS Shares

Obtain an account on the RDS-SSH service!
Ensure all your required CIFS RDS shares are mounted on RDS-SSH by running the following command on the RDS-SSH server:
```
mount-my-cifs
```
If anything is missing contact its-ri-team@manchester.ac.uk for help. Note that your NFS storage will already be mounted, it is only the CIFS storage that you have to mount because it requires your password to mount it – the sysadmins cannot bypass this security.
Log-in to the RDS-SSH service by using an SSH client (on MS Windows this could be Mobaxterm or PuTTY, on linux/MacOS use the command-line ssh), or an SFTP client and copy files from one share to another — see the examples below.

How NOT To Do it!

When copying files from one RDS share to another, it is important to ensure the copy is taking place on RDS-SSH and not via your local desktop/laptop.

At no point should you download the files to your desktop/laptop from one share only to upload it to another. This can be terribly slow because you usually have only a slow network connection to your desktop/laptop (and you actually transfer the files twice over that slow network – download then upload).

In particular, if you have mapped the shares to your desktop using an SSHFS program (e.g., WinSSHFS), do not copy files from one mounted share to another: you may be tempted to mount both shares on your desktop/laptop as networked drives (e.g., drives E: and F: in Windows) then copy files between these drives. This will do the transfer via your desktop even if it appears you are copying from one RDS share to another. This will be slow!

(SSHFS is only useful for transferring files to/from your desktop when you start with the files on your desktop – perhaps a colleague has given you pen-drive from which you want to upload data to the CSF/iCSF – or if you want to copy a file from the CSF on to your desktop).

Example — Using the rds-ssh server’s command-line

Here we log in to the rds-ssh server (using any ssh tool) and do the file copying on the rds-ssh server’s command-line:

ssh username@rds-ssh.itservices.manchester.ac.uk
         #
         # Use your own central username (and password when prompted)

We are now at the rds-ssh home directory. If you have a CSF/iCSF account this will be the same home directory used on those machines.

Your other RDS shares will be available in the /mnt/ directory. The RI Team will have given you the exact path names.

If you have a Windows-style/CIFS storage share then you may need to mount it on the RDS-SSH server first. This is because the CIFS storage requires your password to mount it and the sysadmins cannot bypass this security. So, the following command will check whether you need to do this and, if so, prompt you for your password. If your storage is already mounted it will give you a summary of where the storage is. Hence you can run this command whenever you like to check that your Windows-style (CIFS) storage is available:

Once you’ve logged in to rds-ssh, run:

[username@gorg ~]$  mount-my-cifs

Note that the rds-ssh server is actually made up of two servers named gorg and zola. So you may see zola mentioned in the prompt instead:

[username@zola ~]$  mount-my-cifs

If you log in to rds-ssh.itservices.manchester.ac.uk you will land on one of the above servers. Running mount-my-cifs after logging in will ensure your storage has been mounted on the particular server you have landed on.

Here we show an example of copying from one of our CIFS shares to our CSF home directory:

# Ensure we are at the home directory (same as our CSF home dir)
cd ~

# Copy all .dat files from our mounted CIFS share to a CSF directory (sampledata)
cp /mnt/mxyzabc1-ProjectName/*.dat  ~/sampledata/

# Copy an entire sub-directory from our CIFS share to another of our NFS shares
cp -r /mnt/mxyzabc1-ProjectName/results/  /mnt/fac01-data01/mxyzabc1/

The main thing to notice here is that your CIFS shares will be available via a directory name similar to /mnt/username-ProjectName, your other NFS shares will be in the /mnt/ directory and your CSF/iCSF home directory is also your home directory on rds-ssh. The usual linux command-line file management commands, e.g rsync, can all be used.

Here we show an example of using rsync from one of our CIFS shares to our CSF home directory:

# Ensure we are at the home directory (same as our CSF home dir)
cd ~

# Copy a directory from our mounted CIFS share to a CSF directory (sampledata)
rsync -avz /mnt/mxyzabc1-ProjectName/mydatafiles  ~/sampledata

Note that the rsync command is fussy about / characters at the end of directory names. If you put a / at the end of the source directory name, rsync will copy the files from inside the directory but won’t create an actual directory in your CSF home.

Example — Using the rds-ssh server’s GUI

A graphical file manager named xfe is installed on the rds-ssh server if you prefer not to use command-line tools. You must have an X11 server running on your desktop machine to use this (on Windows we recommend using MobaXTerm to log in to the rds-ssh server because it provides X11 for you without any extra setup).

ssh -X username@rds-ssh.itservices.manchester.ac.uk
         #
         # Use your own central username (and password when prompted)

Once you’ve logged in to rds-ssh, run:

[username@gorg ~]$  mount-my-cifs

Note that the rds-ssh server is actually made up of two servers named gorg and zola. So you may see zola mentioned in the prompt instead:

[username@zola ~]$  mount-my-cifs

Now start the file manager:

xfe

When the GUI pops up, press CTRL+F3 to open two panels (both initially showing the same directory) to make dragging and dropping files between directories easier. You can also open two panels from the menus via View -> Two Panels.

Initially we are at the rds-ssh home directory. If you have a CSF/iCSF account this will be the same home directory used on those machines.

Your other RDS shares will be available in the /mnt/ directory. The RI Team will have given you the exact path names. Navigate to that directory in the GUI and find your shares – CIFS shares will usually be named similar to /mnt/username-ProjectName, and your CSF/iCSF home directory is also your home directory on rds-ssh.

The image shows the xfe browser (opened on rds-ssh from a Windows desktop). Click on the image for a large version.

Last modified on February 28, 2022 at 3:08 pm by George Leaver