Linux/Mac command-line tools
Command-line Tools
Linux and Mac users can use the command-line to transfer files. While GUI programs do exist, we find the command-line more reliable, if a little more difficult to begin with.
Open a Terminal app on your Linux or Mac desktop to give yourself a command-line. The following commands can then be used. Please note, type them carefully (or cut-n-paste). The commands are sensitive to where the spaces occur and additional characters.
In all of the commands below, replace username with your own IT username.
rsync
The rsync command is used to transfer files securely and efficiently between your PC/laptop and CSF. No installation is usually needed on Linux or Mac systems (it is there by default). It has some nice features for resuming interrupted transfers (no need to transfer everything again) and can be used to keep folders on your PC and the CSF in-sync – i.e., it can detect which files are new or have changed and only copy those.
The following commands are run on your PC/laptop, NOT the CSF. Please type them carefully (notice the : between the CSF address and the file or folder names after it).
- Upload a file from your local machine to your CSF home directory (the
~at the end is short-hand for your CSF home folder)rsync -avz myinput.dat username@csf3.itservices.manchester.ac.uk:~
- Upload a file from your local machine to your CSF scratch directory
rsync -avz myinput.dat username@csf3.itservices.manchester.ac.uk:~/scratch/
- Download a file from your CSF home directory to your local machine (don’t forget the
.at the end to mean the current folder on your PC)rsync -avz username@csf3.itservices.manchester.ac.uk:~/myresult.dat .
- Download a file from your CSF scratch directory to your local machine (don’t forget the
.at the end to mean the current folder on your PC)rsync -avz username@csf3.itservices.manchester.ac.uk:~/scratch/myresult.dat .
The -avz flag on the rsync command will run it in archive mode (it will transfer all files in a folder if a folder name is given instead of a filename and it will preserve the timestamps on files), it will be verbose (displaying filenames as it transfers them) and it will compress traffic to try to reduce the transfer time (files will be compressed before transfer and decompressed when received). Please check the rsync documentation using man rsync on your system.
CSF scratch cleanup and timestamps
If transferring files to your CSF scratch area, you may prefer to apply today’s timestamp to the file, rather than using the original timestamp on the source file. This is so that your file will have the full 3-months in the CSF scratch area before the automatic scratch cleanup policy deletes the file! In this case use:
rsync -rlvz myinput.dat username@csf3.itservices.manchester.ac.uk:~/scratch/
The -rl flag replaces the -a flag.
scp or sftp
The commands scp or sftp will both transfer files in a similar manner to rsync, although rsync offers more options.
- Upload a file from your local machine to your CSF home directory (don’t forget the
:at the end)scp myinput.dat username@csf3.itservices.manchester.ac.uk:
- Upload a file from your local machine to your CSF scratch area
scp myinput.dat username@csf3.itservices.manchester.ac.uk:~/scratch/
- Download a file from your CSF home directory to your local machine (don’t forget the
.at the end to mean the current folder on your PC)scp username@csf3.itservices.manchester.ac.uk:results.out .
- Download a file from your CSF scratch area to your local machine
scp username@csf3.itservices.manchester.ac.uk:~/scratch/myresults.dat .
Remember that all of the above commands are run in a terminal on your laptop/desktop that IS NOT logged-in to the CSF i.e. the commands are run on your local machine not on the CSF.
More information
Please see the man for the command for more help (e.g., man rsync). Alternatively have a look at the online docs: