wget

Overview

wget is a command-line program to download files from remote web servers. It can be used on the CSF to download datasets and source code, for example, from remote sites.

The CSF login node does not have a direct connection to the outside world. Hence downloads will be done via the University proxy server. wget has been configured to use this proxy – you don’t need to do anything.

Restrictions on use

All users may use this software. However, please do not download large datasets (50 GB or more) on the CSF login node. This will place a higher than normal load on the network which could affect other users. If you wish to download large datasets please contact us at its-ri-team@manchester.ac.uk and we will provide an alternative.

Set up procedure

It is necessary to load a modulefile in order to be able to connect out of the CSF:

module load tools/env/proxy-wget

The wget command can be run on the CSF login node for small dataset downloads (see above for large datasets).

Running the application

Simply navigate to where you wish to store the downloaded files (e.g., in scratch) and then run wget on the login node to download (small) files. For example:

cd ~/scratch/my_datasets
wget http://www.example.com/files/dataset.zip

Serial batch job submission

wget will not run from the back-end compute nodes. We recommend downloading data prior to submitting a job. If you have an application that insists on downloading data (e.g., a genome processing script) then please contact us.

Further info

• Simply run man wget on the CSF login node for the manual.

Updates

None.