The Computational Shared Facility 3

Proxy (for web downloads)

Overview

The CSF login node does not have a direct connection to the outside world. Hence downloads will be done via the University proxy server. This server does have a connection to the outside world, and so by downloading via this server, we protect the CSF from unwanted outside access while still allowing legitimate download requests initiated by CSF users to talk to the outside world.

Restrictions on use

There are no restrictions on use this modulefile. It does not set up application software. It only adds settings to your environment that other applications may use.

Set up procedure

We now recommend loading modulefiles within your jobscript so that you have a full record of how the job was run. See the example jobscript below for how to do this. Alternatively, you may load modulefiles on the login node and let the job inherit these settings.

Load one of the following modulefiles depending on the requirements of your application. Some apps require the http:// protocol to be included, some don’t:

module load tools/env/proxy         # Uses proxy.man.ac.uk:3128
module load tools/env/proxy2        # Uses http://proxy.man.ac.uk:3128

The above modulefiles simply make environment settings. Other applications that perform the download (such as wget or curl) will use these settings.

Running the application

Please do not run your downloader on the login node. Jobs should be submitted to the compute nodes via batch.

Serial batch job submission

Make sure you have the modulefile loaded then create a batch submission script, for example:

#!/bin/bash --login
#$ -cwd             # Job will run from the current directory
                    # NO -V line - we load modulefiles in the jobscript

# Load the required version
module load tools/env/proxy

my_downloader_app http://example.com/some/large/dataset.dat

Submit the jobscript using:

qsub scriptname

where scriptname is the name of your jobscript.

Updates

None.

Last modified on February 25, 2019 at 3:37 pm by Site Admin