lftp

Overview

LFTP is a file transfer program for downloading files from FTP sites and web sites.

Version 4.8.1 is installed on the CSF.

Restrictions on use

The are no restrictions on accessing this software on the CSF.

Set up procedure

To access the software you must first load the modulefile:

module load tools/gcc/lftp/4.8.1

Running the application

You may run lftp on the login node for small downloads. However, large data downloads should be run as a batch job. This will keep the load off the login node and allow you to leave the download running while not being logged in.

You must also download data through the proxy. For example:

module load tools/env/proxy
module load tools/gcc/lftp/4.8.1

or

module load tools/env/proxy-wget
module load tools/gcc/lftp/4.8.1

You can run lftp interactively – it will give you a prompt where you can then use the usual FTP commands. For example:

lftp -e "set http:proxy $HTTP_PROXY" http://ftp.somearchive.domain/data/release/
cd ok, cwd=/data/release                                                       
lftp ftp.somearchive.domain:/data/release> You can type FTP commands here
lftp ftp.somearchive.domain:/data/release> ls
(shows a directory listing of the remote site)
lftp ftp.somearchive.domain:/data/release> get README.txt
(downloads the README.txt file)
bye
(to exit out of lftp)

Alternatively you can script the commands on the command-line. For example:

To download just the directory listing from an FTP site that can be accessed via HTTP:

lftp -e "set http:proxy $HTTP_PROXY; find -d 2 ; bye" http://ftp.somearchive.domain/data/release/
      #                                #          #
      #                                #          # Exit lftp when done.
      #                                #
      #                                # List the directory contents
      #
      # Ask lftp to one of its commands - in this case to use the proxy

To download a particular file:

lftp -e "set http:proxy $HTTP_PROXY; get README.txt; bye"  http://ftp.somearchive.domain/data/release/

Depending on the setup of the webserver you are trying to download from you may need to change the above settings to use https.

Serial batch job submission

Make sure you have the modulefile loaded then create a batch submission script, for example:

#!/bin/bash
#$ -S /bin/bash
#$ -cwd             # Job will run from the current directory
#$ -V               # Job will inherit current environment settings

lftp -e "set http:proxy $HTTP_PROXY; get README.txt; bye" http://ftp.somearchive.domain/data/release/

Submit the jobscript using:

qsub scriptname

where scriptname is the name of your jobscript.

Further info

• LFTP man page

Updates

None.