{"id":4254,"date":"2017-09-15T13:02:26","date_gmt":"2017-09-15T13:02:26","guid":{"rendered":"http:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/?page_id=4254"},"modified":"2018-02-12T09:59:47","modified_gmt":"2018-02-12T09:59:47","slug":"lftp","status":"publish","type":"page","link":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/software\/applications\/lftp\/","title":{"rendered":"lftp"},"content":{"rendered":"<h2>Overview<\/h2>\n<p><a href=\"https:\/\/linux.die.net\/man\/1\/lftp\">LFTP<\/a> is a file transfer program for downloading files from FTP sites and web sites.<\/p>\n<p>Version 4.8.1 is installed on the CSF.<\/p>\n<h2>Restrictions on use<\/h2>\n<p>The are no restrictions on accessing this software on the CSF.<\/p>\n<h2>Set up procedure<\/h2>\n<p>To access the software you must first load the modulefile:<\/p>\n<pre>\r\nmodule load tools\/gcc\/lftp\/4.8.1\r\n<\/pre>\n<h2>Running the application<\/h2>\n<p>You may run lftp on the login node for small downloads. However, large data downloads should be run as a batch job. This will keep the load off the login node and allow you to leave the download running while not being logged in.<\/p>\n<p>You must also download data through the proxy. For example:<\/p>\n<pre>\r\nmodule load tools\/env\/proxy\r\nmodule load tools\/gcc\/lftp\/4.8.1\r\n<\/pre>\n<p>or<\/p>\n<pre>\r\nmodule load tools\/env\/proxy-wget\r\nmodule load tools\/gcc\/lftp\/4.8.1\r\n<\/pre>\n<p>You can run lftp interactively &#8211; it will give you a prompt where you can then use the usual FTP commands. For example:<\/p>\n<pre>\r\nlftp -e \"set http:proxy $HTTP_PROXY\" http:\/\/ftp.somearchive.domain\/data\/release\/\r\ncd ok, cwd=\/data\/release                                                       \r\nlftp ftp.somearchive.domain:\/data\/release> <strong>You can type FTP commands here<\/strong>\r\nlftp ftp.somearchive.domain:\/data\/release> <strong>ls<\/strong>\r\n(shows a directory listing of the remote site)\r\nlftp ftp.somearchive.domain:\/data\/release> <strong>get README.txt<\/strong>\r\n(downloads the README.txt file)\r\n<strong>bye<\/strong>\r\n(to exit out of lftp)\r\n<\/pre>\n<p>Alternatively you can script the commands on the command-line. For example:<\/p>\n<p>To download just the directory listing from an FTP site that can be accessed via HTTP:<\/p>\n<pre>\r\nlftp -e \"set http:proxy $HTTP_PROXY; find -d 2 ; bye\" http:\/\/ftp.somearchive.domain\/data\/release\/\r\n      #                                #          #\r\n      #                                #          # Exit lftp when done.\r\n      #                                #\r\n      #                                # List the directory contents\r\n      #\r\n      # Ask lftp to one of its commands - in this case to use the proxy\r\n<\/pre>\n<p>To download a particular file:<\/p>\n<pre>\r\nlftp -e \"set http:proxy $HTTP_PROXY; get README.txt; bye\"  http:\/\/ftp.somearchive.domain\/data\/release\/\r\n<\/pre>\n<p>Depending on the setup of the webserver you are trying to download from you may need to change the above settings to use https.<\/p>\n<h3>Serial batch job submission<\/h3>\n<p>Make sure you have the modulefile loaded then create a batch submission script, for example:<\/p>\n<pre>\r\n#!\/bin\/bash\r\n#$ -S \/bin\/bash\r\n#$ -cwd             # Job will run from the current directory\r\n#$ -V               # Job will inherit current environment settings\r\n\r\nlftp -e \"set http:proxy $HTTP_PROXY; get README.txt; bye\" http:\/\/ftp.somearchive.domain\/data\/release\/\r\n<\/pre>\n<p>Submit the jobscript using: <\/p>\n<pre>qsub <em>scriptname<\/em><\/pre>\n<p>where <em>scriptname<\/em> is the name of your jobscript.<\/p>\n<h2>Further info<\/h2>\n<ul>\n<li><a href=\"https:\/\/linux.die.net\/man\/1\/lftp\">LFTP man page<\/a><\/li>\n<\/ul>\n<h2>Updates<\/h2>\n<p>None.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Overview LFTP is a file transfer program for downloading files from FTP sites and web sites. Version 4.8.1 is installed on the CSF. Restrictions on use The are no restrictions on accessing this software on the CSF. Set up procedure To access the software you must first load the modulefile: module load tools\/gcc\/lftp\/4.8.1 Running the application You may run lftp on the login node for small downloads. However, large data downloads should be run as.. <a href=\"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/software\/applications\/lftp\/\">Read more &raquo;<\/a><\/p>\n","protected":false},"author":15,"featured_media":0,"parent":31,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-4254","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/pages\/4254","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/users\/15"}],"replies":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/comments?post=4254"}],"version-history":[{"count":8,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/pages\/4254\/revisions"}],"predecessor-version":[{"id":4416,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/pages\/4254\/revisions\/4416"}],"up":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/pages\/31"}],"wp:attachment":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/media?parent=4254"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}