wget is a command-line program to download files from remote web servers. It can be used on the CSF to download datasets and source code, for example, from remote sites.<\/p>\n
The CSF login node does not have a direct connection to the outside world. Hence downloads will be done via the University proxy server. wget has been configured to use this proxy – you don’t need to do anything.<\/p>\n
All users may use this software. However, please do not download large datasets (50 GB or more) on the CSF login node. This will place a higher than normal load on the network which could affect other users. If you wish to download large datasets please contact us at its-ri-team@manchester.ac.uk<\/a> and we will provide an alternative.<\/p>\n It is necessary to load a modulefile in order to be able to connect out of the CSF:<\/p>\n The Simply navigate to where you wish to store the downloaded files (e.g., in scratch) and then run wget on the login node to download (small) files. For example:<\/p>\n wget will not run from the back-end compute nodes. We recommend downloading data prior to submitting a job. If you have an application that insists on downloading data (e.g., a genome processing script) then please contact us<\/a>.<\/p>\n None.<\/p>\n","protected":false},"excerpt":{"rendered":" Overview wget is a command-line program to download files from remote web servers. It can be used on the CSF to download datasets and source code, for example, from remote sites. The CSF login node does not have a direct connection to the outside world. Hence downloads will be done via the University proxy server. wget has been configured to use this proxy – you don’t need to do anything. Restrictions on use All users.. Read more »<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"parent":31,"menu_order":0,"comment_status":"open","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-2225","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/pages\/2225","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/comments?post=2225"}],"version-history":[{"count":3,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/pages\/2225\/revisions"}],"predecessor-version":[{"id":4412,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/pages\/2225\/revisions\/4412"}],"up":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/pages\/31"}],"wp:attachment":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/media?parent=2225"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Set up procedure<\/h2>\n
\r\nmodule load tools\/env\/proxy-wget\r\n<\/pre>\n
wget<\/code> command can be run on the CSF login node for small dataset downloads (see above for large datasets).<\/p>\nRunning the application<\/h2>\n
\r\ncd ~\/scratch\/my_datasets\r\nwget http:\/\/www.example.com\/files\/dataset.zip\r\n<\/pre>\n
Serial batch job submission<\/h3>\n
Further info<\/h2>\n
\n
man wget<\/code> on the CSF login node for the manual.<\/li>\n<\/ul>\nUpdates<\/h2>\n