{"id":74,"date":"2017-04-04T15:30:24","date_gmt":"2017-04-04T14:30:24","guid":{"rendered":"http:\/\/ri.itservices.manchester.ac.uk\/htccondor\/?page_id=74"},"modified":"2025-02-21T12:08:25","modified_gmt":"2025-02-21T12:08:25","slug":"drop-compute","status":"publish","type":"page","link":"https:\/\/ri.itservices.manchester.ac.uk\/htccondor\/jobs\/drop-compute\/","title":{"rendered":"Drop and Compute"},"content":{"rendered":"<p>The first steps for the drop and compute process is to create the required job submission files as outline on the <a href=\"https:\/\/ri.itservices.manchester.ac.uk\/htccondor\/jobs\/simplejob\/\" target=\"_blank\" rel=\"noopener\">Submitting your first job to Condor<\/a> pages. It is then necessary to place these into a Zip file called <code>submit.zip<\/code>.<\/p>\n<p>There are many ways and utilities to create a zip file. Under Linux\/Mac this can be done using the command line utilities zip\/unzip and under Microsoft Windows, right click on the file, select Send to and then Compressed (Zipped) folder<\/p>\n<p>It is the necessary to mount your home folder located on <code>submitter.itservices.manchester.ac.uk<\/code> using one of the methods described in the pages <a href=\"https:\/\/ri.itservices.manchester.ac.uk\/htccondor\/getting-started\/file-transfer\/\" target=\"_blank\" rel=\"noopener\">File Transfers to and from Condor<\/a>.<\/p>\n<p>It is then necessary to create a directory within the home directory called AutoSubmit. Using the command line this can be completed with the following&#8230;<\/p>\n<pre>mkdir AutoSubmit\r\n<\/pre>\n<p>&#8230;using a SSH-based application this process will differ.<\/p>\n<p>Once the directory has been created you can then just drag the <code>submit.zip<\/code> file to it.<\/p>\n<h2>Differences<\/h2>\n<p>Differences using this approach are&#8230;<\/p>\n<ul>\n<li>you will need to refresh the view of your AutoSubmit folder from time to time; and<\/li>\n<li>you will not see a pop-up when your job concludes (or, indeed, as files are generally updated)<\/li>\n<li>you will have to drag the results back if you don\u2019t want to leave them on <code>submitter.itservices.manchester.ac.uk<\/code><\/li>\n<\/ul>\n<p>To help you spot when your job finishes you can add<\/p>\n<pre>notify_user = your email address\r\n<\/pre>\n<p>to the <code>submit.txt<\/code> file. As you might guess, this means you will receive an email message when your job completes. At that point you can drag the time-stamped folder, now also containing your results, back to your own PC.<\/p>\n<p>It is also possible to submit DAGman jobs this way: simply call the main file <code>submit.dag<\/code> within the zipped up submit folder.<\/p>\n<p>As always, if doing this from Windows, please make sure you are using a text editor that can be set to use the Linux\/Unix end-of-line convention. The rule is that \u2018\\n\u2019 (newline) terminates text lines. The default on Windows is \u2018\\r\\n\u2019 (a return character followed by a newline character). Bash scripts, in particular, will fail if you get this wrong. The free download <a href=\"https:\/\/notepad-plus-plus.org\/&quot;\">notepad++<\/a> lets you set the end-of-line convention.<\/p>\n<h2>Tutorial<\/h2>\n<p>In this tutorial we are going to run a very simple script on the condor pool using Drop and Compute. The script, called <code>myscript.sh<\/code>, is written in the bash scripting language and is as follows<\/p>\n<pre>#!\/bin\/bash\r\nifconfig\r\n<\/pre>\n<p>All this script does is use the <code>ifconfig<\/code> command to return network card information from the machine on which it was run.<\/p>\n<p>In order to run this on the condor pool we also need to supply a condor submit file such as the following, <code>submit.txt<\/code>:<br \/>\n<code><br \/>\nuniverse = vanilla<br \/>\nexecutable = myscript.sh<br \/>\nlog = log.txt<br \/>\nOutput = out.txt<br \/>\nError = err.txt<br \/>\nnice_user = True<br \/>\nnotification = error<br \/>\nRequirements = (OpSys == \"LINUX\" &amp;&amp; Arch == \"X86_64\")<br \/>\nRequest_memory = 10<br \/>\nwhen_to_transfer_output = on_exit<br \/>\nqueue<br \/>\n<\/code><\/p>\n<p>When using Drop and Compute, it is vital that the submit file be called <code>submit.txt<\/code> \u2013 no other filename will work*. Once you have both of these files you should put them both in a folder called <code>submit<\/code> which should then be zipped up to form <code>submit.zip<\/code>. To send this to the condor pool, all you need to do is move <code>submit.zip<\/code> to your Drop and Compute folder.<\/p>\n<ul>\n<li>Download <a href=\"https:\/\/ri.itservices.manchester.ac.uk\/htccondor\/wp-content\/uploads\/myscript.sh\">myscript.sh<\/a> (right click and \u2018save file as\u2019)<\/li>\n<li>Download <a href=\"https:\/\/ri.itservices.manchester.ac.uk\/htccondor\/wp-content\/uploads\/submit.txt\">submit.txt<\/a> (right click and \u2018save file as\u2019)<\/li>\n<\/ul>\n<p>* Advanced users can submit DAGman jobs, in which case the file should be named <code>submit.dag<\/code><\/p>\n","protected":false},"excerpt":{"rendered":"<p>The first steps for the drop and compute process is to create the required job submission files as outline on the Submitting your first job to Condor pages. It is then necessary to place these into a Zip file called submit.zip. There are many ways and utilities to create a zip file. Under Linux\/Mac this can be done using the command line utilities zip\/unzip and under Microsoft Windows, right click on the file, select Send.. <a href=\"https:\/\/ri.itservices.manchester.ac.uk\/htccondor\/jobs\/drop-compute\/\">Read more &raquo;<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"parent":14,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-74","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/htccondor\/wp-json\/wp\/v2\/pages\/74","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/htccondor\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/htccondor\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/htccondor\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/htccondor\/wp-json\/wp\/v2\/comments?post=74"}],"version-history":[{"count":20,"href":"https:\/\/ri.itservices.manchester.ac.uk\/htccondor\/wp-json\/wp\/v2\/pages\/74\/revisions"}],"predecessor-version":[{"id":1246,"href":"https:\/\/ri.itservices.manchester.ac.uk\/htccondor\/wp-json\/wp\/v2\/pages\/74\/revisions\/1246"}],"up":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/htccondor\/wp-json\/wp\/v2\/pages\/14"}],"wp:attachment":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/htccondor\/wp-json\/wp\/v2\/media?parent=74"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}