Drop and Compute

The first steps for the drop and compute process is to create the required job submission files as outline on the Submitting your first job to Condor pages. It is then necessary to place these into a Zip file called submit.zip.

There are many ways and utilities to create a zip file. Under Linux/Mac this can be done using the command line utilities zip/unzip and under Microsoft Windows, right click on the file, select Send to and then Compressed (Zipped) folder

It is the necessary to mount your home folder located on submitter.itservices.manchester.ac.uk using one of the methods described in the pages File Transfers to and from Condor.

It is then necessary to create a directory within the home directory called AutoSubmit. Using the command line this can be completed with the following…

mkdir AutoSubmit

…using a SSH-based application this process will differ.

Once the directory has been created you can then just drag the submit.zip file to it.

Differences

Differences using this approach are…

• you will need to refresh the view of your AutoSubmit folder from time to time; and
• you will not see a pop-up when your job concludes (or, indeed, as files are generally updated)
• you will have to drag the results back if you don’t want to leave them on submitter.itservices.manchester.ac.uk

To help you spot when your job finishes you can add

notify_user = your email address

to the submit.txt file. As you might guess, this means you will receive an email message when your job completes. At that point you can drag the time-stamped folder, now also containing your results, back to your own PC.

It is also possible to submit DAGman jobs this way: simply call the main file submit.dag within the zipped up submit folder.

As always, if doing this from Windows, please make sure you are using a text editor that can be set to use the Linux/Unix end-of-line convention. The rule is that ‘\n’ (newline) terminates text lines. The default on Windows is ‘\r\n’ (a return character followed by a newline character). Bash scripts, in particular, will fail if you get this wrong. The free download notepad++ lets you set the end-of-line convention.

Tutorial

In this tutorial we are going to run a very simple script on the condor pool using Drop and Compute. The script, called myscript.sh, is written in the bash scripting language and is as follows

#!/bin/bash
ifconfig

All this script does is use the ifconfig command to return network card information from the machine on which it was run.

In order to run this on the condor pool we also need to supply a condor submit file such as the following, submit.txt:

universe = vanilla
executable = myscript.sh
log = log.txt
Output = out.txt
Error = err.txt
nice_user = True
notification = error
Requirements = (OpSys == "LINUX" && Arch == "X86_64")
Request_memory = 10
when_to_transfer_output = on_exit
queue


When using Drop and Compute, it is vital that the submit file be called submit.txt – no other filename will work*. Once you have both of these files you should put them both in a folder called submit which should then be zipped up to form submit.zip. To send this to the condor pool, all you need to do is move submit.zip to your Drop and Compute folder.

• Download myscript.sh (right click and ‘save file as’)
• Download submit.txt (right click and ‘save file as’)

* Advanced users can submit DAGman jobs, in which case the file should be named submit.dag