Apologies if you are a DropAndCompute user, but these tricks are Linux command line only.<\/p>\n
Condor_submit is normally used as simply as this:<\/p>\n
condor_submit submit.txt\r\n<\/pre>\nThat is, a submit file is produced that fully describes the Condor submission. An alternative is to leave some lines out of the submit file and supply them on the command line using -append (or -a for short).<\/p>\n
Example 1<\/h2>\n
A trivial example is where sometimes you want to be notified, by email, that your job has finished and sometimes you don\u2019t. The default when not specified in the submit file is that you do get an email notification. But, you could submit as follows:<\/p>\n
condor_submit -a \"notifcation = never\" submit.txt\r\n<\/pre>\nLogically the line is taken as coming just before the
Queue<\/code> line insubmit.txt<\/code>, at least oneQueue<\/code> line being mandatory.<\/p>\nExample 2<\/h2>\n
In this example we change the number of jobs queued dynamically; that is: we supply the details on the command line. Your
Queue<\/code> line in the submit file should be written like this:<\/p>\nQueue $(queuecount)\r\n<\/pre>\nDo not define the macro queuecount in the submit file. Undefined macros default to the empty string, and so the above is equivalent to just<\/p>\n
Queue\r\n<\/pre>\n(which itself means Queue 1).<\/p>\n
When you have, say, finished testing and you want to queue 100 jobs, submit it this way:<\/p>\n
condor_submit -a \"queuecount = 100\" submit.txt\r\n<\/pre>\nNote the quotes around the -a(ppend) argument as it has spaces in it; It can also be written:<\/p>\n
condor_submit -a queuecount=100 submit.txt\r\n<\/pre>\nequivalently. Thanks to Matt Farrellee of Red Hat for this tip. Matt also supplied the next one.<\/p>\n
Example 3<\/h2>\n
What if you have lots of little files to transfer, and they come and go during your work flow. Let\u2019s say they all end in .data, and you would like to write in your submit file:<\/p>\n
transfer_input_files = *.data\r\n<\/pre>\nbut you can\u2019t. Again, the command line -append trick comes to the rescue:<\/p>\n
condor_submit -a \"transfer_input_files = mycode.sh$(for i in *.data; do echo -n \",$i\"; done)\" submit.txt\r\n<\/pre>\nNote that no
transfer_input_files<\/code> line should be used in the submit file. Also, a comma is already provided by the for loop so one isn’t required straight after the first input file.<\/p>\nExample 4<\/h2>\n
This one is a little similar to Example 2 in that you may have different submissions for different purposes: test versus production; or just differing data sets. The idea here is that you switch sub-directory on submission, like so:<\/p>\n
condor_submit -a \"initialdir = Test\" submit.txt\r\n<\/pre>\nAn additional issue encountered here is that it’s likely the original Condor submit file and your executable code are back in the previous directory. It is also possible to script this to allow the submission of a large array of data sets at the same time. A script which achieves this is as follows<\/p>\n
#!\/bin\/bash\r\n\r\nBASE_DIR='\/home\/[your_UoM_username]\/[your_working_directory]'\r\nNUM_OF_INPUTS=2\r\n\r\nfor i in `eval echo {1..$NUM_OF_INPUTS}`\r\ndo\r\n condor_submit -a \"transfer_input_files = $BASE_DIR\/mycode.sh,input.txt,$BASE_DIR\/myscript.sh$(for i2 in $BASE_DIR\/*.data; do echo -n \",$i2\"; done)\" -a \"initialdir = input$i\" $BASE_DIR\/submit.txt\r\ndone\r\n<\/pre>\nFor the above example to work, each of the new data sets needs placing in a directory called inputX, where X is an integer from 1 to the value set for ‘NUM_OF_INPUTS’ provided in the above script. The script transfers the local copy of the file ‘input.txt’ from the inputX directory into the executable being run by the executable line in the Condor submit file. It is assumed these are located directly within your main ‘BASE_DIR’ directory.<\/p>\n
The ‘submit.txt’ file contains, among other things<\/p>\n
\r\nexecutable = \/home\/[your_UoM_username]\/[your_working_directory]\/condor_script.sh\r\n<\/pre>\nWhich is a direct reference to the file containing the actual executable\/script which references the ‘input.txt’ as input to the main executable\/script<\/p>\n
\r\n#!\/bin\/bash\r\n\r\n.\/my_main_script.sh -file input.txt\r\n<\/pre>\nEach time a job is submitted to Condor via the first, master script the working directory is changed and the input.txt file used therein is then used as the input for the task being executed.<\/p>\n
Example 5<\/h2>\n
Another simple example where we decide how we want matching machines to be ranked.<\/p>\n
condor_submit -a \"rank = memory\" submit.txt # go with most memory first\r\n<\/pre>\nor<\/p>\n
condor_submit -a \"rank = kflops\" submit.txt # go with those fastest at floating point first\r\n<\/pre>\nor<\/p>\n
condor_submit -a \"rank = Target.TotalCPUs\" submit.txt # go with most cores first\r\n<\/pre>\nFinally, note that you can add multiple lines with multiple -a (or -append) arguments.<\/p>\n","protected":false},"excerpt":{"rendered":"
condor_submit: command line tricks with the -append argument Apologies if you are a DropAndCompute user, but these tricks are Linux command line only. Condor_submit is normally used as simply as this: condor_submit submit.txt That is, a submit file is produced that fully describes the Condor submission. An alternative is to leave some lines out of the submit file and supply them on the command line using -append (or -a for short). Example 1 A trivial.. Read more »<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"parent":14,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-241","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/htccondor\/wp-json\/wp\/v2\/pages\/241","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/htccondor\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/htccondor\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/htccondor\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/htccondor\/wp-json\/wp\/v2\/comments?post=241"}],"version-history":[{"count":20,"href":"https:\/\/ri.itservices.manchester.ac.uk\/htccondor\/wp-json\/wp\/v2\/pages\/241\/revisions"}],"predecessor-version":[{"id":982,"href":"https:\/\/ri.itservices.manchester.ac.uk\/htccondor\/wp-json\/wp\/v2\/pages\/241\/revisions\/982"}],"up":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/htccondor\/wp-json\/wp\/v2\/pages\/14"}],"wp:attachment":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/htccondor\/wp-json\/wp\/v2\/media?parent=241"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}