Data and File Transfer Summary

Data and File Transfer Summary

In this module:

• Summarise use of remote IO and shared filesystems in Condor.
• Outline how to explicity transfer required input and output files.

Summary of Access to Data Files in Condor

Standard Universe

• Uses remote system calls for IO;
• executable binary and checkpoint files transferred as required.

Other Universes

• Default: assume shared filesystem.
• Can explicitly specify file transfer — if not specced, jobs will run only on local machine!

File Transfer

• Use ShouldTransferFiles;
• Always transfer files to execution site: YES; rely on a shared filesystem: NO.
• Automatically transfer the files if the submit and execute machine are not in the same FileSystemDomain: IF_NEEDED

Example

  Universe   = vanilla

  Executable = my_job
  Log        = my_job.log

  ShouldTransferFiles   = IF_NEEDED

  Transfer_input_files  = dataset_$(Process), common.data
  Transfer_output_files = TheAnswer.dat

  Queue 60