How to use additional R packages in HTCondor
There are two methods of deploying additional R packages for use in R jobs running on Condor:
- Pre-compiling the packages on the Condor submit node and distributing them with the job.
- Downloading, compiling and installing the R packages on-the-fly.
Using pre-compiled R packages
When using pre-compiled R packages it is essential that every install.packages() instruction in the R script is disabled. Otherwise R will attempt to re-compile the packages and fail, and then terminate with an error. If the R script needs to install even a single R package, for example, to optimize the code for the hardware on which the job is deployed then method 2 must be used.
R script using pre-compiled R package fracdiff:
Using R 3.2.4 with a pre-compiled package and submitting to Condor
- module load R/3.2.4
- mkdir R32_packages
- export R_LIBS_USER=R32_packages
- R --no-save <<< 'install.packages("fracdiff")'
- condor_submit submit_precompiled_R32.txt
using the following HTCondor submit file.
Note that the name of the R packages installation folder above (R32_packages) must match the folder name in the HTCondor submit file.
Using R 3.4.3 with a pre-compiled package and submitting to Condor
- module load R/3.4.3
- mkdir R34_packages
- export R_LIBS_USER=R34_packages
- R --no-save <<< 'install.packages("fracdiff")'
- condor_submit submit_precompiled_R34.txt
using the following HTCondor submit file.
Note that the name of the R packages installation folder above (R34_packages) must match the folder name in the HTCondor submit file.
Using R 3.5.3 with a pre-compiled package and submitting to Condor
- module load R/3.5.3
- mkdir R35_packages
- export R_LIBS_USER=R35_packages
- R --no-save <<< 'install.packages("fracdiff")'
- condor_submit submit_precompiled_R35.txt
using the following HTCondor submit file.
Note that the name of the R packages installation folder above (R35_packages) must match the folder name in the HTCondor submit file.
Using R 3.6.0 with a pre-compiled package and submitting to Condor
- module load R/3.6.0
- mkdir R36_packages
- export R_LIBS_USER=R36_packages
- R --no-save <<< 'install.packages("fracdiff")'
- condor_submit submit_precompiled_R36.txt
using the following HTCondor submit file.
Note that the name of the R packages installation folder above (R36_packages) must match the folder name in the HTCondor submit file.
Installing additional R packages on-the-fly
Installing R packages on-the-fly ensures that they can be optimized for the hardware on which the HTCondor job is run. However this will increase the time that R jobs take to complete as each additional R package will need to be downloaded, compiled and installed.
R script using on-the-fly installation of R package fracdiff:
Using R 3.2.4 with on-the-fly package installation and submitting to Condor
- condor_submit submit_onthefly_R32.txt
using the following HTCondor submit file.
Using R 3.4.3 with on-the-fly package installation and submitting to Condor
- condor_submit submit_onthefly_R34.txt
using the following HTCondor submit file.
Using R 3.5.3 with on-the-fly package installation and submitting to Condor
- condor_submit submit_onthefly_R35.txt
using the following HTCondor submit file.
Using R 3.6.0 with on-the-fly package installation and submitting to Condor
- condor_submit submit_onthefly_R36.txt
using the following HTCondor submit file.