Version 5.0 is installed on CSF3. It has been compiled from source using the Intel v19.1.1 compiler and OpenMPI 4.1.1. Currently only the GPU version has been compiled. The Relion user-interface has been enabled. <\/p>\n
Versions 3.0.6, 3.0.8, 3.1-beta, 3.1-beta2 and 3.1.0 are installed on CSF3. They have been compiled from source using the Intel v18.0.3 compilers and OpenMPI 3.1.4. Versions are available for CPU-only usage (with and without CPU Acceleration functions compiled) and GPU (Nvidia v100) usage. All versions have been compiled with the Relion user-interface enabled.<\/p>\n
Older versions 2.1 and 3.0-beta are also installed. They have been compiled from source using the Intel v17.0.7 compilers and OpenMPI 3.1.1. Versions are available for CPU-only usage (non accelerated) and GPU (Nvidia v100) usage. All versions have been compiled with the Relion user-interface enabled.<\/p>\n
Some of the CPU-only versions use the Relion-accelerated CPU code compiled with Intel Compiler multiple code-path optimizations for AVX, AVX2 and AVX512 architectures. This means the application can be run on any of the CSF3 compute nodes and some functions will use optimized versions for the CPU in that compute node.<\/p>\n
Restrictions on use<\/h2>\n
There are no restrictions on accessing this software on the CSF. The software is released under the GNU GPL v2 license. The authors request you cite the use of the software in your publications – please see the Relion website<\/a> for details.<\/p>\n To access the software you must first load one<\/strong> of the following modulefiles, according to whether you wish to run on CPU-only or GPU nodes:<\/p>\n This version has been compiled with the Intel 19.1.2 compiler.<\/p>\n The modulefile This version has been compiled with the Intel 18.0.3 compiler.<\/p>\n The modulefile If you wish to use a different version of the external tools than that provided by the This version has been compiled with the Intel 18.0.3 compiler.<\/p>\n The modulefile If you wish to use a different version of the external tools than that provided by the This version has been compiled with the Intel 18.0.3 compiler.<\/p>\n The modulefile If you wish to use a different version of the external tools than that provided by the This version has been compiled with the Intel 18.0.3 compiler.<\/p>\n The modulefile If you wish to use a different version of the external tools than that provided by the This version has been compiled with the Intel 17.0.7 compiler.<\/p>\n The modulefiles with The modulefile If you wish to use a different version of the external tools than that provided by the All modulefiles will also load automatically the compiler and MPI modulefiles for you.<\/p>\n Relion can be run with a GUI on the login node. This is mainly used to select input files and set up job parameters. The GUI will then submit your job to the batch queues.<\/p>\n Users must always<\/em> run the main processing in batch<\/strong> (where the GUI permits submission to the batch system), never directly on the login node. This requires correct settings in the GUI, as instructed below. <\/p>\n Relion tools other than the GUI found running on the login node will be terminated without warning<\/em>.<\/p>\n Please note, a tutorial on using Relion to process your data is beyong the scope of this page, which only describes how to run the application on the CSF. Please follow the Relion Turorial (pdf)<\/a> on the Relion website if you are new to the software.<\/p>\n Once the modulefile file is loaded you should navigate to your Relion project directory (this is important – it will usually contain a It will display the following GUI (click on image to enlarge): Various operations can be performed by selecting from the list on the left hand side (Micrograph inspection, CTF estimation and so on). These operations allow you to specify use of the batch system, as detailed below.<\/p>\n Note: To re-activate a greyed-out Each Relion processing operation (Micrograph inspection, CTF estimation etc) has a Running<\/em> tab in its GUI page. There are three possible types of Running<\/em> tab:<\/p>\n for the correct number of cores to be calculated if you request more than one thread.<\/li>\n Once you have set the Running<\/em> tab options as indicated above press the The auto-generated jobscript will be written to your current directory (your Relion project directory) and will be named similar to the operation being performed with a The usual output and error output from the job will be written to files with This section is for advanced users familiar with Relion and the CSF batch system. In most cases the default settings will work for you and you should not need to follow this section.<\/p>\n Relion auto-generates jobscripts by filling in place-holders in a qsub template file<\/em> named The default template file contains:<\/p>\nSet up procedure<\/h2>\n
5.0.0 version<\/h3>\n
\r\n# Released 5.0.0 version\r\nmodule load apps\/intel-19.1\/relion\/5.0.0-gpu-sp-all # Single-precision GPU\r\n\r\n<\/pre>\n
apps\/intel-19.1\/relion\/VERSION<\/em>-ARCH<\/em>-all<\/strong><\/code> is a convenience modulefile to load several other modulefile that you will usually need to use relion successfully. It will load the following modulefiles:<\/p>\n\r\napps\/intel-19.1\/relion\/VERSION<\/em>-ARCH<\/em> # Matches the version in the -all name\r\napps\/intel-19.1\/ctffind\/4.1.14 # ctffind3 is also available on the CSF3\r\napps\/intel-19.1\/unblur\/1.0.2\r\napps\/intel-19.1\/summovie\/1.0.2\r\napps\/binapps\/gctf\/1.06\r\n<\/pre>\n
3.1.0 versions<\/h3>\n
\r\n# NOTE: Currently the CSF3 will only do single-node multi-core CPU and GPU jobs. You cannot\r\n# run larger multi-node jobs.\r\n\r\n# Released 3.1.0 version\r\nmodule load apps\/intel-18.0\/relion\/3.1.0-gpu-sp-all # Single-precision GPU\r\nmodule load apps\/intel-18.0\/relion\/3.1.0-cpuacc-sp-all # Single-precision CPU-Accelerated\r\nmodule load apps\/intel-18.0\/relion\/3.1.0-cpu-sp-all # Single-precision CPU\r\n<\/pre>\n
apps\/intel-18.0\/relion\/VERSION<\/em>-ARCH<\/em>-all<\/strong><\/code> is a convenience modulefile to load several other modulefile that you will usually need to use relion successfully. It will load the following modulefiles:<\/p>\n\r\napps\/intel-18.0\/relion\/VERSION<\/em>-ARCH<\/em> # Matches the version in the -all name\r\napps\/intel-18.0\/ctffind\/4.1.13 # ctffind3 is also available on the CSF3\r\napps\/intel-18.0\/unblur\/1.0.2\r\napps\/intel-18.0\/summovie\/1.0.2\r\napps\/binapps\/gctf\/1.06\r\n<\/pre>\n
-all<\/code> modulefile then load the modulefile without<\/em> the -all<\/code> in the name and then load the individual modulefiles needed for each external tool.<\/p>\n3.1 beta versions<\/h3>\n
More info…<\/summary>\n
\r\n# NOTE: Currently the CSF3 will only do single-node multi-core CPU and GPU jobs. You cannot\r\n# run larger multi-node jobs.\r\n\r\n# A new build of the 3.1-beta source (includes relion_convert_to_tiff[_mpi]). Commit 06afb9f.\r\nmodule load apps\/intel-18.0\/relion\/3.1-beta2<\/strong>-gpu-sp-all # Single-precision GPU \r\nmodule load apps\/intel-18.0\/relion\/3.1-beta2<\/strong>-cpuacc-sp-all # Single-precision CPU-Accelerated\r\nmodule load apps\/intel-18.0\/relion\/3.1-beta2<\/strong>-cpu-sp-all # Single-precision CPU\r\n\r\n# Older build of the 3.1-beta source. Commit 7f8d1d6.\r\nmodule load apps\/intel-18.0\/relion\/3.1-beta-gpu-sp-all # Single-precision GPU\r\nmodule load apps\/intel-18.0\/relion\/3.1-beta-cpuacc-sp-all # Single-precision CPU-Accelerated\r\nmodule load apps\/intel-18.0\/relion\/3.1-beta-cpu-sp-all # Single-precision CPU\r\n<\/pre>\n
apps\/intel-18.0\/relion\/VERSION<\/em>-ARCH<\/em>-all<\/strong><\/code> is a convenience modulefile to load several other modulefile that you will usually need to use relion successfully. It will load the following modulefiles:<\/p>\n\r\napps\/intel-18.0\/relion\/VERSION<\/em>-ARCH<\/em> # Matches the version in the -all name\r\napps\/intel-18.0\/ctffind\/4.1.13 # ctffind3 is also available on the CSF3\r\napps\/intel-18.0\/unblur\/1.0.2\r\napps\/intel-18.0\/summovie\/1.0.2\r\napps\/binapps\/gctf\/1.06\r\n<\/pre>\n
-all<\/code> modulefile then load the modulefile without<\/em> the -all<\/code> in the name and then load the individual modulefiles needed for each external tool.
\n<\/details>\n3.0.8 version<\/h3>\n
More info…<\/summary>\n
\r\n# NOTE: Currently the CSF3 will only do single-node multi-core CPU and GPU jobs. You cannot\r\n# run larger multi-node jobs.\r\nmodule load apps\/intel-18.0\/relion\/3.0.8-gpu-sp-all # Single-precision GPU\r\nmodule load apps\/intel-18.0\/relion\/3.0.8-cpuacc-sp-all # Single-precision CPU-Accelerated\r\nmodule load apps\/intel-18.0\/relion\/3.0.8-cpu-sp-all # Single-precision CPU\r\n<\/pre>\n
apps\/intel-18.0\/relion\/3.0.8-ARCH<\/em>-all<\/code> loads:<\/p>\n\r\napps\/intel-18.0\/relion\/3.0.8-ARCH<\/em> # Matches the version in the -all name\r\napps\/intel-18.0\/ctffind\/4.1.13 # ctffind3 is also available on the CSF3\r\napps\/intel-18.0\/unblur\/1.0.2\r\napps\/intel-18.0\/summovie\/1.0.2\r\napps\/binapps\/gctf\/1.06\r\n<\/pre>\n
-all<\/code> modulefile then load the modulefile without<\/em> the -all<\/code> in the name and then load the individual modulefiles needed for each external tool.
\n<\/details>\n3.0.6 version<\/h3>\n
More info…<\/summary>\n
\r\n# NOTE: Currently the CSF3 will only do single-node multi-core CPU and GPU jobs. You cannot\r\n# run larger multi-node jobs.\r\nmodule load apps\/intel-18.0\/relion\/3.0.6-gpu-sp-all # Single-precision GPU code\r\n<\/pre>\n
apps\/intel-18.0\/relion\/3.0.6-gpu-sp-all<\/code> loads:<\/p>\n\r\napps\/intel-18.0\/relion\/3.0.6-gpu-sp # Matches the version in the -all name\r\napps\/intel-18.0\/ctffind\/4.1.13 # ctffind3 is also available on the CSF3\r\napps\/intel-18.0\/unblur\/1.0.2\r\napps\/intel-18.0\/summovie\/1.0.2\r\napps\/binapps\/gctf\/1.06\r\n<\/pre>\n
-all<\/code> modulefile then load the modulefile without<\/em> the -all<\/code> in the name and then load the individual modulefiles needed for each external tool.
\n<\/details>\n3.0-beta version<\/h3>\n
More info…<\/summary>\n
\r\n# NOTE: Currently the CSF3 will only do single-node multi-core CPU and GPU jobs. You cannot\r\n# run larger multi-node jobs.\r\n\r\n# Version 3.0-beta CPU-only + default external tools (see below) - RECOMMENDED\r\nmodule load apps\/intel-17.0\/relion\/3.0-beta-cpu-dp-all # Double-precision CPU-only code\r\nmodule load apps\/intel-17.0\/relion\/3.0-beta-cpu-sp-all # Single-precision CPU-only code\r\n\r\n# ... as above without default external tools\r\nmodule load apps\/intel-17.0\/relion\/3.0-beta-cpu-dp\r\nmodule load apps\/intel-17.0\/relion\/3.0-beta-cpu-sp\r\n\r\n# Version 3.0-beta GPU (nvidia v100) + default external tools (see below) - RECOMMENDED\r\nmodule load apps\/intel-17.0\/relion\/3.0-beta-gpu-dp-all # Double-precision GPU code\r\nmodule load apps\/intel-17.0\/relion\/3.0-beta-gpu-sp-all # Single-precision GPU code\r\n\r\n# ... as above without default external tools\r\nmodule load apps\/intel-17.0\/relion\/3.0-beta-gpu-dp\r\nmodule load apps\/intel-17.0\/relion\/3.0-beta-gpu-sp\r\n\r\n# Version 2.1 CPU and GPU\r\nmodule load apps\/intel-17.0\/relion\/2.1-cpu # CPU-only\r\nmodule load apps\/intel-17.0\/relion\/2.1-gpu-dp # Double-precision GPU code\r\nmodule load apps\/intel-17.0\/relion\/2.1-gpu-sp # Single-precision GPU code\r\n<\/pre>\n
-all<\/code> in their name are helper modulefiles which simply load the main relion modulefile (for the corresponding version) and the modulefiles for the external tools used by relion. To see what is loaded run:<\/p>\n\r\nmodule show apps\/intel-17.0\/relion\/3.0-beta-cpu-dp-all\r\n<\/pre>\n
apps\/intel-17.0\/relion\/3.0-beta-cpu-dp-all<\/code> loads:<\/p>\n\r\napps\/intel-17.0\/relion\/3.0-beta-cpu-sp # Matches the version in the -all name\r\napps\/intel-17.0\/ctffind\/4.1.10 # ctffind3 is also available on the CSF3\r\napps\/intel-17.0\/unblur\/1.0.2\r\napps\/intel-17.0\/summovie\/1.0.2\r\napps\/binapps\/gctf\/1.06\r\n<\/pre>\n
-all<\/code> modulefile then load the modulefile without<\/em> the -all<\/code> in the name and then load the individual modulefiles needed for each external tool.
\n<\/details>\nRunning the application<\/h2>\n
Running the GUI<\/h3>\n
Micrographs<\/code> subdirectory) and then run relion<\/code> on the login node. For example:<\/p>\n\r\ncd ~\/scratch\/my_relion_project\/\r\nrelion\r\n<\/pre>\n
\n![\"Relion](\"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-content\/uploads\/relion-general.png\")
<\/a><\/p>\nRun<\/code> button, press Alt+R<\/code> in the GUI.<\/p>\nUsing the GUI to run in Batch<\/h3>\n
\n
\n![\"Relion](\"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-content\/uploads\/relion-import-red.png\")
<\/a><\/li>\n
\n![\"Relion](\"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-content\/uploads\/relion-ctfest-running-batch-red.png\")
<\/a>
\nThe options should be set as follows:<\/p>\n\n
smp.pe<\/code> and between 2-8 CPU cores per v100 GPU<\/em> or 2-12 CPU cores per A100 GPU<\/em>. Hence if, for example you request 2 v100 GPUs you can request up to 16 CPU cores in smp.pe<\/code>.<\/li>\nsmp.pe<\/code> for single node jobs (2-32 cores) on Intel compute nodes or amd.pe<\/code> for single node jobs (2-168 cores) on AMD compute nodes.<\/li>\nv100=N<\/em><\/code> OR a100=N<\/em><\/code> where N<\/em><\/code> is the number of required GPUs. This should be between 1 and 4 depending on your granted access to the GPU nodes (permission to use the GPUs must be requested<\/a> – without the appropriate settings applied to your CSF account by the sysadmins, GPU jobs will not run.) For the Queue name, you must<\/em> use smp.pe<\/code> and between 2-8 CPU cores per v100 GPU<\/em> or 2-12 CPU cores per A100 GPU<\/em>.<\/li>\nqsub<\/code><\/li>\n\/opt\/apps\/apps\/intel-XY.Z<\/em>\/relion\/version<\/em>\/bin\/relion_qsub.sh<\/pre>\n<\/li>\n
\n![\"Relion](\"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-content\/uploads\/relion-2Dclass-running-batch-red.png\")
<\/a>
\nThe options should be set as follows:<\/p>\n\n
smp.pe<\/code> and between 2-8 CPU cores per v100 GPU<\/em> or 2-12 CPU cores per A100 GPU<\/em>. Hence if, for example you request 2 v100 GPUs you can request up to 16 CPU cores in smp.pe<\/code>.<\/li>\napps\/intel-17.0\/relion\/corestest<\/pre>\n
smp.pe<\/code> for single node jobs (2-32 cores) on Intel compute nodes or amd.pe<\/code> for single node jobs (2-168 cores) on AMD compute nodes.<\/li>\nv100=N<\/em><\/code> OR a100=N<\/em><\/code> where N<\/em><\/code> is the number of required GPUs. This should be between 1 and 4 depending on your granted access to the GPU nodes (permission to use the GPUs must be requested<\/a> – without the appropriate settings applied to your CSF account by the sysadmins, GPU jobs will not run.) For the Queue name, you must<\/em> use smp.pe<\/code> and between 2-8 CPU cores per v100 GPU<\/em> or 2-12 CPU cores per A100 GPU<\/em>.<\/li>\nqsub<\/code><\/li>\n\/opt\/apps\/apps\/intel-XY.Z<\/em>\/relion\/version<\/em>\/bin\/relion_qsub.sh<\/pre>\n<\/li>\n
\n![\"Specify](\"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-content\/uploads\/relion-2Dclass-running-batch-miltigpu-red-1.png\")
<\/a>
\nYou must then indicate the GPU ids to use on the Compute<\/em> tab.
\n![\"Specify](\"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-content\/uploads\/relion-2Dclass-compute-which-gpu-red.png\")
<\/a>
\nThe GPU IDs should always begin with 0<\/code> and further GPUs can be added by separating the IDs with colons. For example, if using two GPUs you would specify 0:1<\/code>. These IDs will be mapped to the physical GPUs assigned to your job.<\/li>\n<\/ol>\nSubmitting the Batch Job from the GUI<\/h3>\n
Run!<\/code> button to submit an auto-generated jobscript to the batch system. It may appear that nothing has happened when you do this. However, you should check on the job using the usual qstat<\/code> command by running it in a command-line window on the CSF. It will report whether your job is queued or running. If you see no output then you job has finished (either successfully or because an error occurred).<\/p>\n.script<\/code> extension. For example:<\/p>\n\r\nrun_ctffind_submit.script\r\n<\/pre>\n
.out<\/code> and .err<\/code> extensions respectively. You should check these for further information about your job. For example:<\/p>\n\r\nrun_ctffind.out\r\nrun_ctffind.err\r\n<\/pre>\n
Summary<\/h3>\n
\n
qstat<\/code> in a terminal window on the CSF.<\/li>\n<\/ul>\nAdvanced: Custom Batch Script Template<\/h2>\n
relion_qsub.sh<\/code>. It is possible for you to take a copy of this file, modify it and have Relion use your new template each time you submit a job via the Relion GUI. The following steps are required:<\/p>\n\n
\r\ncp $RELION_QSUB_TEMPLATE ~\/my_relion_project\/\r\n<\/pre>\n<\/li>\n
\r\ngedit ~\/my_relion_project\/qsub.sh\r\n<\/pre>\n<\/li>\n
\r\nexport RELION_QSUB_TEMPLATE=~\/my_relion_project\/qsub.sh\r\n<\/pre>\n<\/li>\n
\r\nrelion\r\n<\/pre>\n<\/li>\n<\/ol>\n
\r\n#!\/bin\/bash\r\n### Requests XXXcoresXXX in batch which is number of mpiprocs X threads\r\n#$ -pe XXXqueueXXX XXXcoresXXX\r\n#$ -e XXXerrfileXXX\r\n#$ -o XXXoutfileXXX\r\n#$ -cwd\r\n#$ -V\r\n#$ -l \"XXXextra1XXX\" # If using the GPU version will request v100 GPUs\r\n # If using the CPU version, can be blank (\"\") or\r\n # can request an CPU architecture (e.g., skylake)\r\n\r\nmpiexec -n XXXmpinodesXXX XXXcommandXXX\r\n<\/pre>\n