{"id":747,"date":"2017-07-26T15:32:14","date_gmt":"2017-07-26T15:32:14","guid":{"rendered":"http:\/\/ri.itservices.manchester.ac.uk\/icsf\/?page_id=747"},"modified":"2018-11-23T16:14:42","modified_gmt":"2018-11-23T16:14:42","slug":"ukbiobank","status":"publish","type":"page","link":"https:\/\/ri.itservices.manchester.ac.uk\/icsf\/software\/applications\/ukbiobank\/","title":{"rendered":"UKBioBank"},"content":{"rendered":"

Overview<\/h2>\n

These modulefiles set only environment variables to make accessing the centrally hosted UK Bio Bank Full Release Datasets<\/a> easier on the CSF. They allow you to use the environment variables in your jobscripts to access the correct folders that provide the central copy of the UK Bio Bank data.<\/p>\n

The information on this webpage is also valid for the CSF<\/a><\/p>\n

Restrictions on use<\/h2>\n

To access the UK Bio Bank data on the CSF you must<\/em> be added to a group on the system that has the correct access permissions (the dataset-ukbiobank-full<\/code> Unix group). To be added to this group please email its-ri-team@manchester.ac.uk<\/a> with confirmation of your approved UKBioBank\/EGA access to the data – the UKBioBank have advised us that supplying us with a copy of your Material Transfer Agreement (MTA) is suitable proof of access (please remove any passwords from your documentation).<\/p>\n

For details on registering with the UK Bio Bank \/ European Genome Archive please see the UK BioBank Full Release FAQ<\/a> (hosted by the UK Bio Bank).<\/p>\n

Set up procedure<\/h2>\n

Please load the modulefile:<\/p>\n

\r\n# Includes access to the correct v3 imputed dataset EGAD00010001474\r\n\r\nmodule load tools\/env\/ukbiobank-full-release-2018\r\n<\/pre>\n
The original modulefile providing access to the incorrect v2 imputed dataset EGAD00010001225 is still available. This is because you may have been using particular files in that study that the UKBioBank did not need to correct or you may be using the genotyped data in study id EGAD00010001226, which contains entirely correct data,<\/p>\n
\r\n# Includes access to the incorrect<\/strong> v2 imputed dataset EGAD00010001226\r\n\r\nmodule load tools\/env\/ukbiobank-full-release\r\n<\/pre>\n
Environment variables<\/h2>\n
The new 2018 modulefile sets the following environment variables (items in bold<\/strong> have changed since the previous modulefile):<\/p>\n
\r\nUKBIOBANK_BASE            =   \/mnt\/data-sets\/ukbiobank\/full-release\r\nUKBB_BASE                 =   \/mnt\/data-sets\/ukbiobank\/full-release\r\nUKBB_IMPUTATION_STUDYID   =   EGAD00010001474                           # This is the new, correct, v3 imputation data (March 2018)<\/strong>\r\nUKBB_GENOTYPED_STUDYID    =   EGAD00010001497                           # This is identical to the EGAD00010001226 data<\/strong>\r\nUKBB_IMPUTATION_DIR       =   \/mnt\/data-sets\/ukbiobank\/full-release\/EGAD00010001474          # No trailing 001 subdirectory<\/strong>\r\nUKBB_GENOTYPED_DIR        =   \/mnt\/data-sets\/ukbiobank\/full-release\/EGAD00010001497          # Symlink \/ shortcut to EGAD00010001226\/001<\/strong>\r\nUKBB_FILELIST             =   \/mnt\/data-sets\/ukbiobank\/full-release\/filelist.2018.txt        # Includes names of v3 (correct) imputed data files<\/strong>\r\n<\/pre>\n
The original modulefile makes the following settings:<\/p>\n
\r\nUKBIOBANK_BASE            =   \/mnt\/data-sets\/ukbiobank\/full-release\r\nUKBB_BASE                 =   \/mnt\/data-sets\/ukbiobank\/full-release\r\nUKBB_IMPUTATION_STUDYID   =   EGAD00010001225                           # This dataset should NOT be used (March 2018)\r\nUKBB_GENOTYPED_STUDYID    =   EGAD00010001226                           # This dataset can be used (it is identical to EGAD00010001497)\r\nUKBB_IMPUTATION_DIR       =   \/mnt\/data-sets\/ukbiobank\/full-release\/EGAD00010001225\/001\r\nUKBB_GENOTYPED_DIR        =   \/mnt\/data-sets\/ukbiobank\/full-release\/EGAD00010001226\/001\r\nUKBB_FILELIST             =   \/mnt\/data-sets\/ukbiobank\/full-release\/filelist.txt             # Includes names of v2 (incorrect) imputed data files\r\n<\/pre>\n
Basic Usage of Environment Variables<\/h3>\n
You can use any of the above environment variables in your jobscripts and other commands. <\/p>\n
For example, to see all of the files in the GENOTYPED dataset, assuming you have loaded the tools\/env\/ukbiobank-full-release-2018<\/code> modulefile:<\/p>\n
\r\nls -lh $UKBB_GENOTYPED_DIR<\/strong>\r\n\r\n# Note: The compressed .gz files and checksum .md5 files\r\n#       will also be listed.\r\n<\/pre>\n
To use the variables in a jobscript, load the modulefile (either on the login node or in the jobscript) and then you can use the variables to access particular files. For example:<\/p>\n
\r\nMyGenomeApp -input $UKBB_GENOTYPED_DIR\/ukb_cal_chr16_v2.bed -o myresults.dat\r\n<\/pre>\n
Job Arrays (on CSF)<\/h3>\n
The text file given by the $UKBB_FILELIST<\/code> variable contains a list of the datasets available:<\/p>\n
\r\ncat $UKBB_FILELIST\r\nEGAD00010001474\/ukb_imp_chr1_v3.bgen              # Note the v3 imputed files have changed their names compared to the v2 files\r\nEGAD00010001474\/ukb_imp_chr2_v3.bgen\r\nEGAD00010001474\/ukb_imp_chr3_v3.bgen\r\n...\r\nEGAD00010001474\/ukb_mfi_chrX_v3.txt\r\nEGAD00010001474\/ukb_mfi_chrXY_v3.txt\r\n...\r\nEGAD00010001497\/ukb_l2r_chrXY_v2.txt\r\nEGAD00010001497\/ukb_l2r_chrY_v2.txt\r\n   #\r\n   # Notice that the subdirectories EGAD00010001474 and EGAD00010001497 are included in the name\r\n<\/pre>\n
You may use this file when running job arrays<\/a> on the CSF to process all files in a particular dataset. For example:<\/p>\n
Suppose we wish to process all of the ukb_int_chrN<\/em>_v2.bin<\/code> files in the EGAD00010001497<\/code> study (this is the same dataset as the EGAD00010001226 study hence still use v2 in their name). <\/p>\n
    \n
  1. Load the modulefile on the login node:\n
    \r\nmodule load tools\/env\/ukbiobank-full-release-2018\r\n<\/pre>\n<\/li>\n
  2. Check which files we will be processing:\n
    \r\ngrep ukb_int_chr $UKBB_FILELIST\r\n  #\r\n  # 'grep' prints all lines that contain the string 'ukb_int_chr'\r\n\r\nEGAD00010001497\/ukb_int_chr1_v2.bin\r\nEGAD00010001497\/ukb_int_chr2_v2.bin\r\n...\r\nEGAD00010001497\/ukb_int_chrY_v2.bin\r\n   #\r\n   # Notice that the subdirectory EGAD00010001497\/ is included in the name\r\n<\/pre>\n<\/li>\n
  3. Count the number of files and hence number of tasks needed in the job array (you probably expect this to be 26 but let’s check!)\n
    \r\ngrep ukb_int_chr $UKBB_FILELIST | wc -l \r\n                                  #\r\n                                  # 'wc' is a word count utility (-l counts lines) 26\r\n<\/pre>\n<\/li>\n
  4. Write a job array script that will automatically run 26 copies (‘tasks’) of the job where each ‘task’ is given a unique ID 1, 2, …, 26. We will use this to make each task process a different dataset (one of the 26 files listed above):\n
    \r\n#!\/bin\/bash --login\r\n#$ -cwd                # Run in current directory\r\n#$ -t 1-26             # Run a 26-task job array\r\n### No -V line (we load the modulefile in the jobscript, hence added --login above)\r\n\r\n### Load the modulefiles in the jobscript\r\nmodule load tools\/env\/ukbiobank-full-release-2018\r\n# module load name\/of\/my\/app\/1.2.3\r\n\r\n### SGE_TASK_ID is set to 1, 2, ... 26 (one for each of the 26 tasks)\r\n### Read the N-th filename by generating the list as we did earlier\r\nFILENAME=`grep ukb_int_chr $UKBB_FILELIST | awk \"NR==$SGE_TASK_ID {print}\"`\r\n\r\n### Report what we are doing\r\necho \"Processing $UKBB_BASE\/$FILENAME in job array $JOB_ID task $SGE_TASK_ID\"\r\n\r\n### Process the file for this task (change this to use your own real app here!)\r\nmyGenomeApp -input $UKBB_BASE\/$FILENAME -output myResult_$SGE_TASK_ID.data\r\n\r\n<\/pre>\n<\/li>\n
  5. Submit the job once<\/em> using the usual command:\n
    \r\nqsub myjobscript<\/em>\r\n<\/pre>\n
    where myjobscript<\/em><\/code> is the name of your jobscript file.<\/li>\n<\/ol>\n
    Further Dataset Format Information<\/h3>\n
    For a description of the datasets, a text file provided by the UK Bio Bank is available using:<\/p>\n
    \r\nless $UKBB_BASE\/ukb_genetic_file_description.txt\r\n<\/pre>\n
    This file is also available online<\/a> (hosted by the UK Bio Bank).<\/p>\n
    Further info<\/h2>\n