UKBioBank Helper Modulefile

Overview

These modulefiles set only environment variables to make accessing the centrally hosted UK Bio Bank Full Release Datasets easier on the CSF. They allow you to use the environment variables in your jobscripts to access the correct folders that provide the central copy of the UK Bio Bank data.

The information on this webpage is also valid for the iCSF

Restrictions on use

To access the UK Bio Bank data on the CSF you must be added to a group on the system that has the correct access permissions (the dataset-ukbiobank-full Unix group). To be added to this group please contact us with confirmation of your approved UKBioBank/EGA access to the data – the UKBioBank have advised us that supplying us with a copy of your Material Transfer Agreement (MTA) is suitable proof of access (please remove any passwords from your documentation). If you are sending a screenshot of the BioBank Portal which shows who is named on the MTA, please also include a screenshot (if possible) showing which datasets your project is permitted to use.

For details on registering with the UK Bio Bank / European Genome Archive please see the UK BioBank Full Release FAQ (hosted by the UK Bio Bank).

Set up procedure

Access to the UKBioBank corrected v3 dataset

In 2018 the UKBioBank released v3 of the imputed dataset, named EGAD00010001474, to correct some mistakes in the v2 dataset, named EGAD00010001225. Users should use the corrected version. The incorrect version (EGAD00010001225) is no longer available.

The genotyped data was always correct, so is available via the old name EGAD00010001226 and new name EGAD00010001497.

We use the new names for both datasets: EGAD00010001474 (imputed data) and EGAD00010001497 (genotyped data).

Please load the modulefile:

# Access to the corrected v3 imputed dataset EGAD00010001474 and genotype dataset EGAD00010001497

module load tools/env/ukbiobank-full-release-2018

Environment variables

The new 2018 modulefile sets the following environment variables (items in bold have changed since the previous modulefile):

# Settings provided by the tools/env/ukbiobank-full-release-2018 modulefile:
# You can use these environment variables on the command-line or in your jobscripts:
# EG: ls ${UKBB_IMPUTATION_DIR}

UKBIOBANK_BASE            =   /mnt/data-sets/ukbiobank/full-release
UKBB_BASE                 =   /mnt/data-sets/ukbiobank/full-release

# This is the new, correct, v3 imputation data (March 2018)
UKBB_IMPUTATION_STUDYID   =   EGAD00010001474

# This is the genotyped data, identical to the previously released EGAD00010001226 data
UKBB_GENOTYPED_STUDYID    =   EGAD00010001497

# Path to imputed data
UKBB_IMPUTATION_DIR       =   /mnt/data-sets/ukbiobank/full-release/EGAD00010001474

# Path to genotyped data
UKBB_GENOTYPED_DIR        =   /mnt/data-sets/ukbiobank/full-release/EGAD00010001497

# Includes names of v3 (correct) imputed data files
UKBB_FILELIST             =   /mnt/data-sets/ukbiobank/full-release/filelist.2018.txt

To use the variables in a jobscript, load the modulefile (either on the login node or in the jobscript) and then you can use the variables to access particular files. For example:

MyGenomeApp -input $UKBB_GENOTYPED_DIR/ukb_cal_chr16_v2.bed -o myresults.dat

Job Arrays

The text file given by the $UKBB_FILELIST variable contains a list of the datasets available:

cat $UKBB_FILELIST
EGAD00010001474/ukb_imp_chr1_v3.bgen              # Note the v3 imputed files have changed their names compared to the v2 files
EGAD00010001474/ukb_imp_chr2_v3.bgen
EGAD00010001474/ukb_imp_chr3_v3.bgen
...
EGAD00010001474/ukb_mfi_chrX_v3.txt
EGAD00010001474/ukb_mfi_chrXY_v3.txt
...
EGAD00010001497/ukb_l2r_chrXY_v2.txt
EGAD00010001497/ukb_l2r_chrY_v2.txt
   #
   # Notice that the subdirectories EGAD00010001474 and EGAD00010001497 are included in the name

You may use this file when running job arrays on the CSF to process all files in a particular dataset. For example:

Suppose we wish to process all of the ukb_int_chrN_v2.bin files in the EGAD00010001497 study (this is the same dataset as the EGAD00010001226 study hence still use v2 in their name).

  1. Load the modulefile on the login node: 
    module load tools/env/ukbiobank-full-release-2018
  2. Check which files we will be processing: 
    grep ukb_int_chr $UKBB_FILELIST
  #
  # 'grep' prints all lines that contain the string 'ukb_int_chr'

EGAD00010001497/ukb_int_chr1_v2.bin
EGAD00010001497/ukb_int_chr2_v2.bin
...
EGAD00010001497/ukb_int_chrY_v2.bin
   #
   # Notice that the subdirectory EGAD00010001497/ is included in the name
  3. Count the number of files and hence number of tasks needed in the job array (you probably expect this to be 26 but let’s check!) 
    grep ukb_int_chr $UKBB_FILELIST | wc -l 
                                  #
                                  # 'wc' is a word count utility (-l counts lines) 26
  4. Write a job array script that will automatically run 26 copies (‘tasks’) of the job where each ‘task’ is given a unique ID 1, 2, …, 26. We will use this to make each task process a different dataset (one of the 26 files listed above): 
    #!/bin/bash --login
#SBATCH -p serial      # (or --partition) Run on the node used for 1-core jobs
#SBATCH -a 1-26        # Run a 26-task job array
#SBATCH -t 2-0         # Wallclock time limit. Each job-array task will have this amount of time.
                       # In this example 2-0 is two days. Max permitted is 7-0.

### Load the modulefiles in the jobscript
module load tools/env/ukbiobank-full-release-2018

### You will probably want to load modulefiles for bio-inf applications
module load name/of/my/app/1.2.3

### ${SLURM_ARRAY_TASK_ID} is set to 1, 2, ... 26 (one for each of the 26 tasks)
### Read the N-th filename by generating the list as we did earlier
FILENAME=$(grep ukb_int_chr $UKBB_FILELIST | awk "NR==${SLURM_ARRAY_TASK_ID} {print}")

### Report what we are doing
echo "Processing $UKBB_BASE/$FILENAME in job array ${SLURM_ARRAY_JOB_ID} task ${SLURM_ARRAY_TASK_ID}"

### Process the file for this task (change this to use your own real app here!)
myGenomeApp -input $UKBB_BASE/$FILENAME -output myResult_${SLURM_ARRAY_TASK_ID}.data
  5. Submit the job once using the usual command: 
    sbatch myjobscript

    where myjobscript is the name of your jobscript file.

Further Dataset Format Information

For a description of the datasets, a text file provided by the UK Bio Bank is available using:

less $UKBB_BASE/ukb_genetic_file_description.txt

This file is also available online (hosted by the UK Bio Bank).

Further info

Updates

None.

Last modified on June 20, 2025 at 5:03 pm by George Leaver