OpenCL

Overview

The following notes offer some instructions on compiling and running OpenCL code on the CSF Nvidia GPU nodes. Please see the GPU Page for more information on accessing the particular nodes either interactively or in batch.

Compiling

Compiling Interactively on a GPU Node

The Nvidia OpenCL libraries are only installed on the backend nodes containing the Nvidia hardware (in /usr/lib64/). Hence you should compile all code on those nodes. Compiling on the login node will fail because the compiler will not find the libraries (see below for a method of compiling on the login node if obtaining an interactive session on a GPU node is proving difficult).

The Nvidia OpenCL header files are available in the CUDA toolkit (yes, CUDA) installed on the CSF. To assist in writing Makefiles an environment variable (CUDA_HOME) is set by loading the appropriate cuda modulefile.

Hence, to compile OpenCL code, do the following:

1. Log in to CSF with X forwarding enabled (needed if you are going to use a graphical editor, such as gedit, to write your code)

   ssh -X username@csf.itservices.manchester.ac.uk
          #
          # replace username with your own
   

2. On the login node, load the cuda modulefile:

   module load libs/cuda/5.5.22
   
   # Older versions also exist
   module load libs/cuda/4.2.9
   module load libs/cuda/4.1.28
   module load libs/cuda/4.0.17
   module load libs/cuda/3.2.16
   

3. Start an interactive shell on the required backend node (in this example, the nvidia non-IB node – see the GPU Page for others). Note this may take a few seconds to open the xterm window while it waits for the scheduler to run the interactive job. The -V (uppercase V) ensures any environment settings you currently have are also set in the xterm (e.g., settings made by any module load commands). If you omit the -V then you should run the module load command(s) again in the xterm.

   qrsh -V -l inter -l nvidia xterm
   

4. Create a Makefile that looks similar to the one below (download Makefile1.mk and rename it to Makefile)

   # OpenCL headers and library
   OCLINC = -I$(CUDA_HOME)/include
   OCLLIB = -lOpenCL     # In /usr/lib64 so no -L$(CUDA_HOME)/path required
   
   # Other flags
   CC      = gcc     # Could rely on other modulefiles to set this
   DEBUG   = -O2     # Or -g
   CFLAGS  = $(DEBUG) $(OCLINC)
   LDFLAGS = $(OCLLIB)
   
   # Example to build your own vadd.c
   EXES = vadd
   
   # Ensure indented lines start with a tab, not spaces (cut-n-paste warning!)
   .c.o:
            $(CC) -c $(CFLAGS) $<
   
   all: $(EXES)
   
   vadd: vadd.o
           $(CC) vadd.o -o $@ $(LDFLAGS)
   
   clean:
           rm -f *.o $(EXES)
   

5. In the xterm running on the backend node, execute make to compile the code, and then run your code, all interactively. Note that code run from interactive sessions should be short test runs. Long runs of the final code should be submitted via the batch queue for non-interactive execution.

   make
   ./vadd
   

Compiling Interactively on a GPU Node without X11

This is very similar to the above method but should be used if you don't have an X-server running (e.g., you are accessing the CSF from a Windows desktop that doesn't have the Exceed X-Server software installed). In this case you won't be able to use a graphical editor such as gedit but can instead use text-based editors such as vi or emacs -nw. You can still compile your code in the usual manner:

1. Log in to CSF without X11 (hence no -X flag)

   ssh username@csf.itservices.manchester.ac.uk            # replace username with your own
   

2. Start an interactive session in your current terminal window (can't use -V here)

   qrsh -l inter -l nvidia
   

3. Wait for a new prompt to appear. It will refer to a GPU node, e.g., gpu209. Then load the CUDA module file

   module load libs/cuda/5.5.22
   

4. Now compile your code as in the previous method (see the Makefile above) and run it on the GPU node in the interactive session (in this example the code is called vadd). Note that code run from interactive sessions should be short test runs. Long runs of the final code should be submitted via the batch queue for non-interactive execution.

   make
   ./vadd
   

Trick to Compile on Login Node

If requesting an interactive session for compilation is proving difficult (e.g., the system is busy) then it is possible to copy the libOpenCL.so file to somewhere in your $HOME directory so that you can compile your code on the login node. Do the following:

# Start an interactive session on a GPU node if not already in one (only need to do this once)
qrsh -l inter -l nvidia

     # Wait until you get a new prompt on a GPU node then do the following there
     mkdir $HOME/lib64/
     cp /usr/lib64/libOpenCL.so $HOME/lib64/
     exit

# You are now back on the login node where you can develop your code and compile it.
# Modify the line in the Makefile to point to your local lib64 directory.
OCLLIB = -L$(HOME)/lib64 -lOpenCL

Note that if the OpenCL driver is updated on the GPU nodes you'll need to take a new copy of the OpenCL library to ensure you have access to new functions when compiling.

Remember though, you will never be able to run your OpenCL code on the login node - it has no GPUs (and running codes on the login is not permitted anyway).

Executing

Once logged in to the backend node via qrsh you can run your code interactive for short test runs during development. Alternatively, submit the code to the batch system for longer runs. See the GPU Page for more information about batch.

If your code only links against the libOpenCL.so library then your code will run regardless of whether the cuda modulefile is loaded or not because the library is located in /usr/lib64/.

Nvidia SDK

Version 5.5.22

Unfortunately Nvidia have removed all OpenCL samples from CUDA 5 and above. To access OpenCL samples within the CUDA installation you must use CUDA 4.2.9 or earlier - i.e., load the 4.2.9 modulefile.

Version 4.2.9 and earlier

The Nvidia SDK containing OpenCL (and CUDA) sample code is available in a directory pointed to by $CUDA_SDK once the cuda modulefile has been loaded. The OpenCL sample code is in

$CUDA_SDK/OpenCL/src

The instructions for compiling the SDK examples (both CUDA and OpenCL) are given on the GPU Page - see CUDA and OpenCL SDK Examples.

Fortran Code

Nvidia does not provide a fortran implementation of OpenCL. However, it is possible to call C code from fortran. The C code can then provide a wrapper to the OpenCL library. See the GPU Club FAQ on calling OpenCL from Fortran for sample code.

More Info

• University GPU Club
• CSF GPU Page