June 2017: There are a total of 3 Nvidia GPGPUs in production in the CSF.<\/p>\n
We hope to purchase some more GPUs in late 2017\/early 2018 – please get in touch (its-ri-team@manchester.ac.uk<\/a>) if you would like to be involved in that procurement.<\/p>\n The NVida 2050s and 2070s have all been retired due to hardware faults.<\/p>\n Access to the GPGPUs is more restrictive than that for standard compute nodes. Please email its-ri-team@manchester.ac.uk<\/a> before attempting to use these resources<\/em> with brief details of what you wish to use them for.<\/p>\n The K20s are usually only accessible by a specific group from MACE. Once you have emailed its-ri-team@manchester.ac.uk<\/a> and been granted access, set up your environment by loading the appropriate module from the following:<\/p>\n The Nvidia cuDNN libraries are also available via the following modulefiles. Before you load these modulefiles you must<\/em> load one of the cuda modulefiles from above – the list below indicates which versions of cuda can be used with the different cuDNN versions:<\/p>\n The following sections describe how to compile CUDA and OpenCL code on CSF.<\/p>\n CUDA code can be compiled<\/strong> on the login node provided you are using the CUDA runtime library, and not the CUDA driver library. The runtime library is used when you allow CUDA to automatically set up the device. That is, your CUDA code uses the style where you assume CUDA will be set up on the first CUDA function call. For example:<\/p>\n The CUDA driver library allows much more low-level control of the GPU device (and makes CUDA set up more like OpenCL). In that case you must compile on a GPU node<\/strong> because the CUDA driver library is only available on the backend GPU nodes. Driver code will contain something like the following:<\/p>\n No matter where you compile your code you cannot run<\/strong> your CUDA code on the login node because it does not contain any GPUs (see the next section for running your code). <\/p>\n The CUDA libraries and header files are available in the following directories once you have loaded the CUDA module:<\/p>\n It is beyond the scope of this page to give a tutorial on CUDA compilation (there are many possible flags for the nvcc compiler). The CUDA GPU Programming SDK available on CSF in To use the above line in a Makefile, enclose the variable names in brackets as follows<\/p>\n The above to compilation methods use the CUDA runtime libary (libcudart) and so can be used to compile on the login node.<\/p>\n Please see OpenCL programming on CSF<\/a> for compiling OpenCL code.<\/p>\n All work on the Nvidia GPUs must be via the batch system. There are two types of environments which can be used. First, batch, for non-interactive computational work; this should be used where possible. Secondly, an interactive environment for debugging and other necessarily-interactive work.<\/p>\n Maximum job runtime is 14 days. Currently most users are restricted to one job running at any one time. This is due to the small number of GPUs available and the high demand for those GPUs.<\/p>\n As stated above, all jobs must be submitted to the batch system, whether for non-interactive (possibly long) computational runs or for short interactive runs. Jobs should be submitted to the batch system ensuring that the appropriate GPU resources are requested. Examples of jobscripts and commands to access the GPU resources are given below. In all cases ensure you have the appropriate module loaded (see above<\/a>).<\/p>\n Ensure you have the appropriate CUDA module loaded (see above), then use the following jobscript (note the use of the Submit the job in the usual way<\/p>\n If you are familiar with the use of X11 (X-Windows), load the appropriate environment module, then enter<\/p>\n Within the xterm, for example<\/p>\n <\/a><\/p>\n The CUDA SDK contains many example CUDA and<\/strong> OpenCL programs which can be compiled and run. A useful one is In CUDA 5.5 and up there is no separate SDK installation directory. Instead the CUDA toolkit (which provides the The compiled samples are available using<\/p>\n As always, running the samples on the login node won’t work – there’s no GPU there!<\/p>\n In CUDA 4.2.9 the CUDA SDK provides the sample files and is separate to the CUDA toolkit (which provides the The CUDA and OpenCL example programs are just like any other GPU code so please see the instructions earlier on running code either in batch or interactively on a GPU node.<\/p>\n Applications and compilers which can use the Nvidia GPUs are being installed on the CSF. Links to the appropriate documentation will be provided here and will include:<\/p>\n Overview – current GPGPUs June 2017: There are a total of 3 Nvidia GPGPUs in production in the CSF. We hope to purchase some more GPUs in late 2017\/early 2018 – please get in touch (its-ri-team@manchester.ac.uk) if you would like to be involved in that procurement. Three K20s (two hosted in one 12 core compute node, one hosted in one 12 core compute node) Retired GPGPUs The NVida 2050s and 2070s have all been retired.. Read more »<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"parent":31,"menu_order":0,"comment_status":"open","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-108","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/pages\/108","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/comments?post=108"}],"version-history":[{"count":20,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/pages\/108\/revisions"}],"predecessor-version":[{"id":4798,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/pages\/108\/revisions\/4798"}],"up":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/pages\/31"}],"wp:attachment":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf-apps\/wp-json\/wp\/v2\/media?parent=108"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}\n
Retired GPGPUs<\/h2>\n
\n
Seven blade servers each hosting one Nvidia card, two of which are M2070 cards and five are M2050 cards.<\/strike><\/li>\n16 Nvidia M2050 GPUs, two hosted on each of eight Intel compute nodes.<\/strike><\/li>\nThe eight M2050 hosts are connected by Infiniband, so are ideal for computational jobs based on both<\/em> MPI and CUDA.<\/strike><\/li>\n<\/ul>\nHardware and Software Versions<\/h2>\n
\n
Restrictions on who can use these GPUs<\/h2>\n
\n<\/a><\/p>\nSet up procedure<\/h2>\n
\r\n# Load one<\/strong> of the following modulefiles:\r\nmodule load 9.0.176\r\nmodule load 8.0.44\r\nmodule load 7.5.18\r\nmodule load 6.5.14\r\n\r\n# These are very old versions\r\nmodule load 5.5.22\r\nmodule load 4.2.9\r\nmodule load 4.1.28\r\nmodule load 4.0.17\r\nmodule load 3.2.16\r\n<\/pre>\n
Other Libraries<\/h2>\n
\r\nmodule load libs\/cuDNN\/7.0.3 # Load cuda 8.0.44 or 9.0.176 first\r\nmodule load libs\/cuDNN\/6.0.21 # Load cuda 7.5.18 or 8.0.44 first\r\nmodule load libs\/cuDNN\/5.1.5 # Load cuda 7.5.18 or 8.0.44 first\r\n<\/pre>\n
Compiling GPU Code<\/h2>\n
CUDA<\/h3>\n
\r\n#include <cuda_runtime.h>\r\n\r\nint main( void ) {\r\n\r\n \/\/ We assume CUDA will set up the GPU device automatically\r\n cudaMalloc( ... );\r\n cudaMemcpy( ... );\r\n myKernel<<<...>>>( ... );\r\n cudaMemcpy( ... );\r\n cudaFree( ... );\r\n return 0;\r\n}\r\n<\/pre>\n\r\n#include <cuda.h>\r\n\r\nint main( void ) {\r\n\r\n \/\/ Low-level device setup using the driver API\r\n cuDeviceGetCount( ... );\r\n cuDeviceGet( ... );\r\n cuDeviceGetName( ... );\r\n cuDeviceComputeCapability( ... );\r\n ...\r\n\r\n return 0;\r\n}\r\n<\/pre>\n\r\n# All nodes\r\n$CUDA_HOME\/lib64 # CUDA runtime library, CUBlas, CURand etc\r\n$CUDA_HOME\/include\r\n\r\n# On a GPU node only\r\n\/usr\/lib64 # CUDA driver library\r\n<\/pre>\n
$CUDA_SDK<\/code> gives many examples of CUDA programs and how to compile them. However, a simple compile line to run on the command line would be as follows<\/p>\nnvcc -o myapp myapp.cu -I$CUDA_HOME\/include -L$CUDA_HOME\/lib64 -lcudart<\/pre>\n
\r\n# Simple CUDA Makefile\r\nCC = nvcc\r\n\r\nall: myapp\r\n\r\nmyapp: myapp.cu\r\n $(CC) -o myapp myapp.cu -I$(CUDA_HOME)\/include -L$(CUDA_HOME)\/lib64 -lcudart\r\n# note: the preceeding line must start with a TAB, not 8 spaces. 'make' requires a TAB!\r\n<\/pre>\n
OpenCL<\/h3>\n
Running the application<\/h2>\n
Resource Limits<\/h2>\n
K20 GPUs<\/h3>\n
Example Job Submission Scripts and Commands<\/h2>\n
Serial batch job submission to K20 GPUs<\/h3>\n
nvidia_ib<\/code> resource)<\/p>\n\r\n#!\/bin\/bash\r\n#$ -cwd\r\n#$ -V\r\n#$ -l nvidia_k20\r\n\r\n.\/my_gpu_prog arg1 arg2\r\n<\/pre>\n
qsub gpujob.sh<\/pre>\n
Interactive use of the K20 GPUs with X11<\/h3>\n
\r\nqrsh -cwd -V -l inter -l nvidia_k20 xterm\r\n<\/pre>\n
\r\n.\/my_gpu_prog\r\n<\/pre>\n
CUDA and OpenCL SDK Examples (e.g., deviceQuery)<\/h2>\n
deviceQuery<\/code> (and oclDeviceQuery<\/code>) which gives you lots of information about the Nvidia GPU hardware. <\/p>\nVersion 5.5.22 and later<\/h3>\n
nvcc<\/code> compiler, profiler and numerical libraries) also contains a Samples<\/em> directory. The examples have already been compiled but you may also take a copy of the samples so that you can modify them. You can access the samples by loading the CUDA modulefile and then going in to the directory:<\/p>\n\r\ncd $CUDA_SAMPLES\r\n<\/pre>\n
\r\ncd $CUDA_SAMPLES\/bin\/x86_64\/linux\/release\/\r\n<\/pre>\n
Version 4.2.9 and earlier<\/h3>\n
nvcc<\/code> compiler, profiler and numerical libraries). You’ll need to copy the entire SDK to your home (or scratch) area. Compile the SDK on a GPU node<\/strong>, not the login node because some of the examples use the CUDA driver library (e.g., see $CUDA_SDK\/C\/src\/vectorAddDrv\/<\/code>) and OpenCL examples can only be compiled on a GPU node. For example:<\/p>\n\r\n# First start an interactive session on a GPU node\r\nqrsh -l inter -l nvidia\r\n\r\n# Once the interactive session starts:\r\nmodule load libs\/cuda\/4.2.9\r\nexport CUDA_INSTALL_PATH=$CUDA_HOME # Needs adding to the modulefile?\r\nmkdir ~\/cuda-sdk\r\ncd ~\/cuda-sdk\r\ncp -r $CUDA_SDK . # notice the '.' at the end of this command!\r\ncd 4.2.9\r\nmake -k\r\n\r\n# Run one of the examples (deviceQuery) while still on the GPU node\r\n.\/C\/bin\/linux\/release\/deviceQuery\r\n.\/OpenCL\/bin\/linux\/release\/oclDeviceQuery\r\n\r\n# End your interactive session\r\nexit\r\n\r\n# You are now back on the login node\r\n<\/pre>\n
Further info<\/h2>\n
\n