The Computational Shared Facility 3

Nvidia GPU jobs

Access

This page covers the Nvidia v100 and A100 GPUs.

Access to the GPU nodes is not automatic. Jobs will not run unless you have been added to a GPU group. Please read about the different levels of access below then contact us via its-ri-team@manchester.ac.uk to request access.

Contributor Access

If you are a member of a contributing research group (i.e., your PI / Supervisor has funded some GPUs in the system) then please email your request to use the GPUs to the email address below, cc-ing your supervisor.

If your research group is interested in contributing funds for GPU nodes please contact its-ri-team@manchester.ac.uk for more information.

Free at Point of Use Access

There are also free-at-the-point-of-use GPUs available, with job limits in place. These GPUs were funded by the University as part of the Research Lifecycle Programme, in particular Change Project M&K.

If you would like to access these GPUs please email a request to its-ri-team@manchester.ac.uk and provide some brief information about what you wish to use them for.

Free-at-point-of-use job limits: Each person in the free-at-point-of-use group may have two GPUs in use at any one time, provided there are resources available (the limit has been increased from one GPU to two as of December 2020 following further investment by the Research Lifecycle Programme, in particular Change Project M&K.)

The maximum number that can be used by this group in total at any one time is 36, again, provided there are resources available.

May 2022 Update

7th May 2022: The Nvidia GPU driver on all GPU compute nodes has been upgraded to allow use of the CUDA 11.6.x toolkit. The driver version is now 510.47.03.

Nov 2021 Update

The new Nvidia A100 GPU nodes, with AMD EPYC CPU host nodes, have been installed. These nodes have mostly been funded by specific research groups and so access is very restricted. They are currently in testing by the contributing groups. There is a small amount of free-at-point-of-use access available. However, only users with a specific requirement to use A100 GPUs will be granted access. If there is a specific feature your code will be using or you can demonstrate that your code exceeds the GPUs memory of the v100 GPUs then we will consider access. Please see below for further details on the nodes.

Software Applications

A range of GPU capable software is available on the CSF.

List of installed GPU capable software.
List of installed Machine Learning specific software.

GPU Hardware and Driver

The CSF contains 17 GPU nodes each hosting 4 x Nvidia v100 GPUs (16GB GPU RAM) giving a total of 68 v100 GPUs. The node spec is:

  • 4 x NVIDIA v100 SXM2 16GB GPU (Volta architecture – hardware v7.0, compute architecture sm_70)
  • Some GPU hosts: 2 x 16-core Intel Xeon Gold 6130 “Skylake” 2.10GHz
  • Some GPU hosts: 2 x 16-core Intel Xeon Gold 5128 “Cascade Lake” 2.30GHz
  • 192 GB RAM (host)
  • 1.6TB NVMe local storage, 182GB local SSD storage
  • CUDA Driver 510.47.03

The CSF also contains 8 GPU nodes each hosting 4 x Nvidia A100 GPUs (80GB GPU RAM) giving a total of 32 A100 GPUs. The node spec is:

  • 4 x NVIDIA HGX A100 SXM4 80GB GPU (Ampere architecture – hardware v8.0, compute architecture sm_80)
  • 2 x 24-core AMD Epyc 7413 “Milan” 2.65GHz
  • 512 GB RAM (host)
  • 1.6TB local NVMe storage, 364GB local SSD storage
  • CUDA Driver 510.47.03

The very fast local NVMe storage is available as ~/localscratch on each node (you nust access this from your jobscript on the node, not on the login node.) See below for advice on how to use this in your jobs. You MUST also remove your data from ~/localscratch at the end of your job.

Batch jobs running on the GPU nodes have a maximum runtime of 4 days.
Interactive GPU jobs have a maximum runtime of 1 hour.

Job Basics

Batch and interactive jobs can be run. Unlike the CSF2, you must now specify how many GPUs your job requires AND how many CPU cores you need for the host code. A job can use up to 8 CPU cores per GPU. See below for example jobscripts.

#!/bin/bash --login
#$ -cwd

# Choose ONE of the following depending on your permitted access
# The -l nvidia_v100 (or just v100) or -l nvidia_a100 (or just a100) is mandatory.
# M can be 1 to 4 (depending on user limits). If '=M' is missing
# then 1 will be used.
#$ -l nvidia_v100=M
#$ -l nvidia_a100=M

# The -pe line is optional. Number of CPU cores N can be 2..32 (v100 GPU) or 2..48 (A100 GPU)
# (i.e., max 8 per v100 GPU, max 12 per A100 GPU). Will be a serial job if this line is missing.
#$ -pe smp.pe N

Runtime Limits

The maximum runtimes on the GPUs are as follows:

  • batch jobs: 4 days
  • interactive jobs: 1 day

CUDA Libraries

You will most likely need the CUDA software environment for your job, whether your application is pre-compiled (e.g., a python app) or an application you have written yourself and compiled using the Nvidia nvcc compiler. Please see our CUDA libraries documentation for advice on compiling your own code.

To always use the most up-to-date version installed use:

# The main CUDA library and compiler (other libs have separate modulefiles - see below)
module load libs/cuda

# Alternatively use the Nvidia HPC SDK which provides a complete set of CUDA libraries and tools
module load libs/nvidia-hpc-sdk

If your application requires a specific version, or you want to fix on a specific version for reproducibility reasons, use:

module load libs/cuda/11.6.2        # This is now the default for 'module load libs/cuda'
module load libs/cuda/11.2.0
module load libs/cuda/11.1.1
module load libs/cuda/11.0.3
module load libs/cuda/10.1.243
module load libs/cuda/10.1.168
module load libs/cuda/9.2.148  
module load libs/cuda/9.1.85
module load libs/cuda/9.0.176
module load libs/cuda/8.0.61
module load libs/cuda/7.5.18

# To see available versions:
module avail libs/cuda

The Nvidia cuDNN, NCCL and TensorRT libraries are also available. See:

module avail libs/cuDNN
module avail libs/nccl
module avail libs/tensorrt

For more information on available libraries and how to compile CUDA code please see our CUDA page.

Which GPUs to Use (environment variables)

When a job or interactive session runs, the batch system will set the environment variable $CUDA_VISIBLE_DEVICES to a comma-separated list of GPU IDs assigned to your job, where IDs can be one or more of 0,1,2,3 (for example 2 for a single GPU job which is using the GPU with id 2, or 0,4 for a 2-GPU job – the IDs might not be contiguous.) The CUDA library will read this variable automatically and so most CUDA applications already installed on the CSF will simply use the correct GPUs.

You may have to tell your application how many GPUs to use (e.g., with a command-line flag – please check the application’s documentation). The batch system sets the variable $NGPUS to the number of GPUs you requested. Both of the environment variables $CUDA_VISIBLE_DEVICES and $NGPUS can be used in your jobscript. See the example jobscripts below for how this can be useful.

When developing your own CUDA applications the device IDs used in the cudaSetDevice() function should run from 0 to NGPUS-1. An ID of 0 means the “use the GPU who’s ID is listed first in the $CUDA_VISIBLE_DEVICES list” and so on.

A Simple First Job – deviceQuery

Create a jobscript as follows:

#!/bin/bash --login
#$ -cwd
#$ -l v100               # Will give us 1 GPU to use in our job
                         # No -pe line hence a serial (1 CPU-core) job
                         # Can instead use 'a100' for the A100 GPUs (if permitted!)

echo "Job is using $NGPUS GPU(s) with ID(s) $CUDA_VISIBLE_DEVICES and $NSLOTS CPU core(s)"

# Get the CUDA software libraries and applications 
module load libs/cuda

# Run the Nvidia app that reports GPU statistics
deviceQuery

Submit the job using qsub jobscript. It will print out hardware statistics about the GPU device.

See below for more complex jobscripts.

NVMe fast local storage

The GPU host nodes contain a 1.6TB NVMe storage card. This is faster than SSD drive storage (and faster than your scratch area and the home storage area).

This extra storage on the GPU nodes is accessible via the shortcut in your home directory:

~/localscratch

This will access a private directory in the /tmp area (please do not use /tmp directly).

It is highly recommended (especially for machine learning workloads) that you copy your data to ~/localscratch at the start of the job, process it from there, copy any results back to your ~/scratch area and then remove the data from ~/localscratch at the end of the job. If your job performs a lot of I/O (e.g., reading large datasets, writing results) then doing so from ~/localscratch on the GPU nodes will be faster. Even with the cost of copying data to and from the NVMe cards (~/localscratch), using this area during the job usually provides good speed-up.

Remember that ~/localscratch is local to the compute node. So after your job has finished, you will not be able to access any files on the compute node from the login node (the ~/localscratch on the login node points to a different hard-disk to that in the compute nodes). So you must ensure you do any file transfers back to the usual ~/scratch area (or your home area) within the jobscript.

Here is an example of copying data to the ~/localscratch area at the start of the job, processing the data and then cleaning up at the end of the job:

#!/bin/bash --login
#$ -cwd
#$ -l nvidia_v100            # Can instead use 'nvidia_a100' for the A100 GPUs (if permitted!)


# Copy a directory of files from scratch to the GPU node's local NVMe storage
cp -r ~/scratch/dataset1 ~/localscratch

# Process the data with a GPU app, from within the local NVMe storage area
cd ~/localscratch/dataset1
some_GPU_app -i input.dat -o results.dat

# Copy the results back to the main scratch area
cp results.dat ~/scratch/dataset1

# Tidy-up the local NVMe storage on the GPU node
rm -rf ~/localscratch/dataset1

The above jobscript can be in your home or scratch storage. Submit it from there.

Monitoring GPU jobs

We have written a script to help monitor your GPU jobs. You can run the following on the CSF login nodes:

# Get a list of your GPU jobs in the batch system (similar to qstat)
gpustat

# Display the status of the GPUs in use by one of your running jobs (job id needed)
gpustat -j jobid
             #
             # Repace jobid with your job id number (e.g., 12345)

# Display the status of the GPUs in use by one of your job-array tasks (job id & task id needed)
gpustat -j jobid -t taskid
             #        # 
             #        # Replace taskid with your job array task id
             #        # number (e.g., 1)
             #
             # Repace jobid with your job id number (e.g., 12345)

Etiquette

All users are reminded to log out of their interactive GPU session when it is no longer required. This will free up the GPU for other users. If an interactive GPU session is found to be idle for significant periods, making no use of the GPU, it may be killed. Interactive sessions should not be used to reserve a GPU for future use – only request a GPU when you need to use it.

Batch jobs that only use CPU cores should not be submitted to GPU nodes. If such jobs are found they will be killed and access to GPU nodes may be removed. There are plenty of CPU-only nodes on which jobs will run.

Batch Jobs – Example Jobscripts

The following section provides sample jobscripts for various combinations of number of GPUs requested and CPU cores requested.

Note that in the examples below, we load modulefiles inside the jobscript, rather than on the login node. This is so we have a complete record in the jobscript of how we ran the job. There are two things we need to do to make the module command work inside the jobscript: add --login to the first line, then remove any #$ -V line from the jobscript.

Single GPU, Single CPU-core

The simplest case – a single-GPU, single-CPU-core jobscript:

#!/bin/bash --login
#$ -cwd
#$ -l nvidia_v100=1           # If no =N given, 1 GPU will be assumed.
                              # (Can use 'v100' instead of 'nvidia_v100'.)
                              # No $# -pe line give means a 1-core jobs.
                              # Can instead use 'nvidia_a100' for the A100 GPUs (if permitted!)

# Latest version of CUDA (add any other modulefiles you require)
module load libs/cuda

echo "Job is using $NGPUS GPU(s) with ID(s) $CUDA_VISIBLE_DEVICES and $NSLOTS CPU core(s)"

# Run an application (this Nvidia app will report info about the GPU). Replace with your app.
deviceQuery

Single GPU, Multi CPU-cores

Even when using a single GPU, you may need more than one CPU core if your host-code uses OpenMP, for example, to do some parallel processing on the CPU. You can request up to 8 CPU cores per v100 GPU and up to 12 CPU cores per A100 GPU. For example:

#!/bin/bash --login
#$ -cwd
#$ -l v100           # A 1-GPU request (v100 is just a shorter name for nvidia_v100)
                     # Can instead use 'a100' for the A100 GPUs (if permitted!)
#$ -pe smp.pe 8      # 8 CPU cores available to the host code
                     # Can use up to 12 CPUs with an A100 GPU.

# Latest version of CUDA
module load libs/cuda

echo "Job is using $NGPUS GPU(s) with ID(s) $CUDA_VISIBLE_DEVICES and $NSLOTS CPU core(s)"

# This example uses OpenMP for multi-core host code - tell OpenMP how many CPU cores to use.
export OMP_NUM_THREADS=$NSLOTS

./mySimpleGPU_OpenMP_app

Multi GPU, Single CPU-core

A multi-GPU job should request the required number of GPUs and optionally up to 8 CPU cores per v100 GPU and up to 12 CPU cores per a100 GPU.

For example a 2-GPU job that runs serial host code on one CPU core would be:

#!/bin/bash --login
#$ -cwd
#$ -l nvidia_v100=2            # A 2-GPU job. Can use 'v100' instead of 'nvidia_v100'.
                               # Can instead use 'nvidia_a100' for the A100 GPUs (if permitted!)
                               # No -pe smp.pe line means it is a serial job.
# Latest version of CUDA
module load libs/cuda

echo "Job is using $NGPUS GPU(s) with ID(s) $CUDA_VISIBLE_DEVICES and $NSLOTS CPU core(s)"

./myMultiGPUapp.exe

Multi GPU, Multi CPU-cores

Finally a multi-GPU job that also uses multiple CPU cores for the host code (up to 8 CPUs per v100 GPU, up to 12 CPUs per A100 GPU) would be:

#!/bin/bash --login
#$ -cwd
#$ -l v100=4         # A 4-GPU request (v100 is just a shorter name for nvidia_v100)
                     # Can instead use 'a100' for the A100 GPUs (if permitted!)
#$ -pe smp.pe 32     # Let's use the max 8 CPUs per GPU (32 cores in total)
                     # Can use a maximum of 48 CPUs with 4 x A100 GPUs.

# Latest version of CUDA
module load libs/cuda

echo "Job is using $NGPUS GPU(s) with ID(s) $CUDA_VISIBLE_DEVICES and $NSLOTS CPU core(s)"

# This example uses OpenMP for multi-core host code - tell OpenMP how many CPU cores to use.
export OMP_NUM_THREADS=$NSLOTS

./myMultiGPU_OpenMP_app

Multi GPU, Multi CPU-cores for MPI Apps

Multi-GPU applications are often implemented using the MPI library – each MPI process (aka rank) uses a GPU to speed up its computation.

Our GPUs are run in EXCLUSIVE_PROCESS mode, meaning only one process can use a GPU at any one time. This is to protect your GPUs from other jobs – if another job accidentally tries to use a GPU in use by your job, the other job will fail. But it also means that, when running an MPI application, each GPU can only be used by one MPI process. Hence you will usually request the same number of GPUs and CPUs.

The following CUDA-aware version of the OpenMPI libraries are available. This will usually give better performance when your application uses MPI to transfer data from one GPU to another (note that the openmpi modulefile will automatically load the cuda modulefile):

# GCC Compiler
module load mpi/gcc/openmpi/4.0.1-cuda               # CUDA 10.1.168

# Intel Compiler
module load mpi/intel-18.0/openmpi/4.0.1-cuda        # CUDA 10.1.168
module load mpi/intel-17.0/openmpi/3.1.3-cuda        # CUDA 9.2.148
module load mpi/intel-17.0/openmpi/3.1.1-cuda        # CUDA 9.2.148

Note that when running multi-GPU jobs using MPI you usually start one MPI process per GPU. For example:

#!/bin/bash --login
#$ -cwd
#$ -l v100=4         # A 4-GPU request (v100 is just a shorter name for nvidia_v100)
                     # Can instead use 'a100' for the A100 GPUs (if permitted!)
#$ -pe smp.pe 4      # Our MPI code only uses one MPI process (hence one CPU core) per GPU

# MPI library (which also loads the cuda modulefile)
module load mpi/intel-18.0/openmpi/4.0.1-cuda

echo "Job is using $NGPUS GPU(s) with ID(s) $CUDA_VISIBLE_DEVICES and $NSLOTS CPU core(s)"

# In this example we start one MPI process per GPU. We could use $NSLOTS or $NGPUS (both = 4)
# It is assume the application will ensure each MPI process uses a different GPU. For example
# MPI rank 0 will use GPU 0, MPI rank 1 will use GPU 1 and so on.
mpirun -n $NGPUS ./myMultiGPU_MPI_app

# If your application does not map MPI ranks to GPUs correctly, you can try the following method
# where we explictly inform each rank which GPU to use via the CUDA_VISIBLE_DEVICES variable:
mpirun -n $NGPUS -x CUDA_VISIBLE_DEVICES=0 ./myMultiGPU_MPI_app : \
                 -x CUDA_VISIBLE_DEVICES=1 ./myMultiGPU_MPI_app : \
                 -x CUDA_VISIBLE_DEVICES=2 ./myMultiGPU_MPI_app : \
                 -x CUDA_VISIBLE_DEVICES=3 ./myMultiGPU_MPI_app 

Note that it is possible to use a multi-threaded application (implemented using OpenMP for example to create multiple threads). Even though the GPUs are in EXCLUSIVE_PROCESS mode, multiple OpenMP threads can access a different or the same GPU. This is one way to increase the usage of a single GPU if your CUDA kernels do not fully utilise a GPU.

An alternative method, which allows multiple MPI processes to run on the same GPU is now available – please see the section below on the Nvidia MPS facility.

Interactive Jobs

You mainly use interactive jobs to run an app on a GPU node that has a GUI or to log-in to a GPU node to do app development and testing.

Interactive jobs should be done using qrsh (not qsub) from the login node as follows.

Note: Unlike the CSF2, there is no need to add the -l inter flag to the qrsh command-line. The use of qrsh indicates you require an interactive job.

Single GPU, Single CPU-core logging in to GPU node

Here we request an interactive session using 1-GPU and 1-CPU core, logging in to the node

qrsh -l v100 bash
  #            #
  #            # You must supply the name of the shell to login with
  #
  # Wait until you are logged in to a GPU node (e.g. node800).
  # You can now load modulefiles (e.g., libs/cuda) and run your apps.
  # The CUDA_VISIBLE_DEVICES environment variable says which GPUs you have access to.

# Can instead use a100 for the A100 GPUs (if permitted!)
qrsh -l a100 bash

The above command will place you in your home directory when it logs you in to the GPU node. If you wish to remain in the directory from where you run the qrsh, add the -cwd flag to the command:

qrsh -l v100 -cwd bash

# Can instead use a100 for the A100 GPUs (if permitted!)
qrsh -l a100 -cwd bash

GPU qrsh jobs are limited to 24 hours.

Multi GPU, Multi CPU-cores logging in to GPU node

Here we start an interactive session requesting 2-GPUs and 4-CPU cores, logging in to the node:

qrsh -l v100=2 -pe smp.pe 4 bash
  #
  # Wait until you are logged in to a GPU node (e.g. node800).
  # You can now load modulefiles (e.g., libs/cuda) and run your apps.
  # The CUDA_VISIBLE_DEVICES environment variable says which GPUs you have access to.

Multi GPU, Multi CPU-cores running app on GPU node

Here we request 2 x v100 GPUs and 16 CPU cores (max for the v100, it could be up to 24 CPU cores for the A100 GPUs), running my own executable/binary (-b y) from the current directory (-cwd) inheriting the modulefile settings from the login node (-V). My sample application takes some custom flags that it understands:

module load libs/cuda/10.1.168
qrsh -l v100=2 -pe smp.pe 16 -b y -cwd -V ./myMultiGPUapp -in mydata.dat -out myresults.dat

# Can instead use a100 for the A100 GPUs (if permitted!) and up to 24 CPU cores for 2 x A100 GPUs
qrsh -l a100=2 -pe smp.pe 24 -b y -cwd -V ./myMultiGPUapp -in mydata.dat -out myresults.dat

Nvidia Multi-Process Service (MPS)

Our GPUs all use EXCLUSIVE_PROCESS mode – meaning only one process can access a GPU. This protects the GPU allocated to your job from being used accidentally by another job that might be running on the same compute node.

But there are times when you might want to run more than one process (app) on the GPU allocated to your job.

The Nvidia Multi-Process Service (MPS) allows multiple processes to use the same GPU. You might want to do this for small MPI jobs, where each MPI process does not require the resources of an entire GPU. Hence all of the MPI processes could “fit” on a single GPU. Alternatively, if you have a lot of small jobs to run, you might be able to start multiple copies of the executable, all using the same GPU. Using MPI (mpirun) would be one method of doing this, even if the app itself is not an MPI job.

An extra flag is required to start the NVMPS facility on the node allocated to your job. Hence you should add:

#$ -ac nvmps

to your jobscript (or qsub command.)

Note that you should still request enough CPU cores on which to run multiple processes. Even a GPU app does some work on the CPU and so if you are going to run several copies of an app, you should request the correct number of CPU cores so that each instance of your app has its own core(s) to run on. The examples below request 8 CPU cores (-pe smp.pe 8) so that we can run 8 copies of a GPU-capable application.

The following example demonstrates running the simpleMPI example found in the CUDA SDK on a single GPU. Multiple MPI processes are started and they all run on the same GPU. Without MPS, a GPU per MPI process would be required (see later for what happens if we run the same job without using MPS.)

#!/bin/bash --login
#$ -cwd
#$ -l v100=1         # We request only 1 GPU
#$ -pe smp.pe 8      # We want a CPU core for each process (see below)
#$ -ac nvmps         # Extra flag to enable Nvidia MPS

# Load a CUDA-aware MPI modulefile which will also load a cuda modulefile
module load mpi/gcc/openmpi/4.0.1-cuda-ucx

# Let's take a copy of the already-compiled simpleMPI example (the whole folder)
cp -a $CUDA_SDK/0_Simple/simpleMPI/ .
cd simpleMPI

# Now run more than 1 copy of the app. In fact we run with 8 MPI processes
# ($NSLOTS is replaced by the number of cores request on the -pe line above.)
# But we are only using 1 GPU, not 8! So all processes will use the same GPU.
mpirun -n $NSLOTS ./simpleMPI

Submit the above jobscript using qsub jobscript. The job output will be something similar to:

Running on 8 nodes
Average of square roots is: 0.667337
PASSED

You can also use the NV MPS facility with interactive jobs:

# At the CSF login node, start an interactive session, requesting one GPU, 8 CPU cores and enable the NVMPS facility
[username@hlogin1 [csf3] ~]$ qrsh -l v100=1 -pe smp.pe 8 -ac nvmps bash

# Wait until you are logged in to a GPU node, then:
module load mpi/gcc/openmpi/4.0.1-cuda-ucx
cp -a $CUDA_SDK/0_Simple/simpleMPI .
cd simpleMPI

# Run more MPI processes than the 1 GPU we requested. This will only work when
# the interactive session is using the NV MPS facility. The $NSLOTS will be replaced
# with 8 in this example because we requested 8 CPU cores on the qrsh line.
mpirun -n $NSLOTS ./simpleMPI

# Return to the login node
exit

If we run the above job without using the -ac nvmps flag, the job will fail, because the single GPU that we request will only accept one MPI process (because we run our GPUs in EXCLUSIVE_PROCESS mode.) Doing so will give you an error in the job output:

# Job output when trying to run multiple processes on a single GPU without using MPS facility
CUDA error calling "cudaMalloc((void **)&deviceInputData, dataSize * sizeof(float))", code is 46
CUDA error calling "cudaMalloc((void **)&deviceInputData, dataSize * sizeof(float))", code is 46
...
Test FAILED
Test FAILED
...

Profiling Tools

A number of profiling tools are available to help analyse and optimize your CUDA applications. We provide instructions on how to run (start) these tools below. Please note that instructions on how to use these tools are beyond the scope of this webpage. You should consult the Nvidia profiling documentation for detailed instructions on how to use the tools listed below.

We give the command name of each tool below. If running the profiler tool through its graphical user interface (GUI) or interactively on the command-line (i.e., not in a batch job which would be collecting profiling data without any interaction) then you must start an interactive session on a backend GPU node using the commands:

# On the CSF login node, request an interactive session on a GPU node
qrsh -l v100=1 bash
  # Can instead use a100 for the A100 GPUs (if permitted!)
  qrsh -l a100=1 bash

# Wait to be logged in to the node, then run:
module load libs/cuda/10.1.168                # Choose your required version
name-of-profiler-tool                         # See below for the command names

Nsight Compute

The Nvidia Nsight Compute profile tools are installed as of toolkit version 10.0.130 and later. To run the profiler:

nv-nsight-cu        # GUI version
nv-nsight-cu-cli    # Command-line version

Nsight Systems

The Nvidia Nsight Systems performance analysis tool designed to visualize an application’s algorithms is installed as of toolkit version 10.1.168. To run the profiler:

nsight-sys

Nvidia recommend you use the above newer tools for profiling rather than the following older tools, although these tools are still available and may be familiar to you.

Visual Profiler

The Nvidia Visual Profiler is installed as of toolkit version 7.5.18 and later. To run the profiler:

nvvp

Note that the Nvidia Visual Profiler nvvp can be used to view results collected by the nvprof command-line tool (see below). Hence you could use the the nvprof command in a batch job which will save profiling data to file, then view the results at a later time using the nvvp tool.

nvprof Command-line Profiler

The Nvidia command-line nvprof profiler is installed as of toolkit version 7.5.18 and later. To run the profiler:

nvprof

Note that the Nvidia Visual Profiler nvvp (see above) can be used to view results collected by the nvprof command-line tool. Hence you could use the the nvprof command in a batch job which will save profiling data to file, then view the results at a later time using the nvvp tool.

Last modified on November 15, 2022 at 10:18 am by George Leaver