GPU Node (jabberwock)

UPDATE – 2016

The GPU node in the iCSF is no longer available.

The iCSF now has a node named jabberwock containing three AMD FireGL v7800 GPUs. This node can be used to run graphical OpenGL programs such as VMD, Paraview and Abaqus. The iCSF shares your CSF home directory and so this is a convenient place to run these visualization applications (you do not need to download large result files once a simulation running on the CSF has finished, for example).

By running on the jabberwock node of the iCSF your application will use jabberwock’s GPUs to render images, allowing you to interact with your model (e.g., a molecule or finite element mesh) while the application remains responsive.

Please email its-ri-team@manchester.ac.uk to request access to the GPUs in jabberwock. Your username must be added to the correct unix group.

Specification

  • Two 4-core Intel Westmere CPUs
  • 144 GB RAM
  • 3 x AMD FireGL v7800 GPU

Why use the GPU Node?

If you run your application on a non-GPU node then the application will send all of the data needed to render your models back to where the application’s GUI is being displayed. This might be the virtual desktop (e.g., nyx2 or nyx3) or it might be your local desktop. This is often slow because of the large amount of data being sent over the network (e.g., all of the data for a very dense finite element mesh). The local desktop may not have a GPU (the virtual desktops do not have GPUs) and so slower software rendering will be used to render the images. Hence interacting with the model can often feel sluggish.

By connecting to jabberwock, and running your application using the method described below, all of the rendering will be done on the GPUs and only the final rendered image is sent back to your desktop (or the virtual desktop). This is efficient due to the small amount of data in an image.

Follow the steps below to access the iCSF GPU node.

Connect to the nyx3 Virtual Desktop

You must log in to the nyx3 virtual desktop to access the iCSF GPU node. It has a fast, dedicated connection to the iCSF for accessing the GPU node.

Use the X2GO client to connect to the host:

nyx3.itservices.manchester.ac.uk

using your central University username and password.

Connect to the iCSF GPU node jabberwock

To assist with connecting to the iCSF GPU node (named jabberwock) for graphics use, we run an app named vglconnect (do not run ssh):

From the nyx3 virtual desktop, open a shell window (Applications>System Tools>Terminal) then connect to jabberwock using:

vglconnect -s jabberwock

This will ask for your iCSF password twice (it sets up two connections).

The vglconnect program runs a small app on nyx3 named vglclient which receives images from the GPUs. It then logs you in to jabberwock as normal (in fact vglconnect uses ssh).

Running Graphical Application using vglrun

One logged in to jabberwock you should set up to run your application using the usual module load commands (please see the software pages for specific details).

You then run your application using:

vglrun myapp.exe args

where myapp.exe is the command used to run your application and args are any optional additional flags your application may want.

The vglrun command will run your application and force it to use GPU 0 in jabberwock for any rendering.

VMD example

VMD is a graphical molecular dynamics package. We run it as follows:

  1. On your desktop use x2goclient to connect to nyx3.itservices.manchester.ac.uk
  2. On the nyx3 virtual desktop, open a shell window (Applications>System Tools>Terminal) and run vglconnect -s jabberwock (enter your iCSF password twice)
  3. On jabberwock run 
    module load apps/binapps/vmd/1.9.1
vglrun vmd

    This will display the VMD GUI on the virtual desktop but molecule rendering is done on GPU 0 of jabberwock – see diagram below. The sample molecule is available by loading in to VMD the visualization state file: 

    /opt/gridware/apps/binapps/vmd/sample_data/1F45.vmd

VMD on the nyx3 virtual desktop

Paraview Example

Paraview is a powerful scientific data visualization tool that can read many file formats and visualize the data using many different visualization techniques. We run it as follows:

  1. On your desktop use x2goclient to connect to nyx3.itservices.manchester.ac.uk
  2. On the nyx3 virtual desktop, open a shell window (Applications>System Tools>Terminal) and run vglconnect -s jabberwock (enter your iCSF password twice)
  3. On jabberwock run 
    module load apps/binapps/paraview/4.1.0
vglrun -d :0.2 paraview
        #
        # In this example we have used GPU 2 in jabberwock
        # (perhaps GPUs 0 and 1 are in use)

    This will display the Paraview GUI on the virtual desktop but the dataset rendering is done on GPU 2 of jabberwock (see diagram below – you will need to load in your own dataset).

Paraview on the nyx3 virtual desktop

Selecting a GPU

There are three GPUs in jabberwock. They can be selected by running one of the following:

vglrun -d :0.0 myapp.exe
vglrun -d :0.1 myapp.exe
vglrun -d :0.2 myapp.exe

If no GPU number is specified then GPU 0 (:0.0) will be used.

We are working on a method to automatically select a free GPU.

Frequently Asked Questions

Should I always go via nyx3.itservices.manchester.ac.uk if wanting to use the iCSF GPUs?
Yes. nyx3 is the only node that can see the iCSF GPU node (jabberwock). It has a fast dedicated link to jabberwock so that graphics images can be sent to your application as fast as possible.
How do I access nyx3?
Use the X2GO software (x2goclient) on your desktop or laptop which will give you a virtual linux desktop on nyx3. You can run x2goclient on Windows, Mac or Linux desktops. See our X2GO Virtual Desktop instructions.
Why use vglconnect to get on to jabberwock?
vglconnect will run a small program named vglclient on nyx3 which receives the rendered images from the jabberwock GPUs. It then logs you in to jabberwock as normal (using ssh). It creates two connections so will ask for your iCSF password twice. The vglclient program will display the rendered images in your application’s GUI as normal.
What is my jabberwock password?
The same one you use to log in to the iCSF incline node (your central University username and password).
Can I log in to the CSF and iCSF incline nodes from nyx3?
Yes, nyx3 can be used as your single virtual desktop to access all RI computational facilities. Use ssh to log in to the various facilities just like you did from the nyx2 virtual desktop.
My app freezes or doesn’t display any graphics even when I use vglrun. Why?
You probably used ssh to log in to jabberwock from nyx3. You should use 

vglconnect -s jabberwock

to connect to jabberwock from nyx3. Log out of jabberwock and re-connect using the correct method.

Graphics are slow, parts of the model are missing, I get XRequest.147 errors. Why?
You didn’t run your application using vglrun. Hence your application isn’t using the iCSF GPUs but instead uses slow software rendering on nyx3. Re-run your application using 

vglrun app.exe
I think there’s a problem with the vglclient running on nyx3. How do I kill it?
The vglclient is started automatically on nyx3 by vglconnect before it logs you in to jabberwock. It will run in the background until you log out of the nyx3 virtual desktop (logging out of jabberwock does not kill the vglclient – it is perfectly OK to leave it running). Only one copy of vglclient will run even if you log in to jabberwock several times. You shouldn’t need to stop or start the vglclient manually. However, if you wish to kill the vglclient running on nyx3 use the following in a shell window on nyx3

vglclient -kill

When you next run vglconnect to log in to jabberwock it will automatically start a new copy of vglclient.

Last modified on June 29, 2016 at 10:57 am by Site Admin