{"id":299,"date":"2014-04-07T09:41:12","date_gmt":"2014-04-07T09:41:12","guid":{"rendered":"http:\/\/ri.itservices.manchester.ac.uk\/icsf\/?page_id=299"},"modified":"2016-06-29T10:57:14","modified_gmt":"2016-06-29T10:57:14","slug":"gpus","status":"publish","type":"page","link":"https:\/\/ri.itservices.manchester.ac.uk\/icsf\/user-docs\/gpus\/","title":{"rendered":"GPU Node (jabberwock)"},"content":{"rendered":"<h3>UPDATE &#8211; 2016<\/h3>\n<p>The GPU node in the iCSF is <strong>no longer available<\/strong>.<\/p>\n<p>The iCSF now has a node named <em>jabberwock<\/em> containing three AMD FireGL v7800 GPUs. This node can be used to run graphical <em>OpenGL<\/em> programs such as VMD, Paraview and Abaqus. The iCSF shares your CSF home directory and so this is a convenient place to run these visualization applications (you do not need to download large result files once a simulation running on the CSF has finished, for example).<\/p>\n<p>By running on the <em>jabberwock<\/em> node of the iCSF your application will use <em>jabberwock&#8217;s<\/em> GPUs to render images, allowing you to interact with your model (e.g., a molecule or finite element mesh) while the application remains responsive.<\/p>\n<p>Please email <a href=\"&#x6d;&#x61;&#105;l&#x74;&#x6f;&#x3a;&#105;t&#x73;&#x2d;&#x72;&#105;-&#x74;&#x65;&#x61;&#109;&#64;&#x6d;&#x61;&#x6e;&#99;h&#x65;&#x73;&#x74;&#101;r&#x2e;&#x61;&#x63;&#46;u&#x6b;\">&#105;&#x74;s&#45;&#x72;i&#45;&#x74;e&#x61;&#x6d;&#64;&#x6d;a&#110;&#x63;h&#101;&#x73;t&#x65;&#x72;&#46;&#x61;&#x63;&#46;&#x75;k<\/a> to request access to the GPUs in jabberwock. Your username must be added to the correct unix group.<\/p>\n<p>Specification<\/p>\n<ul>\n<li>Two 4-core Intel Westmere CPUs<\/li>\n<li>144 GB RAM<\/li>\n<li>3 x AMD FireGL v7800 GPU<\/li>\n<\/ul>\n<h2>Why use the GPU Node?<\/h2>\n<p>If you run your application on a non-GPU node then the application will send all of the data needed to render your models back to where the application&#8217;s GUI is being displayed. This might be the <em>virtual desktop<\/em> (e.g., <em>nyx2<\/em> or <em>nyx3<\/em>) or it might be your local desktop. This is often slow because of the large amount of data being sent over the network (e.g., all of the data for a very dense finite element mesh). The local desktop may not have a GPU (the <em>virtual desktops<\/em> do not have GPUs) and so slower <em>software rendering<\/em> will be used to render the images. Hence interacting with the model can often feel sluggish.<\/p>\n<p>By connecting to <em>jabberwock<\/em>, and running your application using the method described below, all of the rendering will be done on the GPUs and only the final rendered image is sent back to your desktop (or the <em>virtual desktop<\/em>). This is efficient due to the small amount of data in an image.<\/p>\n<p>Follow the steps below to access the iCSF GPU node.<\/p>\n<h2>Connect to the <em>nyx3<\/em> Virtual Desktop<\/h2>\n<p>You must log in to the <em>nyx<strong>3<\/strong><\/em> virtual desktop to access the iCSF GPU node. It has a fast, dedicated connection to the iCSF for accessing the GPU node.<\/p>\n<p>Use the <a href=\"http:\/\/ri.itservices.manchester.ac.uk\/virtual-desktop-service-x2go\/\">X2GO client<\/a> to connect to the host:<\/p>\n<pre>\r\nnyx3.itservices.manchester.ac.uk\r\n<\/pre>\n<p>using your central University username and password.<\/p>\n<h2>Connect to the iCSF GPU node <em>jabberwock<\/em><\/h2>\n<p>To assist with connecting to the iCSF GPU node (named <em>jabberwock<\/em>) for graphics use, we run an app named <em>vglconnect<\/em> (do <strong>not<\/strong> run ssh):<\/p>\n<p>From the <em>nyx3<\/em> virtual desktop, open a shell window (<em>Applications>System Tools>Terminal<\/em>) then connect to <em>jabberwock<\/em> using:<\/p>\n<pre>\r\nvglconnect -s jabberwock\r\n<\/pre>\n<p>This will ask for your iCSF password <strong>twice<\/strong> (it sets up two connections). <\/p>\n<p>The <em>vglconnect<\/em> program runs a small app on nyx3 named <em>vglclient<\/em> which receives images from the GPUs. It then logs you in to jabberwock as normal (in fact <em>vglconnect<\/em> uses <em>ssh<\/em>).<\/p>\n<h2>Running Graphical Application using <em>vglrun<\/em><\/h2>\n<p>One logged in to <em>jabberwock<\/em> you should set up to run your application using the usual <code>module load<\/code> commands (please see the <a href=\"\/icsf\/software\/\">software pages<\/a> for specific details).<\/p>\n<p>You then run your application using:<\/p>\n<pre>\r\nvglrun myapp.exe <em>args<\/em>\r\n<\/pre>\n<p>where <em>myapp.exe<\/em> is the command used to run your application and <em>args<\/em> are any optional additional flags your application may want.<\/p>\n<p>The <code>vglrun<\/code> command will run your application and force it to use GPU 0 in jabberwock for any rendering.<\/p>\n<h3>VMD example<\/h3>\n<p>VMD is a graphical molecular dynamics package. We run it as follows:<\/p>\n<ol>\n<li>On your desktop use <em>x2goclient <\/em> to connect to <code>nyx3.itservices.manchester.ac.uk<\/code><\/li>\n<li>On the <em>nyx3<\/em> virtual desktop, open a shell window (<em>Applications>System Tools>Terminal<\/em>) and run <code>vglconnect -s jabberwock<\/code> (enter your iCSF password twice)<\/li>\n<li>On <em>jabberwock<\/em> run\n<pre>\r\nmodule load apps\/binapps\/vmd\/1.9.1\r\nvglrun vmd\r\n<\/pre>\n<p>This will display the VMD GUI on the virtual desktop but molecule rendering is done on GPU 0 of jabberwock &#8211; see diagram below. The sample molecule is available by loading in to VMD the visualization state file: <\/p>\n<pre>\/opt\/gridware\/apps\/binapps\/vmd\/sample_data\/1F45.vmd<\/pre>\n<\/li>\n<\/ol>\n<p><a href=\"https:\/\/ri.itservices.manchester.ac.uk\/icsf\/wp-content\/uploads\/x2go-nyx3-icsf-vmd-red.png\"><img decoding=\"async\" src=\"https:\/\/ri.itservices.manchester.ac.uk\/icsf\/wp-content\/uploads\/x2go-nyx3-icsf-vmd-red.png\" alt=\"VMD on the nyx3 virtual desktop\" width=\"500\" class=\"alignnone size-full wp-image-346\" srcset=\"https:\/\/ri.itservices.manchester.ac.uk\/icsf\/wp-content\/uploads\/x2go-nyx3-icsf-vmd-red.png 981w, https:\/\/ri.itservices.manchester.ac.uk\/icsf\/wp-content\/uploads\/x2go-nyx3-icsf-vmd-red-300x231.png 300w\" sizes=\"(max-width: 981px) 100vw, 981px\" \/><\/a><\/p>\n<h3>Paraview Example<\/h3>\n<p>Paraview is a powerful scientific data visualization tool that can read many file formats and visualize the data using many different visualization techniques. We run it as follows:<\/p>\n<ol>\n<li>On your desktop use <em>x2goclient <\/em> to connect to <code>nyx3.itservices.manchester.ac.uk<\/code><\/li>\n<li>On the <em>nyx3<\/em> virtual desktop, open a shell window (<em>Applications>System Tools>Terminal<\/em>) and run <code>vglconnect -s jabberwock<\/code> (enter your iCSF password twice)<\/li>\n<li>On <em>jabberwock<\/em> run\n<pre>\r\nmodule load apps\/binapps\/paraview\/4.1.0\r\nvglrun -d :0.2 paraview\r\n        #\r\n        # In this example we have used GPU 2 in jabberwock\r\n        # (perhaps GPUs 0 and 1 are in use)\r\n<\/pre>\n<p>This will display the Paraview GUI on the virtual desktop but the dataset rendering is done on GPU 2 of jabberwock (see diagram below &#8211; you will need to load in your own dataset).<\/li>\n<\/ol>\n<p><a href=\"https:\/\/ri.itservices.manchester.ac.uk\/icsf\/wp-content\/uploads\/x2go-nyx3-icsf-paraview-red.png\"><img decoding=\"async\" src=\"https:\/\/ri.itservices.manchester.ac.uk\/icsf\/wp-content\/uploads\/x2go-nyx3-icsf-paraview-red.png\" alt=\"Paraview on the nyx3 virtual desktop\" width=\"500\" class=\"alignnone size-full wp-image-343\" srcset=\"https:\/\/ri.itservices.manchester.ac.uk\/icsf\/wp-content\/uploads\/x2go-nyx3-icsf-paraview-red.png 1499w, https:\/\/ri.itservices.manchester.ac.uk\/icsf\/wp-content\/uploads\/x2go-nyx3-icsf-paraview-red-300x181.png 300w, https:\/\/ri.itservices.manchester.ac.uk\/icsf\/wp-content\/uploads\/x2go-nyx3-icsf-paraview-red-1024x619.png 1024w\" sizes=\"(max-width: 1499px) 100vw, 1499px\" \/><\/a><\/p>\n<h2>Selecting a GPU<\/h2>\n<p>There are three GPUs in jabberwock. They can be selected by running one of the following:<\/p>\n<pre>\r\nvglrun -d :0.0 myapp.exe\r\nvglrun -d :0.1 myapp.exe\r\nvglrun -d :0.2 myapp.exe\r\n<\/pre>\n<p>If no GPU number is specified then GPU 0 (<code>:0.0<\/code>) will be used. <\/p>\n<p>We are working on a method to automatically select a free GPU.<\/p>\n<h2>Frequently Asked Questions<\/h2>\n<dl>\n<dt>Should I always go via <em>nyx3.itservices.manchester.ac.uk<\/em> if wanting to use the iCSF GPUs?<\/dt>\n<dd>Yes. <em>nyx3<\/em> is the only node that can see the iCSF GPU node (<em>jabberwock<\/em>). It has a fast dedicated link to jabberwock so that graphics images can be sent to your application as fast as possible.<\/dd>\n<dt>How do I access <em>nyx3?<\/em><\/dt>\n<dd>Use the X2GO software (<em>x2goclient<\/em>) on your desktop or laptop which will give you a virtual linux desktop on <em>nyx3<\/em>. You can run <em>x2goclient<\/em> on Windows, Mac or Linux desktops. See our <a href=\"http:\/\/ri.itservices.manchester.ac.uk\/virtual-desktop-service-x2go\/\">X2GO Virtual Desktop<\/a> instructions.<\/dd>\n<dt>Why use <em>vglconnect<\/em> to get on to jabberwock?<\/dt>\n<dd><em>vglconnect<\/em> will run a small program named <em>vglclient<\/em> on nyx3 which receives the rendered images from the jabberwock GPUs. It then logs you in to jabberwock as normal (using ssh). It creates two connections so will ask for your iCSF password twice. The <em>vglclient<\/em> program will display the rendered images in your application&#8217;s GUI as normal.<\/dd>\n<dt>What is my <em>jabberwock<\/em> password?<\/dt>\n<dd>The same one you use to log in to the iCSF <em>incline<\/em> node (your central University username and password).<\/dd>\n<dt>Can I log in to the CSF and iCSF <em>incline<\/em> nodes from <em>nyx3<\/em>?<\/dt>\n<dd>Yes, <em>nyx3<\/em> can be used as your single virtual desktop to access all RI computational facilities. Use <em>ssh<\/em> to log in to the various facilities just like you did from the <em>nyx2<\/em> virtual desktop.<\/dd>\n<dt>My app freezes or doesn&#8217;t display any graphics even when I use <em>vglrun<\/em>. Why?<\/dt>\n<dd>You probably used <em>ssh<\/em> to log in to <em>jabberwock<\/em> from nyx3. You should use <\/p>\n<pre>vglconnect -s jabberwock<\/pre>\n<p> to connect to jabberwock from nyx3. Log out of <em>jabberwock<\/em> and re-connect using the correct method.<\/dd>\n<dt>Graphics are slow, parts of the model are missing, I get <em>XRequest.147<\/em> errors. Why?<\/dt>\n<dd>You didn&#8217;t run your application using <em>vglrun<\/em>. Hence your application isn&#8217;t using the iCSF GPUs but instead uses slow software rendering on nyx3. Re-run your application using <\/p>\n<pre>vglrun <em>app.exe<\/em><\/pre>\n<\/dd>\n<dt>I think there&#8217;s a problem with the <em>vglclient<\/em> running on nyx3. How do I kill it?<\/dt>\n<dd>The <em>vglclient<\/em> is started automatically on <em>nyx3<\/em> by <em>vglconnect<\/em> before it logs you  in to <em>jabberwock<\/em>. It will run in the background until you log out of the nyx3 virtual desktop (logging out of <em>jabberwock<\/em> does not kill the vglclient &#8211; it is perfectly OK to leave it running). Only one copy of <em>vglclient<\/em> will run even if you log in to jabberwock several times. You shouldn&#8217;t need to stop or  start the <em>vglclient<\/em> manually. However, if you wish to kill the <em>vglclient<\/em> running on <em>nyx3<\/em> use the following in a shell window on <em>nyx3<\/em><\/p>\n<pre>vglclient -kill<\/pre>\n<p>When you next run <em>vglconnect<\/em> to log in to jabberwock it will automatically start a new copy of <em>vglclient<\/em>.<\/dd>\n<\/dl>\n","protected":false},"excerpt":{"rendered":"<p>UPDATE &#8211; 2016 The GPU node in the iCSF is no longer available. The iCSF now has a node named jabberwock containing three AMD FireGL v7800 GPUs. This node can be used to run graphical OpenGL programs such as VMD, Paraview and Abaqus. The iCSF shares your CSF home directory and so this is a convenient place to run these visualization applications (you do not need to download large result files once a simulation running.. <a href=\"https:\/\/ri.itservices.manchester.ac.uk\/icsf\/user-docs\/gpus\/\">Read more &raquo;<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"parent":10,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-299","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/icsf\/wp-json\/wp\/v2\/pages\/299","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/icsf\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/icsf\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/icsf\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/icsf\/wp-json\/wp\/v2\/comments?post=299"}],"version-history":[{"count":20,"href":"https:\/\/ri.itservices.manchester.ac.uk\/icsf\/wp-json\/wp\/v2\/pages\/299\/revisions"}],"predecessor-version":[{"id":660,"href":"https:\/\/ri.itservices.manchester.ac.uk\/icsf\/wp-json\/wp\/v2\/pages\/299\/revisions\/660"}],"up":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/icsf\/wp-json\/wp\/v2\/pages\/10"}],"wp:attachment":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/icsf\/wp-json\/wp\/v2\/media?parent=299"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}