---

The Computational Shared Facility Update

2

The Computational Shared Facility: 2012 December Update

Simon Hood, Research Infrastructure (CSF Service Owner)

simon.hood@manchester.ac.uk

---

The Computational Shared Facility Update

3

CSF Update: Contributions, OS and Hardware

• Who has contributed so far?
• OS upgrade
• New CPU architectures
• How much hardware do we have now?

---

The Computational Shared Facility Update

4

University contrib.	90k	University
Chris Taylor	20k	Imag., Gen. and Prot.
Mike Sutcliffe	70k	CEAS
Ian Hillier	15k	Chemistry
RGF	42k	University
Richard Bryce	22k (+ 8k)	Pharmacy
School contribution	299k	MACE
Various + school contrib.	120k	CEAS
Simon Lovell/Simon Whelan	15k	Bioinf. (FLS)
Jane Worthington	15k	Translational Med.
Nick Higham	54k	Maths
Stephen Welbourne	15k	Psychology
Neil Burton	15k	Chemistry
Paul Popelier	14k (+ 15k)	Chemistry
Richard Henchman	8k	Chemistry
Matthias Heil	13k	Maths
Paul Bowyer	(30k)	Uni Hosp of Sth Man
Faculty contrib	(15k)	MHS
School/faculty contrib.	58k (+ 6k)	FLS
Total:	959k

---

The Computational Shared Facility Update

5

What hardware do we have?

Compute: CPU cores

• HTC: 1332 (+ 252)
• HPC: 2144 cores (all IB-connected)

Nvidia (GP)GPUs

• 16 with Infiniband fast interconnect on hosts
• 7 without

Very High Memory Node

• 12 cores, 0.5 TB RAM

---

The Computational Shared Facility Update

6

OS Upgrade in August to Scientific Linux 6.2

• cf. RHEL 6.2
• Required to support new CPU architectures
• Helpful as SL 5.x libraries were rather old. . .

---

The Computational Shared Facility Update

7

Intel Sandy Bridge

• Soon: 26 SB compute nodes (26 * 12 cores)
• -l sandybridge (or similar)

AMD Bulldozer

• 22 Bulldozer nodes (22 * 64 cores, all IB-connected)
• Code Saturne: 2 * speed, 4 * bang-for-buck vs. old AMD
• Must use optimised executables:
  • compile with Open64 (or PGI? Not GCC or Intel);
  • link to ACML;
  • vendor-supplied binaries rarely optimised for Bulldozer.

---

The Computational Shared Facility Update

8

Isilon, the CSF and the Research Data Network

• Isilon — Compute/Storage "Package"
• Research Data Network
• Shared home-dirs with Redqueen and. . .

---

The Computational Shared Facility Update

9

Isilon is the IT Services research data storage facility

• . . .long-promised. . .
• Can scale up to 15 Petabyes (15 * 1024 * Terabytes)
• Accessible via NFS on managed services like CSF
• Accessible via CIFS on desktops/laptops
• Resilient storage
• Currently accessible on campus only
• Off-campus access to become an IT Services Operational Priority???

---

The Computational Shared Facility Update

10

A Compute Resource and Data Storage Package

• All CSF home-dirs will eventually be on Isilon (two years).
• Replicated and snapped option for home-dirs.
• Unreplicated for data areas?
• Fast 20 Gb dedicated link between Isilon and CSF (can be increased):
  • ideal for those with large amounts of data (e.g., FLS);

---

The Computational Shared Facility Update

11

Dedicated 10Gb Backbone on Campus

• Fast n/w link in place between Isilon & CSF (Reynolds) and Redqueen (Kilburn)
  • Share home-directories (both Isilon and non-Isilon)
  • Extend to other ITS/faculty-managed facilities (only) too?
  • EPS Condor submitter?
• Early 2013: Extended to FLS/Smith Building
  • Kadmon

---

The Computational Shared Facility Update

12

Extras

• SSH Gateway
• Virtual Desktop
• Dashboard

---

The Computational Shared Facility Update

13

Facilitates off-campus SSH access (without VPN)

Easiest off-campus access for Linux users:

• Current VPN difficult(!) to use with Linux
• SSH "hop" instead

In production:

• Uses central ITS credentials
• Email for account

---

The Computational Shared Facility Update

14

Off-campus GUI-based access to the CSF

1. Offers virtual (GNOME) desktop for (primarily) CSF users.
2. Will be accessible on- and off-campus.
3. Stateful.
4. NX-based — heavy compression, works fine on "economy" ISP connections.
5. Requires NX client (free download for Linux, Mac, MS Windows)
6. In testing (weeks)

---

The Computational Shared Facility Update

15

Dashboard

• Work in progress
• Stylesheet issues!
• Temporary home (on-campus only):
• ri.itservices.manchester.ac.uk

---

The Computational Shared Facility Update

16

Policies

• Small contributions
• Scratch storage

---

The Computational Shared Facility Update

17

What is the minimum contribution?

• Standard contrib, C6000 chassis: four C6220 nodes, 13.9 + vat
  • Sandy-Bridge
• ". . .we should be prepared to allow small contributions from existing contributors, whereas new contributors wanting to contribute fractions of a C6100 should receive a less sympathetic hearing, though discretion should prevail."
• C6000/C6220 facilitates half contribs:
  • C6000 + 2*C6220 (slightly over half-price).
• May be able to accept quarter contribs soon.

---

The Computational Shared Facility Update

18

Scratch: not a permanent home for files! Quotas???

Agreed policy:

1. Files greater than three months old may be deleted without notice.
2. Usually give an email warning — not guaranteed.
3. Changing the date-stamp of files is permitted.

High risk:

• Scratch is not backed up; not snapped; minimal resilience.

---

The Computational Shared Facility Update

19

Quotas???

• One user to fill scratch (or individual OST) over the weekend. . .
• Millions of (small) files degrades performance and make it difficult to determine culprits
• Limit capacity and number of files?

---

The Computational Shared Facility Update

20

Future Storage on the CSF

Storage Options

• Scratch (Lustre)
• Home/Isilon

Isilon

• Buy extra home space — 1.5k per TB for five years (tbc)
  • Un-replicated 750
• Possible: some Isilon usage will be free at the point of use
  • ISSC end Jan 2013

---

The Computational Shared Facility Update

21

Finale

Next procurement deadline: 2013 January 31

• Three contribs so far
• EPSRC Small Equipment bids?
• Please let me know as soon as possible!

These slides:

• ri.itservices.manchester.ac.uk