An Example Ecosystem Workflow

Ecosystem Workflow — Data, Compute and Visualisation from the Office, Home and Abroad!

By using the ecosystem computational facilities, storage and virtual desktop, it is easy to prepare, submit and monitor jobs, and post-process results, without transferring any data, all from home, office and abroad (e.g. while at a conference).

Example Workflow

1. Rusty’s data exits his group’s DNA sequencer straight onto storage provided by the RDS, over fast, dedicated networking infrastructure (the RDN). The data is now visible on the CSF and iCSF.
2. Accessing the data on the CSF, by using the Research Virtual Desktop Service (RVDS), he then defines a series of computational jobs to process the data and submits them to the batch system.
3. Later, from home, Rusty re-connects to his RVDS session to monitor his jobs to ensure all is well — or make any necessary tweaks. Over the next few days, from a conference in Barcelona, Rusty checks progress again using the RVDS, from his laptop, and also the SSH Gateway, from his phone; he clears some jobs which have failed and submits additional, corrected work.
4. Back at the office, batch jobs finished, and using the same RVDS session, Rusty starts GUI-based, interactive post-processing on Incline (aka the iCSF) — no need to move data as all the same RDS-based filesystems are available on all ecosystem compute resources.
5. Finally, the results are ready and made available to the public via a Web server running on the Research Virtual Machine Service (RVMS) — accessing the same RDS share.

Another similar scenario

Centralisation of Infrastructure

Before 2010, many small “beowulf” HPC clusters existed on campus. While some were well-run by academics and postgrads, others were not; all took time to administer which was better spent on research; most had many “spare” CPU cycles. Since that time, most campus beowulfs have been decommissioned and contributions made to the CSF instead — research infrastructure has been centralised. Academics now have access to a professionally-run, campus service, with all the benefits that brings.

Following the success of the service provided by the CSF, academics are now encouraged to make use of, and contribute to, other centralised research infrastructure run by IT Services:

The iCSF (aka Incline) — For interactive and GUI-based computation

Many research groups do their main computational work on the CSF, but, currently, purchase powerful workstations for development work and interactive use of GUI-based applications, and for visualization of data. Most such requirements can now be satisfied by use of Incline (the iCSF) in conjunction with the RVDS. The iCSF runs on the same financial model as the CSF: an initial seed has been made by The University and subsequent computational resources are contributed by research groups.

If you interested in finding out more about the iCSF, please contact the IT Services RI team: its-ri-team@manchester.ac.uk.

The Research Data Service — No more USB drives required!

Cheap-n-cheerful NAS boxes and USB drives are commonly used around campus to store important files and data. These systems take time to administer and backup; are less than resilient; and subject to theft. The RDS can satisfy the storage needs of most research groups: some storage is available free-at-the-point-of-use; those requiring greater capacity can pay for additional storage.

To find out more about the RDS, visit the RDS site, or contact the IT Services RI team: its-ri-team@manchester.ac.uk.

Research Virtual Machine Service — Web servers and more

Under-desk Web (and other) servers are frequently to be found in offices around campus. The planned RVMS will offer resilient, professionally-managed, central infrastructure which can be used to replace these. Crucially, academics will have administrator/root privileges if required.