Research Data Storage Documentation

Resilience and Backup Options

When a research data storage share is allocated, choices must be made regarding resilience (replication) and backups (snapshots): each storage share is either replicated or not, and snapshotted or not.

The Research IT team will usually be able to determine the best options based on the use-case information you provide in the request form. By default, storage is both replicated and snapshotted.

What are snapshots? (Recovery of deleted/corrupted files)

A copy of all files and directories is made every hour and these copies are kept for 24 hours; a copy is also taken once a day and these are kept for 35 days. (More accurately, the difference between the current version of a file and the previous version is stored, which helps to reduce the space needed for a copy.) Each such copy is known as a snapshot.

If a file is deleted by accident or corrupted, it can be recovered by using these snapshots.

If data turnover is high, snapshots will take up a significant proportion of a share and hence reduce the available space for current files.

What is replication? (Resilience)

Resilience can be added to the storage by copying it nightly to a second set of storage hardware (in a different physical location). This is known as replication.

Types of share available — example use cases

When a research storage share is requested, the following options are available. Note that most use cases will use snapped & replicated storage for maximum resilience.

snapped & replicated “home” storage (IT Services’ P-drives and central HPC platforms’ home directory (folder) use this combination.)
snapped only Downloaded datasets which are modified locally.
replicated only Valuable read-only data.
neither Scratch space for temporary files.

Is Isilon storage backed up?

Research data storage is not backed up to tape in the traditional fashion. However, resilience is obtained through replication (see above) and use of snapshots means that files deleted by accident or corrupted can be recovered.

Last modified on December 4, 2019 at 2:21 pm by George Leaver