{"id":571,"date":"2018-10-11T11:52:03","date_gmt":"2018-10-11T10:52:03","guid":{"rendered":"http:\/\/ri.itservices.manchester.ac.uk\/csf3\/?page_id=571"},"modified":"2026-11-09T10:45:06","modified_gmt":"2026-11-09T10:45:06","slug":"scratch-cleanup","status":"publish","type":"page","link":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/filesystems\/scratch-cleanup\/","title":{"rendered":"Scratch Cleanup Policy"},"content":{"rendered":"<div class=\"warning\">\n<p><strong>Updated scratch policy (1st July 2021):<\/strong> Any files <em>not accessed<\/em> &#8211; which means files that you, or your jobs, have NOT <em>read<\/em> or <em>written to<\/em> within the last 90 days &#8211; will be deleted.<\/p>\n<p>For example, if your jobs are <em>reading<\/em> dataset files you downloaded a long time ago, then those data-sets will NOT be deleted.<\/p>\n<p>This policy is enforced &#8211; <strong>files the have not been read or written \/ updated within the last 90 days in your <em>scratch<\/em> area will be deleted automatically by the system<\/strong>.<\/p>\n<p>Please ensure you keep a copy of important files (e.g., your results!) or source code or conda environments, in a location other than your <em>scratch<\/em> area (<code>~\/scratch<\/code>). Your <em>home<\/em> folder is suitable for small\/medium files. Groups requiring long term storage for large amounts of data, or lots of files, should ask their <abbr title=\"Principal Investigator\">PI<\/abbr>\/Supervisor to request an allocation of space on the <a href=\"http:\/\/www.rds.itservices.manchester.ac.uk\/\">Research Data Storage service (RDS)<\/a> and for it to be made available on the CSF.<\/p>\n<p><strong>There is NO backup of scratch. If you, or the scratch-tidy, deletes files then they are gone forever!<\/strong><\/p>\n<\/div>\n<h2>Why is there a scratch clean-up policy?<\/h2>\n<p>When the original CSF was first commissioned, the October 2011 CSF User Group accepted that there was a need for an automated clean-up of files in <code>\/scratch<\/code>. This policy still holds &#8211; your scratch area will be cleaned-up automatically!<\/p>\n<ul class=\"gaplist\">\n<li>This area is shared by <em>everyone<\/em> with access to the cluster.<\/li>\n<li>There are no quotas on your usage because quotas reduce performance.<\/li>\n<li>1.2PB may seem a lot of space, but with a large number of users running 1000s of jobs it can easily fill up.<\/li>\n<li>If the filesystem fills up most CSF jobs will fail!<\/li>\n<li>It strongly encourages you to copy important results to more resilient storage. If the scratch filesystem (e.g., the disks) fail there are NO backups.<\/li>\n<\/ul>\n<h2>What does the scratch clean-up policy mean for me?<\/h2>\n<p>Please ensure you aware of the following:<\/p>\n<ul>\n<li><strong>Files which have <em>not<\/em> been read <em>or<\/em> written to in the last 90 days will be deleted automatically.<\/strong><\/li>\n<li><strong>Files which have <em>not<\/em> been read <em>or<\/em> written to in the last 90 days will be deleted automatically.<\/strong><\/li>\n<li><strong>Files which have <em>not<\/em> been read <em>or<\/em> written to in the last 90 days will be deleted automatically.<\/strong><\/li>\n<\/ul>\n<ul class=\"gaplist\">\n<li><strong>Please note:<\/strong> Users are still expected to regularly tidy up their own scratch area &#8211; don&#8217;t just wait for the scratch tidy to do your storage housekeeping. Delete unwanted files and copy important files to <em>home<\/em> or additional <em>research data storage<\/em>.<\/li>\n<li>Unfortunately, the size of the filesystem and University email limits makes it impractical to send warning messages \/ emails prior to deletion. Please consider yourself warned about scratch file deletions here.<\/li>\n<li>There is NO backup of scratch. If you, or the system, deletes files then they are gone forever!<\/li>\n<li>In the rare event of scratch hardware failure, we <strong>cannot<\/strong> recover scratch files. So by having a clean-up policy we are strongly encouraging you to ensure you have copied your important files to backed-up storage (e.g., your <em>home<\/em> storage area or additional <em>research data storage<\/em>).<\/li>\n<li><strong>Please note<\/strong>: user found deliberately circumventing the scratch clean-up policy, thereby using scratch for long-term storage, will have their CSF account suspended.<\/li>\n<\/ul>\n<p>Please read the brief summary of <a href=\"\/csf3\/filesystems\/home-scratch-rds\/\">the three CSF storage areas<\/a> available to you. It is important you understand where <em>your<\/em> files are stored and the different properties of that storage.<\/p>\n<h2>What is my current scratch usage?<\/h2>\n<p>Run the following command on the login node to find out:<\/p>\n<pre>\r\n[<em>username<\/em>@login1[csf3] ~] <strong>scrusage<\/strong>\r\n<\/pre>\n<p>Example output<\/p>\n<pre>Your scratch usage: 450.7GB, 84,053 files\r\n<\/pre>\n<p><a name=\"tardate\"><\/a><\/p>\n<h2>Timestamps when extracting tar and zip archives<\/h2>\n<p>If you have <em>tar<\/em> or <em>zip<\/em> archives (for example downloaded source code or downloaded datasets\/databases or archives of files from an old project), they may contain files with date stamps <em>older<\/em> than 90 days. Extracting the files from the archive will usually result in the <em>original<\/em> data stamp being applied to the files. These files may then be removed without warning because the files considered too old.<\/p>\n<p>To extract the files with <em>today&#8217;s<\/em> date applied to them, which then gives you 90 days, use the following commands:<\/p>\n<pre># Extract a gzip compressed tar file, writing files with today's date (add <code>m<\/code> flag) tar xz<strong>m<\/strong>f my_archive.tar.gz # Extract a zip archive, writing files with today's date (add <code>-D<\/code> flag) unzip <strong>-D<\/strong> my_archive.zip<\/pre>\n<h2>Timestamps when transferring files from your PC to your scratch area<\/h2>\n<p>If you upload files from your PC\/laptop to your scratch area, the following <code>rsync<\/code> command can be used on Linux PCs\/laptops, Macs and in MobaXterm on Windows. It will ensure that the files are written in scratch with <em>today&#8217;s<\/em> date-stamp. This will give the files the full 90 days before the scratch tidy can delete it:<\/p>\n<pre>rsync -rlvz <em>myinput.dat<\/em> <em>username<\/em>@csf3.itservices.manchester.ac.uk:~\/scratch\/\r\n<\/pre>\n<p>The <code>-rl<\/code> flag replaces the more usual <code>-a<\/code> flag when doing this type of transfer.<\/p>\n<h2>Additional Research Data Storage<\/h2>\n<p>Check with your PI if they have RDS, if so they can <a href=\"https:\/\/ri.itservices.manchester.ac.uk\/rds\/\">fill in our form<\/a> asking for you to have access, and if not they can also request RDS via the <a href=\"https:\/\/ri.itservices.manchester.ac.uk\/rds\/\">RDS form<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Updated scratch policy (1st July 2021): Any files not accessed &#8211; which means files that you, or your jobs, have NOT read or written to within the last 90 days &#8211; will be deleted. For example, if your jobs are reading dataset files you downloaded a long time ago, then those data-sets will NOT be deleted. This policy is enforced &#8211; files the have not been read or written \/ updated within the last 90.. <a href=\"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/filesystems\/scratch-cleanup\/\">Read more &raquo;<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"parent":94,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-571","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/571","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/comments?post=571"}],"version-history":[{"count":20,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/571\/revisions"}],"predecessor-version":[{"id":13021,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/571\/revisions\/13021"}],"up":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/94"}],"wp:attachment":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/media?parent=571"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}