{"id":8887,"date":"2025-03-10T11:09:22","date_gmt":"2025-03-10T11:09:22","guid":{"rendered":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/?page_id=8887"},"modified":"2025-06-04T09:43:10","modified_gmt":"2025-06-04T08:43:10","slug":"new-scratch-filesystem","status":"publish","type":"page","link":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/new-scratch-filesystem\/","title":{"rendered":"New Scratch Filesystem – Feb 2025"},"content":{"rendered":"
\nWednesday 4th June 2025, 08:30 – Access to he OLD SCRATCH filesystem has been removed from the login nodes. Any batch jobs (queued or running) that are copying files from the old scratch to the new scratch will continue to run until completion. The content of this page relating to OLD SCRATCH is for reference only.
\n<\/strong><\/span>\n<\/div>\n

Introduction<\/h2>\n

The upgraded CSF3 (running Slurm) has a new<\/em> scratch filesystem, providing approximately 1.9PB storage (an increase of 500TB) as well as improved performance. This replaces the scratch storage on the old CSF3 (SGE) cluster. <\/p>\n

This ensures future operation of the CSF – the current hardware will be going off maintenance, which puts the filesystem at risk. The new hardware will have 5 more years of hardware support.<\/p>\n

Terminology<\/h3>\n

This page will use the following terms:<\/p>\n

\n
“Old scratch”<\/dt>\n
The scratch area from the older CSF3, that was running the SGE batch system (your jobscripts used #$<\/code> flags), that you’ve been accessing so far, possibly for years. It contains your existing scratch files you’ve been working with.<\/p>\n

On the upgraded<\/em> CSF3, this old scratch is available at \/scratch-old\/$USER<\/code> and is READ-ONLY, and is only visible on the login node and file transfer server.<\/dd>\n

“New scratch”<\/dt>\n
The scratch area on the upgraded CSF3, running the Slurm batch system (your jobscripts use #SBATCH<\/code>.) When you login to the upgraded CSF3, this will be your usual day-to-day scratch area on the upgraded system. It will be empty initially!<\/strong><\/p>\n

On the upgraded<\/em> CSF3, this new scratch is available at ~\/scratch<\/code> and is now your scratch area on that system.<\/dd>\n<\/dl>\n

You can only access the scratch filesystems (both old and new) on the ugpraded<\/em> CSF3 (Slurm). It is no longer possible to login to the old CSF3 (SGE) cluster.<\/strong><\/p>\n

Action Required<\/h2>\n
YOU MUST TAKE ACTION<\/strong> to decide which files you want to retain from your OLD SCRATCH, then COPY THEM to your NEW SCRATCH.WE WILL NOT<\/strong> BULK TRANSFER YOUR OLD SCRATCH FILES TO THE NEW SCRATCH. AND NOR SHOULD YOU! Please copy ONLY WHAT YOU NEED (spring-clean your scratch!)<\/strong><\/em><\/div>\n
IF YOU DO NOTHING<\/strong>, YOUR OLD SCRATCH FILES WILL BE LOST FOREVER WHEN WE SWITCH OFF THE OLD SCRATCH HARDWARE!!! From 08:30 Wed 4th June old scratch will no longer be accessible.<\/strong><\/div>\n

Accessing your Scratch Areas on the Upgraded CSF3 (Slurm)<\/h3>\n

These instructions assume you are logged-in to the upgraded<\/em> CSF3 (Slurm) – this is the only place where you can deal with your old and<\/em> new scratch.<\/p>\n

To check that you are on the upgraded CSF3:<\/p>\n

#### Remember: You MUST be on the upgraded CSF3 (Slurm) login node ####\r\n\r\n[mabcxyz1@login1[csf3<\/span>] ~]$              # The 'csf3<\/span>' should be green\r\n\r\nls \/scratch-old\/$USER\r\n  #\r\n  # You should see your 'old' scratch files from the CSF3 that runs SGE. \r\n<\/pre>\n
If you see:<\/p>\n
ls: cannot access '\/scratch-old\/mabcxyz1<\/em>': No such file or directory\r\n  #\r\n  # You're on the old CSF3 - you need to be on the upgraded CSF3!\r\n<\/pre>\n
then you are either on the wrong CSF3 (the old CSF3), or you are a new user that does not have an old scratch area.<\/p>\n
The scratch shortcut in your home on the upgraded CSF3 (Slurm) now points to NEW SCRATCH<\/p>\n
#### Remember: You MUST be on the upgraded CSF3 (Slurm) login node ####\r\n\r\n# NEW SCRATCH<\/strong> is now your default scratch (the three paths below all point to the same place)\r\n~\/scratch                       # Will be empty to begin with!!\r\n$HOME\/scratch\r\n\/scratch\/username<\/em>               # Where username<\/em> is your CSF username\r\n\r\n# OLD SCRATCH is READ ONLY<\/strong> there is no shortcut. The path below takes you your OLD SCRATCH\r\n\/scratch-old\/username<\/em>\r\n   #\r\n   # You CANNOT runs jobs from here - it is READ-ONLY. BATCH JOBS WILL FAIL IF RUN FROM HERE.\r\n<\/pre>\n
REMEMBER<\/strong>: On the upgraded CSF3 (Slurm), the ~\/scratch<\/code> symlink (shortcut) in your home directory points to your NEW SCRATCH area.<\/p>\n
Copying files from OLD to NEW SCRATCH<\/h3>\n
These instructions assume you are logged-in to the upgraded<\/em> CSF3 (Slurm) – this is the only place where you can deal with your old and<\/em> new scratch.<\/p>\n
There are three methods:<\/p>\n
    \n
  1. Copying files on the login node<\/strong> This is fine for smaller transfers – not too many files \/ small files or folders ONLY<\/span> – useful for when you are working on the upgraded CSF3 and you realise you need a few files from your OLD SCATCH area. You will need to remain logged in for the copy to complete successfully, so if the copy is going to take any length of time, see methods 2 and 3 below<\/span>.Use the following commands:\n
    #### Remember: You MUST be on the upgraded CSF3 (Slurm) login node ####\r\n\r\n\r\n# If you are about to copy all<\/em> of your old scratch files to new scratch,\r\n# STOP. Do you really need all<\/em> of your old scratch files? This is a time\r\n# to reduce your scratch usage. Also, DO NOT do this directly on the login\r\n# node. See the batch job below.\r\n<\/strong>\r\n\r\n# All commands should be run from the OLD SCRATCH area. So first do:\r\ncd \/scratch-old\/username<\/em>\r\n\r\n# Copy files from OLD to NEW SCRATCH (remember, on the upgraded CSF3, ~\/scratch is NEW SCRATCH)\r\nrsync -av filename<\/em> ~\/scratch         # Will copy a single file\r\nrsync -av *.dat *.log ~\/scratch      # Copy all files ending with .dat and .log to NEW SCRATCH\r\n\r\n# Copy specific files from a directory to the same directory in the NEW scratch\r\ncd \/scratch-old\/username<\/em>\/run5\/outputs\r\nmkdir -p ~\/scratch\/run5\/outputs\r\nrsync -av *.out ~\/scratch\/run5\/outputs\r\n\r\n# Copy an entire folder from old scratch to new scratch\r\n# (if this will take more than one hour use a batch job instead - see below)\r\nrsync -av myfolder<\/em> ~\/scratch\r\n<\/pre>\n<\/li>\n
  2. Copying files in an interactive session<\/em><\/strong> This is similar to the above login-node copy, but will let you run the commands on a dedicated compute node, where the OLD scratch has been made available. Note that it is NOT available on any other compute node.<\/strong> This is useful when copying a lot of files (or large files) – it keeps the load off the login node, which might otherwise slow down the login node for other users:\n
    #### Remember: You MUST be on the upgraded CSF3 (Slurm) login node ####\r\n\r\n# From the upgraded CSF3 login node, start an interactive session on the transfer<\/em> node.\r\n# This will give you a 2-hour session on the compute node, but you'll need to remain\r\n# logged in to the CSF to run any commands. Max permitted time is 12 hours (0-12).\r\nsrun -p serial -t 0-2 --constraint scratch-old<\/strong> --pty bash\r\n  #                      #\r\n  #                      # Can use -C<\/strong> for a shorter flag!\r\n  #\r\n  # Wait to be logged in to the compute node, then:\r\n\r\n# Go to your OLD scratch\r\ncd \/scratch-old\/username<\/em>\r\n\r\n# Use the commands given in the previous login-node copy method - e.g.,\r\nrsync -av filename<\/em> ~\/scratch\r\nrsync -av myfolder<\/em> ~\/scratch\r\n\r\n# When finished, go back to the login node\r\nexit\r\n<\/pre>\n<\/li>\n
  3. Copying files in a batch job<\/strong> This is preferred for larger copies – entire folders or very large datasets<\/span>. You don’t need to remain logged in once you’ve submitted the batch job. Note that the job must be instructed to run on the dedicated compute node where OLD scratch has been made available. It is NOT available on all compute nodes:Create a batch job in your home-directory (e.g., ~\/my-transfer-job.txt<\/code>) containing the following:\n
    #!\/bin\/bash --login\r\n#SBATCH -p serial                     # A single core (serial) job to do a file transfer \r\n#SBATCH --constraint scratch-old<\/strong>      # Job must run on the dedicated file transfer node\r\n#SBATCH -t 4-0                        # This requests a 4-day time limit. Time is REQUIRED<\/strong>.\r\n                                      # (Max requestable is 7 days: 7-0).\r\n\r\n# Go to your old scratch\r\ncd \/scratch-old\/$USER\r\n\r\n# Now copy files and folders to the new scratch area (see examples above)\r\nrsync -av filename<\/em> ~\/scratch\r\nrsync -av folder<\/em> ~\/scratch\r\n<\/pre>\n
    Submit the batch job using:<\/p>\n
    #### Remember: You MUST be on the upgraded CSF3 (Slurm) login node ####\r\n\r\nsbatch my-transfer-job.txt\r\n<\/pre>\n
    You can check on the job using squeue<\/code><\/li>\n<\/ol>\n
    Can I use NEW SCRATCH but carry on using OLD SCRATCH?<\/h3>\n
    No! Jobs cannot<\/strong> be run from OLD SCRATCH (it is READ-ONLY – jobs will fail if run from here.) You should use NEW SCRATCH for all of your work.<\/p>\n
    Remember: the traditional ~\/scratch<\/code> (or \/scratch\/username<\/em><\/code>) path will take you to your NEW SCRATCH area. You can use the special file-transfer<\/em> node to copy files from OLD SCRATCH to NEW SCRATCH.<\/p>\n
    Should I delete files from OLD SCRATCH?<\/h3>\n
    You can’t delete them from \/scratch\/scratch-old\/<\/code> – it is READ-ONLY on the CSF (Slurm) system.<\/p>\n
    It is no longer possible to login to the old CSF3 (SGE) cluster. So it is now impossible for you to delete your old scratch files. When we switch off the old scratch filesystem, your old files will be gone forever.<\/p>\n","protected":false},"excerpt":{"rendered":"
    Wednesday 4th June 2025, 08:30 – Access to he OLD SCRATCH filesystem has been removed from the login nodes. Any batch jobs (queued or running) that are copying files from the old scratch to the new scratch will continue to run until completion. The content of this page relating to OLD SCRATCH is for reference only. Introduction The upgraded CSF3 (running Slurm) has a new scratch filesystem, providing approximately 1.9PB storage (an increase of 500TB).. Read more »<\/a><\/p>\n","protected":false},"author":12,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-8887","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/8887","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/users\/12"}],"replies":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/comments?post=8887"}],"version-history":[{"count":22,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/8887\/revisions"}],"predecessor-version":[{"id":10279,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/8887\/revisions\/10279"}],"wp:attachment":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/media?parent=8887"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}