File Management FAQ
For the definitions of terms see the section on file systems.
Maintaining Directories
How can I manage my directories?
There a 7 basic commands to inspect and manipulate directories: cp(1) to copy files, ln(1) to create links, ls(1) to list files, mv(1) to rename files, mkdir(1) to create directories, rm(1) to remove files, and rmdir(1) to remove empty directories. For detailed information about their use, see their pages in the manual.
How can I make better use of the file system?
The following strategies can be used to optimise use of the file systems:
- compressing files so that less data needs to be held on the storage medium;
- combining many files into a single file by creating an archive so that the file system has fewer files to index;
- removing empty files for the same reason;
- removing files that are no longer required after a submission has finished so that less data is held and fewer files are indexed;
- storing source files (e.g. data sets), temporary files, or reproducible files (e.g. output files) in the appropriate directory.
Some of my scratch files have been deleted! Where have they gone?
Files on the scratchpad file system that have not been accessed recently are removed. This is because it is to be used for computation and final result is to be stored elsewhere (i.e. the home directory).
A list of candidate files for removal can be constructed with lfs-find(1).
$ lfs find $HOME/scratch --atime +90 ...
The resulting list will be of files that have not been accessed in over 90 days.
Is there a way to keep my scratch files from being deleted by the clean up policy?
No.
Any user found to be trying to systematically retain scratch files beyond the 3 month limit may be banned from the system.
If you need long term storage then your principle investigator (PI) needs to request an allocation on Research Data Storage and clearly state that is is required on the CSF and/or DPSF and/or iCSF when applying.
How do I compress my files?
A file is compressed (and uncompressed) using a program that implements a compression algorithm. Many exist but, at the time of writing, the most common commands are bzip2(1), gzip(1), and xz(1).
$ ls foo $ du -a foo 116 foo $ gzip foo $ ls foo.gz $ du -a foo.gz 36 foo.gz
It is standard behaviour for these commands to remove the name they were given. The output file will have a similar name whose suffix is related to the command used: .bz2, .gz, .xz, etc.
The previously compressed file is uncompressed using a related command. For those listed above, these are: bunzip2(1), gunzip(1), and unxz(1).
$ ls foo.gz $ gunzip foo.gz $ ls foo
How do I tarball files together?
The following command creates a gzipped tarball called my-job-files.tar.gz in the current directory and contains the contents of 3 directories and a text file:
$ tar zcvf my-job-files.tar.gz dir1 dir2 jobscript.txt dir3 ...
You can place create the file somewhere else if you wish. For example, the same command again specifying your directory in a group area:
$ tar zcvf /mnt/zx01-data01/zfgg1234/my-job-files.tar.gz dir1 dir2 jobscript.txt dir3 ...
Note: you have to suffix the name of the file you are creating with .gz so you know it is compressed. You can also check if a file is compressed with:
$ file my-job-files.tar.gz my-job-files.tar.gz: gzip compressed data, from Unix, last modified: Mon Dec 19 15:31:08 2016
This file(1) command will work even if the file does not have .gz.
Does tar delete the original files?
No. Once you have created your tarball you need to delete the original files. For example, based on the above tar command that would be:
$ rm -rf dir1 dir2 jobscript.txt dir3
Warning: Be careful with this – it will delete all the files and folders you specify and anything inside the folders (i.e all files and subdirectories they contain).
I’ve accidentally deleted/overwritten a file. Is there a backup? Is it possible to recover the file?
The possibility of restoring a file (either in entirety or a previous revision) depends on where it is stored. This will be one of three locations: a directory that is part of Research Data Storage (RDS), a home directory, or on the scratchpad file system. It can be done for the first and second, it can not be done for the third.
All this file management stuff takes a lot of time, can I make it any faster?
rsync
rysnc(1) is much faster than cp(1), mv(1) and a little faster than scp(1) or the drag and drop of MobaXterm. It is also clever in that if your transfer gets interrupted it can be restarted using the exact same command and it will figure out where it got up to and start again from there (i.e. it does not do the whole transfer again). It can also copy files that have been previously copied but only send/copy any changes.
You can use the --remove-source-files option which will remove the files from the source once they have been copied to the destination. For example, the below command copies three files from the current directory to a group area and then removes the three files from the current directory
$ rsync --remove-source-files -avz file1 file2 file /mnt/zx01-data01/zfgg1234
saving considerable time if you are trying to in effect move files from one location to another. It is much more reliable than doing a mv(1) or a cp(1) followed by an rm(1).
pigz
You can speed up the creation of tarballs and compressing them with the pigz(1) command. This must be run as a batch job. Some examples are given above and in the detailed pigz CSF documentation.
Querying File Systems
How can I see how much home or group disk space is available?
Home and group file systems are part of the Research Data Storage service. To see how much disk space is allocated, type the following in any directory on the file system you are interested in:
$ df -h . Filesystem Size Used Avail Use% Mounted on ...
The . is important. An example of what a home file system (e.g. /mnt/iusers01/zx01) might report:
Filesystem Size Used Avail Use% Mounted on
nas.isilonr.manchester.ac.uk:/isilon/x/x/x/csf-users01
500G 328G 173G 66% /mnt/iusers01
Here we can see that the group has been allocated 500GB and of that, 328GB is in use, leaving 173GB available.
How can I see how much scratch disk space is available?
The scratch file system uses something called Lustre. You can see how big it is and how much space is available with:
$ df -h /scratch
which would then report something like this:
Filesystem Size Used Avail Use% Mounted on
10.x.x.x@o2ib0:10.x.x.x@o2ib0:/lustre
306T 228T 64T 79% /net/lustre
To see how much you are using on the CSF3 scratch file system run this command:
$ scrusage Your scratch usage: 450.7GB, 84,053 files
Bear in mind that we have a 3 month clean up policy on files in scratch. It may look big, but with over 1000 users on the system there is the potential for it to run out!!
Is there a limit/quota on how much disk space I can use?
In home and group areas you are limited by the quota set on your group. Within that group quota, there are usually no quotas set on individuals, but everyone in the group is expected to make fair usage of the space. If one or two users use up most of the space, it will cause problems for the group and prevent group users from being able to work.
A group can consist of between 2 and 200 users. If someone persistently uses unfair amount of disk, then a per-user quota can be imposed.
Principal investigators (PI) may request that a quota be set on individual directories to help them manage their space. Some groups do this routinely with project areas.
The scratch file system has to be used fairly by everyone on the system.
We will contact people when disk usage on home, group, or scratch file systems gets too high and expect them to tidy up as a matter of priority.
I need a lot of disk space or long term storage, what should I do?
Ask your head of research group to request some Research Data Storage (RDS) and to clearly indicate when they do that it needs to be accessible on the CSF. For further information and the application form please see the RDS website.
How much disk space am I using?
To see how much space you are using in any directory run:
$ du -sh ...
This will report usage for the current directory and all directories below it: e.g
56G .
If you want to see a breakdown for the files and directories in the current directory:
$ du -sh * ...
might return something like:
4.0K jobscript.txt 11G my_jobs 1.4G fileA 43.6G results
Note: typing cd (followed by the enter/return key) will always automatically place you in your home directory.
I’m using a lot of home disk space, but can’t see why.
Not all names in a directory are ordinarily visible. The result of an accident in Unix is that names whose first character is “.” are, normally, hidden. The option -a for ls(1) will list all names.
$ ls baz foo $ ls -a . .. .bar baz foo .quux
If you have been using a tool such as XFE, for example on the RDS-SSH service, to delete files then you may have a trash folder which is rather full and needs empyting. On the command line you will find the size of the trash directory by doing this:
du -sh .Trash
Note: the dot because it is a hidden file.
How can I see the size of a single file?
The command du(1) will print the space consumed by a file.
$ du -a foo 4 foo
The same information can be found in the fifth column of the long format printed by ls(1).
$ ls -l bar -rw-rw---- 1 user user 742 Sep 9 14:39 bar
How do I find large files?
The find(1) command is really good at this. Please run it in batch if you have a large directory for it to work through. Example, to find files in the current directory and directories below it that are bigger than 5GB:
$ find . -size +5G ...
(The . is important.). Example results of this command:
my_model.sim dir1/my_old_model.sim dir1/my_old_model.output dir2/input.txt
To see more detailed information about each file returned by find, including it’s size, you can add a little bit more to the command, for example:
$ find . -size +5G -exec ls -sh {} \;
...
Might return:
5.1G my_model.sim 10.8G dir1/my_old_model.sim 6G dir1/my_old_model.output 5G dir2/input.txt
On the scratchpad file system, the alternative, more specialised, command lfs-find(1) is available.
$ lfs find $HOME/scratch --size +5G ...
Do empty files use disk space?
Yes, they do. For every record of a file on the file system it uses a file system block. This is very small, but when 100s of users have 100s or 1000s of empty files it can add up to a significant amount. Please delete them. Common empty files are those from batch jobs, such as
slurm-123456.out slurm-123456_123.out jobscript.txt.e123456 jobscript.txt.o123456
Many software applications write to their own specific output files and you will have no use for the empty or very small batch output files, but you should check them before deleting. Some applications will write to them, but you will not require the info.
The command find(1) is capable of searching for files whose size is 0.
$ du -a 0 ./baz 28 ./foo 0 ./.quux 8 ./.bar 40 . $ find . -size 0 ./baz ./.quux
The command lfs-find(1) is also available for use on the scratchpad file system.
$ lfs find $HOME/scratch --size 0 ...
Transferring Files
Can I do file management on the CSF login node?
We do see high loads on the CSF login nodes caused by people doing lots of rsync(1), compression or uploading and downloading. Please use batch jobs to manage your files directly on the system and for file transfer of more than a few GB please use the RDS-SSH service.
How can I move files from one file system to another? I’ve heard of something called rsync.
rsync(1) is a fast, versatile tool for copying files between file systems and from one computer to another. An example of copying a file from scratch to your group area:
$ rsync -avz ~/scratch/myjobdir /mnt/zx01-data01/zfgg1234
The above will create an exact copy of myjobdir and all the files it contains in /mnt/zx01-data01/zfgg1234/myjobdir .
We strongly recommend that you do this via a batch job to avoid overloading the login nodes. See below examples.
How do I download/upload to/from the system?
For a few small files you can use scp(1) or rysnc(1) from a linux or mac computer direct to the login nodes. If you are using windows then winscp is a very good tool, or you can use the drag and drop feature of Mobaxterm. All of these methods are described in our file transfer docs.
For large files or many files you can use all of the same tools as for small files, but please do so via the RDS-SSH service. Note, you may have to request an account on this service and it cannot see the scratch file system. Full details are available on the RDS-SSH service webpage.
Can I use a batch script to transfer files to and from the system?
Unfortunately not. Please follow How do I download/upload to/from the system?
Miscellaneous
How do I use the CSF batch system to manage my files, copy, compress, rsync etc?
It is a simple case of putting the same commands you run on the command line into a serial batch job.
Example 1: Create compressed tar and rsync it to RDS space via the batch system
To create a tarball in the current directory and then copy it to a group area:
#!/bin/bash --login #SBATCH -p serial #SBATCH -t 0-1 tar zcvf my-job-files.tar.gz dir1 dir2 jobscript.txt dir3 rsync -avz my-job-files.tar.gz /mnt/zx01-data01/zfgg1234
and submit in the usual way:
sbatch file-tidy.sbatch
where file-tidy.sbatch is the name of your batch submission script.
Once you are happy that everything in the batch job completed without issue remember that depending on which commands you ran (e.g. tar) you may need to delete the original copies of files (via a new batch job).
Example 2: Create compressed tar in RDS space using pigz via the batch system
This example uses pigz (a parallel compression tool) and places the compressed file in your directory in your group’s Research Data Storage:
#!/bin/bash --login #SBATCH -p multicore_small #SBATCH -n 4 # Use 4 cores to do parallel file compression #SABTCH -t 0-4 # Wallclock max allowed time needs to be specified - here 4 hrs # Set up to use pigz module purge module load tools/gcc/pigz ## Note that $SLURM_NTASKS is automatically set to the number of cores requested above ## Compress everything found in a directory named 'my_data' to a compressed tar file named my_data.tar.gz in a group area tar cf - my_data | pigz -p $SLURM_NTASKS > /mnt/zx01-data01/zfgg1234/my_data.tar.gz # # # Note that a '-' here means the output is sent through the # pipe (the | symbol) to the pigz command, not to an intermediate # tar file.
and submit in the usual way:
sbatch file-tidy.sbatch
where file-tidy.sbatch is the name of your batch submission script.
Once you are happy that everything in the batch job completed without issue remember that depending on which commands you ran (e.g. tar) you may need to delete the original copies of files (via a new batch job).
Example 3: Compress individual large files via the batch system
The following will use the find(1) command from earlier to find files of size 5GB or larger in the current directory (and any sub-directories below it). It will compress each file. Here we use the parallel pigz command on the CSF. The jobscript is:
#!/bin/bash --login #SBATCH -p multicore_small #SBATCH -n 4 # Use 4 cores to do parallel file compression #SBATCH -t 0-4 # Wallclock max allowed time needs to be specified - here 4 hrs # Set up to use pigz module purge module load tools/gcc/pigz # Find all 5GB (or larger) files, show the size, compress in parallel, show the new size for fname in `find . -size +5G`; do ls -sh $fname; pigz -p $SLURM_NTASKS $fname; ls -sh $fname.gz; done
Submit the jobscript in the usual way:
sbatch pigz-jobscript
I’ve got 1000s of files in scratch I want to download. What’s the best way?
Downloading a large number of individual files can be very time consuming and will place a lot of strain on the login node.
First consider whether you need to download the files at all. If they are important result files you should consider keeping them in your home area which is on secure backed-up Isilon storage. Your research group may also have additional Isilon areas for specific research projects or data areas. Downloading to a PC that isn’t backed up could result in data loss if the local PCs disks fail. If you don’t have enough space in your Isilon area then consider compressing the files (with zip(1) or gzip(1)).
If you still want to download a copy then a better option would be to zip up the files in to a single compressed archive. Zip files are common on Windows / MacOS so if you want to transfer the files to a local Windows / MacOS computer you can create the zip file on the CSF and then download it. Alternatively if your local PC is running linux you can create a tar.gz file on the CSF and download that. We advise running the zip app as a batch job to prevent the login node from being overloaded. Here’s how:
# In this example we assume the files to be downloaded are in the folder: ~/scratch/my_data/experiment1/ # Go to the parent of the required location in scratch. For example: cd ~/scratch/my_data/ # zip up all the files from a sub-directory named 'experiment1' (job wallclock is 1 day: -t 1-0) sbatch -p serial -t 1-0 --mail-type=END --mail-user=$USER --wrap="zip -r my_stuff.zip experiment1" # Or, to create a .tar.gz file for use on Linux PCs/Laptops (job wallclock is 1 day: -t 1-0) sbatch -p serial -t 1-0 --mail-type=END --mail-user=$USER --wrap="tar czf my_stuff.tar.gz experiment1"
The above command will submit a batch job (without writing a jobscript), run it from the current directory in the short environment (and will email you when it has finished. The job will zip up and compress all the files in the experiment1 sub-directory of the scratch/my_data/ directory (change the names to suit your own directory structure). When the job finished you’ll have a file named my_stuff.zip in your ~/scratch/my_data/ directory which you can then download using WinSCP, scp or other favourite file transfer program from your PC. Alternatively copy the zip file to your home area.
How can I free up some space in my home or scratch area?
The obvious answer is to delete unwanted files (use the rm(1) command or your preferred graphical file browser such as that in MobaXterm). However, deleting results and data files is not always possible. But there are ways to reduce your usage:
- Compress your files. Many applications write out plain text results files and other log files. These can be huge. Do you need the log file? If not, delete it. But the results files will compress will using
gzip myresult.dat(which will create a new smaller file namedmyresult.dat.gz. You can still read the file usingzless myresult.dat.gzor uncompress it usinggunzip myresult.dat.gz - Delete unwanted job
.oNNNNNNand.eNNNNNNoutput files. Every job will produce an output file capturing what would have been printed to screen when your application ran. The files can contain normal output (the.ofile) and error messages (the.efile). Each file will have the unique job number at the end of the name. If you run a lot of jobs (1000s – and many users do!) you will soon have 1000s of files. We’ve seen some directories with millions of these output files! Individually each file is often small but they soon accumulate. They also take up more space on the file system than you think (the minimum block size of the storage system is used even if your file is smaller). Please delete unwanted job output files. The following command can be used:rm -f *.[oe][0-9]*
- Keep your job directories tidy. Deleting files from jobs you ran months ago is never an exciting task – you may have 1000s of output files. Nobody likes looking through old files to see if you need them or not. Deleting unwanted files when the job finishes is the best way to keep your storage areas tidy. You can even but the delete commands (rm(1) in your jobscript to clean up any junk at the end of a job.
I have downloaded a .zip of several datasets but the scratch clean-up keeps deleting them. What can I do?
When you unzip (or tar) the archive of datasets, the files in the archive will be created with their original timestamps. These could be months or years in the past. The scratch-tidy will then see these old files and delete them.
The solution is to ask unzip (or tar) to extract the files and apply today’s date to them. See here for the extra flag/switch you must add to unzip(1) or tar -xf to extract the files correctly.