File Management FAQ
This FAQ is supplementary to other pages we have about filestore.
Introductory notes
Most of the information below applies on all compute systems, but it should be noted that iCSF cannot run batch jobs.
How can I see how much home or group disk space is available?
Home and group filesystems are part of the Research Data Storage service. To see how much disk space is allocated, type the following in any directory on the filesystem you are interested in:
df -h .
The dot is important. An example of what a home filesystem (e.g. /mnt/iusers01/zx01) might report:
Filesystem Size Used Avail Use% Mounted on
nas.isilonr.manchester.ac.uk:/isilon/x/x/x/csf-users01
500G 328G 173G 66% /mnt/iusers01
Here we can see that the group has been allocated 500GB and of that, 328GB is in use, leaving 173GB available.
How can I see how much scratch disk space is available?
The scratch filesystem uses something called lustre. You can see how big it is and how much space is available with:
df -h /scratch
which would then report something like this:
Filesystem Size Used Avail Use% Mounted on
10.x.x.x@o2ib0:10.x.x.x@o2ib0:/lustre
306T 228T 64T 79% /net/lustre
To see how much you are using on the CSF3 scratch filesystem run this command:
sudo lfs quota -h -u $USER /net/scratch2
Bear in mind that we have a 3 month clean up policy on files in scratch. It may look big, but with over 1000 users on the system there is the potential for it to run out!!
Is there a limit/quota on how much disk space I can use?
In home and group areas you are limited by the quota set on your group (as reported by the command in FAQ 1). Within that group quota, there are usually no quotas set on individuals, but everyone in the group is expected to make fair usage of the space. If one or two users use up most of the space, it will cause problems for the group and prevent group users from being able to work.
A group can consist of between two and 200 users. If someone persistently uses unfair amount of disk, then a per-user quota can be imposed.
PIs may request that a quota be set on individual directories to help them manage their space. Some groups do this routinely with project areas.
As pointed out in FAQ 2, the scratch filesystem has to be used fairly by everyone on the system.
We will contact people when disk usage on home, group, or scratch filesystems gets too high and expect them to tidy up as a matter of priority.
I need a lot of disk space or long term storage, what should I do?
Ask your head of research group to request some Research Data Storage (RDS) and to clearly indicate when they do that it needs to be accessible on the CSF. For further information and the application form please see the RDS website.
How much disk space am I using?
To see how much space you are using in any directory run:
du -sh
This will report usage for the current directory and all directories below it: e.g
56G .
If you want to see a breakdown for the files and directories in the current directory:
du -sh *
might return something like:
4.0K jobscript.txt 11G my_jobs 1.4G fileA 43.6G results
Note: typing cd (followed by the enter/return key) will always automatically place you in your home directory.
How can I see the size of a single file?
Use the ls command. For example:
ls -lh resultfile.txt
might return:
-rw-r--r-- 1 zfgg1234 zx01 2.3G Aug 11 06:26 resultfile.txt
The 5th column in the output tells us that the file is 2.3G in size.
How can I make better use of disk space?
- Compress large files with gzip or bzip2.
- Tarball small files into larger files.
- Remove empty files.
- Store temporary files, files that can easily be downloaded (e.g. datasets) or reproduced again in scratch.
- Run batch jobs in scratch and only keep/move the files to home or group storage that are important.
- Tidy up and delete files that are no longer required after each batch job
Is there a way to keep my scratch files from being deleted by the clean up policy?
No. Any user found to be trying to systematically retain scratch files beyond the 3 month limit may be banned from the system. If you need long term storage then your PI needs to request some Research Data Storage and clearly state that is is required on the CSF and /or DPSF and/or iCSF when applying.
How do I find large files?
The find command is really good at this. Please run it in batch if you have a large directory for it to work through. Example, to find files in the current directory and directories below it that are bigger than 5GB:
find . -size +5G
(The dot is important.). Example results of this command:
my_model.sim dir1/my_old_model.sim dir1/my_old_model.output dir2/input.txt
To see more detailed information about each file returned by find, including it’s size, you can add a little bit more to the command, for example:
find . -size +5G -exec ls -sh {} \;
Might return:
5.1G my_model.sim 10.8G dir1/my_old_model.sim 6G dir1/my_old_model.output 5G dir2/input.txt
I’m using a lot of home disk space, but can’t see why.
If you have been using a tool such as XFE, for example on the RDS-SSH service, to delete files then you may have a trash folder which is rather full and needs empyting. On the command line you will find the size of the trash directory by doing this:
du -sh .Trash
Note: the dot because it is a hidden file.
How do I compress my files?
Use gzip or bzip2. For example, the file from our earlier ls example, this file:
-rw-r--r-- 1 zfgg1234 zx01 2.3G Aug 11 06:26 resultfile.txt
can be compressed with the command:
gzip resultfile.txt
and it then becomes much smaller and gets a .gz added to it’s name to indicate it has been compressed:
-rw-r--r-- 1 zfgg1234 zx01 691M Aug 11 06:26 resultfile.txt.gz
How do I tarball files together?
The following command creates a gzipped tarball called my-job-files.tar.gz in the current directory and contains the contents of 3 directories and a text file:
tar zcvf my-job-files.tar.gz dir1 dir2 jobscript.txt dir3
You can place create the file somewhere else if you wish. For example, the same command again specifying your directory in a group area:
tar zcvf /mnt/zx01-data01/zfgg1234/my-job-files.tar.gz dir1 dir2 jobscript.txt dir3
Note: you have to suffix the name of the file you are creating with .gz so you know it is compressed. You can also check if a file is compressed with:
file my-job-files.tar.gz
which returns:
my-job-files.tar.gz: gzip compressed data, from Unix, last modified: Mon Dec 19 15:31:08 2016
This file command will work even if the file does not have .gz.
Does tar delete the original files?
No. Once you have created your tarball you need to delete the original files. For example, based on the above tar command that would be:
rm -rf dir1 dir2 jobscript.txt dir3
Warning: Be careful with this – it will delete all the files and folders you specify and anything inside the folders (i.e all files and subdirectories they contain).
Do empty files use disk space?
Yes, they do. For every record of a file on the filesystem it uses a filesystem block. This is very small, but when 100s of users have 100s or 1000s of empty files it can add up to a significant amount. Please delete them. Common empty files are those from batch jobs, such as
jobscript.txt.pe123456 jobscript.txt.po123456 jobscript.txt.e123456 jobscript.txt.o123456
Many software applications write to their own specific output files and you will have no use for the empty or very small batch output files, but you should check them before deleting. Some applications will write to them, but you will not require the info.
How can I move files from one filesystem to another? I’ve heard of something called rsync.
rsync is a fast, versatile tool for copying files between filesystems and from one computer to another. An example of copying a file from scratch to your group area:
rsync -avz ~/scratch/myjobdir /mnt/zx01-data01/zfgg1234
The above will create an exact copy of myjobdir and all the files it contains in /mnt/zx01-data01/zfgg1234/myjobdir .
We strongly recommend that you do this via a batch job to avoid overloading the login nodes. See below examples.
Can I do file management on the CSF login node?
We do see high loads on the CSF login nodes caused by people doing lots of rsync, compression or uploading and downloading. Please use batch jobs to manage your files directly on the system and for file transfer of more than a few GB please use the RDS-SSH service.
How do I use the CSF batch system to manage my files, copy, compress, rsync etc?
It is a simple case of putting the same commands you run on the command line into a serial batch job.
Example 1: Create compressed tar and rsync it to RDS space via the batch system
To create a tarball in the current directory and then copy it to a group area:
#!/bin/bash #$ -cwd #$ -V tar zcvf my-job-files.tar.gz dir1 dir2 jobscript.txt dir3 rsync -avz my-job-files.tar.gz /mnt/zx01-data01/zfgg1234
and submit in the usual way:
qsub file-tidy.sge
where file-tidy.sge is the name of your batch submission script.
Once you are happy that everything in the batch job completed without issue remember that depending on which commands you ran (e.g. tar) you may need to delete the original copies of files (via a new batch job).
Example 2: Create compressed tar in RDS space using pigz via the batch system
This example uses pigz (a parallel compression tool) and places the compressed file in your directory in your group’s Research Data Storage:
#!/bin/bash --login #$ -S /bin/bash #$ -cwd # Job will run from the current directory #$ -V # Job will inherit current environment settings #$ -pe smp.pe 4 # Number of cores to use for file compression/decompression # Set up to use pigz module load tools/gcc/pigz ## Note that $NSLOTS is automatically set to the number of cores requested above ## Compress everything found in a directory named 'my_data' to a compressed tar file named my_data.tar.gz in a group area tar cf - my_data | pigz -p $NSLOTS > /mnt/zx01-data01/zfgg1234/my_data.tar.gz # # # Note that a '-' here means the output is sent through the # pipe (the | symbol) to the pigz command, not to an intermediate # tar file.
and submit in the usual way:
qsub file-tidy.sge
where file-tidy.sge is the name of your batch submission script.
Once you are happy that everything in the batch job completed without issue remember that depending on which commands you ran (e.g. tar) you may need to delete the original copies of files (via a new batch job).
Example 3: Compress individual large files via the batch system
The following will use the find command from earlier to find files of size 5GB or larger in the current directory (and any sub-directories below it). It will compress each file. Here we use the parallel pigz command on the CSF. The jobscript is:
#!/bin/bash --login #$ -cwd # Find all large (5GB or more) files in current dir and below #$ -pe smp.pe 4 # Use 4 cores to do parallel file compression # Set up to use pigz module load tools/gcc/pigz # Find all 5GB (or larger) files, show the size, compress in parallel, show the new size for fname in `find . -size +5G`; do ls -sh $fname; pigz -p $NSLOTS $fname; ls -sh $fname.gz; done
Submit the jobscript in the usual way:
qsub pigz-jobscript
How do I download/upload to/from the system?
For a few small files you can use scp or rysnc from a linux or mac computer direct to the login nodes. If you are using windows then winscp is a very good tool, or you can use the drag and drop feature of Mobaxterm. All of these methods are described in our file transfer docs.
For large files or many files you can use all of the same tools as for small files, but please do so via the RDS-SSH service. Note, you may have to request an account on this service and it cannot see the scratch filesystem. Full details are available on the RDS-SSH service webpage.
Can I use a batch script to transfer files to and from the system?
Unfortunately not. Please follow the How do I download/upload to/from the system? FAQ.
All this file management stuff takes a lot of time, can I make it any faster?
rsync
rysnc is much faster than cp, mv and a little faster than scp or the drag and drop of MobaXterm. It is also clever in that if your transfer gets interrupted it can be restarted using the exact same command and it will figure out where it got up to and start again from there (i.e. it does not do the whole transfer again). It can also copy files that have been previously copied but only send/copy any changes.
You can use the --remove-source-files option which will remove the files from the source once they have been copied to the destination. For example, the below command copies three files from the current directory to a group area and then removes the three files from the current directory
rsync --remove-source-files -avz file1 file2 file /mnt/zx01-data01/zfgg1234
saving considerable time if you are trying to in effect move files from one location to another. It is much more reliable than doing a mv or a cp followed by an rm .
See an earlier FAQ or the file transfer docs for examples of using rsync.
pigz
You can speed up the creation of tarballs and compressing them with the pigz command. This must be run as a batch job. Some examples are given above and in the detailed pigz CSF documentation.
I’ve accidentally deleted/overwritten a file. Is there a backup? Is it possible to recover the file?
Isilon RDS filesystems including home directories
The following backup policy is in place on Isilon home and RDS areas:
- Snapshots are taken every hour and kept for 24 hours</>
- A full backup is taken each day and kept for 35 days</>
You can recover files yourself. Every directory on isilon has a ‘hidden’ directory called .snapshot. Move to that directory and you will see directories for various dates and times. Choose the one closest to before you deleted the file, and then you should be able to find a copy of the file you have deleted. For example, you have deleted a directory called ‘codes’ from the top level of your home directory:
cd .snapshot ls cd Reynolds_NONFAC_NFS_24hr_2014-01-23_16-30-00 ls cp -Prx codes/ ~
A big advantage of isilon is that, if you only created the files an hour or two before you deleted it, there is a high chance of getting it back.
If the deletion occurred longer than 35 days ago, but you perhaps did not notice it had happened, then it may not be possible to recover the file.
For a more detailed example/description please see the CSF documentation.
Scratch file recovery
There is no backup of scratch, so it is not possible to recover files you accidentally delete from here.