{"id":847,"date":"2022-07-20T11:54:15","date_gmt":"2022-07-20T10:54:15","guid":{"rendered":"https:\/\/ri.itservices.manchester.ac.uk\/csf4\/?page_id=847"},"modified":"2022-07-20T11:54:56","modified_gmt":"2022-07-20T10:54:56","slug":"pigz","status":"publish","type":"page","link":"https:\/\/ri.itservices.manchester.ac.uk\/csf4\/software\/tools\/pigz\/","title":{"rendered":"pigz and unpigz"},"content":{"rendered":"

Overview<\/h2>\n

Pigz<\/a> is a parallel version of gzip, a tool to compress\/uncompress files in to gzip<\/code> or zip<\/code> archives. It uses multiple cores on a compute node to speed up file compression.<\/p>\n

Note that the archive (.gz<\/code>) files written by pigz<\/code> and ordinary gzip<\/code>, installed on many Linux systems, are compatible. You do not<\/em> need to compress and<\/em> uncompress a file using pigz<\/code> and unpigz<\/code>. For example you can compress a file quickly on the CSF using pigz<\/code> (see below), then transfer the file to your local desktop machine and uncompress it using the ordinary gunzip<\/code> command. Conversely if you download a data file from the web that has been compressed with gzip<\/code> you can uncompress it on the CSF using unpigz<\/code>.<\/p>\n

Under no circumstances<\/strong> should pigz<\/code> be run on the login node. If found running it will be killed without warning. It must<\/em> be submitted as a batch job.<\/p>\n

Version 2.4 is installed on the CSF.<\/p>\n

Restrictions on use<\/h2>\n

Under no circumstances should pigz be run on the login node. If found running it will be killed without warning. It must be submitted as a batch job.<\/p>\n

There are no restrictions on accessing pigz on the CSF.<\/p>\n

Set up procedure<\/h2>\n

Under no circumstances should pigz be run on the login node. If found running it will be killed without warning. It must be submitted as a batch job.<\/p>\n

Load the modulefile:<\/p>\n

\r\nmodule load pigz\/2.4-gcccore-9.3.0\r\n<\/pre>\n
Running the application<\/h2>\n
Please do not run pigz on the login node. Jobs should be submitted to the compute nodes via batch.<\/p>\n
Parallel batch job submission – file compression<\/h3>\n
It is recommended you run pigz on files in your scratch area. This is a faster filesystem than your home area:<\/p>\n
\r\ncd ~\/scratch\r\n<\/pre>\n
Create a batch submission script, for example:<\/p>\n
\r\n#!\/bin\/bash --login\r\n#SBATCH -p multicore       # (--partition=multicore) Single compute-node parallel job\r\n#SBATCH -n 8               # (--ntasks=8) Number of cores to use for file compression\/decompression\r\n\r\n# Load the required version\r\nmodule load pigz\/2.4-gcccore-9.3.0\r\n\r\n### Some example compression uses are given below ###\r\n\r\n## Note that $SLURM_NTASKS is automatically set to the number of cores requested above\r\n\r\n## Compress a file named mydatafile.dat - it will be renamed mydatafile.dat.gz once compressed\r\npigz -p $SLURM_NTASKS mydatafile.dat\r\n\r\n## OR Compress everything found in a directory named 'my_data' to a compressed tar file named my_data.tar.gz\r\ntar cf - my_data | pigz -p $SLURM_NTASKS > my_data.tar.gz\r\n       #\r\n       #\r\n       # Note that a '-' here means the output is sent through the\r\n       # pipe (the | symbol) to the pigz command, not to an intermediate\r\n       # tar file.\r\n<\/pre>\n
Submit the jobscript using: <\/p>\n
sbatch scriptname<\/em><\/pre>\n
where scriptname<\/em> is the name of your jobscript.<\/p>\n
Parallel batch job – decompression<\/h3>\n
The pigz manual states:<\/p>\n