{"id":3256,"date":"2019-05-01T13:44:29","date_gmt":"2019-05-01T12:44:29","guid":{"rendered":"http:\/\/ri.itservices.manchester.ac.uk\/csf3\/?page_id=3256"},"modified":"2019-05-01T14:02:17","modified_gmt":"2019-05-01T13:02:17","slug":"zstd","status":"publish","type":"page","link":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/software\/tools\/zstd\/","title":{"rendered":"zstd"},"content":{"rendered":"<h2>Overview<\/h2>\n<p>Zstd \/\u00a0 Zstandard is a real-time compression algorithm, providing high compression ratios. It offers a very wide range of compression \/ speed trade-off, while being backed by a very fast decoder (see benchmarks below). It also offers a special mode for small data, called dictionary compression, and can create dictionaries from any sample set. Zstandard library is provided as open source software using a BSD license.<\/p>\n<p>Version 1.4.0 is installed on the CSF and was compiled with GCC 4.8.5.<\/p>\n<h2>Restrictions on use<\/h2>\n<p>There are no restrictions on accessing zstd on the CSF.<\/p>\n<h2>Set up procedure<\/h2>\n<p>Under no circumstances should zstd be run on the login node. If found running it will be killed without warning. It must be submitted as a batch job.<\/p>\n<p>We now recommend loading modulefiles within your jobscript so that you have a full record of how the job was run. See the example jobscript below for how to do this. Alternatively, you may load modulefiles on the login node and let the job <abbr title=\"add '#$ -V' to your jobscript\">inherit these settings<\/abbr>.<\/p>\n<p>Load the modulefile:<\/p>\n<pre>module load tools\/gcc\/zstd\/1.4.0\r\n<\/pre>\n<h2>Running the application<\/h2>\n<p>Jobs should be submitted to the compute nodes via batch.<\/p>\n<h3>Serial batch job submission &#8211; file compression<\/h3>\n<p>It is recommended you run zstd on files in your scratch area. This is a faster filesystem than your home area:<\/p>\n<pre>cd ~\/scratch\r\n<\/pre>\n<p>Make sure you have the modulefile loaded then create a batch submission script, for example:<\/p>\n<pre>#!\/bin\/bash --login\r\n#$ -cwd             # Job will run from the current directory\r\n\r\n\r\n# Load the required version\r\nmodule load tools\/gcc\/zstd\/1.4.0\r\n\r\n### Some example compression uses are given below ###\r\n\r\n\r\n\r\n## Compress a file named mydatafile.dat - it will be renamed mydatafile.dat.zst once compressed\r\nzstd mydatafile.dat\r\n## OR Compress everything found in a directory named 'my_data' to a compressed tar file named my_data.tar.zst\r\ntar cf - my_data | zstd &gt; my_data.tar.zst\r\n       #\r\n       #\r\n       # Note that a '-' here means the output is sent through the\r\n       # pipe (the | symbol) to the zstd command, not to an intermediate\r\n       # tar file.\r\n<\/pre>\n<p>Submit the jobscript using:<\/p>\n<pre>qsub <em>scriptname<\/em><\/pre>\n<p>where <em>scriptname<\/em> is the name of your jobscript.<\/p>\n<h2>Further info<\/h2>\n<ul>\n<li>On the CSF login node run: <code>zstd -h<\/code>\u00a0to get a list of options.<\/li>\n<li><a href=\"https:\/\/facebook.github.io\/zstd\/\">Zstd<\/a> website.<\/li>\n<\/ul>\n<h2>Updates<\/h2>\n<p>None.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Overview Zstd \/\u00a0 Zstandard is a real-time compression algorithm, providing high compression ratios. It offers a very wide range of compression \/ speed trade-off, while being backed by a very fast decoder (see benchmarks below). It also offers a special mode for small data, called dictionary compression, and can create dictionaries from any sample set. Zstandard library is provided as open source software using a BSD license. Version 1.4.0 is installed on the CSF and.. <a href=\"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/software\/tools\/zstd\/\">Read more &raquo;<\/a><\/p>\n","protected":false},"author":7,"featured_media":0,"parent":144,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-3256","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/3256","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/comments?post=3256"}],"version-history":[{"count":5,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/3256\/revisions"}],"predecessor-version":[{"id":3262,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/3256\/revisions\/3262"}],"up":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/144"}],"wp:attachment":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/media?parent=3256"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}