{"id":2755,"date":"2019-03-19T18:00:59","date_gmt":"2019-03-19T18:00:59","guid":{"rendered":"http:\/\/ri.itservices.manchester.ac.uk\/csf3\/?page_id=2755"},"modified":"2025-12-18T09:32:04","modified_gmt":"2025-12-18T09:32:04","slug":"hdf5","status":"publish","type":"page","link":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/software\/libraries\/hdf5\/","title":{"rendered":"HDF5"},"content":{"rendered":"<h2>Overview<\/h2>\n<p><a href=\"https:\/\/www.hdfgroup.org\/community\/\">HDF5<\/a> (Hierarchical Data Format) is a set of software libraries and self-describing, machine-independent data formats that support the creation, access, and sharing of large complex datasets. <\/p>\n<p>Only the HDF5 standard (not the much older HDF4 standard) is available on CSF3.<\/p>\n<p>Some of the HDF5 versions available on the CSF3 are serial only (non-mpi) and others are parallel (mpi-enabled) &#8211; the ones that the modulefile includes <code>mpi<\/code> in its name. Please see the list in the <a href=\".\/#Set_up_procedure\">set up procedure section<\/a> for details.<\/p>\n<h2>Restrictions on use<\/h2>\n<p>There are no restrictions on access to the HDF5 libraries on CSF. The software is released under a BSD-style Open Source license and all use must adhere to that license. Please consult the <a href=\"https:\/\/support.hdfgroup.org\/products\/licenses.html\">HDF5 license<\/a> for more information.<\/p>\n<h2>Set up procedure<\/h2>\n<p>We now recommend loading modulefiles within your jobscript so that you have a full record of how the job was run. See the example jobscript below for how to do this. <\/p>\n<p>Load <strong>one<\/strong> of the following modulefiles to set up your environment:<\/p>\n<pre>\r\n# Includes MPI I\/O support but does <strong>not<\/strong> have the C++ interface compiled (a limitation of HDF5)\r\nmodule load libs\/gcc\/hdf5\/1.13.0_cudampi4.1.8_gcc11.4.1   # Uses GCC 11.4.1 compiler, OpenMPI 4.1.8, CUDA 12.4.1\r\nmodule load libs\/gcc\/hdf5\/1.13.0_mpi5.0.7_gcc14.2.0       # Uses GCC 14.2.0 compiler, OpenMPI 5.0.7\r\nmodule load libs\/gcc\/hdf5\/1.13.0_mpi4.1.8_gcc14.2.0       # Uses GCC 14.2.0 compiler, OpenMPI 4.1.8\r\nmodule load libs\/gcc\/hdf5\/1.13.0_mpi4.1.8_gcc11.4.1       # Uses GCC 11.4.1 compiler, OpenMPI 4.1.8\r\nmodule load libs\/gcc\/hdf5\/1.12.3_mpi4.1.8-gcc-8.2.0       # Uses GCC 8.2.0 compiler, OpenMPI 4.1.8\r\nmodule load libs\/gcc\/hdf5\/1.12.1_mpi4.1.8_gcc11.4.1       # Uses GCC 11.4.1 compiler, OpenMPI 4.1.8\r\n\r\n# No MPI I\/O support and does <strong>not<\/strong> have the C++ interface compiled.\r\nmodule load libs\/gcc\/hdf5\/1.14.3_gcc11.5.0    # Uses GCC 11.5.0 compiler\r\nmodule load libs\/gcc\/hdf5\/1.13.0_gcc14.2.0    # Uses GCC 14.2.0 compiler\r\nmodule load libs\/gcc\/hdf5\/1.12.1_gcc12.2.0    # Uses GCC 12.2.0 compiler\r\n\r\n# No MPI I\/O support but does have the C++ interface compiled.\r\nmodule load libs\/gcc\/hdf5\/1.14.1              # Uses GCC 8.2.0 compiler\r\n\r\n# Includes MPI I\/O support <strong>and<\/strong> has the C++ interface compiled\r\nmodule load libs\/intel-18.0\/hdf5\/1.10.5_mpi   # Intel 18.0.3 compiler, OpenMPI 4.0.1\r\n\r\n# Includes MPI I\/O support but does <strong>not<\/strong> have the C++ interface compiled (a limitation of HDF5)\r\nmodule load libs\/intel-19.1\/hdf5\/1.14.1_mpi   # Intel 19.1 compiler, OpenMPI 4.1.1\r\nmodule load libs\/intel-17.0\/hdf5\/1.10.5_mpi   # Intel 17.0.7 compiler, OpenMPI 3.1.3\r\n\r\n# Legacy modules that may not work in the upgraded CSF3\r\nmodule load libs\/gcc\/hdf5\/1.10.4_mpi          # Uses system default GCC 4.8.5 compiler, OpenMPI 3.1.3\r\nmodule load libs\/gcc\/hdf5\/1.10.4              # Uses system default GCC 4.8.5 compiler\r\nmodule load libs\/gcc\/hdf5\/1.8.21              # Uses system default GCC 4.8.5 compiler\r\nmodule load libs\/gcc\/hdf5\/1.8.16              # Uses system default GCC 4.8.5 compiler\r\n<\/pre>\n<h2>Compiling an HDF5-capable application<\/h2>\n<p>You will mostly use the HDF5 installation on CSF3 when compiling your own software. This allows you to add HDF5 functionality to your own apps (they&#8217;ll read and write HDF5 files). There are some tools you can also run to process existing HDF5 data files (see below).<\/p>\n<p>The modulefiles will set an environment variable named <code>${HDF5DIR}<\/code> which can then be used in your compilation process (e.g., in a Makefile or directly on the command-line) to access the header and library files:<\/p>\n<ul>\n<li>To inform the compiler of the header file directory use:\n<pre>gcc -I${HDF5DIR}\/include ....<\/pre>\n<\/li>\n<li>To inform the compiler of the library files use:\n<pre>gcc ... -L${HDF5DIR}\/lib -lhdf5<\/pre>\n<\/li>\n<li>In a Makefile ensure you use <code>${HDF5DIR}<\/code> rather than <code>$HDF5DIR<\/code>.<\/li>\n<\/ul>\n<p>An example compilation command could be<\/p>\n<pre>\r\ngcc -I${HDF5DIR}\/include ex_hdf5.c -o ex_hdf5 -L${HDF5DIR}\/lib -lhdf5\r\n #\r\n # Use mpicc if compiling MPI code that uses the hdf5 library\r\n<\/pre>\n<h2>Running an HDF5-capable application<\/h2>\n<p>You must load the HDF5 modulefile before running your HDF5-capable application (unless you have statically linked your code against the HDF5 libraries). <\/p>\n<p>Please do not run HDF5-capable applications on the login node. Jobs should be submitted to the compute nodes via batch.<\/p>\n<h3>Serial batch job submission<\/h3>\n<p>Create a batch submission script (which will load the modulefile in the jobscript), for example:<\/p>\n<pre class='slurm'>\r\n#!\/bin\/bash --login\r\n#SBATCH -p serial\r\n#SBATCH -t 0-4\r\n            \r\n# Load the serial (non-MPI) modulefile\r\nmodule purge\r\nmodule load libs\/gcc\/hdf5\/1.14.3_gcc11.5.0\r\n\r\n# Run my application I compiled earlier\r\n.\/<em>myhdf5app<\/em> <em>arg1<\/em> ...\r\n<\/pre>\n<p>Submit the jobscript using: <\/p>\n<pre>sbatch <em>scriptname<\/em><\/pre>\n<p>where <em>scriptname<\/em> is the name of your jobscript.<\/p>\n<h3>Parallel batch job submission<\/h3>\n<p>Create a batch submission script (which will load the modulefile in the jobscript), for example:<\/p>\n<pre class='slurm'>\r\n#!\/bin\/bash --login\r\n#SBATCH -p multicore     # Job will run from the current directory\r\n#SBATCH -n 8             # An example single-node multi-core job using 8 cores\r\n#SBATCH -t 0-4\r\n\r\n# Load the parallel (MPI) modulefile\r\nmodule load libs\/gcc\/hdf5\/1.13.0_mpi4.1.8_gcc14.2.0:\r\n\r\n# Run my application I compiled earlier using mpirun to run it in parallel.\r\n# $SLURM_NTASKS will be set to the number of cores requested above.\r\nmpirun -n $SLURM_NTASKS .\/<em>myparhdf5app<\/em> <em>arg1<\/em> ...\r\n<\/pre>\n<p>Submit the jobscript using: <\/p>\n<pre>sbatch <em>scriptname<\/em><\/pre>\n<p>where <em>scriptname<\/em> is the name of your jobscript.<\/p>\n<h3>HDF5 Tools<\/h3>\n<p>The HDF5 &#8216;bin&#8217; directory is added to your path so that you can access the <code>h5<\/code> tools. Use <code>ls ${HDF5BIN}<\/code> to see all binary tools which can be used on HDF5 files:<\/p>\n<pre>\r\n# HDF5 tools - must be used in batch jobs, as above, to process large HDF5 data files!\r\n\r\ngif2h5   h5debug           h5import  h5pcc          h5redeploy  h5unjam\r\nh52gif   h5diff            h5jam     h5perf         h5repack    h5watch\r\nh5clear  h5dump            h5ls      h5perf_serial  h5repart    ph5diff\r\nh5copy   h5format_convert  h5mkgrp   h5pfc          h5stat\r\n<\/pre>\n<h2>Further info<\/h2>\n<p>Example projects are available in <code>$HDF5DIR\/share\/hdf5_examples\/<\/code><\/p>\n<p>See the HDF5 website for <a href=\"http:\/\/www.hdfgroup.org\/HDF5\/\">full documentation<\/a>.<\/p>\n<h2>Updates<\/h2>\n<p>None.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Overview HDF5 (Hierarchical Data Format) is a set of software libraries and self-describing, machine-independent data formats that support the creation, access, and sharing of large complex datasets. Only the HDF5 standard (not the much older HDF4 standard) is available on CSF3. Some of the HDF5 versions available on the CSF3 are serial only (non-mpi) and others are parallel (mpi-enabled) &#8211; the ones that the modulefile includes mpi in its name. Please see the list in.. <a href=\"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/software\/libraries\/hdf5\/\">Read more &raquo;<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"parent":140,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-2755","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/2755","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/comments?post=2755"}],"version-history":[{"count":19,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/2755\/revisions"}],"predecessor-version":[{"id":11541,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/2755\/revisions\/11541"}],"up":[{"embeddable":true,"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/pages\/140"}],"wp:attachment":[{"href":"https:\/\/ri.itservices.manchester.ac.uk\/csf3\/wp-json\/wp\/v2\/media?parent=2755"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}