HDF5 compression options
Mon, 02/14/2022 - 09:10 am
I am supporting HDF5 Nexus file format in my Nika package and it is common that X-ray instruments are using compression to reduce the file size to reduce load on network communication and storage. We ourselves are using Gzip and all works fine... Unluckily, compression of choice for large X-ray area detectors seems to be LZ4 which is available in Nexus Github depository with few other compressions used by Nexus (https://github.com/nexusformat/HDF5-External-Filter-Plugins).
My argument against using "private" compression in user files (= files provided to user to take home) had no impact. Instrument admins argue either "just install the compression" or "Matlab is reading this fine, so where is the problem?". And I cannot read these files in Igor, so there is my problem...
I would be happy to provide users with link to installer and let them install it, but there is none on the GIthub - you need to compile it on your computer and the instructions are bit cryptic and more or less guaranteed not to work as presented. And it is not reasonable suggestion for my user community to compile Github code on their computers. They are using Igor to have GUI driven toolbox and not have to compile anything.
Unluckily, HDF5 allows nearly infinite number of compressions. What a nightmare, see: https://confluence.hdfgroup.org/display/support/Registered+Filter+Plugi…
Anyone has suggestion how to address this? Could we agree on some subset of compressions most used and could Wavemetrics include them (at least for reading)? I am sure there is no way Igor can (and should) include too many compressions. But we should have some solution.
Or would someone have installers for some of these compressions ?
Actually, digging more through this, HDF5 packages common plugins (including LZ4) in their plugin package released with each HDF5 release, see here: https://confluence.hdfgroup.org/display/support/HDF5+Filter+Plugins, it would be enough to package these with the HDF5 included in IP9.
February 14, 2022 at 01:29 pm - Permalink
I would be interested in other compression algorithms than zlib as well. I think that shipping IP with the packaged plugins from the HDFGroup and allowing users to add their own plugins (~/Documents/WaveMetrics/Igor Pro 9 User Files/HDF5 Plugins) would be good.
February 16, 2022 at 01:04 am - Permalink
I have spent the day investigating HDF5 dynamically loaded filters. Consequently my brain is fried. I may have more to say later, but here is an interim report.
You can find background documentation on HDF5 dynamically loaded filters here.
The following supports reading only, not writing, using HDF5 dynamically loaded filters
=== Windows Setup ===
You can get the dynamically loaded filters to work on Windows with Igor Pro 8 and Igor Pro 9, 64-bit only, as follows:
Download the Windows binary plugin package from here. This creates hdf5plugin-1.12.1-win10_64-vs15.zip in your Downloads folder.
Unzip hdf5plugin-1.12.1-win10_64-vs15.zip to create an "hdf" folder.
Open hdf folder, double-click h5pl-1.12.1-win64.msi, and install to the default location. This creates "C:\Program Files\HDF_Group\HDF5\1.12.1\lib\plugin".
Copy the plugin folder to "C:\ProgramData\hdf5\lib\plugin" which is the default location in which the HDF5 library looks for plugins.
(The ProgramData folder is hidden by default. You can find instructions for viewing hidden folders here.)
Now you can run Igor Pro 8 64-bit or Igor Pro 9 64-bit and you should be able to load HDF5 files compressed using BLOSC, BSHUFFLE, BZ2, JPEG, LZ4, LZF, and ZFP compression.
You can get a LZ4-compressed HDF5 file for testing here. The DS1 dataset is LZ4-compressed.
These are 64-bit libraries so this will not work with the 32-bit version of Igor Pro.
(Instead of copying to "C:\ProgramData\hdf5\lib\plugin", advanced users can create an environment variable named HDF5_PLUGIN_PATH pointing to "C:\Program Files\HDF_Group\HDF5\1.12.1\lib\plugin".)
=== Macintosh Setup ===
On Macintosh you need special libraries that have been tweaked to work with the HDF5 library shipped with Igor Pro 9. (For experts, the tweak changes where the plugins look for the HDF5 library so that they find the HDF5 library that ships with Igor Pro 9.)
These libraries do not work with Igor Pro 8 or before.
This worked with Igor Pro 9 on Mac OS 10.14 (Mojave). In 10.15 or later there are additional hoops to jump through, described below.
Download the plugins from here.
Unzip to create "hdf5/lib/plugin".
Move this to /usr/local to create /usr/local/hdf5/lib/plugin which is the default location in which the HDF5 library looks for plugins.
(/usr/local is normally hidden. To see it, in Finder, choose Go->Go To Folder, enter "/usr/local" and click Go.)
Now you can run Igor Pro 9 and you should be able to load HDF5 files compressed using BLOSC, BSHUFFLE, BZ2, JPEG, LZ4, LZF, and ZFP compression.
You can get a LZ4-compressed HDF5 file for testing from here. The DS1 dataset is LZ4-compressed.
This worked on Mac OS 10.14 (Mojave). In 10.15 or later, you are likely to get security-related errors. You can avoid them by removing the quarantine status of the downloaded files. Because this is not sanctioned by Apple, you will have to figure out how to do it on your own and understand that you are taking a risk by defeating Apple's security system.
February 16, 2022 at 03:12 pm - Permalink
Hello Howard,
this is great! Thank you! I have been looking into HDF5 and plugin installation in the last few days and my head hurts - and I am basically confused. I feel like HDF5 versions, distributions, and instructions are intended to confuse me. So I really appreciate you solving this for us.
I installed the libraries on macOS 10.15.7 and can confirm, that my own LZ4 compressed data now load perfectly fine in IP9. I will setup my Windows VM later today.
These are, at the end, very user friendly instructions. On Mac : download zip file, unzip, place in ... folder, done. Windows is very similar. That is as simple as it gets. I can provide this to users in manual and problem solved.
How robust is this going to be to IP upgrades? 9.0x or 10.0? Will we need new versions of libraries?
February 17, 2022 at 09:29 am - Permalink
It will be robust in 9.0x and you should not need new versions of the libraries. I don't know about 10.
I am still investigating other solutions and will report back.
February 17, 2022 at 10:12 am - Permalink
For Igor Pro 9.01 (shipping in a month or two?), I have added built-in support for the HDF5 dynamically loaded plugins provided by The HDF Group in precompiled form as described at https://portal.hdfgroup.org/display/support/HDF5+Filter+Plugins (BZIP2, JPEG, LZF, BLOSC, LZ4, Bitshuffle, and ZFP).
(Although that web page lists MAFISC as a dynamically loaded plugin filter, empirically I find that you can load an MAFISC-encoded file without the dynamically loaded plugins.)
This supports reading (decoding) only. It does not support writing (encoding). Write support is more involved and therefore a possiblity for Igor Pro 10.
On both Macintosh and Windows, Igor will ship with its own copies of these libraries so no user setup will be required.
Because The HDF Group provides pre-compiled libraries in 64 bits only, this feature will not be available for the 32-bit Windows version of Igor Pro 9.
February 21, 2022 at 03:49 pm - Permalink
That is great news, thank you Howard!
February 21, 2022 at 03:57 pm - Permalink