FAQ

HDF5

My results files are large, can I compress them?

There is an HDF5 tool that allows you to “repack” and compress your datasets called h5repack. A good ratio of compression to time is using GZIP level 2 but read the documentation to customize for yourself:

h5repack -f GZIP=2 original.wepy.h5 compressed.wepy.h5

There are other more involved mechanisms, but this should cut your size significantly.

I get a ‘valid identifiers’ error when accessing HDF5 data.

If you get something like this:

ValueError: Type names and field names must be valid identifiers: "b'decision_id'"

When accessing your WepyHDF5 data its likely because you have a version of h5py that is too new. Simply downgrade to before version 3.0.

You can check this by running:

pip list | grep h5py

And install an ealier version:

pip install 'h5py<3'

See I think I have the wrong versions of dependencies for more info on fixing dependencies.

Clearing HDF5 Write Locks

Sometimes if processes don’t shut down correctly there will be open locks on the dataset. If you are absolutely sure that no processes are still active you can manually clear these flags using the HDF5 toolkit:

h5clear -s -m $file

Dependencies

I think I have the wrong versions of dependencies

All known version constrained dependencies are recorded both in the setup.py file (this is what gets installed) as well as the requirements.in file. The requirements.in file allows for more complex expressions of dependencies and may have notes as to why they are constrained.

You can check info/changelog.org to find when and why a constraint was added as well.