Get Matlab Cell about h5wasm HOT 20 CLOSED

Matlab is storing Matlab-specific hints for how to interpret the data in the attributes, so their own interface is able to seamlessly read/write these files. It wouldn't be too hard to write a post-processor in javascript for a limited number of cases - for the most part h5wasm already knows how to handle all the datatypes in https://pythonhosted.org/hdf5storage/storage_format.html#how-data-is-stored - the special case seems to be strings.

from h5wasm.

Can you attach an example file to this issue so I can have a look?

from h5wasm.

Sure thing, here you go. Let me know if you need anything else, and thanks a lot for your help.
some list.zip

from h5wasm.

This seems to describe how values are encoded from MATLAB into an HDF5 file: https://pythonhosted.org/hdf5storage/storage_format.html

In the file above, the dataset of interest seems to be at '/StratList', and it has the Reference datatype, shape = (15,2). It has an attribute "MATLAB_class" which has value "cell". Each value of this dataset is a reference to another object, in this case all of them are datasets in the '/#refs#' group at the root level. In that group there are 31 datasets named ["a", "b", "c" ... "A", B", "C", "D", "E"]. Of those, some are of type '<f8', (float64) while some of are of type '<H' (uint16). For example, dataset "/#refs#/b" has dtype uint16 and these attributes:
{"H5PATH": "/#refs#/b", "MATLAB_class": "char", "MATLAB_int_decode": 2}
This indicates that it is a string type in MATLAB, and should be interpreted as a utf-16 string ('some other report')

Unfortunately, I had to use h5py or h5dump to do this investigation, as support for Reference data is not built into h5wasm at this time. Even if this support was added, you would still have to build some post-processing tools in order to interpret the cell data (since e.g. strings are recorded as uint16 arrays with 2 dimensions)

Support for references would take a little while to build - there are multiple different types of reference, and you don't know which kind each reference is without calling the HDF5 API to check it. So you could have a dataset of references that are a mixture of object references (any of file, group, dataset, named datatype, or attribute) and dataset region references.

from h5wasm.

Thanks for looking into it, and wow, I never thought it would be so intricate to figure out what's in that file, considering that opening it with matlab is a single line of code. I guess I don't have many options at the moment besides keeping using matlab, right? There's no easy way to read these files otherwise

from h5wasm.

So do you think that based on the analysis you shared before I should be able to figure out how to get the strings from the file?

from h5wasm.

Yes, there are functions in javascript that make it straightforward, once you know the data is utf-16 - e.g.

new TextDecoder('utf-16').decode(new Uint16Array( [115, 111, 109, 101, 32, 111, 116, 104, 101, 114, 32, 114, 101, 112, 111, 114, 116] ))
// 'some other report'

Still, I would first need to add support for References to h5wasm. I'm not sure when I'll be able to do that.

from h5wasm.

Oh ok, so there are some missing features that wouldn't make it possible at the moment, right?

from h5wasm.

Yes - but I'll update this ticket when I implement support for References.

from h5wasm.

Thanks a lot 👌

from h5wasm.

Ok, this is now possible with the newly published v0.7.1 h5wasm. See: https://observablehq.com/@bmaranville/reading-matlab-hdf5-files-with-h5wasm

from h5wasm.

Thanks @bmaranville , I'll try it out!

from h5wasm.

I see errors at the link you shared though:

from h5wasm.

Apologies - getting a notebook set up so all the async bits run in the right order is a little tricky. I edited the notebook - if you reload, it should be working.

from h5wasm.

Nice, I can see that working now! Is there a way to tell the dimensions of the cell so I know how long each row is? Because at the moment it's just a flat array

from h5wasm.

Of course - the shape is stored in the metadata and is available as dset.shape, which returns [15,2]. You can also see it in the expanded metadata in the notebook:

from h5wasm.

That's great, thanks a lot for this!

from h5wasm.

Feel free to close this issue of course!

from h5wasm.

@bmaranville I've come across another file which I cannot seem to be able to parse correctly. Can I send it to you privately?

The file is generated using the same Matlab software that I used to generate the one I shared previously, but I cannot explain why the file size in this case is extremely reduced compared to the previous one. It seems like the same Matlab software I'm using, at some point and for reasons I cannot explain started generating this file in a different format, and a much smaller size. I don't have Matlab but using Matlab online I can open both files just fine, and I can't explain the difference in size.

Thanks!

from h5wasm.

Sure - feel free - my email address is public in my profile.

from h5wasm.