failed to store DataFrame with column multi-index about mdsynthesis HOT 4 CLOSED

StackOverflow Pandas HDFStore select from nested columns not overly helpful...

from mdsynthesis.

Ah...this is not something we can fix, as it's a limitation of the pandas.HDFStore object, which handles conversion of pandas objects into PyTables objects. What it's saying is that having a multi-index on the columns isn't something it can handle.

It can, however, handle a multi-index on the rows, so doing:

s.data.add('multi', df.transpose())

works just fine. Does that help you, though?

A problem you might have despite this is that appends can only be done on the rows, not on columns. So if the code you're writing to generate the data needs to keep appending to the already-stored DataFrame then you'll have to drop the multi-index on the columns entirely. :/

If you don't care about appends, then you shouldn't lose anything from this workaround. However, depending on the dimensions (number of columns) read/write performance may not be great. DataFrame storage tends to be better when there are < 1000 columns.

from mdsynthesis.

Thanks. I suppose I could live with the workaround; in the mean time I've done aggregating before converting to a DataFrame.

But it is an inconvenience (from pandas' side) that one cannot store everything that one can build. (I could probably redesign my data structures but part of the appeal here is that I can do relatively quick and dirty analysis in a sane framework).

I am closing this issue because MDS/datreant cannot do anything about it.

from mdsynthesis.

Yeah, I don't think there is any persistence format supported in pandas that can support all of pandas own data structures, unfortunately.

from mdsynthesis.