Support NumPy for tensorflow-io about io HOT 8 CLOSED

@areeh PR #407 has been opened which should support your case. The PR allows to read a local numpy through the same address space.

It may have limitations but if your process is local then performance might be good for large numpy array (as there is no serialization overhead before hand).

The dict/tuple of the features has been added as well.

from io.

@yongtang Arrow supports reading numpy arrays to record batches. So I don't think it would be much effort to add this, but there would be a limit on dimensionality - for now at least. Hopefully that will change in the future.

from io.

@BryanCutler I think there are two issues, one is the conversion between Apache Arrow and Numpy in memory, another is to read data from npy or npz file format. I haven't find out a way to open npy or npz from Arrow library. However, it seems like the npy or npz file format are very much straightforward, a simple parser should be enough. I will look into it and see if I could come up with something quick.

from io.

Actually the conversion from Numpy to Arrow is zero copy, so it wouldn't consume any more memory, but you're right, Arrow doesn't support reading these files. If you're able to read them directly in the op that would be cool!

from io.

Performant solutions using tf.data and disk are quite painful right now, so this would be a welcome addition. I just want to mention that it's particularly useful if you support a case where data looks like:

{ 'feature1': ([...], dtype=np.float32), 'feature2': ([...], dtype=np.int16), }
etc. Somewhat similar to the example from the guide in the split between feature and label, if I'm understanding correctly

from io.

@yongtang The PR looks great, it supports everything I had in mind when I wrote the comment. Thank you

from io.

@yongtang can this be closed?

from io.

@kvignesh1420 Ah yes thanks for the reminder 👍

from io.