zhangxiangxiao / tunnel Goto Github PK

Currently it is unsafe to use tunnel.Share, tunnel.Atomic, tunnel.Vector and tunnel.Hash as values for these data structures themselves. It would be nice if we can do something like this

a = tunnel.Vector()
b = tunnel.Vector()
-- Nested vector
a[1] = b

I have not yet figured out why. Opening an issue to mark it.

Using threads.sharedserialization assumes that the object will be deserialized only once (hence calling :retain() in the serialize call). However, if a share serialized data is stored in tunnel.Vector or tunnel.Hash, there is no guarantee that it will be deserialized only once.

We need to track the tunnel operations on whether the deserialization is done so as to remove it from the data structures. If not, retain must be called. Until then, it is not memory-safe to use tunnel.Vector and tunnel.Hash for share-serializable data (like *Tensor, *Storage, tds.Hash, tds.Vec etc).

Tunnel seems to get into deadlock with cutorch often. Such cases happen when the main thread is executing cutorch computation functions and multiple threads are waiting for items in tunnel.Vector.