Timers lost on Shutdown occasionally about skyring HOT 8 CLOSED

I can look into this one. Just to be sure, you have each node using a different Mongo collection?

from skyring.

Yes, each one is using a different Mongo collection

from skyring.

If this.disconnect(cb) gets commented out the node won't exit but the timers never drop. I believe a race condition is occurring here and the node is disconnecting too soon. I'm looking more into it

from skyring.

Ya. Sounds like a race with the last write ack from mongo

from skyring.

Ya. Sounds like a race with the last write ack from mongo

from skyring.

I believe the issue is with nats not being flushed out completely. #25 resolves this bug.
Update: Bug is only present in the Kubernetes chart now so I believe it's the way I handle the load balancer. It's resolved in just docker images

from skyring.

I agree that there still is a race condition that exists with the mongo storage. I'm not sure how to find it however. I can only reproduce this bug in Kubernetes using a large amount of requests and killing the node during this chain. Sometimes I will get +1 extra timer which I assume is from timers firing as they are trying to transfer them off (more acceptable explanation). Other times I will lose a large amount of timers from killing a node with no clear explanation as to why. And even rarer I will receive the following error messages stating that a timer with an id already exists. I think the post you made in the PR about the driver could be promising and I will try to follow up on that.
2018-11-30T00:52:15.418Z skyring:error Error: Timer with id 231fcc23-3057-476a-9d17-7499e6e057dd already exists 2018-11-30T00:52:15.419Z skyring:error Error: Timer with id 226d3015-4a3b-465d-a58c-2c74de221b42 already exists

from skyring.

Timers firing during a re-balance likely, and fixable. Right now a rebalance is a pretty anonymous operation. Transports don't know a rebalance is happening & vice versa. But That can be fixed. Sounds like it needs to be.

Can you give some specifics around "large amount". I'll investigate.
You only see this problem while running on kubernetes? only with a mongo backend?

from skyring.