Locks may be left in redis over restart? about sidekiq-limit_fetch HOT 3 CLOSED

Hi,

yes, stale locks can appear because of a forceful termination of sidekiq process.

To cleanup such locks sidekiq-limit_fetch checks if process which should be responsible for locks is around every once in a while. If process is dead then locks are terminated.

Before 1.3 version a detection of dead processes was once in 9 minutes. It was obviously too long.
So starting with 1.3 version a required time to detect such situation is 2 minutes.

But I think even shorter interval is necessary for better experience, so I will release a new version with 30 seconds time interval.

from sidekiq-limit_fetch.

I have a question about these locks. Are you referring to the "pause" or "block" keys in redis, or the "busy" queues?

On my server, I can see that for a given queue called "critical" the pause and block are nil, but the "busy" queue always contains 1 item, even when I can confirm that no critical job is being processed:

redis 127.0.0.1:6379> llen limit_fetch:busy:critical
(integer) 1

I would have expected this llen to be zero, since no workers are working on that queue. Is this expected behaviour, or do I have a stale lock in my critical queue?

from sidekiq-limit_fetch.

Yes, currently it's an expected behavior.

pause:_, *_block:*** keys indicate user values, values that were set by you.
busy:* keys indicate how many workers at the moment potentially work on a queue.

Why potentially work? Because sidekiq-limit_fetch passes a list of a possible queues to sidekiq, which sidekiq then queries for a task. After task was picked we release all other given to this worker queues. So currently there is no separation whether queue is probed by a worker or whether queue is worked on by a worker.

I'm going to improve this behavior and distinguish these two phases in the next release of sidekiq-limit_fetch.

from sidekiq-limit_fetch.