veeqo / activejob-uniqueness Goto Github PK

View Code? Open in Web Editor NEW

256.0 256.0 28.0 113 KB

Unique jobs for ActiveJob. Ensure the uniqueness of jobs in the queue.

Home Page: https://devs.veeqo.com/job-uniqueness-for-activejob/

License: MIT License

Ruby 100.00%

activejob distributed-locks lock rails redis uniqueness

activejob-uniqueness's People

Contributors

Stargazers

Watchers

activejob-uniqueness's Issues

The default value of retry_count for redlock-rb causes extra delays

The default value of retry_count of redlock-rb is 3. This setting stands for a number of attempts to set the lock, not the number of retries on Redis connection error.

It adds extra delay on jobs enqueuing if jobs have uniqueness strategies until_executing, until_executed, until_expired, or until_and_while_executing. While some retries might be helpful for locking on execution, the activejob-uniqueness is more about jobs uniqueness and it should process lock conflicts as fast as possible in order not to slow jobs enqueuing down.

Currently the activejob-uniqueness allows to set custom options for redlock client. Basically, I suggest to set the default value for v0.2.0. The setting could still be amended.

`RedisClient::CommandError` on retry

Hi and first of all thank you for your hard work on this gem! ❤️

My problem

I'm having trouble retrying jobs using sidekiq UI. Whenever I navigate to /sidekiq/retries and click on Retry Now, I get a 500 with the following:

So it's a RedisClient::CommandError with the message "ERR syntax error".
If I set a breakpoint before the command is sent, here is what's going on:

["SCAN", "0", "0", "match", "activejob_uniqueness:path/to/job:3aa7381f4be2b444874cacf24f6c5c9e*"]

And it's coming from:

redis-client (0.20.0) lib/redis_client/connection_mixin.rb:35:in `call'
redis-client (0.20.0) lib/redis_client.rb:257:in `block (2 levels) in call'
redis-client (0.20.0) lib/redis_client/middlewares.rb:16:in `call'
redis-client (0.20.0) lib/redis_client.rb:256:in `block in call'
redis-client (0.20.0) lib/redis_client.rb:677:in `ensure_connected'
redis-client (0.20.0) lib/redis_client.rb:255:in `call'
redis-client (0.20.0) lib/redis_client.rb:639:in `scan_list'
redis-client (0.20.0) lib/redis_client.rb:359:in `scan'
activejob-uniqueness (0.3.1) lib/active_job/uniqueness/lock_manager.rb:28:in `each'
activejob-uniqueness (0.3.1) lib/active_job/uniqueness/lock_manager.rb:28:in `block (2 levels) in delete_locks'
[...]

Which is the following line:

conn.scan(0, match: wildcard).each { |key| conn.call('DEL', key) }

What am I missing? (I'm pretty sure it's kind of obvious so apologies if it is).
Thank you in advance for your help 🙏

Config

activejob-uniqueness (0.3.1)
redlock (2.0.6)
redis-client (0.20.0)

# config/initializers/active_job_uniqueness.rb
ActiveJob::Uniqueness.configure do |config|
  config.on_conflict = :log
  config.redlock_servers = [RedisClient.new(url: ENV.fetch('REDIS_URL', 'redis://localhost:6379/0'), ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE })]
end

Turn off test_mode! for some tests only?

Hi,

The doc mentions:

Most probably you don't want jobs to be locked in tests. Add this line to your test suite (rails_helper.rb):

ActiveJob::Uniqueness.test_mode!

What if I want it turned off, but need to turn it on for some specific tests where I want to test that jobs are not getting enqueued twice?

I took a look in activejob-uniqueness-0.2.3/lib/active_job/uniqueness.rb, and I don't see an obvious way to reset the @lock_manager instance var after having set it with test_mode!.

Wouldn't it make more sense to use a similar pattern to activejob's? (ActiveJob::Base.queue_adapter = :some_adapter).
Something like ActiveJob::Uniqueness.lock_manager = :test, so you can switch it back easily, passing the name of the default manager.

Random redis errors on delete_lock

I experienced random redis errors, all having origin in the delete_lock method:

ArgumentError
invalid value for Integer(): "" (ArgumentError)

      Integer(io.gets_chomp)

RedisClient::CommandError
NOAUTH Authentication required. (RedisClient::CommandError)

Redlock::LockAcquisitionError
failed to acquire lock on 'Too many Redis errors prevented lock acquisition' (Redlock::LockAcquisitionError)

NoMethodError
undefined method `chr' for nil:NilClass (NoMethodError)

        raise UnknownType, "Unknown sigil type: #{type.chr.inspect}"
                                                      ^^^^

After a closer look into redlock, I noticed that they introduced a monitor to handle redis client connection in version 2.0.4. The delete_lock is not using it. I made a patch in our app and all the errors are gone. In order to get rid of the patch on my side I opened #67

Got these errors with following gem versions:

activejob-uniqueness (0.3.0)
rails (7.0.8)
redis-client (0.18.0)
redlock (2.0.4)
sidekiq (7.1.6)

Clear a queue

We are really struggling with queue clearing, particularly if queues have gotten big. Usually when it happens, we don't care about the locks because they have timed out.

Our current plan is to just copy the base sidekiq code for clearing a queue and monkey-patching it back into Sidekiq next to the normal clear code that has been modified by the patch (which we want in other cases), and then just call that. I am posting for two reasons:

This feels like a common-enough need that it might make sense to modify this gem to rename the base clear method and leave it there so that others can do this without patching
Confirm if you see other gotchas.

Thanks for the great gem!

ActiveJob::Uniqueness::StrategyNotFound: Strategy 'until_executed' is not found. Is it declared in the configuration?

I would like to know Why is happen this error, the another jobs is ok but this specific job is launching this error. thanks

  queue_as :tracker
  unique :until_executed

  def perform(args)
    ....
  rescue StandardError => e
    ....
  end```

Deleting many items in the queue results in timeouts

Hi,

Thanks for creating this awesome gem.

We've been using this for a while but when I upgraded to a recent version, I noticed that the deletion of jobs now also removes the lock. That's great, however when you delete a full queue of 1000+ items this causes timeouts on sidekiq admin and it actually doesn't seem to be clearing the queue.

My previous method was always to clear the queue and then unlock all items.
This method is much faster.
Perhaps a similar method can be used to unlock all items when you delete a full queue?

Support for Rails 7.0.0

Rails 7 has recently shipped and it looks like activejob-uniquesss cannot be installed currently alongside it.

Bundler could not find compatible versions for gem "activejob":
  In Gemfile:
    activejob-uniqueness was resolved to 0.2.2, which depends on
      activejob (>= 4.2, < 7)

    rails (~> 7.0.0) was resolved to 7.0.0, which depends on
      activejob (= 7.0.0)

NoMethodError: undefined method `unique'

Anyone been facing the issues for the gem recently.

My job name is WelcomeNotificationJob.

we been facing the issue recently like -> NoMethodError: undefined method `unique' for WelcomeNotificationJob

for the line which we have -> unique :until_executed, on_conflict: :log, lock_ttl: 5.minutes

Clarification on Sidekiq patch

Hi there, thank you for this useful gem!

I'm using it both with Resque and Sidekiq, and as the README says, in order to properly make Sidekiq queue cleanup and job death working with this gem, the patch needs to be required.

Can you please better explain how this patch works and why is it necessary?
More particularly, why is it always calling ActiveJob::Uniqueness.unlock!, that delete locks using the wildcard?

activejob-uniqueness/lib/active_job/uniqueness.rb

Lines 34 to 36 in e86c621

    
           def unlock!(**args) 
        
             lock_manager.delete_locks(ActiveJob::Uniqueness::LockKey.new(**args).wildcard_key) 
        
           end

I can understand that wildcard is useful when cleaning a queue, but why is it used also when a job (single job) death?

activejob-uniqueness/lib/active_job/uniqueness/sidekiq_patch.rb

Lines 92 to 96 in e86c621

    
           # Global death handlers are introduced in Sidekiq 5.1 
        
           # https://github.com/mperham/sidekiq/blob/e7acb124fbeb0bece0a7c3d657c39a9cc18d72c6/Changes.md#510 
        
           if sidekiq_version >= Gem::Version.new('5.1') 
        
             Sidekiq.death_handlers << ->(job, _ex) { ActiveJob::Uniqueness.unlock_sidekiq_job!(job) } 
        
           end

Thank you in advance!

retry_on does not work with until_executed strategy

I have the following ActiveJob declaration with Sidekiq backend, and noticed that the job is not re-enqueued when an error is raised that has a retry_on declaration. I would have expected the job to retry.

class ExampleJob < ApplicationJob
  unique :until_executed on_conflict: :log
  retry_on ExampleError, wait: :polynomially_longer, attempts: 5
  
 end

A failing job "blocks" any future similar jobs from running

Not sure if its intentional, but if a job fails (e.g. network timeout) it impossible to queue additional similar jobs.

jobs are dropped or re-enqueued ?

Hi , I'm lookiing to use this library but is not clear to me if the conflicting jobs are just dropped or re-enqueued ,
it seems that it only drops jobs? is there an effective strategy to re-enqueue conflict jobs?

Any possibility of bringing over the lock_args from sidekiq-unique-jobs?

Really appreciate the gem, it's been a total savior while using activejob with a lot of job processing!

Being able to control what args are used for locking would be very useful.

https://github.com/mhenrixon/sidekiq-unique-jobs#finer-control-over-uniqueness handles it very well.

I'd be happy to take a shot at a PR if you'd consider the functionality?

Kill old job if runs too long?

Is there a way to do this. I looked at the strategy and it seems like until_and_while_executing is the right one with on_runtime_conflict: :raise .

However, does it raise and kill the old job or simply doesn't start a new job?

Great library btw for us rails users. Thanks for making this 🙏

Support Sidekiq 7

Looks like I can't run my tests with Sidekiq 7 and activejob-uniqueness:

rails aborted!00:02
NoMethodError: undefined method `death_handlers' for Sidekiq:Module00:02
  Sidekiq.death_handlers << ->(job, _ex) { ActiveJob::Uniqueness.unlock_sidekiq_job!(job) }00:02
         ^^^^^^^^^^^^^^^00:02
/home/semaphore/app/vendor/bundle/ruby/3.1.0/gems/activejob-uniqueness-0.2.4/lib/active_job/uniqueness/sidekiq_patch.rb:95:in `<main>'00:02
/home/semaphore/app/vendor/bundle/ruby/3.1.0/gems/bootsnap-1.13.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'00:02
/home/semaphore/app/vendor/bundle/ruby/3.1.0/gems/bootsnap-1.13.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'00:02
/home/semaphore/app/config/application.rb:7:in `<main>'00:02
/home/semaphore/app/Rakefile:6:in `require_relative'00:02
/home/semaphore/app/Rakefile:6:in `<main>'00:02
/home/semaphore/app/vendor/bundle/ruby/3.1.0/gems/railties-7.0.4/lib/rails/commands/rake/rake_command.rb:20:in `block in perform'00:02
/home/semaphore/app/vendor/bundle/ruby/3.1.0/gems/railties-7.0.4/lib/rails/commands/rake/rake_command.rb:18:in `perform'00:02
/home/semaphore/app/vendor/bundle/ruby/3.1.0/gems/railties-7.0.4/lib/rails/command.rb:51:in `invoke'00:02
/home/semaphore/app/vendor/bundle/ruby/3.1.0/gems/railties-7.0.4/lib/rails/commands.rb:18:in `<main>'00:02
/home/semaphore/app/vendor/bundle/ruby/3.1.0/gems/bootsnap-1.13.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'00:02
/home/semaphore/app/vendor/bundle/ruby/3.1.0/gems/bootsnap-1.13.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'00:02
bin/rails:4:in `<main>'00:02
(See full trace by running task with --trace)

I can't see any relevant mention in the changelog, but a quick look for death_handlers in the code seems to indicate we'd need to call Sidekiq.config.death_handlers or Sidekiq.default_config.death_handlers instead.

Looking at the method definition for death_handlers it appears it might just be a matter of wrapping the call in a Sidekiq.configure_server block. It might just require a change in sidekiq_patch.rb as simple as this:

if sidekiq_version >= Gem::Version.new('7.0')
  Sidekiq.configure_server do |config|  
    config.death_handlers << ->(job, ex) { ActiveJob::Uniqueness.unlock_sidekiq_job!(job) }
  end
elsif sidekiq_version >= Gem::Version.new('5.1')
  Sidekiq.death_handlers << ->(job, _ex) { ActiveJob::Uniqueness.unlock_sidekiq_job!(job) }
end

Can PR if you think that's the right approach.

Duplicate ActiveJob log entries

Hello, thanks for the hard work on this gem! It's been very useful. I was hoping if you could help with the following issue.

Gem Versions

rails (6.0.6.1)
activejob (6.0.6.1)
activejob-uniqueness (0.3.1)
rails_semantic_logger (4.14.0)
sidekiq (7.2.4)
sidekiq-pro (7.1.3)

Rails Logging Config

  config.log_level = :info
  config.i18n.fallbacks = true

  # Send deprecation notices to registered listeners.
  config.active_support.deprecation = :notify

  config.colorize_logging = false

  config.rails_semantic_logger.started    = true
  config.rails_semantic_logger.processing = true
  config.rails_semantic_logger.rendered   = false

The Issue

We are getting duplicate ActiveJob log entries when activejob-uniqueness is present in the Gemfile. The issue resolves as soon as we remove the gem from our code. Please note how both the jobs below have two log entries each for Performed.

Neither of these jobs use unique. Logs for all of our jobs are getting duplicated as soon as we include the gem.

2024-05-01 19:58:22.544696 I [1:sidekiq.default/processor] [PullLookalikePercentilesJob] [cd86b4d4-433f-4a2f-9584-de34a232f1e0] Rails -- Performed PullLookalikePercentilesJob (Job ID: cd86b4d4-433f-4a2f-9584-de34a232f1e0) to Sidekiq(default) in 5.89ms -- {:event_name=>"perform.active_job", :adapter=>"Sidekiq", :queue=>"default", :job_class=>"PullLookalikePercentilesJob", :job_id=>"cd86b4d4-433f-4a2f-9584-de34a232f1e0", :provider_job_id=>"51558e2c9e7da60762338772", :duration=>5.89, :arguments=>"[\n\n]"}

2024-05-01 19:58:22.544717 I [1:sidekiq.default/processor] [PullLookalikePercentilesJob] [cd86b4d4-433f-4a2f-9584-de34a232f1e0] Rails -- Performed PullLookalikePercentilesJob (Job ID: cd86b4d4-433f-4a2f-9584-de34a232f1e0) to Sidekiq(default) in 5.93ms -- {:event_name=>"perform.active_job", :adapter=>"Sidekiq", :queue=>"default", :job_class=>"PullLookalikePercentilesJob", :job_id=>"cd86b4d4-433f-4a2f-9584-de34a232f1e0", :provider_job_id=>"51558e2c9e7da60762338772", :duration=>5.93, :arguments=>"[\n\n]"}

2024-05-01 19:58:26.132405 I [1:sidekiq.default/processor] [ImportInventoryPackageSiteListJob] [a7018f7b-15b6-4e93-a4d3-d868a24cd91c] Rails -- Performed ImportInventoryPackageSiteListJob (Job ID: a7018f7b-15b6-4e93-a4d3-d868a24cd91c) to Sidekiq(default) in 1397.53ms -- {:event_name=>"perform.active_job", :adapter=>"Sidekiq", :queue=>"default", :job_class=>"ImportInventoryPackageSiteListJob", :job_id=>"a7018f7b-15b6-4e93-a4d3-d868a24cd91c", :provider_job_id=>"859c39a03444316f10b93ce5", :duration=>1397.53, :arguments=>"[\n\n]"}

2024-05-01 19:58:26.132448 I [1:sidekiq.default/processor] [ImportInventoryPackageSiteListJob] [a7018f7b-15b6-4e93-a4d3-d868a24cd91c] Rails -- Performed ImportInventoryPackageSiteListJob (Job ID: a7018f7b-15b6-4e93-a4d3-d868a24cd91c) to Sidekiq(default) in 1397.89ms -- {:event_name=>"perform.active_job", :adapter=>"Sidekiq", :queue=>"default", :job_class=>"ImportInventoryPackageSiteListJob", :job_id=>"a7018f7b-15b6-4e93-a4d3-d868a24cd91c", :provider_job_id=>"859c39a03444316f10b93ce5", :duration=>1397.89, :arguments=>"[\n\n]"}

Each enqueued, performing, performed etc log entry is repeated twice.

SSL error with sidekiq and heroku redis premium

I am using activejob-uniqueness together with sidekiq as the background worker as well as redis, deployed on heroku.
When using activejob-uniqueness with heroku redis on the mini plan, everything works fine.
When using it with heroku redis on the premium-0 plan, I get the following error when trying to enqueue a job that is using activejob-uniqueness's unique :until_executed:

A Redis::CannotConnectError occurred in background at 2023-08-09 15:50:42 +0200 :

 SSL_connect returned=1 errno=0 peeraddr=[omitted] state=error: certificate verify failed (self signed certificate in certificate chain)
 /app/vendor/bundle/ruby/3.2.0/gems/redis-client-0.15.0/lib/redis_client/ruby_connection.rb:138:in `connect_nonblock'

I believe this is the case because on Heroku's premium-0 plan TLS is enforced. This is a known issue when using Sidekiq and Heroku and Heroku's recommendation is to configure the SSL connection to VERIFY_NONE:

  Sidekiq.configure_server do |config|
    config.redis = {
      url: ENV["REDIS_URL"],
      ssl_params: { verify_mode: OpenSSL::SSL::VERIFY_NONE }
    }
  end

In fact, when enqueuing other jobs that are not using activejob-uniqueness I do not see the described issue.

I assume the reason is that when activejob-uniqueness is initialised, the Redis configuration for Sidekiq is somehow ignored or overwritten.

I tried to fix this by setting a similar configuration in config/initializers/active_job_uniqueness.rb as follows:

config.redlock_servers = [RedisClient.new(:url => ENV["REDIS_URL"], :ssl_params => { verify_mode: OpenSSL::SSL::VERIFY_NONE }), 'redis://localhost:6379']

but for this I get the following error:

A NoMethodError occurred in background at 2023-08-09 15:38:12 +0200 :

 undefined method `evalsha' for #<RedisClient [REDIS_URL]>
 /app/vendor/bundle/ruby/3.2.0/gems/redlock-1.3.2/lib/redlock/client.rb:171:in `block (2 levels) in lock'

0.1.2 bombs when initializing rake commands for a Rails app with Sidekiq

After updating from 0.1.1 to 0.1.2, any rake/rails or bundle exec command bombs with the following error:

/Users/jarkko/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/activejob-uniqueness-0.1.2/lib/active_job/uniqueness/patch.rb:79:in `block in <module:Uniqueness>': undefined method `to_sym' for nil:NilClass (NoMethodError)

This seems to be because the newly added on_load hook is already run in the Bundler.require(*Rails.groups) phase, i.e. before the queue adapter (or any other actual config) has been set in config/application.rb.

Sidekob::Job renamed to Sidekiq::JobRecord

It appears as the latest Sidekiq release - 6.2.2 - renames an internal class Sidekiq::Job to Sidekiq::JobRecord.

This change breaks the Sidekiq patch here.

I'm not entirely sure how to fix this while still supporting previous releases. Below would be a very rough guess and I'm happy to PR if it's suitable.

if Gem::Version.new(Sidekiq::VERSION) >= Gem::Version.new('6.2.2')
  Sidekiq::Job.prepend ActiveJob::Uniqueness::SidekiqPatch::Job
else
  Sidekiq::JobRecord.prepend ActiveJob::Uniqueness::SidekiqPatch::Job
end

Incompatible with Rails 7.1.0

I'm getting this error when trying to upgrade rails from 7.0.8 to 7.1.0.

➜  my_app git:(main) ✗ bundle update rails
Fetching gem metadata from https://rubygems.org/........
Resolving dependencies...
Could not find compatible versions

Because activejob-uniqueness >= 0.1.3, < 0.2.3 depends on activejob >= 4.2, < 7
  and activejob-uniqueness >= 0.2.3 depends on activejob >= 4.2, < 7.1,
  activejob-uniqueness >= 0.1.3 requires activejob >= 4.2, < 7.1.
Because rails >= 7.1.1 depends on activejob = 7.1.1
  and rails >= 7.1.0, < 7.1.1 depends on activejob = 7.1.0,
  rails >= 7.1.0 requires activejob = 7.1.0 OR = 7.1.1.
Thus, activejob-uniqueness >= 0.1.3 is incompatible with rails >= 7.1.0.
So, because Gemfile depends on rails ~> 7.1.0
  and Gemfile depends on activejob-uniqueness ~> 0.2,
  version solving has failed.
➜  shopify_app git:(main) ✗

It seems like this gem restricts itself to activejob versions < 7.1 here

activejob-uniqueness/activejob-uniqueness.gemspec

Line 31 in 96e207a

spec.add_dependency 'activejob', '>= 4.2', '< 7.1'

Is there a reason for this or would a simple fix to remove this restriction be OK? I haven't ran the specs without it to be sure.

Redlock::LockAcquisitionError

I am getting this error when running my specs

Redlock::LockAcquisitionError:
failed to acquire lock on 'Too many Redis errors prevented lock acquisition:
RedisClient::CommandError: NOSCRIPT No matching script. Please use EVAL. (redis://redis:6379/3)'

Everything works fine in development mode. Any ideas what the issue might be?

Support for re-enque later, or drop-previous

Hello, I love this gem, thank you.
I have a use case where when a job is enqueued I'd like to keep pushing back the time of a scheduled job until it is executed without being re-requested.
ie.

job is scheduled to run in 1 hour.
before that job runs it is re-requested
I'd like to either push it back to 1 hour from the new request, or, drop it and schedule a new one in 1 hour.

Perhaps I could do this with context to the job that was locking out the requested job?
unique :until_executed, on_conflict: ->(job, locked_job) { locked_job.drop(); job.enqueue(wait: 1.hour) }
or
unique :until_executed, on_conflict: ->(job, locked_job) { locked_job.reenque(wait: 1.hour) }

Any advice on how to best do this?

Sporadic RedisClient::ConnectionError: Broken pipe

I have not been able to reproduce this, but I'm seeing this error daily when a job gets enqueued from a model callback:

RedisClient::ConnectionError: Broken pipe
/usr/local/bundle/gems/redis-client-0.18.0/lib/redis_client/ruby_connection.rb:76:in `rescue in write'
/usr/local/bundle/gems/redis-client-0.18.0/lib/redis_client/ruby_connection.rb:73:in `write'
/usr/local/bundle/gems/redis-client-0.18.0/lib/redis_client/connection_mixin.rb:30:in `call'
/usr/local/bundle/gems/redis-client-0.18.0/lib/redis_client.rb:256:in `block (2 levels) in call'
/usr/local/bundle/gems/redis-client-0.18.0/lib/redis_client/middlewares.rb:16:in `call'
/usr/local/bundle/gems/redis-client-0.18.0/lib/redis_client.rb:255:in `block in call'
/usr/local/bundle/gems/redis-client-0.18.0/lib/redis_client.rb:675:in `ensure_connected'
/usr/local/bundle/gems/redis-client-0.18.0/lib/redis_client.rb:254:in `call'
/usr/local/bundle/gems/activejob-uniqueness-0.3.1/lib/active_job/uniqueness/lock_manager.rb:13:in `block (2 levels) in delete_lock'
/usr/local/bundle/gems/redlock-2.0.6/lib/redlock/client.rb:200:in `block (2 levels) in synchronize'
/usr/local/bundle/gems/redis-client-0.18.0/lib/redis_client.rb:219:in `with'
/usr/local/bundle/gems/redlock-2.0.6/lib/redlock/client.rb:200:in `block in synchronize'
/usr/local/bundle/gems/redlock-2.0.6/lib/redlock/client.rb:200:in `synchronize'
/usr/local/bundle/gems/redlock-2.0.6/lib/redlock/client.rb:200:in `synchronize'
/usr/local/bundle/gems/activejob-uniqueness-0.3.1/lib/active_job/uniqueness/lock_manager.rb:35:in `synced_redis_connection'
/usr/local/bundle/gems/activejob-uniqueness-0.3.1/lib/active_job/uniqueness/lock_manager.rb:12:in `block in delete_lock'
/usr/local/bundle/gems/activejob-uniqueness-0.3.1/lib/active_job/uniqueness/lock_manager.rb:11:in `each'
/usr/local/bundle/gems/activejob-uniqueness-0.3.1/lib/active_job/uniqueness/lock_manager.rb:11:in `delete_lock'
/usr/local/bundle/gems/activejob-uniqueness-0.3.1/lib/active_job/uniqueness/strategies/base.rb:30:in `unlock'
/usr/local/bundle/gems/activejob-uniqueness-0.3.1/lib/active_job/uniqueness/strategies/until_executed.rb:12:in `after_perform'
/usr/local/bundle/gems/activejob-uniqueness-0.3.1/lib/active_job/uniqueness/active_job_patch.rb:52:in `block (2 levels) in <module:ActiveJobPatch>'

Another similar error I see is

Redlock::LockAcquisitionError: failed to acquire lock on 'Too many Redis errors prevented lock acquisition:
RedisClient::ConnectionError: Broken pipe'
/usr/local/bundle/gems/redlock-2.0.6/lib/redlock/client.rb:323:in `lock_instances'
/usr/local/bundle/gems/redlock-2.0.6/lib/redlock/client.rb:275:in `block in try_lock_instances'
/usr/local/bundle/gems/redlock-2.0.6/lib/redlock/client.rb:271:in `times'
/usr/local/bundle/gems/redlock-2.0.6/lib/redlock/client.rb:271:in `try_lock_instances'
/usr/local/bundle/gems/redlock-2.0.6/lib/redlock/client.rb:79:in `lock'
/usr/local/bundle/gems/activejob-uniqueness-0.3.1/lib/active_job/uniqueness/strategies/base.rb:24:in `lock'
/usr/local/bundle/gems/activejob-uniqueness-0.3.1/lib/active_job/uniqueness/strategies/base.rb:59:in `before_enqueue'
/usr/local/bundle/gems/activejob-uniqueness-0.3.1/lib/active_job/uniqueness/active_job_patch.rb:50:in `block (2 levels) in <module:ActiveJobPatch>'
/usr/local/bundle/gems/activesupport-7.0.8/lib/active_support/callbacks.rb:467:in `instance_exec'

This does not happen if I downgrade to 0.2.5 and redlock 1.x. I'm using rails 7.0.8

Can I set the uniqueness based on job class only?

Hi, I'm interested to only have one job running at any given time based on job's class name and not the arguments.
Any idea how to do so?
or alternatively, can we filter by queue?

Thanks.

Error when seeding

class Notification < ApplicationRecord
  ...
  after_create_commit :after_create

  def after_create
    case notification_type
    when 'incoming_friend_request'
      ::BroadcastIncomingFriendRequestNotificationJob.set(wait: 1.seconds).perform_later(self)
    when 'accepted_friend_request'
      ...
    end
  end
end

class BroadcastIncomingFriendRequestNotificationJob < ActiveJob::Base
  unique :until_executed

  def perform(notification)
    if notification.unread?(notification.notificationable)
      notification.broadcast
    end
  end
end

-------------------------------------------------------------
Seeding!
-------------------------------------------------------------
rails aborted!
ActiveJob::Uniqueness::JobNotUnique: Not unique BroadcastIncomingFriendRequestNotificationJob (Job ID: a62bc78e-e835-4578-b5a9-3c152c269308) (Lock key: activejob_uniqueness:broadcast_incoming_friend_request_notification_job:0e593936c1e000761d743910639b1be4) [#<Notification id: 1, user_id: 2, notification_type: "incoming_friend_request", actor_type: "User", actor_id: 1, notificationable_type: "User", notificationable_id: 1, created_at: "2022-08-12 14:17:59.782068000 +0000", updated_at: "2022-08-12 14:17:59.782068000 +0000">]
/Users/frexuz/www/travclub-api/app/models/notification.rb:41:in `after_create'
/Users/frexuz/www/travclub-api/app/models/friendship.rb:61:in `notify_incoming_friend_request'
/Users/frexuz/www/travclub-api/lib/tasks/seeds/seed_friends.rake:8:in `block (2 levels) in <main>'
/Users/frexuz/www/travclub-api/db/seeds.rb:8:in `<main>'

Any idea? (this is in development)

I thought perform_later(self) would create a unique job id :)

gem 'rails', '6.1.6'
gem 'sidekiq', '6.5.1'
gem 'activejob-uniqueness', '0.2.4'

ActiveJob perform_now results in crash when the patch tries to delete the resource

I saw this when deploying to a live/staging enviroment, as during development, the server was on localhost, and the connection issue will not appear.

To reproduce:
Create a simple job
Use sidekiq
Run redis on non-localhost

Run 'SimpleJob.perform_now'

Strategy is set to until_executed

It will trigger this:

 (Error connecting to Redis on localhost:6379 (Errno::ECONNREFUSED)):
/usr/local/bundle/gems/redis-4.7.1/lib/redis/client.rb:398:in `rescue in establish_connection'
/usr/local/bundle/gems/redis-4.7.1/lib/redis/client.rb:379:in `establish_connection'
/usr/local/bundle/gems/redis-4.7.1/lib/redis/client.rb:115:in `block in connect'
/usr/local/bundle/gems/redis-4.7.1/lib/redis/client.rb:344:in `with_reconnect'
/usr/local/bundle/gems/redis-4.7.1/lib/redis/client.rb:114:in `connect'
/usr/local/bundle/gems/redis-4.7.1/lib/redis/client.rb:417:in `ensure_connected'
/usr/local/bundle/gems/redis-4.7.1/lib/redis/client.rb:269:in `block in process'
/usr/local/bundle/gems/redis-4.7.1/lib/redis/client.rb:356:in `logging'
/usr/local/bundle/gems/redis-4.7.1/lib/redis/client.rb:268:in `process'
/usr/local/bundle/gems/redis-4.7.1/lib/redis/client.rb:161:in `call'
/usr/local/bundle/gems/redis-4.7.1/lib/redis.rb:269:in `block in send_command'
/usr/local/bundle/gems/redis-4.7.1/lib/redis.rb:268:in `synchronize'
/usr/local/bundle/gems/redis-4.7.1/lib/redis.rb:268:in `send_command'
/usr/local/bundle/gems/redis-4.7.1/lib/redis/commands/keys.rb:192:in `del'
/usr/local/bundle/gems/activejob-uniqueness-0.2.4/lib/active_job/uniqueness/lock_manager.rb:13:in `block (2 levels) in delete_lock'
/usr/local/bundle/gems/redlock-1.2.2/lib/redlock/client.rb:152:in `with'
/usr/local/bundle/gems/activejob-uniqueness-0.2.4/lib/active_job/uniqueness/lock_manager.rb:12:in `block in delete_lock'
/usr/local/bundle/gems/activejob-uniqueness-0.2.4/lib/active_job/uniqueness/lock_manager.rb:11:in `each'
/usr/local/bundle/gems/activejob-uniqueness-0.2.4/lib/active_job/uniqueness/lock_manager.rb:11:in `delete_lock'
/usr/local/bundle/gems/activejob-uniqueness-0.2.4/lib/active_job/uniqueness/strategies/base.rb:30:in `unlock'

Not sure how to solve this the best way.

It looks like the parameters for the redlock client isn't passed properly at some point

Issue unlocking Jobs manually

First of great work on the gem 👍

We are experiencing some issues when trying to unlock jobs manually. When running normal operations we have not noticed any issues with either locking or unlocking, but we experienced this during a restart of Sidekiq. Probably Sidekiq restarted when a lock was active and this was then never unlocked. We noticed the job not being processed and tried unlocking without any success. Had to manually remove the key from Redis, in order to release the lock. I did notice what might be the issue for this behaviour, explanation:

Setup
Sidekiq 6.2.1
Redis 4.2.5
Rails 6.1.3.1
Ruby 3.0.1
Activejob Uniqueness 0.2.2 (gem 'activejob-uniqueness', require: 'active_job/uniqueness/sidekiq_patch')

    sidekiq (6.2.1)
      connection_pool (>= 2.2.2)
      rack (~> 2.0)
      redis (>= 4.2.0)
    activejob-uniqueness (0.2.2)
      activejob (>= 4.2, < 7)
      redlock (>= 1.2, < 2)
    redis (4.2.5)
    redlock (1.2.2)
      redis (>= 3.0.0, < 5.0)

Custom Initializer

ActiveJob::Uniqueness.configure do |config|
  config.redlock_servers = ["redis://#{REDIS_HOST}:6379/0"]
end

Reproducing Issue
The class for our job is named WorkflowExecutionPerformerJob and it's pretty straight forward.
It locks using the second argument while_executing and on conflicts moves the job back to another queue where we also throttle jobs using the Sidekiq API. The lock key might lock as this organization-x1x2x3.

WorkflowExecutionPerformerJob < ActiveJob::Base

unique :while_executing, on_conflict: ->(job) { job.schedule_job_later }

 def lock_key
    arguments.second
  end

The schedule_job_later function will enqueue another job on our throttle queue, so not really related.

I tried to unlocking the jobs manually using:

WorkflowExecutionPerformerJob.unlock!('organization-x1x2x3') => true
WorkflowExecutionPerformerJob.unlock!("other argument", "organization-x1x2x3") => true
WorkflowExecutionPerformerJob.unlock! => true
ActiveJob::Uniqueness.unlock! => true

Aslo tried to remove the job causing the lock from Sidekiq schedule (using the Sidekiq web GUI).
They all returned true but looking in Redis nothing was actually released.

redis = Redis.new(host: REDIS_HOST
redis.exists(WorkflowExecutionPerformerJob.new("other argument", "organization-x1x2x3").lock_key) => true

After looking at the code for the gem I tried the following:

config = ActiveJob::Uniqueness::Configuration.new
lock_manager = ActiveJob::Uniqueness::LockManager.new(config.redlock_servers, config.redlock_options)

After running this I did notice that the config.redlock_servers set in the initializer was not set correctly here so I ran:

config = ActiveJob::Uniqueness::Configuration.new
config.redlock_servers = ["redis://#{REDIS_HOST}:6379/0"]
lock_manager = ActiveJob::Uniqueness::LockManager.new(config.redlock_servers, config.redlock_options)

I then tried to release the lock using:

lock_manager.delete_locks(ActiveJob::Uniqueness::LockKey.new(job_class_name: 'WorkflowExecutionPerformerJob', arguments: ["other argument", "organization-x1x2x3"]).wildcard_key) => true

However the following still returned true:

redis.exists(WorkflowExecutionPerformerJob.new("other argument", "organization-x1x2x3").lock_key)

To finally release the lock I simply removed the key in Redis:

redis = Redis.new(host: REDIS_HOST
redis.del(""other argument", "organization-x1x2x3")

After that the job did process as expected.

Does using ActiveJob::Uniqueness::Configuration.new not respect the initializer here?
And what might be the issue causing the manually method to remove the job not working?

My initial guess is that it might be something with the initializer since trying to use the LockManager did not give me the correct config, however as I stated before normal operations obviously sets and unlocks the lock as expected.

Using Sidekiq Web to delete enqueued jobs - doesn't remove lock

Hey,
Using Sidekiq Web to delete enqueued jobs, lock remain active.

Steps to reproduce:

Create a Job with until_and_while_executing
Enqueue job
Dequeue the job using Sidekiq Web
Enqueue the same job again

Expected result: Should be able to enqueue the job
Actual result: I recieve an error Not unique ClassNameJob (Job ID: ce10cf03-d2c4-4369-a4ec-5f50316acf66) (Lock key: activejob_uniqueness:..

Support Redis 6+ via Redlock 2

Supports redis versions 6 or later using RedisClient: https://github.com/leandromoreira/redlock-rb/releases/tag/2.0.0

Rails 7.2 support

The gemspec is locking the dependency on activejob < 7.2, it would be great if that was upgraded to support Rails 7.2.

spec.add_dependency 'activejob', '>= 4.2', '< 7.2'

Would you like me to submit a PR to handle the latest activejob version on the CI?

Unlock on dead jobs in the UI

Hi!
Love the gem!

We just realized that since adding the Sidekiq patch, clearing out the dead jobs list is being quite slow too; any idea if that could indeed be because unlock is happening on dead jobs? Stopping that could be a materially perf improvement for a common maintenance activity.

Thanks!

Upgrading `0.2.5` to `0.3.1` hit some "Redlock::LockAcquisitionError" issues

I was getting these errors after upgrading:

Redlock::LockAcquisitionError: failed to acquire lock on 'Too many Redis errors prevented lock acquisition:
Redis::CommandError: NOSCRIPT No matching script. Please use EVAL.'

Which eventually got be bumbling around to this related issue in the redlock gem: leandromoreira/redlock-rb#124

Even though it is not technically this gem's fault, it might be worth calling out somewhere in the upgrade notes that the config.redlock_servers can no longer receive a Redis.new instance, and now must seem to be a RedisClient.new instance instead.

undefined method `unique' on calling `Rails.application.eager_load!` in rails initializer of other gem

Calling of Rails.application.eager_load! in an Rails initializer (/config/initializers/foobar.rb) raises error

NoMethodError: undefined method `unique' for MyJob:Class

The Railtie initializer of ActiveJob::Uniqueness runs later. Therefore the ActiveJob::Uniqueness::Patch is not applied yet.

There is no benefits to have 'active_job_uniqueness.patch_active_job' Railitie initializer. Switching to plain AS callback would resolve the problem

Quieting logs

This might be more a doc issue than a code issue but...

The message ActiveJob gives when it does not enqueue something because of non-uniqueness is pretty nasty (Failed enqueuing XXX to Sidekiq(default), a before_enqueue callback halted the enqueuing execution.). Is there a way to silence that from the logs?

Check if a job is locked

Great gem!

Is there any way to check (without enqueueing) if a job with some specific arguments is already locked?

	def unlock!(**args)
	lock_manager.delete_locks(ActiveJob::Uniqueness::LockKey.new(**args).wildcard_key)
	end

	# Global death handlers are introduced in Sidekiq 5.1
	# https://github.com/mperham/sidekiq/blob/e7acb124fbeb0bece0a7c3d657c39a9cc18d72c6/Changes.md#510
	if sidekiq_version >= Gem::Version.new('5.1')
	Sidekiq.death_handlers << ->(job, _ex) { ActiveJob::Uniqueness.unlock_sidekiq_job!(job) }
	end