Comments (9)
Personally, I've moved to Sidekiq Capsules in one of my apps as it does what I need, which is limiting the number of jobs that execute in a queue
from sidekiq-limit_fetch.
Thanks for the reply. I think your needs are different then than ours. Am I correct in assuming you only have a single worker/server with a single Sidekiq process (but multiple threads) then? And you use capsules to limit the amount of threads certain queues have (while other queues have more threads available)?
Yep, I just use a single server. And with SolidQueue being released, I may end up moving to that instead.
I'm still happy to maintain this gem though. I just am unable to reproduce the issue. If anyone can reproduce it and create a PR, I'd be happy to merge and create a release.
from sidekiq-limit_fetch.
@kreintjes I think your hypothesis is correct. My use case consists of a single Sidekiq process with multiple threads, processing several queues and limit_fetch was used to process one of the queues strictly sequentially (by limiting concurrency to 1 with limit_fetch) without having to run multiple Sidekiq processes.
This worked with Sidekiq 6 and broke in Sidekiq 7, so ended moving to the capsules.
Thank you for the tips and your work on this project guys!
from sidekiq-limit_fetch.
Maybe because of this: https://github.com/sidekiq/sidekiq/blob/main/docs/7.0-Upgrade.md#capsules
from sidekiq-limit_fetch.
Ah, interesting. So it looks like Sidekiq basically has limiting built in, which is great to see.
from sidekiq-limit_fetch.
Will capsules also work when having multiple Heroku workers each running a separate Sidekiq instance with concurrency 1? Since limiting through capsules seems to be focussed on threads within one Sidekiq instance, while with sidekiq-limit_fetch you can set a maximum amount of workers per queue.
from sidekiq-limit_fetch.
Will capsules also work when having multiple Heroku workers each running a separate Sidekiq instance with concurrency 1? Since limiting through capsules seems to be focussed on threads within one Sidekiq instance, while with sidekiq-limit_fetch you can set a maximum amount of workers per queue.
@deanpcmad can you say anything about this? We are currently wondering whether to update sidekiq-limit_fetch or switch to Sidekiq Capsules.
from sidekiq-limit_fetch.
Would've been nice to change >= 6 sidekiq dependency to ~> 6 if v7 support is broken and the gem is no longer maintained.
Ran into a nasty race condition in production after upgrading to latest sidekiq and sidekiq-limit_fetch as the limit silently stopped working.
I checked the commit history before updating, which made an impression that v7 is supported but didn't think to look into the Issues.
from sidekiq-limit_fetch.
Personally, I've moved to Sidekiq Capsules in one of my apps as it does what I need, which is limiting the number of jobs that execute in a queue
Thanks for the reply. I think your needs are different then than ours. Am I correct in assuming you only have a single worker/server with a single Sidekiq process (but multiple threads) then? And you use capsules to limit the amount of threads certain queues have (while other queues have more threads available)?
In our situation we run multiple Heroku workers (amount is autoscaled) with a (single) Sidekiq process with the same configuration (concurrency 1 so a single thread per Heroku worker). We removed the limit_fetch gem and tried Sidekiq 7's capsules to see if we could limit the amount of concurrent jobs for some specific queues, but I did not work. As I feared, capsules only work to limit the concurrency within a single Sidekiq process, but not over multiple Sidekiq processes (on different servers). When I used capsules to say for example the heavy queue has concurrency 1, each of the Heroku workers had single thread to pick up jobs from the heavy queue next to a number of threads (default 5) to pick up jobs from the other queues. So the concurrency for the heavy queue would equal the amount of Heroku workers present at that moment, meaning we could still have multiple heavy jobs running at the same time (on different workers). So I'm afraid Capsules isn't a feasible solution for us, at least not without rethinking our complete Heroku sidekiq configuration and autoscaling set-up and maybe introducing extra (read) databases to limit/spread the load on our database.
Long story short: we still need something like sidekiq-limit_fetch to limit the concurrency of certain queues over all processes/servers. I tried updating both sidekiq (7.2.0) and sidekiq-limit_fetch (4.4.1) to the latest versions and it seems to still work for us. It makes sure only one Heroku worker picks up jobs from the super_heavy queue, while we have some available for the heavy queue and more for the other queues.
@JoshuaMart @paul-at Not sure why it doesn't work in your set-up. Maybe it only works for us because we have concurrency of 1 in our sidekiq.yml for our global configuration (and use multiple workers to scale horizontally instead of threads).
@deanpcmad could you say anything about the continuity of the sidekiq-limit_fetch gem? Are you the only maintainer currently? And are you stopping maintenance now that you no longer need it?
Ps. I also found the sidekiq-throttled gem which also seems to solve this problem. Unfortunately however, it has no ActiveJob integration at all it seems. We use ActiveJob so it is not really an option for us, but maybe it is for others who are (or want) to use Sidekiq::Job classes.
from sidekiq-limit_fetch.
Related Issues (20)
- undefined local variable or method `options' for #<Sidekiq::Manager> HOT 5
- NoMethodError: undefined method `logger' for Hash HOT 1
- Sidekiq 6.5.1 deprecation warning options use HOT 5
- Error starting Sidekiq Ruby 2.7 Sidekiq 6.5.x HOT 2
- Allow dynamic exclusion
- redis.brpop Passing the timeout as a positional argument is deprecated HOT 1
- Redis 5.0.0 compatibility: Redis#sadd will always return an Integer HOT 1
- ERROR: Error fetching job: undefined method `map' for nil:NilClass HOT 12
- Digest::Base cannot be directly inherited in Ruby HOT 4
- Broken with Sidekiq 7? HOT 7
- lowest priority vs weighted?
- Limit is not being enforced? HOT 7
- no jobs executing with limit 1 HOT 8
- Sidekiq 7 tests failing HOT 1
- undefined method `to_sym' for ["active_storage_analysis", 4]:Array HOT 3
- Sidekiq 7 Embedding results in eval error: undefined method `map' for nil:NilClass
- Queue leak HOT 2
- ignore
- Setting Priority on Sidekiq Dynamic Queues
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sidekiq-limit_fetch.