Comments (10)
Exq polls redis for each and every queue independently using RPOPLPUSH. Let's say your poll_timeout
is 50ms, it will poll redis 20 times per second, 20 * 5600 = 112000 ops. Redis usually can handle this, but there is a limit. I think up to 1 million you can easily scale, after that, it might get tricky. I am also talking about the job polling, it does a lot of other things. I would suggest you start looking at Redis ops per second metrics first, you can do a redis-cli monitor
(not safe for production, run it locally) to get a rough idea about the commands executed by exq.
I think you are trying to design a queue per user? If the count could go up to 100k, I would suggest not to go this route. Also, Exq implementation itself might not be optimized. Most deployments I have seen are < 100 queues.
from exq.
Agreed with @ananthakumaran.
There's also some performance tests you can try here and adapt (we did some optimization a while back but not related to this case): https://github.com/akira/exq/blob/master/test/performance_test.exs
In the case you don't need stats, you can also disable those which would reduce the qps on Redis.
from exq.
@akira @ananthakumaran Thank you for your insights on this. I appologize that I didn't respond sooner, but I have been thinking about this problem, especially as the application continues to grow and place more load on Exq. I am noticing more "gremlins" with regards to jobs being processed predictably. Sometimes, I notice that a queue which is subscribed to, simply doesn't execute the jobs within.
I have the following thought for a potential workaround to buy me some more time. Basically, out of the thousands of queues which exist in the application, only around 10 - 20 ever have any jobs queued up at a given time. I'm thinking that I can "prune" the queues which don't have any jobs every 5 minutes. Then, when I need to queue up a job for that client later on, the application will create the new queue and subscribe to it dynamically.
I expected I could make the "prune" function as follows:
def prune() do
{:ok, queues} = Exq.Api.queue_size(Exq.Api)
Enum.each(queues, fn {queue, jobs} ->
if jobs == 0 do
Exq.Api.remove_queue(Exq.Api, queue)
end
end)
end
The problem I am facing here though, is that redis-cli monitor
still shows RPOPLPUSH
commands being run for the queue, even after the queue has been removed. I also tried this with Exq.unsubscribe Exq.Api, "queue-name"
but found the same result.
It seems like the queues remain in the Exq cache even after they are removed, causing the application to continue executing the RPOPLPUSH
somehow.
from exq.
Okay, quick update. After switching the above code to use Exq.unsubscribe Exq.Api, "queue-name"
, it seems to be working much better. However, I'm curious if you have any insight as to why this might be a bad idea? seems like I can keep my qps much lower on average if I run this worker every few minutes.
from exq.
Exq.Api.remove_queue(Exq.Api, queue)
This is not necessary and might delete actual jobs. unsubscribe is all you need, though it needs to be run on all worker nodes
if I run this worker every few minutes.
From what I understand, you are delaying the job execution (after enqueue) by a few minutes for queues with infrequent jobs. If this is ok, then unsubscribe might work.
from exq.
Hey @ananthakumaran, thank you for your thoughtful response. I enqueue jobs for users immediately based on incoming webhooks, with the following logic:
def enqueue_job(user) do
# Check for an existing subscription for this user.
{:ok, existing_subscriptions} = Exq.subscriptions(Exq)
subscription_already_exists? = Enum.find(existing_subscriptions, &(&1 == user.name))
unless subscription_already_exists? do
# Subscribe to the queue if a subscription is not already in place.
Exq.subscribe(Exq, user.name)
end
# Enqueue the job immediately, with the subscription is in place.
Exq.enqueue(Exq, user.name, Foo.Worker, [])
end
def prune() do
# Get list of queues along with their queue size
{:ok, queues} = Exq.Api.queue_size(Exq.Api)
{:ok, subscriptions} = Exq.subscriptions(Exq)
Enum.each(queues, fn {queue, jobs} ->
if queue in subscriptions and jobs == 0 do
# If there are no jobs in the subscribed queue, unsubscribe from that queue
Exq.unsubscribe(Exq, queue)
end
end)
end
The prune()
worker gets run every 5 minutes, removing all queues which don't have any jobs. Enqueuing a job always checks for the presence of a queue, creating it if it doesn't exist.
I ran some tests and this really improved things a lot, since 99% of the queues which were created dynamically have 0 pending jobs at any given moment. This pruning approach keeps the ~10,000 subscriptions which were in place previously (making 20 RPOPLPUSH
requests per second for each subscription) down below 100 - 200 subscriptions on average.
With this in mind, do you see anything which I might have failed to consider? Once again, I really appreciate your thoughts.
Eric
from exq.
If you know when jobs are getting enqueued, then this approach would work, though I haven't given much thought about how it would play with multiple worker nodes.
Exq also has a Dequeue behaviour which can be overridden. This was added to support rate limiter (see https://github.com/ananthakumaran/exq_limit), you might be able to re-purpose it for your use case.
from exq.
Please forgive my ignorance, but what is the reason Exq uses polling as opposed to Pub/Sub?
from exq.
Pub/Sub is not persistent. There is a blocking variant of rpoplpush called brpoplpush, but it doesn't support multiple lists redis/redis#1785. Exq also needs to be compatible with Sidekiq, which restricts what data structures are allowed
If I were to write a Job processing library today, I would use redis streams, but that's not possible with Exq due to Sidekiq compatibility.
from exq.
That makes sense, thanks!
from exq.
Related Issues (20)
- Proposal: add middleware hook just before apply(mod, fun, args) in Task started by worker HOT 2
- Can't find running job HOT 1
- How to configure a worker at runtime? HOT 4
- Order of middleware execution HOT 1
- Busy Tab is not working HOT 2
- Busy Tab is not working
- no function clause matching in Exq.Manager.Server.handle_call/3 HOT 2
- `Exq.enqueue_in` doesn't work in `mode: [:enqueuer, :api]` HOT 3
- Exq vs Oban? HOT 1
- Redis 6.0 ACL is supported ? HOT 1
- Dynamic subscription enqueued workers never run HOT 2
- [Question] How do I get job score? HOT 3
- Logging middleware doesn't give support for enqueueing logs HOT 5
- expected :timeout to be an integer of :infinity, got: nil HOT 5
- Unique jobs feature is not working HOT 2
- expected :timeout to be an integer of :infinity HOT 2
- Feature Request: enqueue a batch of jobs atomically HOT 3
- Missing assign `:worker_module` HOT 2
- How to configure Exq to push jobs to Sidekiq? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from exq.