Pub/Sub option? about php-resque HOT 10 CLOSED

It seems to me that pub/sub is a bit orthogonal to queuing for your intended purpose for a couple reasons.

1. If you have multiple workers subscribed to the same channel, they will all receive the job and act on it
2. If no workers are subscribed to a channel, then those messages will be lost (at least this is my understanding from the Redis documentation, it doesn't seem to have a replay mechanism).

Rather than use the pub/sub mechanism to pass the actual messages to workers, if I were going down this route (which seems to be a reasonable one), I would use the pub/sub mechanism to trigger the workers to go out and look for new messages in their respective queues. The workflow would look something like this:
worker 1 & 2) Starts and subscribes to the 'slow-process' channel
client) Pushes new work item onto 'slow-process' queue
client) Publishes message to the 'slow-process' channel (the content of this message isn't particularly important)
redis) Pushes message to all workers subscribed to the 'slow-process' channel
worker 1) Receives the new message on the 'slow-process' channel
worker 1) Pops a work item off the 'slow-process' queue and works on it
worker 2) Receives the new message on the 'slow-process' channel
worker 2) Tries to pop a work item off the 'slow-process' queue, but worker 1 has already grabbed the work item
worker 1 & 2) Both are now idle until they receive another message on the 'slow-process' channel

However, if your plan was already to distribute the workers across a number of servers, it may prove unnecessary to implement a pub/sub mechanism for triggering workers. I'd take a look at the average response time first since while certainly doable (Redis is already taking care of all the hard parts) the benefit that this gives you might not be particularly significant.

from php-resque.

My idea of using pub-sub was mostly to maybe have each worker listening on their channel and "blocked" waiting for the next job, that way you dont cycle jobs every x seconds, you take care of them immediately as they come... that was the main idea...

operationally it has its quirks as you mentioned, multiple workwrs would get the same if they werent all on different lists...

I'd like to have more info on what you end up doing if you do something in php, unfortunately we are a php-only shop and changing is not a viable option for the forseeable future...

from php-resque.

Would definitely be implementing this in PHP for the time being, however, the Sinatra (Ruby) frontend is entirely standalone (as long as your server has Ruby installed on it which most do) so I'd probably look into implementing any frontend in that rather than starting over from scratch.

from php-resque.

I see, could you please explain a bit further? I am a bit lost... I understand you need a semi-failproof way of sending messages (not banking critical, but not "hey that didnt get sent, no problem, itl get sent tomorrow") - just like me :), I also understood that you would be basing most of it (at least workers and queuers) in php, right?

Now are you talking about something you are working on? is it a "fork" or version of php-resque? is it something that will be available?

about the ruby part, it would be the admin part? or webview? I meant to get the resque interface running but didnt have time yet, is that what you were referring to?? just getting that part running and getting some of the bugs or quirks of php-resque ironed out? If so then that is something i'd be interested in helping test and eventually work together with!

all the best!

from php-resque.

I think the two issue threads are getting a bit crossed. Where I'm at now is primarily a PHP place as well (at least on the frontend) so I like to keep that aspect unified. There are two main issues I have with resque (these are related to both the Ruby version as well as the PHP port, although I'm currently just doing work on the PHP port).

1. Jobs can be lost without any indication (I'm more concerned with the without any indication than the lost job part for my uses)
2. Reliance on polling when Redis provides a nice built in Pub/Sub mechanism (this is mostly for my own learning/enjoyment since I know that the polling mechanism actually works quite well even with high volumes as long as you distribute enough workers)

Both of these would be forks (the two issues are orthogonal so I'd work on them in separate branches) off of Chris' port. I'm just starting to play around with both so no timeframe on either, but the goal would be to make transitioning to either transparent. I'll definitely push things up as I get them working so that you can check them out.

I was referring to the Ruby admin part, since I quite like the Sinatra framework and since none of the admin side was ported to PHP, I'd likely extend the admin interface of the Ruby version of Resque rather than built up a new version from scratch in PHP.

from php-resque.

If you want to take a look at the starts to a pub/sub version of php-resque, I just pushed up a branch to my fork (https://github.com/andrewjshults/php-resque/tree/pubsub). I've really only done the most basic testing (workers wait until something is published, enqueing a new job publishing a new message to a channel with the same name as the queue).

I've only tested this with a single worker on a single queue (therefore only a single channel as well) but I wanted to push this up so that other people can start to take a look and let me know if they have any suggestions.

from php-resque.

Hey Andrew, good work! I will try and have a better look tomorrow as its quite late. Most probably Ill be able to look at it in more depth on monday though... but Ill try during the weekend.

I'll try and contribute if at all possible too!

from php-resque.

I believe the blocking-pop approach which has since been proposed is probably a cleaner fix than the pub/sub approach mentioned here - though this approach would absolutely work. One thing of note, that may not have been clear at the time, is that PHP-Resque doesn't pause between every check - it only pauses if the queue is empty. So the first task takes up to the full delay (5s by default) to start, but all tasks after that (until the queue is empty again) are executed as soon as the previous job is complete.

As to "lost" jobs - under what conditions do these jobs disappear? Or has that been patched in the main fork already?

from php-resque.

@andrewjshults - Are you still seeing jobs disappear without indication? That seems like a pretty big bug, and it would be good if we could iron it out quickly.

@RVN-BR et al - The blocking pop approach has been merged into master, so this is probably no longer a problem, yes? We should probably close this issue.

from php-resque.

@danhunsaker I'm no longer working on that code base, and looking back at my comment, I can't remember the exact situation. It may have been when a job was picked up by a worked, but something causes the worker to fatal error (i.e., the failed job code does not get run since PHP doesn't have a way to handle all fatal error situations). Granted, this should be a fairly uncommon situation (since exceptions are handled properly), but I think I ran into it a couple times with jobs that had very high memory requirements.

Blocking pop is a much cleaner approach than using pub/sub to handle blocking

from php-resque.