Support multiple adapters, map adapters to queues about activejob HOT 25 CLOSED

I'm working on a branch to make it look like this

class MyJob < ActiveJob::Base
  queue_as :my_jobs
  set_adapter :sidekiq


  def perform(record)
    record.do_work
  end
end

MyJob will use sidekiq instead of the default adapter.

But i'm not sure about the naming yet.

from activejob.

Interesting. I wonder about putting the adapter into the code though, as this makes it more difficult to swap out backends.

Should a job know what adapter it is related to?

Have you thought about using some kind of configuration instead?
I did this before by mapping the queue names to adapters in the config, so perhaps something like:

ActiveJob::Base.queue_adapter = { default: :inline, my_jobs: :sidekiq }

from activejob.

But that will restrict from using same queue name for different adapter.

from activejob.

yes it would, I prefer that trade-off.

from activejob.

To elucidate more, when naming queues, which you generally have control over, I would think it would generally be a bad idea to have multiple queues with the same logical name across different backends. It is hard for me to imagine a case where that is a good idea, do you have one?

from activejob.

I think it more dirty that way.

ActiveJob::Base.queue_adapter = { default: :inline, 
                                  my_jobs: :sidekiq,
                                  mailer: :resque,
                                  paperclip: :sidekiq,
                                  notification: :sucker_punch,
                                  s3: :sidekiq,
                                  foo: :bar
}

And you will have to restart your app every-time you do a change in the initializer.
What if another dev removed the s3 queue from the initializer by mistake, you will have hard time debugging why you app is suddenly slow(since it started to process everything inline).

from activejob.

It is hard for me to imagine a case where that is a good idea, do you have one?

Have a mailer queue for delayed_job that send not so important emails (newletter, daily digests, ect)
Have a mailer queue in sucker_punch that send very important emails (password reset, login notification, invoices, ect)

from activejob.

That is an example of multiple similar queues, but does not seem like queues with the same exact name.

In that case I would be name them something like :express_mail and :standard_mail, that reflects that these are not going to the same queue (regardless of which backend that queue lives on). If you actually used the same name for both these, that would seem more confusing.

from activejob.

I just think that in the long term , that initialized might be forgotten and headaches will start happen. And everything someone create a new job with different back-end, he has to touch that file.
Just imagine if @dhh, used the same technique for AR instead of establish_connection.

while we are at it maybe it will be alright to have it in config/activejobs.yml

from activejob.

From the README:
"And you'll be able to switch between them without having to rewrite your jobs."

Obviously just changing one declaration is not a rewrite, so it doesn't exactly apply, but to me this implied there was some design direction towards decoupling the choice of job infrastructure from the job code itself, so I think it makes sense to have the backend choice remain in configuration

There is no doubt a better way to config this that would be cleaner than what I suggested - I jotted off that example as just a modest suggestion of how the queue could be related to an adapter, and the job abstracted from this.

I am not at all married to my particular example - more interested in the decoupling.

re: activejobs.yml, yes in my own project, I broke this kind of config out into its own config file where where each destination/queue was declared with its adapter and other options, but that seemed a bit heavy for ActiveJob, so I didn't want to suggest that immediately, but perhaps it would make sense, just as database.yml declares specifics.

from activejob.

I like the idea of having an external configuration that says which jobs run where. One example:

config.active_jobs.running_on[:sidekick] = [ MyJob, YourJob, HerJob ]
config.active_jobs.running_on[:stomp] = [ HisJob, TheirJob ]

I think selecting which queue you're using is a deployment configuration issue. Just like config/database.yml is a deployment configuration issue. You should be able to use the same ARs, without change, on SQLite or MySQL. And you should be able to use the same jobs on whatever adapter. Without changing their declaration.

from activejob.

Do you use two background jobs professors? Do you know anyone who does? I think this is a nice thing to have but overengineering at this point. We should keep this in a wiki page and re-evaluate at a later point. Maybe have an icebox page with this kind of ideas--
Cristian Bica

On Thu, May 29, 2014 at 8:29 PM, Andrew Kuklewicz
[email protected] wrote:

From the README:
"And you'll be able to switch between them without having to rewrite your jobs."
Obviously just changing one declaration is not a rewrite, so it doesn't exactly apply, but to me this implied there was some design direction towards decoupling the choice of job infrastructure from the job code itself, so I think it makes sense to have the backend choice remain in configuration
There is no doubt a better way to config this that would be cleaner than what I suggested - I jotted off that example as just a modest suggestion of how the queue could be related to an adapter, and the job abstracted from this.
I am not at all married to my particular example - more interested in the decoupling.

re: activejobs.yml, yes in my own project, I broke this kind of config out into its own config file where where each destination/queue was declared with its adapter and other options, but that seemed a bit heavy for ActiveJob, so I didn't want to suggest that immediately, but perhaps it would make sense, just as database.yml declares specifics.

Reply to this email directly or view it on GitHub:
#74 (comment)

from activejob.

I'd like to see a real example before going down this path as well.

from activejob.

@cristianbica , 👍 for the icebox .

from activejob.

AJ should solve the 80% problem. If you want to use a different queue for a few special job type, don't use AJ for those jobs.

from activejob.

I would go for 99% :)--
Cristian Bica

On Thu, May 29, 2014 at 8:41 PM, Mike Perham [email protected]
wrote:

AJ should solve the 80% problem. If you want to use a different queue for a few special job type, don't use AJ for those jobs.

Reply to this email directly or view it on GitHub:
#74 (comment)

from activejob.

I do this; I don't think many apps do.

I have some simple jobs my own system executes using a local queue, like mailing or updating a search index, and more complex jobs that are shipped off to a different system for things like long running media processing via SQS.

from activejob.

I’d be happy to evaluate a PR on the subject. I’m sympathetic because we have ARs that are running against different databases in Basecamp. So know that feeling.

On May 29, 2014, at 7:44 PM, Andrew Kuklewicz [email protected] wrote:

I do this; I don't think many apps do.

I have some simple jobs my own system executes using a local queue, like mailing or updating a search index, and more complex jobs that are shipped off to a different system for things like long running media processing via SQS.

—
Reply to this email directly or view it on GitHub.

from activejob.

@mperham using a different system to queue other jobs is always an option, but as AJ is meant to abstract connections to different queuing systems, then I could see it doing this, just as a database.yml can detail multiple connections even if most people only use one database.

from activejob.

@dhh I'll put a PR where my mouth is, if folks don't mind keeping this out of the icebox for a bit.

from activejob.

@kookster , go for it. If you need help ping me.

@dhh : I like the idea of activejobs.yml, we could have different adapters for different environments. In test we could use inline for example.

from activejob.

@seuros, I don't even think we need activejobs.yml out the gate. The API I showed can just be in config/environments/production.rb for example.

from activejob.

@dhh , i saw an application with hundreds of workers.

from activejob.

I like the queue -> adapter solution.

from activejob.

If someone decides to pick this up and work on it, open a PR with the actual implementation.

from activejob.