Code Monkey home page Code Monkey logo

minion's Introduction

Minion: super simple job queue over amqp

Minion makes processing jobs over AMQP simple and easy.

Setup

Minion pulls the AMQP credentials out the environment via AMQP_URL.

$ export AMQP_URL="amqp://johndoe:abc123@localhost/my_vhost"

Alternativly you can explicitly set it programmatically like this:

Minion.amqp_url = "amqp://johndoe:abc123@localhost/my_vhost"

If no URL is supplied, Minion defaults to “amqp://guest:guest@localhost/” which is the default credentials for Rabbitmq running locally.

Principles

Minion treats your jobs with respect. The queues are durable and not autodelete. When popping jobs off the queue, they will not receive an ack until the job is done. You can rest assured that once queued, the job will not be lost.

Sends are done synchronously and receives are done asynchronously. This allows you to Minion.enqueue() from the console, or in a mongrel and you don’t need to worry about eventmachine. It also means that when enqueue returns, the AMQP server has received your message. Daemons set to receive messages however use eventmachine.

Message processing is done one at a time (prefetch 1). If you want tasks done in parallel, run two minions.

Push a job onto the queue

Its easy to push a job onto the queue.

Minion.enqueue("make.sandwich", { "for" => "me", "with" => "bread" })

Minion expects a queue name (and will create it if needed). The second argument needs to be a hash.

Processing a job

require 'minion'

include Minion

job "make.sandwich" do |args|
  Sandwich.make(args["for"],args["with"])
end

Chaining multiple steps

If you have a task that requires more than one step just pass an array of queues when you enqueue.

Minion.enqueue([ "make.sandwich", "eat.sandwich" ], "for" => "me")

job "make.sandwich" do
  ## this return value is merged with for => me and sent to the next queue
  { "type" => "ham on rye" }  
end

job "eat.sandwich" do |args|
  puts "I have #{args["type"]} sandwich for #{args["for"]}"
end

Conditional Processing

If you want a minion worker to only subscribe to a queue under specific conditions there is a :when parameter that takes a lambda as an argument. For example, if you had a queue that makes sandwiches but only if there is bread on hand, it would be.

job "make.sandwich", :when => lambda { not Bread.out? } do
  Sandwich.make
end

Error handling

When an error is thrown in a job handler, the job is requeued to be done later and the minion process exits. If you define an error handler, however, the error handler is run and the job is removed from the queue.

error do |e|
  puts "got an error! #{e}"
end

Logging

Minion logs to stdout via “puts”. You can specify a custom logger like this:

logger do |msg|
  puts msg
end

Meta

Created by Orion Henry

Patches contributed by Adam Wiggins, Kyle Drake

Released under the MIT License: www.opensource.org/licenses/mit-license.php

minion's People

Contributors

day avatar fordhurley avatar jonathantron avatar orionz avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

minion's Issues

How to pass that :when

Hello,
I would like to do something like this:


job "destroy.copy", :when => lambda { args["file_server"] == APP_CONFIG['server']['domain'] } do |args|
...

How should this look?

Publish / Subscribe (fanout) support

What do people think about adding support for Pub/Sub type jobs to Minion? Currently jobs are only processed once, but with AMQP fanout messaging, jobs could be sent to multiple subscribers.

These types of jobs are slightly different and require the worker identifying themselves in a unique way.

Here's an example:

Minion.name = "app-1"

subscribe "unsubscribes" do |args|
  Unsubscribe.perform(args)
end

Minion.name = "app-2"

subscribe "unsubscribes" do |args|
   User.unsubscribe_by_email(args["email"])
end

Now any publisher can send a message to the unsubscribes queue and it will be processed by both applications. You can load up multiple workers with the same name and they will only process the message once.

We have something similar to this already in production, if others are interested we can clean up the code and prepare a pull request.

next_job still has an issue

Should you not check for: queue.nil? as well ?

enqueue(queue,args.merge(response)) unless queue.empty?

should be:

enqueue(queue,args.merge(response)) unless queue.nil? or queue.empty?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.