Hello,
I was trying to test @user.delay.destroy instead of creating a worker just for that.

I have few issues with this approach:

• testability: apparently does not respect this: https://github.com/mperham/sidekiq/blob/master/lib/sidekiq/testing.rb
• the first timeUser.last.delay.destroy runs it returns true, but it raises an exception[1]
• the subsequential same calls will always return false and occasionally return the same exception. the only thing I see in redis is 1331396520.563867 "get" "f328ddaba1aa4e76ca4e6fa1461160df"

Exception[1]

E, [2012-03-10T17:18:08.585763 #64926] ERROR -- : Sidekiq::Processor crashed!
NameError: uninitialized constant Sidekiq::Extensions
/Users/kain/.rvm/gems/ruby-1.9.3-p125/bundler/gems/sidekiq-e2d65c11b1e7/lib/sidekiq/util.rb:36:in `block in constantize'
/Users/kain/.rvm/gems/ruby-1.9.3-p125/bundler/gems/sidekiq-e2d65c11b1e7/lib/sidekiq/util.rb:35:in `each'
/Users/kain/.rvm/gems/ruby-1.9.3-p125/bundler/gems/sidekiq-e2d65c11b1e7/lib/sidekiq/util.rb:35:in `constantize'
/Users/kain/.rvm/gems/ruby-1.9.3-p125/bundler/gems/sidekiq-e2d65c11b1e7/lib/sidekiq/processor.rb:36:in `process'
/Users/kain/.rvm/gems/ruby-1.9.3-p125/gems/celluloid-0.9.0/lib/celluloid/calls.rb:93:in `dispatch'
/Users/kain/.rvm/gems/ruby-1.9.3-p125/gems/celluloid-0.9.0/lib/celluloid/actor.rb:202:in `block in handle_message'
/Users/kain/.rvm/gems/ruby-1.9.3-p125/gems/celluloid-0.9.0/lib/celluloid/task.rb:45:in `block in initialize'

Using Rails 3.2.2, sidekiq (master)

https://gist.github.com/1838040

I'm just running rake on the latest commit.

∴ uname -a
Linux maynard 3.0.0-16-generic #28-Ubuntu SMP Fri Jan 27 17:44:39 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

Hello,
there's already support for Airbrake and Exceptional, can't see why we can't support ExceptionNofifier as well.

Something along those lines:

class MyNotifier
  def initialize(options=nil)
  end
  def call(worker, msg, queue)
    yield
  rescue => e
    ::ExceptionNotifier::Notifier.background_exception_notification(e) if defined?(::ExceptionNotifier)
  end
end

I looked at #51, but the re-queueing doesn't seem to work on Heroku (using HEAD). This is from the logs:

2012-03-06T08:21:42+00:00 app[worker.1]: 2012-03-06T08:21:42Z 1 TID-aeam4 INFO: Shutting down
2012-03-06T08:21:42+00:00 app[worker.1]: 2012-03-06T08:21:42Z 1 TID-ow6lw INFO: Pausing 5 seconds to allow workers to finish...
2012-03-06T08:21:52+00:00 heroku[worker.1]: Error R12 (Exit timeout) -> Process failed to exit within 10 seconds of SIGTERM
2012-03-06T08:21:52+00:00 heroku[worker.1]: Stopping process with SIGKILL
2012-03-06T08:21:54+00:00 heroku[worker.1]: Process exited with status 137

Unfortunately, nothing was re-queued. Every restart/deploy on heroku terminates the workers, meaning we are just loosing all running threads. Any ideas?

E, [2012-02-20T13:07:05.661890 #9710] ERROR -- : Sidekiq::Processor crashed!
NoMethodError: undefined method `log' for #<XXXX:0x00000006bcb000>

After updating to 0.6 we have had some troubles with log... because it is no longer available. Now we have to use logger.debug, etc. A warning sign would be awesome (changelog, api...)

Thank you for the work on sidekiq, it's great and the code is a pleasure to read! :D

On line #45 in cli.rb:

Sidekiq.redis ||= RedisConnection.create(:url => options[:server], :namespace => options[:namespace])

Sidekiq.redis will never be nil since sidekiq.rb (line 55) defines .redis as:

@redis ||= Sidekiq::RedisConnection.create

The fix appears to change ||= to just = on line 45.

...or now that there is Sidekiq.configure_server to remove namespace (and other options?) from the cli option list (I can update the wiki if this is the plan since it recommends -n).

Thanks,
Scott

While switching my app to use sidekiq for background jobs, I wrote a gem to make it easier to send mail through sidekiq: https://github.com/indirect/sidekiq-mailer. It implements delayed mail with pretty much the same API as delayed_job does, just without monkeypatching ActionMailer. Is there somewhere on the wiki that it's appropriate to mention that it exists, so other people can find it instead of thinking they need to re-implement it? Thanks!

Hi,

I am not sure why it's implemented this way (https://github.com/mperham/sidekiq/blob/master/lib/sidekiq/cli.rb#L51-55), but would it be possible to support sidekiq with sinatra as well?

Thanks,
Owen

A line should be emitted by default for each message started and finished. Something like:

PID / ThreadId / Timestamp / Request Id / Worker Class / payload size / [start/finish]

RequestId should log Rails 3.2's RequestId feature or be a Sidekiq-generated GUID.

Running the following worker using ruby 1.9.3p0 ( mac os x lion ) and then exiting with ctrl-c sometimes exits ( although you have to give it a second ctrl-c ), and other times you have to kill -9 the worker as it will not exit gracefully

module Worker
  class TestWorker
    include Sidekiq::Worker

    def perform(key)
      while true 
        puts "x"
        sleep 1
      end
    end
  end
end

Am I doing something obviously wrong? e.g. you can not use while loops or is it a bug?

would minimize the lag between enqueue and dequeue

http://redis.io/commands/blpop

I'm trying to setup sidekiq w/ resque integration. I have tried setting the namespace via the -n cli option (now deprecated), a yml config file (passed in with -C) and a Sidekiq.configure_client block in my bootstrap file.

When starting a sidekiq process, it registers itself in the "workers" key in redis. When I start jobs, they are in the correct namespace, but can't get worked on, since the processor doesn't look for it in the right place.

I'm currently running the latest code from master, but it also breaks with the latest published version of the gem too.

This is a standalone resque installation, no rails or sinatra. However, that shouldn't matter that much, since I have a "boostrap" file that modified the $LOAD_PATH and loads up all our libraries, gems, Mongoid models and Worker files. This boostrap file is passed to sidekiq via the -r option. The sidekiq processor starts up fine, but doesn't seem to respect the namespace.

After switching to Sidekiq, we started seeing lots of CURSOR_NOT_FOUND errors in production in our Rails 3.1/Ruby 1.9.2 app on Heroku (Cedar stack).

The same queries work fine when run from the remote heroku console, but when they are run inside the sidekiq worker I get CURSOR_NOT_FOUND at random.

Something going on inside the mongo driver (https://github.com/mongodb/mongo-ruby-driver) that sidekiq doesn't like or vice versa?

I got Sidekiq working with resque-web on Heroku today. Things are working really well so far. The only thing that doesn't seem to work are failed jobs.

I see zero failed jobs right now, but I've seen several "Processor death" messages in the logs where jobs failed. Seems like a bug?

Hi,

Is it possible to perform a single job from a single queue as a synchronous operation? It’d be useful for testing workers closer to their actual environment, as well as testing that middleware actually behaves the way they should.

I’ve temporarily built a helper for this, but it’s not an ideal solution:

  def process_one_job(queue)
    actor = double(:processor_done! => true)
    work = MultiJson.decode(Sidekiq.redis.lpop("queue:#{queue}"))
    Sidekiq::Processor.new(actor).process(work, queue)
  end

For the above solution, if a job raises an error, a whole wall of not-so-nice warnings.

Jobs that get terminated when a worker gets shut down should get re-queued.

Sidekiq has new support for plain Ruby integration. We need a reasonably non-trivial example for users to reference and a wiki update.

I'm trying to keep all the Sidekiq configuration in config/initializers/sidekiq.rb and accessing Sidekiq.server_middleware raises uninitialized constant Sidekiq::Processor exception for client process. I assume we should just wrap the server configuration in if defined?(Sidekiq::Processor) clause? Or there is better way to do that?

PS. Thanks for the recent updates, trying out these now.

It would be great if sidekiq could support a few unix signals - here is a list similar to Resque:

• QUIT - Wait for sidekiq to finish processing current jobs then exit
• TERM / INT - Immediately kill sidekiq then exit
• USR1 - Don't start to process any new jobs
• CONT - Start to process new jobs again after a USR1

Perhaps allow a job max execution time config variable and use that as the timeout for a QUIT to exit

I'm getting that error about twice per day on heroku. I'm running on ruby 1.9.3, and sidekiq with the default config (bundle exec sidekiq).

It doesn't seem to be critical—AFAICT, all jobs are being processed, though I could be wrong.

Looking around to see what could cause this, I found Girl Friday's #29, but the ruby bug mentioned said it was fixed in 1.9.3, so I don't know what's going on.

I'm seeing the following Mongo connection errors in some workers:

Mongo::OperationFailure: errors HostAndPort: bad config string
/app/vendor/bundle/ruby/1.9.1/gems/mongo-1.3.1/lib/mongo/cursor.rb:101:in `next_document'
/app/vendor/bundle/ruby/1.9.1/gems/mongoid-2.4.3/lib/mongoid/cursor.rb:49:in `block in each'
/app/vendor/bundle/ruby/1.9.1/gems/mongoid-2.4.3/lib/mongoid/collections/retry.rb:29:in `retry_on_connection_failure'
/app/vendor/bundle/ruby/1.9.1/gems/mongoid-2.4.3/lib/mongoid/cursor.rb:48:in `each'
/app/vendor/bundle/ruby/1.9.1/gems/mongoid-2.4.3/lib/mongoid/contexts/mongo.rb:257:in `iterate'
/app/vendor/bundle/ruby/1.9.1/gems/mongoid-2.4.3/lib/mongoid/criteria.rb:145:in `block in each'
/app/vendor/bundle/ruby/1.9.1/gems/mongoid-2.4.3/lib/mongoid/criteria.rb:145:in `tap'
/app/vendor/bundle/ruby/1.9.1/gems/mongoid-2.4.3/lib/mongoid/criteria.rb:145:in `each'
/app/vendor/bundle/ruby/1.9.1/gems/mongoid-2.4.3/lib/mongoid/relations/targets/enumerable.rb:153:in `each'
/app/vendor/bundle/ruby/1.9.1/gems/mongoid-2.4.3/lib/mongoid/relations/referenced/many.rb:441:in `none?'
/app/vendor/bundle/ruby/1.9.1/gems/mongoid-2.4.3/lib/mongoid/relations/referenced/many.rb:441:in `method_missing'
/app/app/workers/hard_worker.rb:5:in `perform'
/app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-9dbaad8c1538/lib/sidekiq/processor.rb:39:in `block (2 levels) in process'
/app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-9dbaad8c1538/lib/sidekiq/middleware/chain.rb:71:in `call'
/app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-9dbaad8c1538/lib/sidekiq/middleware/chain.rb:71:in `block in invoke'
/app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-9dbaad8c1538/lib/sidekiq/middleware/server/failure_jobs.rb:10:in `call'
/app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-9dbaad8c1538/lib/sidekiq/middleware/chain.rb:73:in `block in invoke'
/app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-9dbaad8c1538/lib/sidekiq/middleware/server/active_record.rb:6:in `call'
/app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-9dbaad8c1538/lib/sidekiq/middleware/chain.rb:73:in `block in invoke'
/app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-9dbaad8c1538/lib/sidekiq/middleware/server/unique_jobs.rb:10:in `call'
/app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-9dbaad8c1538/lib/sidekiq/middleware/chain.rb:73:in `block in invoke'
/app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-9dbaad8c1538/lib/sidekiq/middleware/server/airbrake.rb:9:in `call'
/app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-9dbaad8c1538/lib/sidekiq/middleware/chain.rb:73:in `block in invoke'
/app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-9dbaad8c1538/lib/sidekiq/middleware/chain.rb:76:in `call'
/app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-9dbaad8c1538/lib/sidekiq/middleware/chain.rb:76:in `invoke'
/app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-9dbaad8c1538/lib/sidekiq/processor.rb:38:in `block in process'
/app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-9dbaad8c1538/lib/sidekiq/processor.rb:67:in `stats'
/app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-9dbaad8c1538/lib/sidekiq/processor.rb:37:in `process'
/app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.8.0/lib/celluloid/calls.rb:93:in `dispatch'
/app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.8.0/lib/celluloid/actor.rb:181:in `block in handle_message'
/app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.8.0/lib/celluloid/task.rb:45:in `block in initialize'

Is there a work around for this? Is this related to this issue: https://github.com/mongoid/mongoid/issues/1302 ?

It would be nice to start Sidekiq with a config file, especially on EY. Since Sidekiq doesn't kick out a PID file, it's hard/impossible to manage with monit. I got something working today on my current project that I'll be able to publish, but as of now I wouldn't be able to easily pass in configuration options.

#!/bin/sh
#
# This script starts and stops the Sidekiq daemon
# This script belongs in /engineyard/bin/sidekiq
#

PATH=/bin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin:$PATH
CURDIR=`pwd`

usage() {
  echo "Usage: $0 <appname> {start|stop|term|quit|pause|cont|abort} <environment> <conf_file>"
  echo -e "\nstop)     is a synonym for quit"
  echo "quit)     issues -QUIT to request the worker to stop"
  echo "term)     issues -TERM to request the worker to stop"
  echo "pause)    issues -USR2 when the current job finishes, worker remains running, but doesn't start any new jobs"
  echo "cont)     issues -CONT to continue after pausing"
  echo "abort)    issues -USR1 kills current job, or shuts down if there is no job" 
  echo "see https://github.com/mperham/sidekiq for more details"
  echo -e "\nSet the GRACE_TIME env variable to configure how long we should wait for after issuing quit, before asumming a problem and killing the worker and job"
  exit 1
}

...

#Baisc Setup of default values
APP=$1 ; ACTION=$2; RACK_ENV=$3; CONF_FILE=$4;

APP_DIR="/data/${APP}"
APP_ROOT="${APP_DIR}/current"
APP_SHARED="${APP_DIR}/shared"
APP_CONFIG="${APP_SHARED}/config"

if [ -e "${APP_CONFIG}/${CONF_FILE}" ]; then
  logger -t "sidekiq_${APP}" -s "Good, found a conf file. Proceeding..."
else
  logger -t "sidekiq_${APP}" -s "/data/${APP}/shared/config/${CONF_FILE} not found for app: ${APP}"
  exit 1
fi

WORKER_REF=`echo $CONF_FILE | sed s/.conf//`
LOG_FILE="$APP_ROOT/log/$WORKER_REF.log"
LOCK_FILE="/tmp/$WORKER_REF.monit-lock"
PID_FILE="/var/run/engineyard/sidekiq/$APP/$WORKER_REF.pid"
GEMFILE="$APP_ROOT/Gemfile"
SIDEKIQ="sidekiq"
if [ -f $GEMFILE ];then
  SIDEKIQ="bundle exec $APP_ROOT/ey_bundler_binstubs/sidekiq"
fi

if [ -d $APP_ROOT ]; then
  USER=$(stat -L -c"%U" $APP_ROOT)
  export HOME="/home/$USER" 

  # Fix for SD-3786 - stop sending in VERBOSE= and VVERBOSE= by default
  if declare -p VERBOSE >/dev/null 2>&1; then export V="VERBOSE=$VERBOSE"; fi
  if declare -p VVERBOSE >/dev/null 2>&1; then export VV="VVERBOSE=$VVERBOSE"; fi

  # Older versions of sudo need us to call env for the env vars to be set correctly
  COMMAND="/usr/bin/env $V $VV APP_ROOT=${APP_ROOT} RACK_ENV=${RACK_ENV} RAILS_ENV=${RACK_ENV} MERB_ENV=${RACK_ENV} QUEUE=${QUEUE} $SIDEKIQ -e ${RACK_ENV}"

...

The conf file argument to the script is used by monit to find the PID. I don't see a way around having this conf file, so, it would be nice to pass that conf file in to Sidekiq to set the CLI options. Otherwise I'm going to have to parse the conf file in bash, which I'd rather not do.

If you're ok with this, I'd like to work on it. It'd be nice to contribute more to this project than just some examples. 😄

Hi,

So after getting the cap task running, i'm seeing an issue after deploy when sidekiq:restart is called, I suspect the first kill call succeeds which means the second kill -9 cannot find a process, so the pid file is later not cleared, and start is never called.

Any plans on adding MongoDB support? I'd be happy to create a pull request if you have any suggestions on how to approach it.

What's the recommended approach for testing sidekiq? An in memory processor would be ideal.

After deploying/restarting my Cedar app on Heroku, the worker count in my Resque dashboard will double. Any chance this can be addresses in sidekiq?

This is most likely the same issue resque has:
https://github.com/defunkt/resque/issues/319

The client should include the Rails RequestId GUID in the payload so it can be logged by Sidekiq when processed.

After a while running I get these errors in the logs:

Processor death:  Got 'O' as initial reply byte. If you're running in a multi-threaded environment, make sure you pass the :thread_safe option when initializing the connection. If you're in a forking environment, such as Unicorn, you need to connect to Redis after forking.

Fatal error in sidekiq, dispatch loop died
input must be a string or IO

Any clues?

Hello,
I'm trying the new web ui, in the queues tab when clicking on a worker the request ends up with route not found (I'm mounting under /queues)

http://0.0.0.0:5000/queues/workers/kain-osx.local:1513-70170907144940:default

No route matches [GET] "/queues/workers/kain-osx.local:1513-70170907144940:default"

Just to let you know.

Sidekiq::Client.redis and Sidekiq::Manager.redis is the suck. Refactor to Sidekiq.redis which is always available.

How far off is stats support to enable resque's dashboard?

I've just started to play around with sidekiq a bit, and after generating a new Rails 3.1.2 app on Ruby 1.9.3 I toyed around with the following trivial worker:

class FetchWorker
  include Sidekiq::Worker

  def perform(url)
    puts url
    sleep 0.5
  end
end

Running this...

rails r '100.times {|i| FetchWorker.perform_async("http://www.example.com?n=#{i}") }'

...resulted in quite a few messages like the following to the terminal where sidekiq was running:

E, [2012-02-08T22:34:59.972354 #58448] ERROR -- : Sidekiq::Processor crashed!
NoMethodError: undefined method `clear_active_connections!' for ActiveRecord::Base:Class
/Users/kmarsh/.rvm/gems/ruby-1.9.3-p0/gems/sidekiq-0.5.1/lib/sidekiq/middleware.rb:86:in `call'
/Users/kmarsh/.rvm/gems/ruby-1.9.3-p0/gems/sidekiq-0.5.1/lib/sidekiq/processor.rb:26:in `block in invoke_chain'
/Users/kmarsh/.rvm/gems/ruby-1.9.3-p0/gems/sidekiq-0.5.1/lib/sidekiq/middleware.rb:64:in `call'
/Users/kmarsh/.rvm/gems/ruby-1.9.3-p0/gems/sidekiq-0.5.1/lib/sidekiq/processor.rb:26:in `block in invoke_chain'
/Users/kmarsh/.rvm/gems/ruby-1.9.3-p0/gems/sidekiq-0.5.1/lib/sidekiq/processor.rb:29:in `call'
/Users/kmarsh/.rvm/gems/ruby-1.9.3-p0/gems/sidekiq-0.5.1/lib/sidekiq/processor.rb:29:in `invoke_chain'
/Users/kmarsh/.rvm/gems/ruby-1.9.3-p0/gems/sidekiq-0.5.1/lib/sidekiq/processor.rb:16:in `process'
/Users/kmarsh/.rvm/gems/ruby-1.9.3-p0/gems/celluloid-0.8.0/lib/celluloid/calls.rb:93:in `dispatch'
/Users/kmarsh/.rvm/gems/ruby-1.9.3-p0/gems/celluloid-0.8.0/lib/celluloid/actor.rb:181:in `block in handle_message'
/Users/kmarsh/.rvm/gems/ruby-1.9.3-p0/gems/celluloid-0.8.0/lib/celluloid/task.rb:45:in `block in initialize'

I'm not sure if it resulted in any loss of jobs, it seems like they all ran... but there might be something up. I'm going to familiarize myself a bit more with the sidekiq code and see what I can make of it.

I'd love to use sidekiq in lieu of Resque, but I'm currently also using resque-loner. It would be nice to see a way to create jobs in sidekiq that would only be queued up once per payload.

If I can, I'll look through resque-loner's code and see what would be involved in adapting it for use in sidekiq, but I also wanted to open this issue to get input.

https://gist.github.com/1865906

In manager.rb#L103 queues_idx is reset to 0 on every dispatch, which means that queue will be unfairly weighted. queue_idx should be @queue_idx and act like a ring buffer index so we start from where we left off in the last dispatch.

ruby 1.9.3-p0
rails 3.2.1

Interesting is that changing the line from manager.rb:34
@ready = @count.times.map { Processor.new_link(current_actor) }
To:

@count.times.each { @ready << Processor.new_link(current_actor) }

makes the error go away completely

The back-trace is:

sidekiq -v
/Users/briannorton/.rvm/gems/ruby-1.9.3-p0@super-node/bundler/gems/sidekiq-1393f99dc16a/lib/sidekiq/version.rb:2: warning: already initialized constant VERSION
You did not specify how you would like Rails to report deprecation notices for your developemnt environment, please set config.active_support.deprecation to :log, :notify or :stderr at config/environments/developemnt.rb
I, [2012-02-14T17:48:45.350551 #18528]  INFO -- : Booting sidekiq 0.6.0 with Redis at localhost:6379
D, [2012-02-14T17:48:45.350709 #18528] DEBUG -- : {:queues=>["default"], :processor_count=>25, :require=>".", :environment=>nil}
/Users/briannorton/.rvm/gems/ruby-1.9.3-p0@super-node/gems/sidekiq-0.6.0/lib/sidekiq/manager.rb:34: [BUG] cfp consistency error - call0
ruby 1.9.3p0 (2011-10-30 revision 33570) [x86_64-darwin11.1.0]

-- Control frame information -----------------------------------------------
c:0009 p:---- s:0025 b:0025 l:000024 d:000024 CFUNC  :times
c:0008 p:---- s:0023 b:0023 l:000022 d:000022 CFUNC  :each
c:0007 p:---- s:0021 b:0021 l:000020 d:000020 CFUNC  :map
c:0006 p:0139 s:0018 b:0018 l:0005e0 d:0005e0 METHOD /Users/briannorton/.rvm/gems/ruby-1.9.3-p0@super-node/gems/sidekiq-0.6.0/lib/sidekiq/manager.rb:34
c:0005 p:0038 s:0014 b:0014 l:000013 d:000013 METHOD /Users/briannorton/.rvm/gems/ruby-1.9.3-p0@super-node/gems/celluloid-0.8.0/lib/celluloid/calls.rb:47
c:0004 p:0014 s:0007 b:0007 l:000fb0 d:000006 BLOCK  /Users/briannorton/.rvm/gems/ruby-1.9.3-p0@super-node/gems/celluloid-0.8.0/lib/celluloid/actor.rb:181
c:0003 p:0106 s:0005 b:0005 l:001970 d:000004 BLOCK  /Users/briannorton/.rvm/gems/ruby-1.9.3-p0@super-node/gems/celluloid-0.8.0/lib/celluloid/task.rb:45
c:0002 p:---- s:0003 b:0003 l:000002 d:000002 FINISH
c:0001 p:---- s:0001 b:-001 l:000000 d:000000 ------

-- Ruby level backtrace information ----------------------------------------
/Users/briannorton/.rvm/gems/ruby-1.9.3-p0@super-node/gems/celluloid-0.8.0/lib/celluloid/task.rb:45:in `block in initialize'
/Users/briannorton/.rvm/gems/ruby-1.9.3-p0@super-node/gems/celluloid-0.8.0/lib/celluloid/actor.rb:181:in`block in handle_message'
/Users/briannorton/.rvm/gems/ruby-1.9.3-p0@super-node/gems/celluloid-0.8.0/lib/celluloid/calls.rb:47:in `dispatch'
/Users/briannorton/.rvm/gems/ruby-1.9.3-p0@super-node/gems/sidekiq-0.6.0/lib/sidekiq/manager.rb:34:in`initialize'
/Users/briannorton/.rvm/gems/ruby-1.9.3-p0@super-node/gems/sidekiq-0.6.0/lib/sidekiq/manager.rb:34:in `map'
/Users/briannorton/.rvm/gems/ruby-1.9.3-p0@super-node/gems/sidekiq-0.6.0/lib/sidekiq/manager.rb:34:in`each'
/Users/briannorton/.rvm/gems/ruby-1.9.3-p0@super-node/gems/sidekiq-0.6.0/lib/sidekiq/manager.rb:34:in `times'

-- C level backtrace information -------------------------------------------

   See Crash Report log file under ~/Library/Logs/CrashReporter or
   /Library/Logs/CrashReporter, for the more detail of.

-- Other runtime information -----------------------------------------------
- Loaded script: /Users/briannorton/.rvm/gems/ruby-1.9.3-p0@super-node/bin/sidekiq
- Loaded features:
  
  0 enumerator.so
  1 /Users/briannorton/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/x86_64-darwin11.1.0/enc/encdb.bundle
  2 /Users/briannorton/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/x86_64-darwin11.1.0/enc/trans/transdb.bundle
  3 /Users/briannorton/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/site_ruby/1.9.1/rubygems/defaults.rb
  4 /Users/briannorton/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/1.9.1/x86_64-darwin11.1.0/rbconfig.rb
  5 /Users/briannorton/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/site_ruby/1.9.1/rubygems/deprecate.rb
  6 /Users/briannorton/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/site_ruby/1.9.1/rubygems/exceptions.rb
  7 /Users/briannorton/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/site_ruby/1.9.1/rubygems/custom_require.rb
  8 /Users/briannorton/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/site_ruby/1.9.1/rubygems.rb
  9 /Users/briannorton/.rvm/rubies/ruby-1.9.3-p0/lib/ruby/site_ruby/1.9.1/rubygems/version.rb

.......................... it just keeps going

Assuming we're processing long-running jobs (many slow API calls, etc.), and Heroku decides to shutdown the worker as described in [1] what will happen? Specifically

1. Will the exception notification be send?
2. Will the job be requeued (e.g. as Qu does it [2]) or put into failed queue?

I have made some tests and it seems like the job is just gone as it was when we were using Resque, but I can't be sure because 1) probably Airbrake setup wasn't picked up correctly 2) not really understanding the Celluloid internals to have a good idea on what should happen.

And yes, we're trying to restructure our jobs to execute faster, but it still seems like these SIGTERM should get 'special treatment', especially in Heroku env (which they don't in Resque [3])

[1] http://devcenter.heroku.com/articles/ps#graceful_shutdown_with_sigterm
[2] https://github.com/bkeepers/qu/blob/master/lib/qu/payload.rb#L27
[3] https://github.com/defunkt/resque/issues/319

I got this exception on Heroku running Ruby 1.9.2 after a processor died due to an exception.
After this happened, no further processing occurred and I had to manually restart the sidekiq process.

Processor death: Waited 1 sec
/app/vendor/bundle/ruby/1.9.1/gems/connection_pool-0.1.0/lib/timed_queue.rb:25:in `block (2 levels) in timed_pop'
/app/vendor/bundle/ruby/1.9.1/gems/connection_pool-0.1.0/lib/timed_queue.rb:22:in `loop'
/app/vendor/bundle/ruby/1.9.1/gems/connection_pool-0.1.0/lib/timed_queue.rb:22:in `block in timed_pop'
<internal:prelude>:10:in `synchronize'
/app/vendor/bundle/ruby/1.9.1/gems/connection_pool-0.1.0/lib/timed_queue.rb:21:in `timed_pop'
/app/vendor/bundle/ruby/1.9.1/gems/connection_pool-0.1.0/lib/connection_pool.rb:57:in `checkout'
/app/vendor/bundle/ruby/1.9.1/gems/connection_pool-0.1.0/lib/connection_pool.rb:48:in `method_missing'
/app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-4c7ae756c8b5/lib/sidekiq/middleware/server/unique_jobs.rb:12:in `call'
/app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-4c7ae756c8b5/lib/sidekiq/middleware/chain.rb:73:in `block in invoke'
/app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-4c7ae756c8b5/lib/sidekiq/middleware/server/airbrake.rb:6:in `call'
/app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-4c7ae756c8b5/lib/sidekiq/middleware/chain.rb:73:in `block in invoke'
/app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-4c7ae756c8b5/lib/sidekiq/middleware/chain.rb:76:in `call'
/app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-4c7ae756c8b5/lib/sidekiq/middleware/chain.rb:76:in `invoke'
/app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-4c7ae756c8b5/lib/sidekiq/processor.rb:37:in `block in process'
/app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-4c7ae756c8b5/lib/sidekiq/processor.rb:66:in `stats'
/app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-4c7ae756c8b5/lib/sidekiq/processor.rb:36:in `process'
/app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.8.0/lib/celluloid/calls.rb:93:in `dispatch'
/app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.8.0/lib/celluloid/actor.rb:181:in `block in handle_message'
/app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.8.0/lib/celluloid/task.rb:45:in `block in initialize'
Processor death: Waited 1 sec
/app/vendor/bundle/ruby/1.9.1/gems/connection_pool-0.1.0/lib/timed_queue.rb:25:in `block (2 levels) in timed_pop'
/app/vendor/bundle/ruby/1.9.1/gems/connection_pool-0.1.0/lib/timed_queue.rb:22:in `loop'
/app/vendor/bundle/ruby/1.9.1/gems/connection_pool-0.1.0/lib/timed_queue.rb:22:in `block in timed_pop'
<internal:prelude>:10:in `synchronize'
/app/vendor/bundle/ruby/1.9.1/gems/connection_pool-0.1.0/lib/timed_queue.rb:21:in `timed_pop'
/app/vendor/bundle/ruby/1.9.1/gems/connection_pool-0.1.0/lib/connection_pool.rb:57:in `checkout'
/app/vendor/bundle/ruby/1.9.1/gems/connection_pool-0.1.0/lib/connection_pool.rb:48:in `method_missing'
/app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-4c7ae756c8b5/lib/sidekiq/middleware/server/unique_jobs.rb:12:in `call'
/app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-4c7ae756c8b5/lib/sidekiq/middleware/chain.rb:73:in `block in invoke'
/app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-4c7ae756c8b5/lib/sidekiq/middleware/server/airbrake.rb:6:in `call'
/app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-4c7ae756c8b5/lib/sidekiq/middleware/chain.rb:73:in `block in invoke'
/app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-4c7ae756c8b5/lib/sidekiq/middleware/chain.rb:76:in `call'
/app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-4c7ae756c8b5/lib/sidekiq/middleware/chain.rb:76:in `invoke'
/app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-4c7ae756c8b5/lib/sidekiq/processor.rb:37:in `block in process'
/app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-4c7ae756c8b5/lib/sidekiq/processor.rb:66:in `stats'
/app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-4c7ae756c8b5/lib/sidekiq/processor.rb:36:in `process'
/app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.8.0/lib/celluloid/calls.rb:93:in `dispatch'
/app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.8.0/lib/celluloid/actor.rb:181:in `block in handle_message'
/app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.8.0/lib/celluloid/task.rb:45:in `block in initialize'
E, [2012-02-19T09:03:29.704046 #1] ERROR -- : Sidekiq::Manager crashed!
Timeout::Error: Waited 1 sec
/app/vendor/bundle/ruby/1.9.1/gems/connection_pool-0.1.0/lib/timed_queue.rb:25:in `block (2 levels) in timed_pop'
/app/vendor/bundle/ruby/1.9.1/gems/connection_pool-0.1.0/lib/timed_queue.rb:22:in `loop'
/app/vendor/bundle/ruby/1.9.1/gems/connection_pool-0.1.0/lib/timed_queue.rb:22:in `block in timed_pop'
<internal:prelude>:10:in `synchronize'
/app/vendor/bundle/ruby/1.9.1/gems/connection_pool-0.1.0/lib/timed_queue.rb:21:in `timed_pop'
/app/vendor/bundle/ruby/1.9.1/gems/connection_pool-0.1.0/lib/connection_pool.rb:57:in `checkout'
/app/vendor/bundle/ruby/1.9.1/gems/connection_pool-0.1.0/lib/connection_pool.rb:48:in `method_missing'
/app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-4c7ae756c8b5/lib/sidekiq/processor.rb:30:in `initialize'
/app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.8.0/lib/celluloid/calls.rb:47:in `dispatch'
/app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.8.0/lib/celluloid/actor.rb:181:in `block in handle_message'
/app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.8.0/lib/celluloid/task.rb:45:in `block in initialize'
E, [2012-02-19T09:03:29.704315 #1] ERROR -- : Sidekiq::Manager: ERROR HANDLER CRASHED!
ThreadError: Attempt to unlock a mutex which is not locked
/app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.8.0/lib/celluloid/mailbox.rb:101:in `unlock'
/app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.8.0/lib/celluloid/mailbox.rb:101:in `ensure in shutdown'
/app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.8.0/lib/celluloid/mailbox.rb:101:in `shutdown'
/app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.8.0/lib/celluloid/actor.rb:212:in `cleanup'
/app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.8.0/lib/celluloid/actor.rb:205:in `handle_crash'
/app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.8.0/lib/celluloid/actor.rb:144:in `rescue in run'
/app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.8.0/lib/celluloid/actor.rb:147:in `run'
/app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.8.0/lib/celluloid/actor.rb:89:in `block in initialize'
/app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.8.0/lib/celluloid/thread_pool.rb:46:in `call'
in `block in create'

It looks like this may actually be partially a celluloid issue, Although it appears that mutex.unlock is meant to raise
an exception if the mutex wasn't already locked. I'm unsure if this is how Celluloid's mailbox is meant to work.

Perhaps Sidekiq just needs some sort of ability to restart a crashed Manager.
Or at the very least it needs to exit the process if the Manager crashes so that a monitoring process can restart Sidekiq.

When trying to run sidekiq I get:

Booting sidekiq 0.5.1 with Redis at redis://localhost:6379/0
/Users/Kyle/.rvm/gems/ruby-1.9.2-p290@skateparks/gems/celluloid-0.8.0/lib/celluloid/task.rb:56: [BUG] Bus Error
ruby 1.9.2p290 (2011-07-09 revision 32553) [x86_64-darwin11.2.0]

-- control frame ----------
c:0010 p:---- s:0034 b:0034 l:000033 d:000033 CFUNC :resume
c:0009 p:0018 s:0030 b:0030 l:000029 d:000029 METHOD /Users/Kyle/.rvm/gems/ruby-1.9.2-p290@skateparks/gems/celluloid-0.8.0/lib/celluloid/task.rb:56
c:0008 p:0081 s:0025 b:0025 l:001df0 d:001df0 METHOD /Users/Kyle/.rvm/gems/ruby-1.9.2-p290@skateparks/gems/celluloid-0.8.0/lib/celluloid/actor.rb:181
c:0007 p:0052 s:0021 b:0021 l:000020 d:000020 METHOD /Users/Kyle/.rvm/gems/ruby-1.9.2-p290@skateparks/gems/celluloid-0.8.0/lib/celluloid/actor.rb:130
c:0006 p:0061 s:0015 b:0015 l:000010 d:000014 BLOCK /Users/Kyle/.rvm/gems/ruby-1.9.2-p290@skateparks/gems/celluloid-0.8.0/lib/celluloid/actor.rb:89
c:0005 p:---- s:0013 b:0013 l:000012 d:000012 FINISH
c:0004 p:---- s:0011 b:0011 l:000010 d:000010 CFUNC :call
c:0003 p:0021 s:0008 b:0008 l:001748 d:000007 BLOCK /Users/Kyle/.rvm/gems/ruby-1.9.2-p290@skateparks/gems/celluloid-0.8.0/lib/celluloid/thread_pool.rb:46
c:0002 p:---- s:0004 b:0004 l:000003 d:000003 FINISH

c:0001 p:---- s:0002 b:0002 l:000001 d:000001 TOP

-- Ruby level backtrace information ----------------------------------------
/Users/Kyle/.rvm/gems/ruby-1.9.2-p290@skateparks/gems/celluloid-0.8.0/lib/celluloid/thread_pool.rb:46:in block in create' /Users/Kyle/.rvm/gems/ruby-1.9.2-p290@skateparks/gems/celluloid-0.8.0/lib/celluloid/thread_pool.rb:46:incall'
/Users/Kyle/.rvm/gems/ruby-1.9.2-p290@skateparks/gems/celluloid-0.8.0/lib/celluloid/actor.rb:89:in block in initialize' /Users/Kyle/.rvm/gems/ruby-1.9.2-p290@skateparks/gems/celluloid-0.8.0/lib/celluloid/actor.rb:130:inrun'
/Users/Kyle/.rvm/gems/ruby-1.9.2-p290@skateparks/gems/celluloid-0.8.0/lib/celluloid/actor.rb:181:in handle_message' /Users/Kyle/.rvm/gems/ruby-1.9.2-p290@skateparks/gems/celluloid-0.8.0/lib/celluloid/task.rb:56:inresume'
/Users/Kyle/.rvm/gems/ruby-1.9.2-p290@skateparks/gems/celluloid-0.8.0/lib/celluloid/task.rb:56:in `resume'

-- C level backtrace information -------------------------------------------
0 libruby.1.9.1.dylib 0x000000010676651e rb_vm_bugreport + 110
1 libruby.1.9.1.dylib 0x000000010665a533 report_bug + 259
2 libruby.1.9.1.dylib 0x000000010665a6a1 rb_bug + 161
3 libruby.1.9.1.dylib 0x00000001066fcfb2 sigbus + 18
4 libsystem_c.dylib 0x00007fff8db09cfa _sigtramp + 26
5 libruby.1.9.1.dylib 0x0000000106704809 st_lookup + 137

[NOTE]
You may have encountered a bug in the Ruby interpreter or extension libraries.
Bug reports are welcome.
For details: http://www.ruby-lang.org/bugreport.html

[1] 433 abort sidekiq

Resque provides "inline" mode (activated with Resque.inline = true) in which the job is serialized/deserialized and processed immediately after enqueue'd or perform_async (in Sidekiq speak). Is there something similar available for Sidekiq?

Would be super helpful to get New Relic instrumentation in Sidekiq. Would it be possible to implement using middleware or would have to be added to the rpm_contrib gem?

Here's the documentation:
http://newrelic.com/docs/ruby/monitoring-ruby-background-processes-and-daemons

It seems that when the env config is nil then activerecord looks for a config that is not there and errors

How do I specify a worker to only process jobs from a certain queue? Will it always check for default jobs?

Getting this error after a while of running sidekiq:

2012-02-25T00:10:53+00:00 app[urgent.2]: E, [2012-02-24T16:10:53.099717 #1] ERROR -- : Fatal error in sidekiq, dispatch loop died
2012-02-25T00:10:53+00:00 app[urgent.2]: E, [2012-02-24T16:10:53.099940 #1] ERROR -- : attempted to call a dead actor (Celluloid::DeadActorError)
2012-02-25T00:10:53+00:00 app[urgent.2]: /app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.9.0/lib/celluloid/actor.rb:33:in `rescue in call'
2012-02-25T00:10:53+00:00 app[urgent.2]: /app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.9.0/lib/celluloid/actor.rb:30:in `call'
2012-02-25T00:10:53+00:00 app[urgent.2]: /app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.9.0/lib/celluloid/actor_proxy.rb:80:in `method_missing'
2012-02-25T00:10:53+00:00 app[urgent.2]: /app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-f2da18176527/lib/sidekiq/manager.rb:101:in `find_work'
2012-02-25T00:10:53+00:00 app[urgent.2]: /app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-f2da18176527/lib/sidekiq/manager.rb:117:in `block (3 levels) in dispatch'
2012-02-25T00:10:53+00:00 app[urgent.2]: /app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-f2da18176527/lib/sidekiq/manager.rb:116:in `times'
2012-02-25T00:10:53+00:00 app[urgent.2]: /app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-f2da18176527/lib/sidekiq/manager.rb:116:in `block (2 levels) in dispatch'
2012-02-25T00:10:53+00:00 app[urgent.2]: /app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-f2da18176527/lib/sidekiq/manager.rb:113:in `loop'
2012-02-25T00:10:53+00:00 app[urgent.2]: /app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-f2da18176527/lib/sidekiq/manager.rb:113:in `block in dispatch'
2012-02-25T00:10:53+00:00 app[urgent.2]: /app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-f2da18176527/lib/sidekiq/util.rb:33:in `watchdog'
2012-02-25T00:10:53+00:00 app[urgent.2]: /app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-f2da18176527/lib/sidekiq/manager.rb:109:in `dispatch'
2012-02-25T00:10:53+00:00 app[urgent.2]: /app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-f2da18176527/lib/sidekiq/manager.rb:81:in `block in processor_done'
2012-02-25T00:10:53+00:00 app[urgent.2]: /app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-f2da18176527/lib/sidekiq/util.rb:33:in `watchdog'
2012-02-25T00:10:53+00:00 app[urgent.2]: /app/vendor/bundle/ruby/1.9.1/bundler/gems/sidekiq-f2da18176527/lib/sidekiq/manager.rb:72:in `processor_done'
2012-02-25T00:10:53+00:00 app[urgent.2]: /app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.9.0/lib/celluloid/calls.rb:93:in `dispatch'
2012-02-25T00:10:53+00:00 app[urgent.2]: /app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.9.0/lib/celluloid/actor.rb:202:in `block in handle_message'
2012-02-25T00:10:53+00:00 app[urgent.2]: /app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.9.0/lib/celluloid/task.rb:45:in `block in initialize'
2012-02-25T00:10:53+00:00 app[urgent.2]: E, [2012-02-24T16:10:53.101333 #1] ERROR -- : /app/vendor/bundle/ruby/1.9.1/gems/celluloid-0.9.0/lib/celluloid/actor.rb:33:in `rescue in call'

Any thoughts on adding queue configuration inside each worker class?

I'd prefer something shorter than the current verbose syntax:
Sidekiq::Client.push('foo', 'class' => 'HardWorker', 'args' => ['bob', 5])

Running 'sidekiq' inside my Rails 3.1.3 directory fails due to it defaulting to production environment. Running 'sidekiq -e development' works.

Need to use the latest gem:

gem 'sidekiq',  '~> 0.9.0', :git => "git://github.com/mperham/sidekiq.git"

Then add sprockets as a dependency for the webview to load (for older versions of Rails)

Anything planned in terms retrying failed jobs using sidekiq? Should I be looking at resque-retry for this?

@robgleeson recently wrote a gem that provides new middleware for executing jobs at a particular time in the future. Unfortunately I don't think this is appropriate as middleware since it involves re-scheduling the job back into the 'scheduled' queue if it's not the appropriate time to run the job. You can see this here:

https://github.com/robgleeson/sidekiq-scheduler/blob/master/lib/sidekiq-scheduler/middleware.rb

Furthermore, the gem makes direct use of Sidekiq classes that aren't meant to be utilized outside of the core library.

After talking to @robgleeson, we think it makes sense to try and add this support directly to Sidekiq. This is quite similar to delayed_job's ability to run jobs in the future, e.g.:

Delayed::Job.enqueue(:payload_object => ['foo'], :run_at =>  5.days.from_now)

Mike, any thoughts on this? Is this something you'd like to be handled by sidekiq core? I'm thinking that Sidekiq::Manager#find_work could handle jobs in the special "scheduled" queue a bit differently, and instead of actually removing them from the queue, it essentially just peeks at them to see if they should be popped off the queue and executed. It probably doesn't need to support the same weight semantics that we currently have in place, but would just be a special worker (or set of workers) that always check this special queue for items that should be executed by doing a simple date comparison like in the link above.

Thoughts?

Need a cap recipe a la delayed_job to start/stop/restart Sidekiq upon deployment.