sidekiq-scheduler / sidekiq-scheduler Goto Github PK

Hello,

i try to figure-out how to pass args to ActiveJob class

for example:

Sidekiq.set_schedule('job_name', 'SomeJob', cron: '* * * * *', args: { message_id: 23121, regenerate: true } )

but in method enqueue_job is

      if defined?(ActiveJob::Enqueuing) && config['class'].included_modules.include?(ActiveJob::Enqueuing)
        config['class'].new.enqueue(config)
      else

ActiveJob::Enqueuing.enqueue doesn't accept job arguments but initialize does.

Passing config['args'] to initialize works:
Just for example:

    def self.enqueue_job(job_config)
      config = job_config.dup

      config['class'] = config['class'].constantize if config['class'].is_a?(String)
      if config['args'].is_a?(Hash)
        config['args'].symbolize_keys!
      else
        config['args'] = Array(config['args'])
      end

      if defined?(ActiveJob::Enqueuing) && config['class'].included_modules.include?(ActiveJob::Enqueuing)
        config['class'].new(config['args']).enqueue
      else
        Sidekiq::Client.push(config)
      end
    end

Right now the released version of gem (latest on rubygems.org is 0.4.1) is broken, runner script outputs:

undefined method `start!' for #SidekiqScheduler::Manager:0x007f83f4f24c50

As i found in sidekiq-scheduler/cli.rb, there are still bang method calls:

      scheduler = SidekiqScheduler::Manager.new(scheduler_options)
      scheduler.start!
      run_manager
      scheduler.stop!

Is it possible to release fixed version 0.5 of the gem anytime soon?

In the main readme:

The schedule is a list of Resque worker classes with arguments and a schedule frequency (in crontab syntax). The schedule is just a hash, but is most likely stored in a YAML like so:

I'm assuming that should refer to Sidekiq jobs? (I'm brand new to sidkiq, just doing early investigations, so I don't really know)

When I run following from Rails console

BasicWorker.perform_in(10.seconds, 'bob', 5)

I expect this BasicWorker runs every 10 seconds..

But I noticed it runs only once..

I see my recurring jobs schedule in the Sidekiq UI. Even though my schedule is listed, it's still not very clear to me exactly when they will be run. It would be nice to know when the next job is scheduled to run. It would help a lot with debugging.

I'm not familiar with the internals to know how to make this possible. Is it possible to display next run time in web UI?

The latest version available for this gem on rubygems is 0.4.1. There are bugs that completely prevent the use of this gem but are fixed in the current master (such as #26).

It looks like the VERSION file has already been updated within the repo and a gem just never got pushed.

https://github.com/jmettraux/rufus-scheduler#everyjob-methods

Every jobs use a time duration between each start of their execution, while interval jobs use a time duration between the end of an execution and the start of the next.

any equivalent feature in sidekiq-scheduler ?

jruby-1.7.0 :020 > Sidekiq::Scheduler.enqueue_from_config(Sidekiq.schedule['Store-61474574'])
NoMethodError: undefined method `constantize' for Sidekiq::Scheduler:Class

need change：

job_config['class'] = job_config['class'].constantize if job_config['class'].is_a?(String) # Sidekiq expects the class to be constantized.

My redis connection is configured using "config/initializers/sidekiq.rb", reading from rails-config:

connection = proc {
  redis_from_url(Settings.sidekiq.redis.url)
}

Sidekiq.configure_server do |config|
  config.redis = ConnectionPool.new(size: 20, &connection)
end

This worked fine until I added sidekiq-scheduler. Now I get this error:

Redis::CannotConnectError: Error connecting to Redis on 127.0.0.1:6379 (Errno::ECONNREFUSED)

I don't even have redis running locally.

If I rename "config/initializers/scheduler.rb" to "config/initializers/sidekiq_scheduler.rb" (which is loaded after the sidekiq initializer), it gives a different bizarre error.

Any information on load order for these guys or similar problems?

we currently have 5 machines to run sidekiq with sidekiq-scheduler enabled. but the situation is every worker will start a scheduled job at time which means that there will be 4 duplicated jobs created.
how can I solve this?

Hey guys, I'd love it if, as part of your deploy process you could keep the tags up to date. It's super handy for devs to be able to see the latest version without having to go to Rubygems (since we're typically looking at the README anyway).

We were stuck at sidekiq-scheduler 2.0.0 due to the requirement that tilt be at version 2.0. We're currently using sass-rails 4.0.2, which requires tilt version 1.4.1. Upgrading tilt would require us to upgrade sass-rails, and upgrading sass-rails would require a good amount of effort on our part due to some major changes. As a last ditch effort, we forked the sidekiq-scheduler repo and dropped the tilt version requirement to >= 1.4.0 and pointed our gemfile to our forked repo. The gem installs fine, and, from what I can tell, functions as expected. It seems like you were only using the gem to parse erb files, and the web interface functions perfectly. Was there any reason you had tilt pegged to 2.0? If not, would you mind dropping the version requirement so that we can switch back to the official repo?

Thanks!

Hi,

I am using sidekiq-scheduler and noticed a strange behaviour in web UI. the schedules were only visible for every alternative sidekiq process restarts. so if its visible for the 1st time then i would have to restart the process 2 more times to see the schedules in web UI again. The culprit seems be the following line :

https://github.com/moove-it/sidekiq-scheduler/blob/master/lib/sidekiq-scheduler/schedule.rb#L46

def schedule=(schedule_hash)
schedule_hash = prepare_schedule(schedule_hash)
to_remove = (get_all_schedules || {}).keys - schedule_hash.keys

  schedule_hash.each do |name, job_spec|
    set_schedule(name, job_spec)
  end

  to_remove.each do |name|
    remove_schedule(name)
  end

  @schedule = schedule_hash
end

The schedules are removed after they are added to the redis. It happens alternatively because the second time they are cleared making the way for third start to set it properly. If this is the correct approach then i am assuming the schedules should have been properly cleared when the sidekiq exited.

Can you please point out if this issue is genuine or if my understanding is wrong.

Thanks,
Vikram

I got no cron / every / at / in / interval found for schedule (schedule) - skipping following the starting guide.

This trips new people as the recommended initializer will parse the YML expecting to find the individual job configs right away (relevant line).

When I click the new "enqueue now" button in the UI, I get an error:

ArgumentError at /sidekiq/recurring-jobs/test_notification/enqueue
Message must include a Sidekiq::Worker class, not class name: [PushNotifications::SendNotificationJob, PushNotifications::BaseJob, ActiveJob::Base, ActiveJob::Translation, ActiveJob::Logging, ActiveJob::Callbacks, ActiveSupport::Callbacks, ActiveJob::Execution, ActiveSupport::Rescuable, ActiveJob::Enqueuing, ActiveJob::QueueName, ActiveJob::QueueAdapter, ActiveJob::Core, Object, PP::ObjectMixin, ActiveSupport::Dependencies::Loadable, JSON::Ext::Generator::GeneratorMethods::Object, Kernel, BasicObject]

The whole reason I'm trying to manually enqueue the job is because I can't get any of my recurring jobs to run at all.

Here is my configuration yaml:

test_notification:
  every: 3m
  class: PushNotifications::SendNotificationJob
  args: ["2c56d016-ddbd-4537-8a19-4bf4d2a2e462", "Test Notification!"]

I'm using the latest version of Sidekiq (4.0.2) and the latest commit on the master branch of this repo. I'm trying to test this in my development environment. So I started sidekiq with bundle exec sidekiq. The output shows:

2016-01-07T21:04:14.777Z 23500 TID-ox04q3th4 INFO: Running in ruby 2.2.3p173 (2015-08-18 revision 51636) [x86_64-darwin15]
2016-01-07T21:04:14.777Z 23500 TID-ox04q3th4 INFO: See LICENSE and the LGPL-3.0 for licensing details.
2016-01-07T21:04:14.778Z 23500 TID-ox04q3th4 INFO: Upgrade to Sidekiq Pro for more features and support: http://sidekiq.org
2016-01-07T21:04:14.778Z 23500 TID-ox04q3th4 INFO: Booting Sidekiq 4.0.2 with redis options {:url=>nil}
2016-01-07T21:04:14.780Z 23500 TID-ox04q3th4 INFO: Loading Schedule
2016-01-07T21:04:14.780Z 23500 TID-ox04q3th4 INFO: Schedule empty! Set Sidekiq.schedule
2016-01-07T21:04:14.781Z 23500 TID-ox04q3th4 INFO: Schedules Loaded
2016-01-07T21:04:14.787Z 23500 TID-ox04q3th4 INFO: Starting processing, hit Ctrl-C to stop

Am I doing something wrong?

The README mentions that it's possible to store the schedules in config/sidekiq.yml, but it doesn't tell you what key to use. So it would be nice to have an example.

Eg. a config might look like:

---
:concurrency: 5
:pidfile: tmp/pids/sidekiq.pid
:queues:
  - default
  - process_asset
staging:
  :concurrency: 5
  :tag: myapp_staging
  :queues:
    - default
    - export
    - process_asset
    - import
production:
  :concurrency: 25
  :tag: myapp_production
  :queues:
    - default
    - export
    - process_asset
    - import

It would make sense to be possible to define schedules like this:

---
:concurrency: 5
:pidfile: tmp/pids/sidekiq.pid
:queues:
  - default
  - process_asset
:schedules:
  AssetWorker:
    every: 15m
staging:
  :concurrency: 5
  :tag: myapp_staging
  :queues:
    - default
    - export
    - process_asset
    - import
  :schedules:
    AssetWorker:
      every: 60m
    ImportWorker:
      every: 1m
    ExportWorker:
      every: 1m
production:
  :concurrency: 25
  :tag: myapp_production
  :queues:
    - default
    - export
    - process_asset
    - import
  :schedules:
    AssetWorker:
      every: 2h
    ImportWorker:
      every: 15m
    ExportWorker:
      every: 15m

The idea is that you can define a default schedule and override per environment.

regarding this issue
#8

is both sidekiq and sidekiq-scheduler supposed to be using tmp/pids/sidekiq.pid?

this is what is happening in my capistrano log when i deploy
is this supposed to be happening?

• 2013-10-14 17:05:35 executing `sidekiq:stop'
  • executing "if [ -d /app/current ] && [ -f /app/current/tmp/pids/sidekiq.pid ] && kill -0 cat /app/current/tmp/pids/sidekiq.pid> /dev/null 2>&1; then cd /app/current && bundle exec sidekiqctl stop /app/current/tmp/pids/sidekiq.pid 10 ; else echo 'Sidekiq is not running'; fi"
    • 2013-10-14 17:05:36 executing `sidekiq:start'
  • executing "cd /app/current ; nohup bundle exec sidekiq -e production -C /app/current/config/sidekiq.yml -i 0 -P /app/current/tmp/pids/sidekiq.pid >> /app/current/log/sidekiq.log 2>&1 &"
  • 2013-10-14 17:05:37 executing `sidekiq_scheduler:restart'
    then sidekiq-scheduler stops sidekiq which was just started
  • 2013-10-14 17:05:37 executing `sidekiq_scheduler:stop'
  • executing "cd /app/current && if [ -f /app/current/tmp/pids/sidekiq.pid ]; then bundle exec sidekiqctl stop /app/current/tmp/pids/sidekiq.pid 10 ; fi"
  • 2013-10-14 17:05:38 executing `sidekiq_scheduler:start'
  • executing "cd /app/current ; nohup bundle exec sidekiq-scheduler -e production -C /app/current/config/sidekiq.yml -P /app/current/tmp/pids/sidekiq.pid >> /app/current/log/sidekiq.log 2>&1 &"

hi,
I'm not sure I understand if sidekiq-scheduler runs separately or together with sidekiq.

For example, do I need to run sidekiq and sidekiq-scheduler separately? or just run sidekiq-scheduler by it self and will do both jobs?

I ask because in resque we have to run both separately loading 2 copies of the application in Ram, such as:

bundle exec rake resque:work
bundle exec rake resque:scheduler

I'm running sidekiq-scheduler with a default of 25 workers. I've noticed that whenever I restart sidekiq-scheduler, the number of workers displayed on the web increments by 25 (ie, restarting 3 times makes sidekiq web says 75 workers).

The log file reports that it shut down 25 workers, so I'm thinking it is a red herring, or perhaps I'm misinterpreting what that number means. I wanted to post this in sidekiq-scheduler before I ask over at sidekiq proper.

This very well may be a non-issue.

Thanks.

Hello, I noticed that you won't be supporting sidekiq-scheduler anymore. I understand the reasons for that but we are still using the scheduled jobs part of sidekiq-scheduler and while I could create a new gem for that purpose I would prefer (if you have no problems with that) to maintain this gem myself.
What I would do is remove the parts of that are already present in new versions of sidekiq and update the remaining code.
If you are okay with this could you put a message indicating that the gem is being maintained in https://github.com/Moove-it/sidekiq-scheduler?

I'll be waiting your response.
Thanks in advance.
Adrian.

https://github.com/mperham/sidekiq/blob/master/4.0-Upgrade.md

I'm using version 2.17.7 of sidekiq and can't use your plugin (trying to use the develop branch currently) as your spec requires ~> 2.12.0.

It looks like you are preparing for a new release, are you planning on updating the sidekiq version as well?

Thanks!

Whatever I do, I keep getting this error:

19:35:55 web.1     | [24973] ! Unable to load application: LoadError: cannot load such file -- sidekiq-scheduler/web

My load in config/routes.rb looks like this:

require 'admin_constraint'
require 'sidekiq/web'
require 'sidekiq-scheduler/web'

BattleHack::Application.routes.draw do
  ...
  mount Sidekiq::Web => '/admin/sidekiq', constraints: AdminConstraint.new
  ...

λ sidekiq-scheduler
2012-04-11T17:28:36Z 10635 TID-4q4 INFO: Booting sidekiq scheduler 0.2.1 with Redis at localhost:6379
2012-04-11T17:28:36Z 10635 TID-4q6 INFO: Booting sidekiq 1.0.0 with Redis at localhost:6379
2012-04-11T17:28:36Z 10635 TID-4q6 INFO: Running in jruby 1.6.7 (ruby-1.9.2-p312) (2012-02-22 3e82bc8) (Java HotSpot(TM) 64-Bit Server VM 1.6.0_29) [darwin-x86_64-java]
2012-04-11T17:28:36Z 10635 TID-4q4 ERROR: Fatal error in sidekiq, scheduler loop died2012-04-11T17:28:36Z 10635 TID-4q4 ERROR: wrong number of arguments (2 for 1)
2012-04-11T17:28:36Z 10635 TID-4q4 ERROR: /Users/anil.wadghule/.rvm/gems/jruby-1.6.7@test_sidekiq/gems/sidekiq-scheduler-0.2.1/lib/sidekiq-scheduler/manager.rb:52:in find_scheduled_work' org/jruby/RubyKernel.java:1410:inloop'
/Users/anil.wadghule/.rvm/gems/jruby-1.6.7@test_sidekiq/gems/sidekiq-scheduler-0.2.1/lib/sidekiq-scheduler/manager.rb:48:in find_scheduled_work' /Users/anil.wadghule/.rvm/gems/jruby-1.6.7@test_sidekiq/gems/sidekiq-scheduler-0.2.1/lib/sidekiq-scheduler/manager.rb:72:inschedule'
org/jruby/RubyKernel.java:1410:in loop' /Users/anil.wadghule/.rvm/gems/jruby-1.6.7@test_sidekiq/gems/sidekiq-scheduler-0.2.1/lib/sidekiq-scheduler/manager.rb:70:inschedule'
/Users/anil.wadghule/.rvm/gems/jruby-1.6.7@test_sidekiq/gems/sidekiq-1.0.0/lib/sidekiq/util.rb:42:in watchdog' /Users/anil.wadghule/.rvm/gems/jruby-1.6.7@test_sidekiq/gems/sidekiq-scheduler-0.2.1/lib/sidekiq-scheduler/manager.rb:66:inschedule'
/Users/anil.wadghule/.rvm/gems/jruby-1.6.7@test_sidekiq/gems/sidekiq-scheduler-0.2.1/lib/sidekiq-scheduler/manager.rb:30:in start' org/jruby/RubyBasicObject.java:1698:insend'
org/jruby/RubyKernel.java:2097:in send' /Users/anil.wadghule/.rvm/gems/jruby-1.6.7@test_sidekiq/gems/celluloid-0.10.0/lib/celluloid/calls.rb:98:indispatch'
/Users/anil.wadghule/.rvm/gems/jruby-1.6.7@test_sidekiq/gems/celluloid-0.10.0/lib/celluloid/actor.rb:212:in handle_message' /Users/anil.wadghule/.rvm/gems/jruby-1.6.7@test_sidekiq/gems/celluloid-0.10.0/lib/celluloid/task.rb:45:ininitialize'
2012-04-11T17:28:37Z 10635 TID-4rq INFO: Starting processing, hit Ctrl-C to stop

We have an issue with Sidekiq 4 and Sidekiq-Scheduler where our jobs are running 2x as often as they should.

Hi

i'm using Sidekiq 3.1.4 with Sidekiq-scheduler 1.0 with Ruby 2.0 and i'm having issues Reloading my schedules. my basic sidekiq.rb looks like:

#./sidekiq.rb
Sidekiq.configure_server do |config|
config.redis = { url: 'REDIS_URL', namespace: "sidekiq" , :size => 5}
end
Dir["./myworker.rb"].each { |file| require file }
Sidekiq::Scheduler.dynamic = true
Sidekiq.schedule = YAML.load_file(File.expand_path('../schedule.yml',FILE))
#./sidekiq.yml
:concurrency: 25
:pidfile: ./sidekiq.pid
:logfile: ./sidekiq.log
:queues:
- default
- myqueue
when loading my schedule from the rakefile using "bundle exec sidekiq -r ./sidekiq.rb -C ./sidekiq.yml -d" i see on the log that it loaded the schedules and he enqueue them according to the cron arguments.

what i dont fully understand is how flexible the scheduling can be. i want to have the ability to update a job, for example argument or different cron time, and want to add additional jobs while system is running, without the need to restart sidekiq process.

found few posts that suggest to search for the jid of the job and update it, although indicate performance issues on that solution.

now i saw on the Sidekiq API that there is a method like "reload_schedule!"
i i've tried to do something like

config = {
:class => 'MyWorker',
:cron => '*/1 * * * * ',
:args => 'myargs'
}
Sidekiq.set_schedule('testSystem1',config)
Sidekiq.reload_schedule!
schedules =Sidekiq.get_all_schedules
#does show the new schedule but it doesnt really loaded it for actual work of the workers.
i though that i should see on the sidekiq.log something like "Schedule loaded" as it wrote at the startup, and now with the new addition.

any help will be appriciated.

Thanks!

Levi

Hey, not sure if sidekiq-scheduler is responsible for this or sidekiq itself, so I'm going to open this issue both places.

I've written a unix init script to launch sidekiq using start-stop-daemon. When I try to start sidekiq using this script, I get this error when I pass in the option to daemonize.

Celluloid cannot be required until here, or it will break Sidekiq's daemonization
(...)/sidekiq-2.8.0/lib/sidekiq/cli.rb:127:in `load_celluloid'

This script fails:

start-stop-daemon --start --pidfile ${pidfile} \
    -N 2 --chuid ${run_as_user} --chdir ${app_dir} \
    --startas /usr/local/bin/bundle -- exec \
    sidekiq-scheduler -d -P ${pidfile} -e ${environment} -L ${stdout_log} -C ${config_file}

This script works:

start-stop-daemon --start --pidfile ${pidfile} \
    -N 2 --chuid ${run_as_user} --chdir ${app_dir} \
    --startas /usr/local/bin/bundle -- exec \
    sidekiq-scheduler -P ${pidfile} -e ${environment} -L ${stdout_log} -C ${config_file} &

You see I used a & in the second one instead of relying on sidekiq's internal daemonization (-d).

I'm wondering what we're doing wrong, any ideas?

Thanks!

Not sure if this is something with my config, but I get timeout errors in the scheduler log after it runs for a few days straight. It happened today, and beforehand it was running fine for about a week.

Today the log showed this:

2014-11-30_21:02:23.84590 2014-11-30T21:02:23.845Z 29383 TID-dr6wc INFO: queueing JobName1 ()
2014-11-30_21:02:23.84762 2014-11-30T21:02:23.846Z 29383 TID-dq2ng INFO: queueing JobName2 ()
2014-11-30_21:02:23.84803 2014-11-30T21:02:23.846Z 29383 TID-dq7ao INFO: queueing JobName3 ()
2014-11-30_21:02:23.84858 2014-11-30T21:02:23.846Z 29383 TID-dq5ew INFO: queueing JobName4 ()
2014-11-30_21:02:23.84860 2014-11-30T21:02:23.846Z 29383 TID-dq41g INFO: queueing JobName5 ()
2014-11-30_21:02:23.84884 2014-11-30T21:02:23.846Z 29383 TID-dq3bw INFO: queueing JobName6 ()
2014-11-30_21:02:23.84887 2014-11-30T21:02:23.847Z 29383 TID-dq80s INFO: queueing JobName7 ()
2014-11-30_21:02:25.84791 2014-11-30T21:02:25.847Z 29383 TID-dr6wc INFO: Timeout::Error: Waited 2 sec
2014-11-30_21:02:25.84812 2014-11-30T21:02:25.848Z 29383 TID-dq2ng INFO: Timeout::Error: Waited 2 sec
2014-11-30_21:02:25.84830 2014-11-30T21:02:25.848Z 29383 TID-dq7ao INFO: Timeout::Error: Waited 2 sec
2014-11-30_21:02:25.84884 2014-11-30T21:02:25.848Z 29383 TID-dq41g INFO: Timeout::Error: Waited 2 sec
2014-11-30_21:02:25.84885 2014-11-30T21:02:25.848Z 29383 TID-dq5ew INFO: Timeout::Error: Waited 2 sec
2014-11-30_21:02:25.84898 2014-11-30T21:02:25.848Z 29383 TID-dq3bw INFO: Timeout::Error: Waited 2 sec
2014-11-30_21:02:25.84921 2014-11-30T21:02:25.849Z 29383 TID-dq80s INFO: Timeout::Error: Waited 2 sec

I've set my sidekiq client/server connection pool size to 27, timeout to 2 seconds. Not sure what else it could be. Jobs that are manually kicked off are queueing and processed fine.

undefined method `start!' for #SidekiqScheduler::Manager:0x007f8df13854b0

lib/sidekiq-scheduler/cli.rb:24:in `run_scheduler'

Hello, I noticed that you won't be supporting sidekiq-scheduler anymore. I understand the reasons for that but we are still using the scheduled jobs part of sidekiq-scheduler and while I could create a new gem for that purpose I would prefer (if you have no problems with that) to maintain this gem myself.
What I would do is remove the parts of that are already present in new versions of sidekiq and update the remaining code.
If you are okay with this could you put a message indicating that the gem is being maintained in https://github.com/Moove-it/sidekiq-scheduler?

I'll be waiting your response.
Thanks in advance.
Adrian.

Hey, just tried the new gem release. This is my sidekiq.yml:

---
:timeout: 8
:queues:
  - priority_emails
  - emails
  - releases
  - default

:schedule:
  process_emails_due:
    cron: "* * * * *"
    queue: emails
    class: "EmailsDueWorker"

  process_releases_due:
    cron: "* * * * *"
    queue: releases
    class: "ReleasesDueWorker"

  process_releases_expire_on_start:
    cron: "* * * * *"
    queue: releases
    class: "ReleasesExpireOnStartWorker"

The tasks are scheduled but somehow I'm not seeing them in the dashboard

The current release enqueues jobs without taking into account if the jobs queue is being listened on by sidekiq.

PR #96 changed this behaviour so a job will be pushed only if its queue is active; as it is a breaking change, it's a good idea to activate the new behavior based on the value of a new config flag.

it was removed here:

e173071#diff-6706bb7d3bf023a73d5dd7443cb89459L25

but there is still a dependency on tilt in the code base.

When sidekiq.yml has the following:

:schedule:
  :remove_temp_files: # This key is loaded as a Symbol
    cron: " ..." 

The schedule :remove_temp_files is stored in Redis during sidekiq startup, while during the next startup is removed from redis. In the next startup the shedule will stored in Redis.

This occurs because SidekiqScheduler::Schedule#get_all_schedules fetches a Hash of schedules from Redis, and that hash's keys are strings.

The fetch/remove/add occurs here:
https://github.com/moove-it/sidekiq-scheduler/blob/master/lib/sidekiq-scheduler/schedule.rb#L40

When I schedule a job it is processed on each queue I have.

For example the job will be scheduled and run three times with the following code. Even though it should only run on the worker queue it runs on a worker queue on both the reports and payment sidekiqs

Scheduler:

:schedule:
  DailyCheckWorker:
    cron: "0 20 00 * * *"
    class: DailyCheckWorker
    queue: worker
    description: "abcdef"

Procfile:

web: bundle exec puma -C config/puma.rb
reports: bundle exec sidekiq -q reports
worker: bundle exec sidekiq -q worker
payment: bundle exec sidekiq -q payment

Hi

i'm using Sidekiq 3.1.4 with Sidekiq-scheduler 1.2 with Ruby 2.3 and i'm having issues Reloading my schedules.
my basic sidekiq.rb looks like:

#./sidekiq.rb
Sidekiq.configure_server do |config|
    config.redis = {  url: 'REDIS_URL', namespace: "sidekiq" , :size => 5}
end
Dir["./myworker.rb"].each { |file| require file }
Sidekiq::Scheduler.dynamic = true
Sidekiq.schedule = YAML.load_file(File.expand_path('../schedule.yml',__FILE__))

#./sidekiq.yml
:concurrency: 25
:pidfile: ./sidekiq.pid
:logfile: ./sidekiq.log
:queues:
 - default
 - myqueue

when starting sidekiq service from the rakefile using "bundle exec sidekiq -r ./sidekiq.rb -C ./sidekiq.yml -d" i see on the log that it loaded the schedules of shcedule.yml and he enqueue them according to the cron arguments.

what i dont fully understand is how flexible the scheduling can be. i want to have the ability to update a job, for example argument or different cron time, and want to add additional jobs while system is running, without the need to restart sidekiq process.

i couldnt find anything working on the net, but i'v found few methods on the sources code which seems should do the job, but still not quite sure how its really should work.

i've tried to use load_scheudle, update_scheudle and reload_schedule!
for example i've tried to do something like the following for loading additional job dynamicaly

 config =  {
     :class => 'MyWorker',
     :cron => '*/1 * * * * ',
     :args => 'myargs'
 }
 Sidekiq.set_schedule('testSystem1',config)
 Sidekiq.reload_schedule!  # and also tried with Sidekiq::Scheduler.update_schedule
 schedules =Sidekiq.get_all_schedules
 #does show the new schedule but it doesnt really loaded it for actual work of the workers.

i though that i should see on the sidekiq.log something like "Schedule loaded" as it wrote at the startup, and now with the new addition, but it didnt.

using the update_schedule sometimes just give an exception that the class of "@@scheduled_jobs" isnt initalized.

what is the right way to work with Sidekiq-scheduler?
how i can load when string the sidekiq process few jobs as we do with sidekis.schedule= and continue add additional jobs when the system is running?

any help will be appriciated.

Thanks!

Levi

Try ti run SidekiqScheduler and get it.
Need this fix? : 83e862a#diff-40be30c8fe2e0931896251ca948fed78L24

Allow to define scheduled jobs based on fixed schedule like ResqueScheduler do:

schedule.yml:

CancelAbandonedOrders:
  cron: "*/5 * * * *"

queue_documents_for_indexing:
  cron: "0 0 * * *"
  class: QueueDocuments
  queue: high
  args:
  description: "Job"

Somewhere in application:

Sidekiq.schedule = YAML.load_file("schedule.yml")

Please let me know if it's implemented. I looked through documentation and source code but found nothing.

Hi,

I'm trying to run a job with the cron: "0 0 19 * * *" but it doesn't ever run.

I'f I click the enqueue button on the web interface it runs my job though.

Any ideas where I am going wrong?

Would be nice to have this functionality that is available in https://github.com/ondrejbartas/sidekiq-cron

We switched away because sidekiq-cron doesn't appear to be working properly with high volume in Sidekiq 4, but this feature was definitely nice so you don't have to redeploy if you just need to disable a scheduled task.

I currently migrate from sidetiq gem, which now looks unsupported and sometimes behaves unexpected and I concerned by missing buttons for manually starting jobs through web ui. This feature will be very helpful for development.

my schedule in config/sidekiq.yml

:schedule:
  authenticom_daily_files_import:
    every: 4h
    class: AuthenticomDailyFilesImportWorker
    queue: default
    description: "This job pull data from FTP server and import it to database"

  check_alerts:
    cron: 0 2 * * *
    class: CheckAlertsWorker
    queue: default
    description: "Should check alerts"

With dynamic option set to true I would expect my Sidekiq proces to regularly pull changed schedules from Redis.

Sidekiq::Scheduler.�update_schedule looks like it is supposed to do so but it's never called (except for a unit test).

d8b66fa

This changes the logic so that if you explicitly set enabled=false in your initializers it is always overwritten. If enabled was set prior, then it should not be overwritten here.

We have an environment where we have multiple worker processes and each of this process loads up the scheudler
ie we have:
scheduler: bundle exec sidekiq -c 5 -q scheduler -q default
worker: bundle exec sidekiq -c 5 -q default

So we end up with duplicate schedules , how can we avoid it?

I'm seeing an issue where restarting the Rails app does not reload the schedule.

Following the README example, I have an initializer that loads config/schedule.yml

require 'sidekiq/scheduler'
Sidekiq.schedule = YAML.load_file(File.expand_path('../../../config/schedule.yml', __FILE__))

If the following is my schedule.yml

vagrant@precise64:/vagrant$ cat config/schedule.yml
############################################################################
# Defines the schedule of recurring Sidekiq jobs
# See https://github.com/Moove-it/sidekiq-scheduler for more information
############################################################################

# run, every minute, the job to look for delayed orders
queue_delayed_order_notifications:
  every: ["1m"]
  class: DelayedOrderNotifications
  description: "This job looks for delayed orders and triggers the appropriate notifications"

It loads fine:

vagrant@precise64:/vagrant$ bundle exec rails c
Loading development environment (Rails 4.1.2)
[1] pry(main)> Sidekiq.schedule
=> {"queue_delayed_order_notifications"=>
  {"every"=>["1m"],
   "class"=>"DelayedOrderNotifications",
   "description"=>"This job looks for delayed orders and triggers the appropriate notifications"}}

However, if I change the schedule - so changing from 1m to 1h:

vagrant@precise64:/vagrant$ cat config/schedule.yml
############################################################################
# Defines the schedule of recurring Sidekiq jobs
# See https://github.com/Moove-it/sidekiq-scheduler for more information
############################################################################

# run, every minute, the job to look for delayed orders
queue_delayed_order_notifications:
  every: ["1h"]
  class: DelayedOrderNotifications
  description: "This job looks for delayed orders and triggers the appropriate notifications"

Then I reload Rails:

vagrant@precise64:/vagrant$ bundle exec rails c
Loading development environment (Rails 4.1.2)
[1] pry(main)> Sidekiq.schedule
=> {"queue_delayed_order_notifications"=>
  {"every"=>["1m"],
   "class"=>"DelayedOrderNotifications",
   "description"=>"This job looks for delayed orders and triggers the appropriate notifications"}}

The schedule has not changed. It's still 1m in Sidekiq, even though I've completely reloaded the app.

In fact, in my Vagrant development environment, the only thing that I found to reload the schedule is actually to reboot the VM.

Is this expected? I've used Resque scheduler before, so I'm expecting behavior similar to that - with Resque scheduler, the schedule was loaded fresh every time the Rails app was restarted.

Hi. It's possible add or change scheduled job runtime?

For now we use model for store scheduled jobs (created or deleted with change in other model). Scheduled job is correctly showed (or missing) in web interface after add/update/delete model.

Jobs work as expected on server restart.

Here is model:

# == Schema Information
#
# Table name: schedulers
#
#  id               :uuid             not null, primary key
#  job_code         :string           not null
#  job_class        :string           not null
#  job_arguments    :jsonb
#  cron_expression  :string           not null
#  schedulable_id   :uuid
#  schedulable_type :string
#  state            :string           not null
#  disabled         :boolean          not null
#  created_at       :datetime         not null
#  updated_at       :datetime         not null
#

class Scheduler < ActiveRecord::Base
  include Enableable
  strip_attributes
  has_paper_trail

  belongs_to :schedulable, polymorphic: true

  validates :job_code,        presence: true
  validates :job_class,       presence: true
  validates :cron_expression, presence: true

  after_create  :create_cron_job
  after_update  :update_cron_job
  after_destroy :destroy_cron_job

  before_save do
    self.state = :scheduled
  end

  def self.schedules
    schedules = {}

    where(disabled: false).each do |row|
      schedules[row.job_code] = {
        "cron"  => row.cron_expression,
        "class" => 'Monitoring::AutomaticFinalizeJob',
        "queue" => 'low_priority',
        "args"  => Array(row.job_arguments)
      }
    end

    schedules
  end

  def job_arguments
    return super.symbolize_keys if super.is_a?(Hash)

    super
  end

  def create_cron_job
    Sidekiq.set_schedule(self[:job_code], cron: self[:cron_expression], class: self[:job_class], queue: 'low_priority', args: self[:job_arguments])
    Sidekiq.reload_schedule!
  end

  def destroy_cron_job(another_job_code = nil)
    another_job_code ||= self[:job_code]

    Sidekiq.remove_schedule(another_job_code)
    Sidekiq.reload_schedule!
  end

  def update_cron_job
    destroy_cron_job(job_code_was) if job_code_changed? || self[:disabled]

    create_cron_job unless self[:disabled]
    Sidekiq.reload_schedule!
  end

  def run_job
    Sidekiq::Scheduler.enqueue_job(cron_job)
  end

  def cron_job
    Sidekiq.get_schedule(self[:job_code])
  end

  def exist?
    cron_job.present?
  end
end

Here is sidekiq_scheduler.rb initializer:

if Rails.env == 'production'
  Sidekiq.configure_server do |config|
    config.on(:startup) do
      if LOCAL_HOSTNAME == 'app2.local'
        schedules = {}

        # Load predefined schedules from YAML file.
        schedules.merge!(YAML.load_file(Rails.root.join('config', 'scheduler.yml')))

        # Load predefined schedules from database.
        schedules.merge!(Scheduler.schedules)

        # Set schedules.
        Sidekiq.schedule = schedules

        Sidekiq::Scheduler.enabled = true
        Sidekiq::Scheduler.listened_queues_only = true
        Sidekiq::Scheduler.reload_schedule!
      end
    end
  end
else
  Sidekiq::Scheduler.enabled = false
end

As far as I can tell, this simply isn't working. Tested in development and production, the schedules load but never process. And that's after tinkering the code to remove the bugs (including the bang still).

If this is not maintained, do you think it would be better to remove completely so more people don't have to go through the pain of using (trying to use) something broken.