Comments (7)
Note that it also works fine if I run:
$ heroku run 'QUE_WORKER_COUNT=1 rake que:work'
from que.
I was able to find a workaround. Since this is possibly related to #47, I decided to try putting the Que config lines in an initializer file rather than directly in application.rb.
# /config/initializers/que.rb
Que.worker_count = 1
Que.wake_interval = 10.minutes
Que.mode = :async
This appears to be working.
As a follow-up question, is there a way to prevent normal rake tasks from starting up Que workers? Otherwise I get Que output every time I run rake db:migrate
on Heroku.
from que.
Hm, now I'm getting another intermittent problem.
$ heroku run 'rake db:migrate'
Running `rake db:migrate` attached to terminal... up, run.9547
{"lib":"que","hostname":"51647c12-81b9-41f9-a6c7-32793d95d389","pid":2,"thread":70331113231360,"event":"worker_count_change","value":"1"}
{"lib":"que","hostname":"51647c12-81b9-41f9-a6c7-32793d95d389","pid":2,"thread":70331113231360,"event":"mode_change","value":"async"}
/app/vendor/bundle/ruby/2.1.0/gems/que-0.8.1/lib/que/adapters/base.rb:69: [BUG] Segmentation fault at 0x00000000000114
ruby 2.1.2p95 (2014-05-08 revision 45877) [x86_64-linux]
And lots more: https://gist.github.com/jamonholmgren/e863f6402d5407eedacb
It appears that this happens when I run a rake task at the same time that Que is waking up the workers (I changed the wakeup to 10.seconds).
I am also noticing that perhaps the order of the config settings affects this. When changing the order to the following:
Que.wake_interval = 10.seconds
Que.mode = :async
Que.worker_count = 1
...I haven't been able to replicate the above issue. Not sure if this is a coincidence or not.
Anyway, the idea behind Que is great and I have high hopes for it, but just need to figure out how to make it work well on Heroku.
from que.
Segfaults are bugs in Ruby, and there's not a lot we can do other than make a reproducible test case and report it. Que does run into them more often than other gems, because it tends to stress Ruby's threading system, which seems to be more bug-prone. If you can make a reproducible case that reliably results in a segfault, I'd certainly follow it with great interest. I've tried to make reproductions a few times and always given up before sinking too much time into them - they're tricky. I would wager that any apparent effects of one process on another are coincidental, though - their only mechanism of interaction would be via the job locking query, and its behavior is very well defined.
The last time I checked, Rails didn't offer a good way to check whether the currently running process is a server or a rake task or a console or a runner or whatever else. This is one of the reasons why the Railtie is the single most unreliable and prone-to-breakage part of the gem, and why I'm continuously tempted to break it off into its own gem and try to dump it on someone who's more interested in Rails internals than I am.
But none of that is helpful to you ;) I'll try to dig into this when I can, but I expect most of my free time over the next two weeks to be taken up by moving into a new place, so it may not be soon. If you or anyone else wants to take the time to dig into it, feel free, and I'll be happy to offer advice and answer questions when I can. If it helps, the only reason I can think of why setting up Que in an initializer would be functionally different than setting it up in application.rb is that those two configurations would be processed at different points in the application boot process. But again, I don't know enough about Rails internals to know what why you're seeing that specific effect from it.
from que.
Thanks, @chanks , very helpful reply. I'll keep playing with it.
In the meantime, I've written a short blog post about getting up and running quickly on Que: http://jamonholmgren.com/easy-background-tasks-on-heroku-with-que
Let me know if I've missed anything important on there if you do get a chance in between moving. :)
from que.
You can have the job automatically requeue itself, there's a document in
the docs folder that explains how.
On Friday, August 22, 2014, Jamon Holmgren [email protected] wrote:
Thanks, @chanks https://github.com/chanks , very helpful reply. I'll
keep playing with it.In the meantime, I've written a short blog post about getting up and
running quickly on Que:
http://jamonholmgren.com/easy-background-tasks-on-heroku-with-queLet me know if I've missed anything important on there if you do get a
chance in between moving. :)—
Reply to this email directly or view it on GitHub
#54 (comment).
from que.
I did read that. I kind of like the scheduled task, though...feels cleaner and more like how I'm used to things working.
from que.
Related Issues (20)
- Need support for draining queues HOT 3
- Support JRuby
- best way to re-activate a job HOT 2
- Que::ActiveRecord::Model.by_args incompatible with ActiveJob JobWrapper HOT 1
- Flaky Errors HOT 2
- Proper way of removing Que from project? HOT 3
- Schema version comment doesn't end up in Rails structure.sql HOT 3
- Que v1 should state that it's incompatible with ruby 3 HOT 2
- Tests fail in Docker on Apple Silicon
- Timeline for v2.0.0. release HOT 1
- Remove unnecessary trigger that notifies about job state transitions HOT 2
- Deprecated method used in connection.rb HOT 1
- Check that all que migrations have been applied at startup HOT 1
- Detect Que schema version missing due to bug in Rails schema dump - affects all versions of Rails 7 prior to 7.0.3
- Migration to v. 2.2.0 failed HOT 2
- Stats about failing jobs HOT 2
- upgrading from v0 to v2 when already on ruby 3 HOT 5
- Migrating from 1.0.0.beta4 to 1.4.1 HOT 5
- Que fails to calculate args and kwargs on Ruby 3.2 with ActiveJob HOT 1
- Migration fails when adding Que to a new project (with rspec) HOT 8
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from que.