"ThreadError: can't create thread (11)" on Heroku about sidekiq HOT 12 CLOSED

Oh, I have only one worker, but it's usually a long one that takes between 4 and 60 seconds to run for each job, that does a lot of HTTP requests to a couple external services, and it's getting around 20 calls every 10 minutes.

from sidekiq.

How many threads are you spinning up? Could be that Heroku limits your thread quantity. I remember having issues with this a while back (not with Sidekiq, but in general) then I lowered the thread quantity.

from sidekiq.

I'm using the default, which is 25, I think.

2012/2/25 Michael van Rooijen
[email protected]:

How many threads are you spinning up? Could be that Heroku limits your thread quantity. I remember having issues with this a while back (not with Sidekiq, but in general) then I lowered the thread quantity.

---

Reply to this email directly or view it on GitHub:
#54 (comment)

from sidekiq.

If your worker requires a lot of memory (e.g. lots of big ActiveRecord queries), you can run into dyno memory limits. Consider reducing your concurrency count and see if that helps.

from sidekiq.

Do you have a stack trace? Which version of Ruby and Rails? Cedar stack?

from sidekiq.

Yep, it's on cedar, Ruby 1.9.3-p0, Rails 3.2.1. This is the backtrace I get on Airbrake:

vendor/bundle/ruby/1.9.1/gems/sidekiq-0.5.1/lib/sidekiq/middleware.rb:73:in `send_to_airbrake'
vendor/bundle/ruby/1.9.1/gems/sidekiq-0.5.1/lib/sidekiq/middleware.rb:66:in `rescue in call'
vendor/bundle/ruby/1.9.1/gems/sidekiq-0.5.1/lib/sidekiq/middleware.rb:64:in `call'
vendor/bundle/ruby/1.9.1/gems/sidekiq-0.5.1/lib/sidekiq/processor.rb:26:in `block in invoke_chain'
vendor/bundle/ruby/1.9.1/gems/sidekiq-0.5.1/lib/sidekiq/processor.rb:29:in `call'
vendor/bundle/ruby/1.9.1/gems/sidekiq-0.5.1/lib/sidekiq/processor.rb:29:in `invoke_chain'
vendor/bundle/ruby/1.9.1/gems/sidekiq-0.5.1/lib/sidekiq/processor.rb:16:in `process'
vendor/bundle/ruby/1.9.1/gems/celluloid-0.8.0/lib/celluloid/calls.rb:93:in `dispatch'
vendor/bundle/ruby/1.9.1/gems/celluloid-0.8.0/lib/celluloid/actor.rb:181:in `block in handle_message'
vendor/bundle/ruby/1.9.1/gems/celluloid-0.8.0/lib/celluloid/task.rb:45:in `block in initialize'

But then, I'm noticing this is the same stack trace I get for all exceptions from sidekiq, which makes sense looking at the code that sends things to airbrake, which never sends a backtrace, but leaves it to the default of caller. My guess is that when you call a different method from rescue caller loses the backtrace info. I'll open a new ticket for this and investigate :)

from sidekiq.

FYI, the default Cedar stack is Ruby 1.9.2, could that be the issue?

from sidekiq.

Shouldn't matter whether you're on 1.9.2 or 1.9.3. Although as @mperham mentioned you might be hitting memory limits. Heroku provides you with 512mb ram per dyno. I believe this is a hard limit. So, once you reach it, Heroku will SIGTERM your process and restart it. Pretty sure that that's what is happening here since "garbage" memory grows over time (pretty rapidly I might add, especially with ORM/ODM objects like ActiveRecord, Mongoid, etc) and Ruby's GC does a poor job at collecting it. Eventually you will hit the 512mb ram wall and will get a force restart from Heroku.

from sidekiq.

If you see the following errors, then yes it's memory limit being reached, otherwise it's a bug:

2012-02-26T09:34:51+00:00 heroku[worker.1]: Process running mem=551M(107.8%)
2012-02-26T09:34:51+00:00 heroku[worker.1]: Error R14 (Memory quota exceeded)

from sidekiq.

Also, Heroku will only kill the process after 1gb, process performance will degrade after 512mb until finally sending SIGTERM at 1gb.

from sidekiq.

Papertrail tells me there were no R14 errors ever in my app.

BTW, I'm running 15 processors now instead of the default 25, and I
haven't seen this error in Airbrake since.

On Tue, Feb 28, 2012 at 5:30 PM, Fredrik Björk
[email protected]
wrote:

If you see the following errors, then yes it's memory limit being reached, otherwise it's a bug:

2012-02-26T09:34:51+00:00 heroku[worker.1]: Process running mem=551M(107.8%)
2012-02-26T09:34:51+00:00 heroku[worker.1]: Error R14 (Memory quota exceeded)

---

Reply to this email directly or view it on GitHub:
#54 (comment)

from sidekiq.

Oh, neat! Didn't know that. Last time I read through their docs I didn't see any mention of that, only the 512mb memory limit. Good to know though, that's pretty generous too, letting it climb up to 1gb before killing it. 512mb is actually quite a lot for a single process, unless you're using something like JRuby or Rubinius (in which case 512mb is still enough in most cases). My MRI processes however have never exceeded 120mb ram (on initial boot), but did climb up to maybe 200-250mb ram per process with Rails. Even more if it does A LOT of ActiveRecord/Mongoid object generation, in which case I fork off a child (if possible, like with background processes) to do all the dirty work, then once the child is finished, kill it and respawn it in a "clean" state, removing all garbage objects from the memory.

I guess it's not a memory issue then if that Heroku message isn't being sent. Not should what it would be then either.. Hmm.

from sidekiq.