Add configuration option for maximum job run time about que HOT 13 CLOSED

I had someone email me earlier today asking how to cancel a job early. I
told him that there wasn't a safe way to implement that that I'm aware of,
and I don't think this is doable for the same reason.

The fundamental problem is that Que needs to yield to arbitrary user code
for potentially very long periods of time, and there isn't a foolproof way
in Ruby to stop that user code early and return control to Que (for an
example of how I tried to use Thread#kill and hack my way through its
pitfalls, look back through the commit history at the minor debacle that
was Que.stop!).

I'm open to the notion if anyone can find a good way to do it, but I think
the best we may be able to do is to have good, accessible documentation on
how to use timeouts and other appropriate tools to manage your own jobs
safely.

On Thu, Aug 14, 2014 at 1:56 PM, Joe Van Dyk [email protected]
wrote:

Had a que job that hanged forever, possibly because of a problem unrelated
to que.

Got me to thinking, would it be useful to have an option for setting the
maximum length a job could run? Obviously could be handled by client code
easily, but could be a useful option to include in que.

—
Reply to this email directly or view it on GitHub
#51.

from que.

Is it this problem? https://coderwall.com/p/1novga If so, fwiw, both bugs in that post were fixed.

from que.

It's along those lines. The fundamental problem is that threads targeted
with Thread#kill in Ruby will still run ensure blocks, but if the kill hits
while the thread is inside an ensure block the remainder of it will be
ignored. This means that even with proper use of ensure blocks, Thread#kill
can leave your application in an unpredictable state if you don't know
exactly what the thread is doing when you kill it.

(On a side note, take a look at my attempt to get ActiveRecord to care
about killed threads: rails/rails#13656)

One thing I've considered is supporting a simple messaging system for
workers. You could set some flag saying that a particular job or worker
pool needs to stop. Then, if you have a worker that is, for example,
looping through 100 API calls, it could call a private helper method in the
worker titled stop_if_necessary or something similar, which would check
for the existence of that flag and then cancel the job early if it is
present.

I think there are things we can do to make it easier for people to write
reliable jobs, but I don't think it's possible to support something like
this in a universal and safe way.

On Thu, Aug 14, 2014 at 2:11 PM, Joe Van Dyk [email protected]
wrote:

Is it this problem? https://coderwall.com/p/1novga If so, fwiw, both bugs
in that post were fixed.

—
Reply to this email directly or view it on GitHub
#51 (comment).

from que.

@chanks are you aware of a bug filed with Ruby for the Thread#kill/ensure problem?

from que.

No, sorry.

On Thu, Aug 14, 2014 at 2:53 PM, Joe Van Dyk [email protected]
wrote:

@chanks https://github.com/chanks are you aware of a bug filed with
Ruby for the Thread#kill/ensure problem?

—
Reply to this email directly or view it on GitHub
#51 (comment).

from que.

Wanna add one? :)

from que.

Ha, do you? On further reflection, I'm not sure you could even call it a
bug. If code running in an ensure block hangs indefinitely, wouldn't you
want a thread running it to be killable?

On Tue, Aug 19, 2014 at 5:54 PM, Joe Van Dyk [email protected]
wrote:

Wanna add one? :)

—
Reply to this email directly or view it on GitHub
#51 (comment).

from que.

I'm still confused as to what the exact problem is, or what consequences happen when a thread is killed in the ensure block.

Is it that killing a thread in that state can prematurely commit a transaction? Isn't that worked around in AR, or will about to be?

from que.

With ActiveRecord's transaction logic, killing a thread anywhere in its transaction will cause the transaction to be committed prematurely. The solution in my PR, which just checks whether the thread is aborting and rolls back the transaction if so, is the same thing Sequel does, and should prevent premature commits the vast majority of the time.

The issue they raised is that killing the thread while it's in an ensure block skips the remaining logic in that ensure block, so things will still be unpredictable. I accepted that at the time, but looking at it again now I'm not so sure, because I believe the logic that will issue the commit will also be skipped, which is really all that's necessary, right? There might be some complicating issue I haven't noticed, but this feels solvable to me.

But regardless, they don't seem to be that interested in it, and I only have so much patience for trying to get a patch into ActiveRecord. When I have more time I might dig into Sequel's transaction logic and try to verify that it's not prone to this sort of error.

from que.

Correction, looking at that thread again I actually raised that objection at the time, but it seemed to be ignored. Oh well.

from que.

I don't get why AR commits a transaction inside an ensure block. :( Why do we want to always ensure the transaction commits?

from que.

For future searchers, answered over here: rails/rails#13656 (comment)

from que.

Closing this for now. It's up to Ruby core to solve, if there is a solution.

from que.