Possible to have an ordered queue? about verk HOT 7 CLOSED

Maybe you can enqueue the second job from inside the first job, once all work is done?
Something like

defmodule Worker1 do
  def perform(arg1, arg2) do
    # do some work
    Verk.enqueue(%Verk.Job{queue: :default, class: "Worker2", args: [some, args], max_retry_count: 5})
  end
end

from verk.

Thanks for your suggestions.
In our case the jobs are triggered by events which should be handled in the order they arrive. So unfortunately it is not possible to enqueue the next job when the first one has finished.
So maybe verk is not quite the right tool for this use case and it's better to use some kind of event queue / event bus.

from verk.

@krasio's suggestion would be the ideal. If jobs are linked one job can start another job once it's done.

And even if there was such feature, if the first job from the queue fails and goes to the retry set, should the second job be started? In this case it's better to start jobs only if their dependencies were achieved as @krasio suggested.

from verk.

Do you think there is much work to do to use verk as a basis to achieve the sorted job behavior?

from verk.

It isn't entirely clear to me how you would want Verk to behave in that case, but it sounds like it would be quite complicated to achieve given the current implementation. Right now Verk job processing control is entirely based on FIFO queue logic.

Imagine the following situation:

• Job A is enqueued
• Job A is dequeued but fails during processing, it is added to the retry queue
• Job B is enqueued (we want this to run only after A)
• Job C is enqueued (this is independent of jobs A/B and should run FIFO)

How can we know which jobs need to run when? Or that job C doesn't need to wait on other jobs? Or which job B is waiting on? When job A fails does it prevent all further jobs from executing indefinitely?

from verk.

Yeah as @keyan said it's not just a sorted behaviour but a sort of dependency between jobs or something like this.

from verk.

Yeah you are both right, it's not quite easy. The approach we will give a try which works without modification of verk itself is to

• limit the queue to one worker
• Now the processing order is guaranteed unless there is a failure and retries
• To failure we implement the retry logic in the worker itself and only when the event was dispatched the worker finishes the job and processes the next one

from verk.