Question: What would be the best way to push thousands of jobs at once? about cloudtasker HOT 2 CLOSED

Hey @bcgianni ! I've been digging into the Cloud Tasks API docs (both REST and gRPC) once again and it seems that Cloud Task does not provide a way to create tasks in bulk at this stage.

The batch capability of Cloudtasker won't really give you an edge here considering it enqueues jobs sequentially.

From there you have several options:

Option 1: sequential enqueuing
Assuming you have an array of [WorkerClass, args], you can simply do something like this:

job_list.each { |(klass, args)| klass.perform_async(*args) }

If you prefer you can also initialize the workers with MyWorker.new(job_args: args) then invoke schedule on them.

job_list.each(&:schedule)

Option 2: use a gem like parallel
Taking the first example above, you should be able to do:

Parallel.each(job_list) { |(klass, args)| klass.perform_async(*args) }

I haven't tried it but it should work just fine. In any case, feel free to reply here with any issue you find!

from cloudtasker.

Thank you @alachaum for the answer, that is exactly the way we went with, and it has worked pretty well actually.

Phenomenal job on the lib and documentation btw, we didn't have any issues to have it rolling in just a few days.

from cloudtasker.