Batch processing? about verk HOT 2 CLOSED

Hey @heyitsjames ,

Maybe you could try writing workers that simply "synchronise" work by talking to a single process accumulating inside an ETS table?

This single process would simply hold an ETS table.

Once 500 is reached (your work adding jobs directly to this ETS table can find out if 500 was reached) or the time is up (5 seconds) then execute the work and reply back to all workers that are waiting? The jobs will be done once the batch is processed?

There are issues with this implementation as if you reach say 1500 jobs very quickly you may have a bottleneck.

This is not trivial to be solved as there's a lot of coordination. Maybe some redis scripts ? 😆 Feel free to look at the existing ones inside Verk 1.* (Verk 2.0 is coming one day and it uses Redis Streams)

from verk.

@edgurgel Thank you for the response!

As an update, I ended up using a Verk queue with a max of 500 jobs, and then using GenStateMachine to handle batching and timeout/flush operations.

Ended up working quite nicely, is very performant, and can scale nicely with adding more queues and linking them to their own batcher.

Each of my worker processes declares a receive block that the batcher can send success or failure, which allows me to use all the retry/requeue niceties that Verk affords me.

If I have time, it honestly doesn't seem like a stretch to add this into Verk itself. Instead of perform, there could be a batch_perform behaviour with timeout options.

Closing this now, if that's ok! Thanks for your help.

from verk.