registerWorker() performance over WAN about abraxas HOT 11 CLOSED

I remember @iarna blogging about the fact that the gearman protocol allows multiple concurrent GRAB_JOB's even for a single connection.

I have understood that this is why the abraxas registerWorker has the maxJobs option to limit how many GRAB_JOB calls it should concurrently send. Haven't tested that in practice though.

If you are running in an event loop environment like node.js, you could also manually register several worker objects to run in parallel with their individual connections.

from abraxas.

Yes, that's right, that's exactly what maxJobs does, check out https://github.com/iarna/abraxas/blob/master/example/load.js for an example of using it.

https://github.com/iarna/abraxas/blob/master/example/multicore.js has an example of using cluster to spread work across CPU cores too.

from abraxas.

abraxas supports concurrent JOB_ASSIGN_UNIQ but not concurrent JOB_GRAB_UNIQ

The sequence for 2 maxJobs is "grab-assign-grab-assign" instead of "grab-grab-assign-assign" and it harms worker performance on connections with large bandwidth-latency product.

For example,

Should it be while instead of if?

from abraxas.

@nponeccop Ah, I see what you're saying.

It will totally handle multiple concurrent jobs on a single connection, because it PRE_SLEEPs after accepting a job if there is room for more work. But that means that the concurrent job network flow looks like:

(assuming maxJobs = 2)

-> PRE_SLEEP
<- NOOP
-> GRAB_JOB_UNIQ
<- ASSIGN_JOB_UNIQ
-> PRE_SEEP
<- NOOP
-> GRAB_JOB_UNIQ
<- ASSIGN_JOB_UNIQ
-> WORK_COMPLETE
-> PRE_SLEEP
-> WORK_COMPLETE

Where as you would like:

-> PRE_SLEEP
<- NOOP
-> GRAB_JOB_UNIQ
-> GRAB_JOB_UNIQ
<- ASSIGN_JOB_UNIQ
<- ASSIGN_JOB_UNIQ
-> WORK_COMPLETE
-> PRE_SLEEP
-> WORK_COMPLETE

Am I understanding correctly?

from abraxas.

If we just change the if to a while, it looks like it would mostly be ideal. The only "issue" I would anticipate would be, if only one job was actually available, there would be an extraneous PRE_SLEEP– and that's A) easy to guard and B) maybe not an issue?

-> PRE_SLEEP
<- NOOP
-> GRAB_JOB_UNIQ
-> GRAB_JOB_UNIQ
<- ASSIGN_JOB_UNIQ
<- NO_JOB
-> PRE_SLEEP
-> WORK_COMPLETE
-> PRE_SLEEP

from abraxas.

@iarna Yes you understand it correctly. However, waiting for NOOP is a different issue #19 and could be fixed too if proved useful.

from abraxas.

If you wanted to put together PRs, I'd take them, if not, I will likely get to these, though I'm not sure on what timeline at the moment.

from abraxas.

@streamcode9 and I implemented a benchmark in Typescript:

benchmark/app.ts

And it shows that if if is replaced by while, and both the worker and the client are connected by WAN with 150 ms latency, the performance is increased twofold. I'll show few whisker charts later.

How do we clean the commit so it can be accepted as a PR? Now the repo contains Typescript type declarations. Is it ok? Or it's better to submit the compiled javascript version of the benchmark? Or should we submit the while PR separately from the bench?

The change of the library code is minimal: worker.js

from abraxas.

oh wow, that's super cool... I'd prefer the PR by itself for just the code change if that's not too much work.

from abraxas.

Is there any reason why #24 isn't merged? It's just the code change

from abraxas.

I didn't see it =D

from abraxas.