kiernanmcgowan / forerunner Goto Github PK

When we leave our forerunner system on for days on end, the logs being recorded by the manager for heartbeats to each worker really add up.

It would be great to be able to silence them or to make them much less frequent (heartbeats should still happen, but successful heartbeats should be recorded less frequently). Maybe bundle all the successful heartbeats into summary logs every 30 seconds or every minute?

Ex.
"Last 30 heartbeats from X were successful."

It would be really useful to have an API on the manager that could return information about all workers connected to that manager.

Misc information that would be useful:

• Worker host name
• Worker IP address.
• Worker host port.
• Information about the job that the worker thinks is assigned to it (if any).
• Any other useful info you can think of.

If the very first task in a composed job is a progress event the job will fail with the error:

TypeError: Object #<Object> has no method 'progress:assigned'
    at Object.editor_upload (/averylongpath/node_modules/forerunner/lib/worker.js:113:48)

Some event hooks are being called before the store callbacks, which is in appropriate if the store is a single source of truth in an application

I think it should be forerunner's responsibility to track and log failures in a way that makes it trivial to restart the job at the point where it failed. In this vein it would also be good for forerunner to die or at least do something when a job of a certain type fails too many times in a row, or to have an option for it.

Need to create a event hook for an 'aggregate failure' which is called when a job is marked as failed, but AFTER the store calls back

There needs to be an option to not use a cumulative data object; that is for the input field on task 1 to get deleted before the data object gets passed to task2. Otherwise when the object gets parsed by socket.io it'll exceed its maxBuffer size and the process will die with a websocket parser forced user kick: max buffer size reached error. It'd also be nice when JSON.stringifying/doing anything that loads the string into memory, because that would make it easier to debug individual tasks.

it might also be advantageous for check for maxBuffer when sending job event messages (in particular I'm looking at lib/utils/connection.js), and/or add resiliency so that when the websocket kills the connection or any other forerunner-based error it doesn't kill the entire job runner.

I think composed jobs could be improved if they were treated internally as separate jobs and handled the same way by forerunner. I'm not sure of the best way to handle optional passthroughs though. I was thinking maybe if the object passed to the callback included an extra field purge or something with the names of the fields that shouldn't be passed. Then forerunner could delete those fields as well as the purge field before giving it to the next task. Do you have any ideas?