juliaparallel / dagscheduler.jl Goto Github PK

Bug in reference counting of intermediate results causes occasional errors.

• make the scheduler/executor in Dagger configurable and isolate it to a separate module
• make DagScheduler conform to the common interface
• make Dagger PR with final changes

• an execution node consists of a broker and some executors on one physical node
• execution nodes can join/drop off the cluster
• master node, from where broad level scheduling is done
  • maps critical dag nodes to physical nodes
  • monitors progress, and reallocates dag nodes if brokers fail/drop off
  • allows new brokers to join dynamically

Partition the dag into sub graphs at points where there are cross dependencies (multiple dag nodes depend on it) or strong dependencies on a certain nodes and schedule them first. This is done at top level, and local brokers take care of executing assigned dag nodes using multiple local executor processes.