zxxia / deft Goto Github PK

• Components

  • Containers(multi-containers in a pod)
    • Hook lib is attached to each container
  • DEFT Controller (Can be in a sidecar container)
    • Feat 1: Preempt low-priority jobs when high-priority jobs arrive
    • Feat 2: Send predicted sync points to containers/pods
  • Predictor for synchronization points
    • Algo 1: Offline-profiling
    • Algo 2: Mathematical modeling of execution delay for preempted jobs and preemption delay for preempting jobs

• Communication between the server in the container and the controller

  • HTTP (Inference request)
    • Opt 1: Flask
    • Opt 2: Python HTTP class

• Communication between hook lib in the container and the controller

  • Opt 1: HTTP
  • Opt 2: Pipe
  • Opt 3: Shared Memory

• Communication among server containers
  Not allowed.

Currently, kernel count is not zeroed when cudasync is encountered. This causes extra sync points to be inserted and each inference has sync points inserted at different positions.