ds3lab / dt-fm Goto Github PK

It seems two files imported in dist_pp_utils.py are missing:

from .dist_gpipe_pipeline_async_offload import GpipeAsyncOffload
from .dist_1f1b_pipeline_async import Pipe1F1BAsync

Hi, I really enjoyed your paper!

I was hoping you could provide some insight into how much pipeline communication and data parallel communication separately contributed to the communication costs. For example, as a percentage how much data transmission was needed for pipeline communication vs how much was needed for data parallel communication for a given device?

I want to do this but not on AWS, rather on consumer hardware.
All my friends have GPUs and we want to make a pool of our resource to train large models.
Is it possible?

First of all, thanks for the paper!
It was very intriguing to view model parallelism as an optimization problem in itself.

I wonder how would such scheduling work in a fully decentralized system?
Naively, you could run it concurrently on all nodes in hope that they find the same solution.

However, this naive option may be difficult to implement in geographically distributed networks: if nodes observe slightly different network bandwith, or if they take network measurements at a different time, they may end up with different solutions.

Is there a way to guarantee such network is consistent?
I mean, you can always elect a "leader" or let nodes vote on the solution, but perhaps there are more natural way to approach this.
What would you suggest?

p.s. another group that i'm in close contact faced similar issue their paper, and they ended up with a heuristic load-balancing rule where nodes greedily switch pipeline stages. However, unlike your work, they do not prove that such rule leads to optimal throughput.