Unable to Replicate MFU for 7B on 80gb A100 about fms-fsdp HOT 3 CLOSED

Hi, if you get same throughput, then MFU should be same/similar.

For an easier calculation on MFU:

Leveraging the formula we can get model TFLOP for roughly 192 for 7b model per one bs. We run a batch size of 2, thus each step is running 192 * 2 TFLOP, and our step time is 2.2s (I assume you should have something same/similar given you observed same throughput), so we can get Model TFLOPS = 192 * 2 / 2.2 = 175.

MFU is calculated by the ratio of this number with theoretical peak (for A100, this was 312), so 175/312 = 0.56

from fms-fsdp.

Thank you, Linsong! Makes sense - must be a measurement error on my part. Appreciate the quick response :)

from fms-fsdp.

Noting why my measurement was wrong on the off chance it saves others time.

When you do sum([p.numel() for p in model.parameters()]) for an FSDP model, it only counts the parameters that are in the local FSDP shard. To count the full number of parameters in the FSDP model you can use:

with FSDP.summon_full_params(model):
    n_params = sum([p.numel() for p in model.parameters()])

from fms-fsdp.