Thunder is 10**6 times slower than Lightning about lightning-thunder HOT 3 CLOSED

😁 Right but AI helps us defy the laws of physics (Karpathy) 😅

or more simply, the first run is slower due to compilation times (@nikitaved)

from lightning-thunder.

But on a serious note :) @kurmukovai , as @lantiga pointed out, the initial run could be slow because of the compilation/program analysis step. Note, however, that changes in inputs' meta-data (like device, shapes, but not the data itself) may trigger re-compilation.

Once meta-data is fixed and is not expected to change, the goal is to suffer a single initial warm-up perf hit for faster subsequent runs (for example, during inference).

If meta-data is expected to vary (say, like in the streaming setting), but this set is finite, one could cache compiled models to avoid non-stop re-compilations.

from lightning-thunder.

Thunder, feel the thunder
Lightning, then the thunder

(Imaging Dragons - Thunder)

from lightning-thunder.