sinanakkoyun / lit-llama-fp8 Goto Github PK

Did you use the padding in the commented code when doing FP8 precision text generation inference? Would it cause errors if used?

Your Comments
"""
# SINAN: FP8 token multiple of 8:
# Ensure the encoded sequence's length is a multiple of 8
remainder = len(encoded) % 8
if remainder != 0:
padding_needed = 8 - remainder
padding_tensor = torch.full((padding_needed,), tokenizer.pad_id, dtype=torch.long, device=fabric.device)
encoded = torch.cat((encoded, padding_tensor))
"""