finetune.py segmentation fault about slowllama HOT 6 CLOSED

Thanks, I did and it works now! This is super. Great work! Some points I noted:

1. Not all snapshots were written to disk. Some were skipped. I will re run and check.
2. GPU usage during finetune was optimum. However during inferencing, it is using fairly low, <10%. Is this expected?
   Will try more things in coming days. Thanks again.

from slowllama.

1. Which version of pytorch do you use? I saw some issues with pytorch 2.0.1 (pytorch/pytorch#110975)
2. did you update the slowllama repo recently?

Thank you!

from slowllama.

Thanks. In that case I should upgrade?

1. Yes.

% pip freeze | egrep -i 'torch|numpy|sentence|fewlines'
fewlines==0.0.9
numpy==1.25.2
sentence-transformers==2.2.2
sentencepiece==0.1.99
torch==2.0.1
torchvision==0.15.2

2. Just couple of hours ago.

from slowllama.

Yes, try 2.1.0 please. Also please make sure you ran prepare_model and finetune at the same version of slowllama - as it is pretty early/experimental there's no backwards compatibility.

from slowllama.

1. The logic to save is to only save if loss is lower. We can change that (https://github.com/okuvshynov/slowllama/blob/main/finetune.py#L53-L58).
2. Yes, I put close to no effort to inference optimization. I think there are other libraries focusing on that specifically (e.g. llama.cpp)

from slowllama.

Thanks for the clarifications, I will dive into the code too. Though I am not an expert at this but I will give it my best shot. llama.cpp works great for me, however I am unable to get the finetune to work on it on my mac.

from slowllama.