trapoom555 / language-model-sts-cft Goto Github PK

For ease of development

--logging true
# or
--logging false

• Make training process faster
• All training sample should be padded to the same max_length

With this approach, the batch size can linearly scale with the number of GPUs

Hi,

Niels here from the open-source team at Hugging Face. Congrats on your work! I discovered your work through the paper page: https://huggingface.co/papers/2408.00690 (feel free to claim the paper so that it appears at your HF account!).

It's great to link the 🤗 Collection to the paper, and add the paper itself to the collection.

See here on how to do that: https://huggingface.co/docs/hub/en/paper-pages#linking-a-paper-to-a-model-dataset-or-space.

Cheers,

Niels
ML Engineer @ HF 🤗

• STS Benchmark (validation set) can be used for evaluation during training
• This benchmark can imply how well the model create an embedding vector given a sentence
• Score will be calculated using Spearman Correlation

The result of training with batch_size_per_gpu = 2 is unstable.

• The key success of contrastive learning heavily depends on batch size
• batch_size_per_gpu = 2 has already consumed ~45% of GPU memory (RTX3090)
• Now using a normal DDP strategy + bf16 Mixed Precision + LoRA for training