Implementation of ConGen: Unsupervised Control and Generalization Distillation For Sentence Representation (Finding of EMNLP 2022).

@inproceedings{limkonchotiwat-etal-2022-congen,
    title = "{ConGen}: Unsupervised Control and Generalization Distillation For Sentence Representation",
    author = "Limkonchotiwat, Peerat  and
      Ponwitayarat, Wuttikorn  and
      Lowphansirikul, Lalita and
      Udomcharoenchaikit, Can  and
      Chuangsuwanich, Ekapol  and
      Nutanong, Sarana",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2022",
    year = "2022",
    publisher = "Association for Computational Linguistics",
}

• We have a new version of ConGen: SCT (published at TACL2023).
• The SCT method outperforms ConGen on distillation settings.
• This method is also effective for a small model to learn sentence embedding without the teacher model!

git clone https://github.com/KornWtp/ConGen.git
cd ConGen
pip install -e .

• ConGen-BERT-Tiny
• ConGen-BERT-Mini
• ConGen-TinyBERT-L4
• ConGen-MiniLM-L3
• ConGen-MiniLM-L6
• ConGen-BERT-Small
• ConGen-MiniLM-L12
• ConGen-TinyBERT-L6
• ConGen-BERT-base
• ConGen-RoBERTa-base
• ConGen-Multilingual-DistilBERT
• ConGen-Multilingual-MiniLM-L12

We use the training data from BSL's paper: monolingual version and multilingual version.

We use sts-b development set from sentence transformer.

The full model parameters:

Please set the model's parameter before training.

>> bash train_congen.sh

For finetuning model parameters:

learning_rate_all=(3e-4 5e-4 1e-4 3e-5 5e-5 1e-5)
queue_sizes=(262144 131072 65536 16384 1024)
teacher_temps=(0.01 0.03 0.05 0.07 0.09 0.1)
student_temps=(0.01 0.03 0.05 0.07 0.09 0.1)

Our evaluation code for sentence embeddings is based on a modified version of SentEval and SimCSE.

Before evaluation, please download the evaluation datasets by running

cd SentEval/data/downstream/
bash download_dataset.sh

Please see https://github.com/KornWtp/ConGen/tree/main/notebook

Then come back to the root directory, you can evaluate any sentence transformers models using SimCSE evaluation code. For example,

python evaluation.py \
    --model_name_or_path "your-model-path" \
    --task_set sts \
    --mode test

In our paper, we average score over three models and shown as follows:

• Unsupervised learning: ConGen-simcse-model-roberta-base-thai. Teacher model: simcse-model-roberta-base-thai. Student model: WangchanBERTa
• Weakly supervised learning: ConGen-paraphrase-multilingual-mpnet-base-v2. Teacher model: paraphrase-multilingual-mpnet-base-v2. Student model: WangchanBERTa
• Training data: we do backtranslatation from TH-to-EN-to-TH from scb_mt_enth_2020's model. The translation dataset: back translated machine translation of SCB
• We evaluate on two task benchmark tasks, such as Thai semantic textual similarity benchmark and Thai transfer benchmark