aub-mind / arabic-empathetic-chatbot Goto Github PK

can u please tell me the steps to run the code on colab ?

First, I would like to thank you for the dataset. It did help me a lot in my thesis, good work. Second, when I run the model on Colab it crashes with output Your session crashed after using all available RAM .

I would like to know if the model requires specific computational specs since it is not mentioned.

Best Regards.

Thanks for your great work!

I have difficulty in reproducing the bleu score on the test set for the BERT2BERT model, which should be 5.58 according to your paper:

what I have done is:

1. I downloaded your pretrained BERT2BERT model from huggingface and then did the prediction on the test set. I also tried to tune the top_k, top_p and other args during generation, according to your instruction. The highest bleu score I could get on the test set is 0.91:

2. I also tried to train the model from scratch, using the train-bert2bert.ipynb file, however, the highest bleu score I could get is less than 2.

As you can see, it is far from 5.58. So could you please share with us the prediction code, as well as the values of all args, hyper-parameters (e.g. top_k, top_p, temperature, length_penalty...) that could reproduce your test result?

Thanks a lot!

Hi,

Thanks for sharing the code and models for your great work!!!

I tried to reproduce the BLUE scores using your code but could not get the 5% BLUE score you reported in the paper.
Based on your code and experiments we did internally, it seems that you calculate the bleu score on the segmented text (with farasa segment). This roughly gives a BLUE score which is x2.5 times larger than what it should be. We found that in the best case (after extensive hyperparameters tuning (>40 experiments) the best BLUE score is around 1.6-1.9.

The bug is in the compute_metrics method:

def compute_metrics(pred):
  labels_ids = pred.label_ids
  #pred_ids = torch.argmax(pred.predictions,dim=2)
  pred_ids = pred.predictions  

  # all unnecessary tokens are removed
  pred_str = tokenizer.batch_decode(pred_ids, skip_special_tokens=True)
  labels_ids[labels_ids == -100] = tokenizer.pad_token_id
  label_str = tokenizer.batch_decode(labels_ids, skip_special_tokens=True)

#########################################################
### You should add these lines to calculate the BLUE score correctly  ######
#########################################################
  response = response.replace("\'", '')
  response = response.replace("[[CLS]", '')
  response = response.replace("[SEP]]", '')
  response = str(arabert_prep.desegment(response)) # this is the most important one
########################################


  return {"bleu": round(corpus_bleu(pred_str , [label_str]).score, 4)}

Here is an example of how BLUE scores vary between the same segmented vs. desegmented text:

t1 = "أنا متأكد من أنها ستكون بخير."
ppt1 = arabert_prep.preprocess(t1)
print(ppt1) # أنا متأكد من أن +ها س+ تكون ب+ خير .

t2 = "أنا متأكد ستكون بخير."
ppt2 = arabert_prep.preprocess(t2)
print(ppt2) # أنا متأكد س+ تكون ب+ خير .


print("Wrong BLUE on segmenented text", sacrebleu.sentence_bleu(ppt2, [ppt1]).score) # 51.51425457345961
print("Correct BLUE on unsegmenented text", sacrebleu.sentence_bleu(t2, [t1]).score) # 33.51600230178196

Thanks,