I modify it and then predict the test.tsv and submit to kaggle sentiment competition. The score is quite low, much lower than BERT, so how much epoch will be better. For Bert, I just use 3 epoch, but increase epoch in this roberta seems not work.

I noticed the fine-tuned roberta script saved the fine-tuned model locally by

model_to_save = model
torch.save(model_to_save, output_model_file)
tokenizer.save_vocabulary(output_vocab_file)

How to load this model from the local folder?
I try this:

from transformers import RobertaModel, RobertaTokenizer
# from the local folder
roberta_model = RobertaModel.from_pretrained("./")
roberta_tokenizer = RobertaTokenizer.from_pretrained('./', truncation=True, do_lower_case=True)

But failed with error: OSError: does not appear to have a file named config.json. Checkout 'https://huggingface.co/.//main' for available files.

The current code is unable to support multi-GPU environment.

I parallelized the model using torch.nn.DataParallel(model).cuda()

I got this error when I try :

Thank you. In the Transformers multilabel distillbert, in the function "train(epoch)", you clear gradients after the model feed-forward operation and before loss backward calculation, and you uses "optimizer.zero_grad()" twice. Will this clear the gradients that the backward calculation needs? Shall it be like this where we only need to clear gradients once before model feed-forward operation:

    optimizer.zero_grad()
    outputs = model(ids, mask, token_type_ids)

    loss = loss_fn(outputs, targets)
    if _%5000==0:
        print(f'Epoch: {epoch}, Loss:  {loss.item()}')

    loss.backward()
    optimizer.step()

To make predictions, how to load the model which were saved during the training process?
After load model, we can inference testing data.
thank you :)

May I know the data location please!

The code I'm referring to is in cell 8 in https://github.com/DhavalTaunk08/NLP_scripts/blob/master/Transformers_multilabel_distilbert.ipynb
I tried to print before and after this line, and I see no difference.

print(text)
print(repr(text))
text = " ".join(text.split())
print(text)
print(repr(text))

Output:

proving there is no cure for stupidity.
'proving there is no cure for stupidity.'
proving there is no cure for stupidity.
'proving there is no cure for stupidity.'

Maybe it makes a difference in certain cases? Or can I just remove this line?