Problem Loading Bert Weights about bert-toxic-comments-multilabel HOT 3 CLOSED

Self-reply to the first part: you need to compress both files together, like:

tar cvzf bert_model.tar.gz bert_config.json pytorch_model.bin

I still get the "weights not initialized" message though.

from bert-toxic-comments-multilabel.

Sel-reply to the second part: looking closely at the error it makes sense: it states that it is missing weights for the classifier layers at the end of the model. These layers have been added on top of the Bert encoder, so obviously we don't have pre-trained weights for them.

A bit more disconcerting is another warning complaining that some of the weights in the source have not been used. Hopefully these correspond to the final Bert layers in the original model (those that have been swapped by the new classifier layers: it makes sense that we lack weights for the new layers while we are not using our weights for the discarded layers).

For the record, this is the list of unused layers:
'cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias'

And this is the list of layers with no pre-trained weights:
'classifier.weight', 'classifier.bias'

from bert-toxic-comments-multilabel.

I met this issue too. The problem is might I download the Tensforflow version pre-train model, instead of the pytorch's version. Here is the source code.

Download the model from below links, which contains the .bin file:

PRETRAINED_MODEL_ARCHIVE_MAP = {
        'bert-base-uncased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased.tar.gz",
        'bert-large-uncased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased.tar.gz",
       'bert-base-cased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased.tar.gz",
        'bert-base-multilingual': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual.tar.gz",
        'bert-base-chinese': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-chinese.tar.gz",
    }

from bert-toxic-comments-multilabel.