Hi, I am trying to run basic masked word prediction in pytorch transformers to compare

Masked worked prediction returning "unused" tokens,about digitalepidemiologylab/covid-twitter-bert

Comments (3)

mar-muel commented on June 17, 2024 1

Hi again!

There was indeed a problem when preparing the model for Huggingface. I have now converted our model into a native PyTorch model, you can now use it like any other model:

from transformers import pipeline
import json

pipe = pipeline(task='fill-mask', model='digitalepidemiologylab/covid-twitter-bert')
out = pipe(f"I think I might have {pipe.tokenizer.mask_token}")
print(json.dumps(out, indent=4))

Which gives me:

[
    {
        "sequence": "[CLS] i think i might have it [SEP]",
        "score": 0.48932939767837524,
        "token": 2009,
        "token_str": "it"
    },
    {
        "sequence": "[CLS] i think i might have. [SEP]",
        "score": 0.14833217859268188,
        "token": 1012,
        "token_str": "."
    },
    {
        "sequence": "[CLS] i think i might have corona [SEP]",
        "score": 0.12335461378097534,
        "token": 21887,
        "token_str": "corona"
    },
    {
        "sequence": "[CLS] i think i might have died [SEP]",
        "score": 0.029644010588526726,
        "token": 2351,
        "token_str": "died"
    },
    {
        "sequence": "[CLS] i think i might have to [SEP]",
        "score": 0.02588510327041149,
        "token": 2000,
        "token_str": "to"
    }
]

Let me know if this works for you! And thanks again for letting us know about this!

from covid-twitter-bert.

mar-muel commented on June 17, 2024

Hi ogencoglu! Thanks for bringing this to our attention! I'll look into this within the following days. For now, the way to go is to use the TF models (e.g. through TFHub).

from covid-twitter-bert.