Infill a sentence as a continuation of a conditioning token? about ilm HOT 4 OPEN

Sorry for the delayed response. I don't think this would be a huge job. All you would have to do is create a new mask function which does exactly this masking task of preserving the first word of a sentence but masking out the rest.

For example, for the sentence:

Therefore, the idea was straightforward.

You would want to produce a training example of the form:

Therefore, [blank] [sep] the idea was straightforward. [answer]

Does this help?

from ilm.

Ah, right—a custom mask function... makes sense. I can't dig into this for another week, or so, but will give it a go when I'm back on this task. Thanks!

This is a very cool tweak on applied Transformers, btw! 👍

from ilm.

Thank you!

Also, I just thought about this again and realized that you absolutely don't need a custom mask function for this. You just need to train a standard ILM to perform sentence infilling, and then condition the generation on the initial word.

E.g., pass the model [blank_sentence] [sep] Therefore, as input and have it generate until it produces an [answer] token.

from ilm.

Ah, okay! I'll give this a go first.

I can't remember what I tried, but I think I was just trying things in the Jupyter notebook and wasn't sure how infill_word or infill_sentence would do this. Sounds like it would just be a matter of writing a new "infill" function(?).

from ilm.