Does GLM-130 B support [sMASK] for sentence generation? about glm-130b HOT 4 CLOSED

@lomizandtyd Hi,

Thanks for your interest in GLM-130B!

Currently I'm using "[MASK]" at the end of dialogue for bot's response generation.

That's cool! Looking forward to your application online, and please contact as we can also help to promote it for you.

Your GLM repo https://github.com/THUDM/GLM reports [sMASK] could be used for sentence generation.
But I didn't find any doc in this repo. Does GLM-130 B support [sMASK] for sentence generation?

Unfortunately, we didn't implement [sMASK] in GLM-130B since it harms the efficiency of pre-training. However, we are planning to train GLM-130B for a few steps on [sMASK] objectives to enable this feature. Please wait for a few days as we are working on it : )

Do you have any plan to export more API of GLM-130 B ? Such as compute LM perplexity / Multiple choice selection or any other features? Since you have already test the model on Few-CLUE, there must be ways to utilize those features.

That sounds great. But it may not be our first priority since we are now having a hard time sparing any machines for such a goal. Thanks for your suggestions.

from glm-130b.

Could you provide some guidelines about how to compute PPL/logprobs of input sentence on GLM-130 B?

from glm-130b.

Hello @lomizandtyd, we have recently implemented evaluation code for language modeling tasks such as WikiText103 and PTB, you can check out the eval-lm branch and the code will be merged into the main branch after further testing. Here are our preliminary results for your reference:

If you have any other questions, please feel free to ask. Thank you again for your interest in GLM-130B!

from glm-130b.

Thanks, that's what I want.

from glm-130b.