Special Tokens Are Wrong about llama3-tokenizer-js HOT 4 CLOSED

For instance, counting the special token "<|end_of_text|>" is wrong.

Are you inserting the string <|end_of_text|> into normal input text and expecting it to be parsed as a special token? I have implemented the special tokens in a different manner. I wrote into the README an example of how to work with special tokens. Here is the example from README adapted into this particular token:

const tokens = []
tokens.push(llama3Tokenizer.getSpecialTokenId('<|end_of_text|>'))
console.log(tokens.length) // Count is one

I implemented it this way because I am not sure what to do with all these special tokens, and the official LLaMA 3 repo has similar Python implementation: https://github.com/meta-llama/llama3/blob/main/llama/tokenizer.py#L202-L229

Thinking more about this, maybe I should also parse the special tokens from normal input text?

It might be better to also visualize the token id in the website.

Thank you for the suggestion. I can see how there would be a need for this, though the website is just a quick demo for the library.

from llama3-tokenizer-js.

Got it! Makes much more sense now~!

from llama3-tokenizer-js.

I have now added the features you requested:

1. Visualize token ids on the demo website
2. Allow passing special tokens inside normal input text, no longer requiring any special function calls

from llama3-tokenizer-js.

@belladoreai Awesome! Thanks so much, I appreciate it!

from llama3-tokenizer-js.