joeng100 / gpt-tokenizer Goto Github PK
View Code? Open in Web Editor NEWThis project forked from niieani/gpt-tokenizer
JavaScript BPE Tokenizer Encoder Decoder for OpenAI's GPT-2 / GPT-3 / GPT-4. Port of OpenAI's tiktoken with additional features.
Home Page: https://gpt-tokenizer.dev
License: MIT License