vamseekm / rust-tokenizers Goto Github PK
View Code? Open in Web Editor NEWThis project forked from guillaume-be/rust-tokenizers
Rust-tokenizer offers high-performance tokenizers for modern language models, including WordPiece, Byte-Pair Encoding (BPE) and Unigram (SentencePiece) models
License: Apache License 2.0