Token count about mochidiffusion HOT 4 CLOSED

A helpful rule of thumb is that one token generally corresponds to ~4 characters of text for common English text. This translates to roughly ¾ of a word (so 100 tokens ~= 75 words).
Source: https://beta.openai.com/tokenizer

from mochidiffusion.

The gpt-3-encoder package provides a python implementation as well as a javascript one.

Both of which give me a more accurate token count than what Mochi provides me with currently. Unfortunately I have never wrote a single line of Swift so I have no idea how I would implement this but I thought I'd leave the info out there.

Token counts differ based on the model's vocab list. Also after I made the initial commit for the naive token counter, @CarterLombardi improved it by actually calculating the real token count (commit: 034bbe7)

from mochidiffusion.

So the token≈word isn't correct. I can see the text being truncated in Xcode output but parts of words are getting truncated if it goes over the "token" length. I've tried looking at webui's implementation of their token counter but the code is messy and isn't very straight forward. I'll be open to a PR but I can't figure out exactly what a token is.

from mochidiffusion.

From the same source,

If you need a programmatic interface for tokenizing text, check out the transformers package for python or the gpt-3-encoder package for node.js.

The gpt-3-encoder package provides a python implementation as well as a javascript one.

Both of which give me a more accurate token count than what Mochi provides me with currently. Unfortunately I have never wrote a single line of Swift so I have no idea how I would implement this but I thought I'd leave the info out there.

from mochidiffusion.