[Bug] Max tokens parameter is being incorrect set about chatgpt-next-web HOT 3 CLOSED

Really thx for the clarification!

from chatgpt-next-web.

A PR has been submitted to fix it: #4597

from chatgpt-next-web.

The max_tokens setting is often misunderstood, and its description in this project's settings is incorrect. To clarify, max_tokens does not refer to the LLM's context, which is the combination of input and output tokens. Instead, max_tokens limits the number of output tokens generated by the LLM.

To illustrate this, consider the gpt-4-turbo model, which has a context size of 128,000 tokens. However, its output is capped at 4,096 tokens. To obtain the maximum output, you should set max_tokens to 4,096. If you set it to 128,000, you'll encounter an error, as you won't leave sufficient tokens for the input.

Note that max_tokens is disabled by default in this project, but enabled for vision models with a setting of 4,000. If you're using a custom fork, be sure to enable max_tokens accordingly.

from chatgpt-next-web.