Comments (4)
Are there resources about token counting that someone could read to start learning about this topic? What context can we provide so the difficulty is understood?
from mattermost-plugin-ai.
Hi @azigler, @crspeller I'm the maintainer of LiteLLM https://github.com/BerriAI/litellm we allow you to do cost tracking for 100+ LLMs
Usage
Docs: https://docs.litellm.ai/docs/#calculate-costs-usage-latency
from litellm import completion, completion_cost
import os
os.environ["OPENAI_API_KEY"] = "your-api-key"
response = completion(
model="gpt-3.5-turbo",
messages=[{ "content": "Hello, how are you?","role": "user"}]
)
cost = completion_cost(completion_response=response)
print("Cost for completion call with gpt-3.5-turbo: ", f"${float(cost):.10f}")
We also allow you to create a self hosted OpenAI Compatible proxy server to make your LLM calls (100+ LLMs), track costs, token usage
Docs: https://docs.litellm.ai/docs/simple_proxy
I hope this is helpful, if not I'd love your feedback on what we can improve
from mattermost-plugin-ai.
This isn't really about cost it's about token counting for fitting text into the LLM context window. We also can't rely on something like LiteLLM being in the middle for all deployments.
from mattermost-plugin-ai.
Hey, Mattermost AI community! 👋
We're looking for help on this one. 👀 In this issue, we want to improve the plugin's token counting capabilities and accuracy for common LLM providers. Understanding token counting is vital for communicating with LLMs programmatically. Check out this help topic from OpenAI to learn more.
For those interested in tackling this, here are some starting points:
- Review the current implementation for OpenAI's token counting using this reference and improve the existing method.
- Evaluate and account for the other token limits for other LLM providers like Anthropic and Ask Sage.
Dive in and let us know if you need any guidance along the way. 👍
Happy contributing! ✨
from mattermost-plugin-ai.
Related Issues (20)
- 📑 docs: Update README and issues ahead of v1.0 release
- Webapp bad export of AdvancedCreateComment
- 🐛 bug: Wrong file added to the 0.5.0 pre release HOT 1
- 💡 idea: Ollama support HOT 7
- Bug: Unable to generate plugin webapp bundle HOT 2
- Feature Idea: Support multiple AI models HOT 12
- Add apps bar icon for AI plugin HOT 1
- Add ability to go back to a thread from the plugin RHS HOT 4
- Add call summarization button to the calls recording post HOT 1
- Enhaced meeting summarization flow HOT 1
- chore: Add 'hacktoberfest' to repository topics
- Feature Idea: Allow image input HOT 3
- Feature Idea: Specific context by linking to chat messages HOT 5
- Bug: crash with mattermost-server 9.0.0 and LocalAI backend HOT 8
- Feature Idea: Image Generation HOT 2
- Enhancement: Provide image as input to the model - gpt-4-vision-preview HOT 2
- Bug: RPC call MessageHasBeenPosted to plugin failed. HOT 3
- Feature Request: Gracefully handle when conversation contexts become too long HOT 3
- Feature Idea: Add in a "copy to clipboard" function onto AI messages HOT 3
- Bug: Plugin crashes when API request faces network failure
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mattermost-plugin-ai.