any-llm is a collection of Python tools to make LLM APIs plug-and-play for fast, easy experimentation. I use this in my own projects today, but it's still very much a work in progress. I'm trying to strike the balance between simplicity and modularity, so if you have ideas feel free to message me or submit a PR!
- Chat Completion: Mostly a wrapper around jxnl/instructor, supports a/sync chat completion and streaming for structured and unstructured chat completions.
- Embeddings/Rerankers: Easily use a slew of embedding and reranking models
- Asynchronous Workloads: Run all chat completion and embedding workloads in massively parallel fashion without worrying about rate limits. Nice for ETL pipelines.
- OpenAI Batch Jobs: Run large scale batch jobs with OpenAI's batch API.
All types of chat completions are made easy!
- Make a client
from any_llm import from_any, Provider
client = from_any(
provider=Provider.OPENAI,
model_name="gpt-4o",
async_client=False
)- Choose between a/sync un/structured response with/without streaming
# synchronous, unstructured response without streaming
# (aka a standard chat completion
response = client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "system", "content": "You are a friend."},
{"role": "user", "content": "Tell me about your day."}
],
response_model=None # No response model means unstructured response
)from any_llm import Provider, embed_from_any
# Create a Cohere embedding client
client = embed_from_any(Provider.COHERE)
# Example text to embed
text = "This is an example sentence to embed using Cohere."
# Get the embedding
embeddings, total_tokens = client.create(input=text, model="embed-english-v3.0", input_type="clustering")
print(f"Number of embeddings: {len(embeddings)}")
# > Number of embeddings: 1
print(f"Embedding dimension: {len(embeddings[0])}")
# > Embedding dimension: 1536
print(f"Usage: {total_tokens}")
# > Usage: 13
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent