Code Monkey home page Code Monkey logo

agentops's Introduction

Observability and DevTool platform for AI Agents

๐Ÿฆ Twitter ย ย โ€ขย ย  ๐Ÿ“ข Discord ย ย โ€ขย ย  ๐Ÿ–‡๏ธ Dashboard ย ย โ€ขย ย  ๐Ÿ“™ Documentation

Dashboard Banner

AgentOps helps developers build, evaluate, and monitor AI agents. From prototype to production.

๐Ÿ“Š Replay Analytics and Debugging Step-by-step agent execution graphs
๐Ÿ’ธ LLM Cost Management Track spend with LLM foundation model providers
๐Ÿงช Agent Benchmarking Test your agents against 1,000+ evals
๐Ÿ” Compliance and Security Detect common prompt injection and data exfiltration exploits
๐Ÿค Framework Integrations Native Integrations with CrewAI, AutoGen, & LangChain

Quick Start โŒจ๏ธ

pip install agentops

Session replays in 2 lines of code

Initialize the AgentOps client and automatically get analytics on all your LLM calls.

Get an API key

import agentops

# Beginning of your program (i.e. main.py, __init__.py)
agentops.init( < INSERT YOUR API KEY HERE >)

...

# End of program
agentops.end_session('Success')

All your sessions can be viewed on the AgentOps dashboard

Agent Debugging Agent Debugging
Session Replays Session Replays
Summary Analytics Summary Analytics

First class Developer Experience

Add powerful observability to your agents, tools, and functions with as little code as possible: one line at a time.
Refer to our documentation

# Automatically associate all Events with the agent that originated them
from agentops import track_agent

@track_agent(name='SomeCustomName')
class MyAgent:
  ...
# Automatically create ToolEvents for tools that agents will use
from agentops import record_tool

@record_tool('SampleToolName')
def sample_tool(...):
  ...
# Automatically create ActionEvents for other functions.
from agentops import record_action

@agentops.record_action('sample function being record')
def sample_function(...):
  ...
# Manually record any other Events
from agentops import record, ActionEvent

record(ActionEvent("received_user_input"))

Integrations ๐Ÿฆพ

CrewAI ๐Ÿ›ถ

Build Crew agents with observability with only 2 lines of code. Simply set an AGENTOPS_API_KEY in your environment, and your crews will get automatic monitoring on the AgentOps dashboard.

pip install 'crewai[agentops]'

AutoGen ๐Ÿค–

With only two lines of code, add full observability and monitoring to Autogen agents. Set an AGENTOPS_API_KEY in your environment and call agentops.init()

Langchain ๐Ÿฆœ๐Ÿ”—

AgentOps works seamlessly with applications built using Langchain. To use the handler, install Langchain as an optional dependency:

Installation
pip install agentops[langchain]

To use the handler, import and set

import os
from langchain.chat_models import ChatOpenAI
from langchain.agents import initialize_agent, AgentType
from agentops.partners.langchain_callback_handler import LangchainCallbackHandler

AGENTOPS_API_KEY = os.environ['AGENTOPS_API_KEY']
handler = LangchainCallbackHandler(api_key=AGENTOPS_API_KEY, tags=['Langchain Example'])

llm = ChatOpenAI(openai_api_key=OPENAI_API_KEY,
                 callbacks=[handler],
                 model='gpt-3.5-turbo')

agent = initialize_agent(tools,
                         llm,
                         agent=AgentType.CHAT_ZERO_SHOT_REACT_DESCRIPTION,
                         verbose=True,
                         callbacks=[handler], # You must pass in a callback handler to record your agent
                         handle_parsing_errors=True)

Check out the Langchain Examples Notebook for more details including Async handlers.

Cohere โŒจ๏ธ

First class support for Cohere(>=5.4.0). This is a living integration, should you need any added functionality please message us on Discord!

Installation
pip install cohere
import cohere
import agentops

# Beginning of program's code (i.e. main.py, __init__.py)
agentops.init(<INSERT YOUR API KEY HERE>)
co = cohere.Client()

chat = co.chat(
    message="Is it pronounced ceaux-hear or co-hehray?"
)

print(chat)

agentops.end_session('Success')
import cohere
import agentops

# Beginning of program's code (i.e. main.py, __init__.py)
agentops.init(<INSERT YOUR API KEY HERE>)

co = cohere.Client()

stream = co.chat_stream(
    message="Write me a haiku about the synergies between Cohere and AgentOps"
)

for event in stream:
    if event.event_type == "text-generation":
        print(event.text, end='')

agentops.end_session('Success')

Anthropic ๏นจ

Track agents built with the Anthropic Python SDK (>=0.32.0).

Installation
pip install anthropic
import anthropic
import agentops

# Beginning of program's code (i.e. main.py, __init__.py)
agentops.init(<INSERT YOUR API KEY HERE>)

client = anthropic.Anthropic(
    # This is the default and can be omitted
    api_key=os.environ.get("ANTHROPIC_API_KEY"),
)

message = client.messages.create(
        max_tokens=1024,
        messages=[
            {
                "role": "user",
                "content": "Tell me a cool fact about AgentOps",
            }
        ],
        model="claude-3-opus-20240229",
    )
print(message.content)

agentops.end_session('Success')

Streaming

import anthropic
import agentops

# Beginning of program's code (i.e. main.py, __init__.py)
agentops.init(<INSERT YOUR API KEY HERE>)

client = anthropic.Anthropic(
    # This is the default and can be omitted
    api_key=os.environ.get("ANTHROPIC_API_KEY"),
)

stream = client.messages.create(
    max_tokens=1024,
    model="claude-3-opus-20240229",
    messages=[
        {
            "role": "user",
            "content": "Tell me something cool about streaming agents",
        }
    ],
    stream=True,
)

response = ""
for event in stream:
    if event.type == "content_block_delta":
        response += event.delta.text
    elif event.type == "message_stop":
        print("\n")
        print(response)
        print("\n")

Async

import asyncio
from anthropic import AsyncAnthropic

client = AsyncAnthropic(
    # This is the default and can be omitted
    api_key=os.environ.get("ANTHROPIC_API_KEY"),
)


async def main() -> None:
    message = await client.messages.create(
        max_tokens=1024,
        messages=[
            {
                "role": "user",
                "content": "Tell me something interesting about async agents",
            }
        ],
        model="claude-3-opus-20240229",
    )
    print(message.content)


await main()

LiteLLM ๐Ÿš…

AgentOps provides support for LiteLLM(>=1.3.1), allowing you to call 100+ LLMs using the same Input/Output Format.

Installation
pip install litellm
# Do not use LiteLLM like this
# from litellm import completion
# ...
# response = completion(model="claude-3", messages=messages)

# Use LiteLLM like this
import litellm
...
response = litellm.completion(model="claude-3", messages=messages)
# or
response = await litellm.acompletion(model="claude-3", messages=messages)

LlamaIndex ๐Ÿฆ™

AgentOps works seamlessly with applications built using LlamaIndex, a framework for building context-augmented generative AI applications with LLMs.

Installation
pip install llama-index-instrumentation-agentops

To use the handler, import and set

from llama_index.core import set_global_handler

# NOTE: Feel free to set your AgentOps environment variables (e.g., 'AGENTOPS_API_KEY')
# as outlined in the AgentOps documentation, or pass the equivalent keyword arguments
# anticipated by AgentOps' AOClient as **eval_params in set_global_handler.

set_global_handler("agentops")

Check out the LlamaIndex docs for more details.

Time travel debugging ๐Ÿ”ฎ

Time Travel Banner

Try it out!

Agent Arena ๐ŸฅŠ

(coming soon!)

Evaluations Roadmap ๐Ÿงญ

Platform Dashboard Evals
โœ… Python SDK โœ… Multi-session and Cross-session metrics โœ… Custom eval metrics
๐Ÿšง Evaluation builder API โœ… Custom event tag trackingย  ๐Ÿ”œ Agent scorecards
โœ… Javascript/Typescript SDK โœ… Session replays ๐Ÿ”œ Evaluation playground + leaderboard

Debugging Roadmap ๐Ÿงญ

Performance testing Environments LLM Testing Reasoning and execution testing
โœ… Event latency analysis ๐Ÿ”œ Non-stationary environment testing ๐Ÿ”œ LLM non-deterministic function detection ๐Ÿšง Infinite loops and recursive thought detection
โœ… Agent workflow execution pricing ๐Ÿ”œ Multi-modal environments ๐Ÿšง Token limit overflow flags ๐Ÿ”œ Faulty reasoning detection
๐Ÿšง Success validators (external) ๐Ÿ”œ Execution containers ๐Ÿ”œ Context limit overflow flags ๐Ÿ”œ Generative code validators
๐Ÿ”œ Agent controllers/skill tests โœ… Honeypot and prompt injection detection (PromptArmor) ๐Ÿ”œ API bill tracking ๐Ÿ”œ Error breakpoint analysis
๐Ÿ”œ Information context constraint testing ๐Ÿ”œ Anti-agent roadblocks (i.e. Captchas) ๐Ÿ”œ CI/CD integration checks
๐Ÿ”œ Regression testing ๐Ÿ”œ Multi-agent framework visualization

Why AgentOps? ๐Ÿค”

Without the right tools, AI agents are slow, expensive, and unreliable. Our mission is to bring your agent from prototype to production. Here's why AgentOps stands out:

  • Comprehensive Observability: Track your AI agents' performance, user interactions, and API usage.
  • Real-Time Monitoring: Get instant insights with session replays, metrics, and live monitoring tools.
  • Cost Control: Monitor and manage your spend on LLM and API calls.
  • Failure Detection: Quickly identify and respond to agent failures and multi-agent interaction issues.
  • Tool Usage Statistics: Understand how your agents utilize external tools with detailed analytics.
  • Session-Wide Metrics: Gain a holistic view of your agents' sessions with comprehensive statistics.

AgentOps is designed to make agent observability, testing, and monitoring easy.

Star History

Check out our growth in the community:

Logo

Popular projects using AgentOps

Repository Stars
ย  geekan / MetaGPT 42787
ย  run-llama / llama_index 34446
ย  crewAIInc / crewAI 18287
ย  camel-ai / camel 5166
ย  superagent-ai / superagent 5050
ย  iyaja / llama-fs 4713
ย  BasedHardware / Omi 2723
ย  MervinPraison / PraisonAI 2007
ย  AgentOps-AI / Jaiqu 272
ย  strnad / CrewAI-Studio 134
ย  alejandro-ao / exa-crewai 55
ย  tonykipkemboi / youtube_yapper_trapper 47
ย  sethcoast / cover-letter-builder 27
ย  bhancockio / chatgpt4o-analysis 19
ย  breakstring / Agentic_Story_Book_Workflow 14
ย  MULTI-ON / multion-python 13

Generated using github-dependents-info, by Nicolas Vuillamy

agentops's People

Contributors

areibman avatar bboynton97 avatar howieg avatar siyangqiu avatar albertkimjunior avatar atomsilverman avatar emdoyle avatar suryavanshi avatar sprajosh avatar abhinavmir avatar avdiuandin avatar eltociear avatar mbarnathan avatar the-praxs avatar andkoma avatar stateofkate avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.