Do you know of any packages or frameworks similar to promptfoo? about promptfoo HOT 8 CLOSED

I also would like to know.

from promptfoo.

Unfortunately not really. I built this because there wasn't anything else out there that did what I needed it to do. OpenAI does have an Evals framework you can take a look at. Its focus is on testing OpenAI models with heavier test cases, and some of the more advanced test cases require Python implementation.

from promptfoo.

The main thing I need is this but in python with langchain compatibility. It might be worth cloning and converting.

from promptfoo.

As far as I know, QAEvalChain in langchain module might be useful to me. I'm still looking to see if there are other alternatives.

from promptfoo.

@typpo Thanks for the link reference.

from promptfoo.

For those of you working in Python, have a look at the end-to-end LLM chain testing documentation.

Specifically, I've created an example that shows how to evaluate a Python LangChain implementation.

The example compares raw GPT-4 with LangChain's LLM-Math plugin by using the exec provider to run the LangChain script:

# promptfooconfig.yaml
# ...
providers:
  - openai:chat:gpt-4-0613
  - exec:python langchain_example.py
# ...

The result is a side-by-side comparison of GPT-4 and LangChain doing math:

Hope this helps your use cases. If not, interested in learning more.

Side note - QAEvalChain is similar in approach to the llm-rubric assertion type of promptfoo. It can help evaluate whether a specific answer makes sense for a specific question.

from promptfoo.

It looks like it was released recently.
hegelai/prompttools: Open-source tools for prompt testing and experimentation

from promptfoo.

@typpo : First of all congratulations on the great work in building this library. It would be great if we can have some way to directly compare and contrast promptfoo with prompttools and evals by OpenAi. This will make life easier for consumers to pick & choose best among these based on the usecase.

from promptfoo.