Comments (3)
Picking this one up soon - thinking about ways to clean up:
- the number of configs (enabling multiple test suites)
- the number of test cases (enabling better importing/organization for test cases and vars)
from promptfoo.
Appreciate it!
Striving to achieve a cleaner config with a lot of tests has been the most challenging aspect so far. We used $ref
in our config extensively.
I definitely like the idea of being able to run multiple TestSuite
objects, one for each dataset. All those test suites could use the exact same configuration.
from promptfoo.
Closing this as complete! Scenarios are documented here: https://promptfoo.dev/docs/configuration/scenarios
from promptfoo.
Related Issues (20)
- Add support for Mistral.ai HOT 2
- Feature Request: Use Promptfoo to generate DPO datasets HOT 5
- Unable to set config for google ai studio gemini api HOT 2
- How to properly set perplexity model to run via a YAML file? HOT 15
- Do we have native support for cohere AI models? HOT 3
- Allow progress bar in debug/verbose mode HOT 4
- View server HOT 3
- Invalid option: a HOT 1
- python prompt display HOT 2
- Add bedrock:completion:anthropic.claude-v2.1 HOT 6
- Bedrock anthropic Claude v2 - `llm-rubric produced malformed response:`
- Response from Claude v2 (Bedrock) causes errors HOT 8
- A way to prevent vars array auto-expansion HOT 4
- Customizing the grader provider doesn't work HOT 2
- Make the Config description visible on the view page HOT 1
- error with `promptfoo view` HOT 2
- Context relevance test broken
- HuggingFace api key token configuration HOT 1
- Cannot set environment variables in self-hosted share server HOT 2
- Problem using is-valid-openai-tools-call assertion
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from promptfoo.