Comments (3)
Thanks for catching this. #361 will disable cache if repeat > 1
from promptfoo.
I understand that #361 will make impossible to use cache with repeat. It would still be nice to cache the outputs of each repeat, although not sure how hard would be to implement that.
from promptfoo.
Fix is in 0.34.0!
I'm going to close this for now, as I think it would be tricky to implement repeat
and cache
simultaneously (and with marginal benefit, because the main use of repeat
is to capture the randomness in outputs caused by temperature)
from promptfoo.
Related Issues (20)
- How to generate separate report or a breakdown report per test suite instead of just one single report for all test suites? HOT 2
- Feature Request: Add max_tokens config to Azure OpenAI provider HOT 3
- Self-Hosting via provided Dockerfile does not work HOT 3
- Mistral provider is hard-coded to https://api.mistral.ai/v1/chat/completions HOT 1
- Use promptfoo for output evaluation/visualisation but not inferencing? HOT 6
- OpenRouter integration errors HOT 1
- Feature Request: CLI command to retrieve results after manual labelling in csv file HOT 2
- Integration of Google VertexAI gcloud auth HOT 4
- Save evals directly in self-hosted machine HOT 2
- Nunjucks Custom Filters not working HOT 1
- Save evals in self hosted machine when running it from another machine HOT 2
- VertexAI 0.49 release causing issues for VertexAI provider HOT 2
- Option for `contains: False` in assertions HOT 3
- Throw an error if config file not found HOT 1
- Python assertions not working with external.py HOT 7
- Deploying Promptfoo to Railway HOT 14
- llm-rubric with VertexAI Gemini Pro HOT 3
- Add support for threaded tests HOT 2
- Add support for picking up variables from .env file HOT 4
- Run External File from `tests.csv` HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from promptfoo.