Comments (2)
evaline-ju
commented on July 25, 2024
Currently detectors that expect a threshold but not provided one through the orchestrator API will result in a 500 (this part is indicative of a 🐛 )
There are likely two parts that have to happen for this:
- Any detector servers will have to give an expected
4xxerror code if there are fixable missing parameters, which will allow the orchestrator to propagate the error (as necessary) - The orchestrator will do filtering of detector results based on user-provided thresholds - a default threshold for each detector will be provided through the config for now.
from fms-guardrails-orchestrator.
evaline-ju
commented on July 25, 2024
Closing with #52 , the orchestrator will do detector score thresholding now but a case like “model_name”: {} without “threshold” will continue to break if any particular detector implementations error when no parameters are passed, even if they are not needed or used in final results returned. Calling this out in case of confusion, but I don’t think this is particularly worth fixing if we will be updating the detector API soon with #37
from fms-guardrails-orchestrator.
Related Issues (20)
- v2.0 not returning tokens HOT 4
- Add unit test to verify parameter massaging for the detectors HOT 1
- NLP client not working on tokenize at least
- Add tests for orchestrator response with text generation edge cases HOT 1
- Failed to deserialize the JSON body into the target type: missing field `models` HOT 2
- Detected PII word's "start" and "end" are returning the wrong positions HOT 5
- Add a check to verify if the len of input and output match for detectors
- Add ADR for orchestrator API
- Change `get_test_context()` to have a default GenerationClient HOT 1
- Add error handling for streaming
- Response's classification results ordering by span behaving inconsistently HOT 1
- Orchestrator image does not work
- Integration tests for `/api/v1/task/classification-with-text-generation` endpoint HOT 1
- Pass along include_stop_sequence to all client for generation request
- Implementation of `/api/v1/text/task/generation-detection` HOT 2
- Implementation of `/api/v1/text/task/detection/context-docs`
- Implementation of `/api/v1/text/task/detection/content`
- Last event appears missing on streaming w.r.t. finish reason
- Expected seed not returned on streaming events
- Empty generated text can return 500s
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fms-guardrails-orchestrator.