Comments (6)
A fix can be to rework the 'post_process' function in run.py:
def post_process(self, output: str) -> str:
output = output.strip()
# if output.startswith("```"):
# output = output[3:]
# if output.startswith("json"):
# output = output[4:]
# if output.endswith("```"):
# output = output[:-3]
output = output.replace(" False,","'False'").replace(" True,","'True'").replace(" None,","'None'")
output = output.replace("json","JSON").split("JSON")[-1]
output = "{" + output.split("{",1)[-1].rsplit("}",1)[0] + "}"
return output
from guardrails.
Changing the high_level_reask_prompt to explicitly mention that it can only return the corrected JSON also improves the behaviour
<high_level_reask_prompt>
I was given the following JSON response, which had problems due to incorrect values.
{previous_response}
Help me correct the incorrect values based on the given error messages. Only return the corrected JSON, don't repeat the given JSON
</high_level_reask_prompt>
(if you add this, you do not need the following line in the post_process function anymore:
output = output.replace("json","JSON").split("JSON")[-1]
)
from guardrails.
Hey, thanks for the improvement suggestions.
First off, I'm unable to replicate your example. engine="chatgpt"
is invalid. Running it with model="gpt-3.5-turbo"
worked for me. Could you double-check how you ran it?
Let me run through your suggestions one by one.
output = output.replace(" False,","'False'").replace(" True,","'True'").replace(" None,","'None'")
This seems to change the booleans and Nones to string equivalents. Their equivalents should be false
, true
and null
(JSON uses javascript equivalents). This would be a reasonable change, if we can find a way to write this without it affecting the values within strings. Maybe there's a good regex solution?
output = output.replace("json","JSON").split("JSON")[-1]
If I'm reading this correctly, it assumes that the output will be annotated by a triple-quote json
. If it's not, it'll parse the whole thing as it would otherwise. This sounds like a good change, except I'm worried it might hit a "json" substring in the json itself.
output = "{" + output.split("{",1)[-1].rsplit("}",1)[0] + "}"
I like this one a lot, and it would meaningfully replace the lines you commented out.
<high_level_reask_prompt>
I was given the following JSON response, which had problems due to incorrect values.
{previous_response}
-Help me correct the incorrect values based on the given error messages.
+Help me correct the incorrect values based on the given error messages. Only return the corrected JSON, don't repeat the given JSON.
</high_level_reask_prompt>
This change seems reasonable to me, though I've never personally encountered it repeating the given JSON.
The cases you're proposing to handle are weird, I haven't really seen openai's models make these errors. Could you please:
- provide what model you used (the runtime arguments you provided are unreplicable),
- submit a sample
doctors_notes
, - show some examples of the failed generations?
from guardrails.
Okay, let me provide some more context:
-
I renamed my "gpt-3.5-turbo" engine to "chatgpt" in the Azure openAI service. So it is the same as the one you used (Sorry I should have clarified this).
-
the doctor notes that I used are the same ones as in the getting-started guide:
doctors_notes = """49 y/o Male with chronic macular rash to face & hair, worse in beard, eyebrows & nares. Itchy, flaky, slightly scaly. Moderate response to OTC steroid cream"""
-
An example where guardrails errored because the output from OpenAI was unexpected:
output openAI:
{
"patient_info": {
"symptoms": [
{
"affected area": {
"incorrect_value": "face & hair",
"error_message": "Value face & hair is not in choices ['head', 'neck', 'chest']."
}
},
{
"affected area": {
"incorrect_value": null,
"error_message": "Value None is not in choices ['head', 'neck', 'chest']."
}
},
{
"affected area": {
"incorrect_value": null,
"error_message": "Value None is not in choices ['head', 'neck', 'chest']."
}
},
{
"affected area": {
"incorrect_value": null,
"error_message": "Value None is not in choices ['head', 'neck', 'chest']."
}
}
]
}
}
The corrected JSON response is:
{
"patient_info": {
"symptoms": [
{
"affected area": "face & hair"
},
{
"affected area": null
},
{
"affected area": null
},
{
"affected area": null
}
]
}
}
Which cannot be processed by the JSON decoder and therefore the output_as_dict=None
. This leads to errors in the reask functions.
from guardrails.
Another example:
output openAI:
{
"patient_info": {
"symptoms": [
{
"affected area": {
"incorrect_value": "face & hair",
"error_message": "Value face & hair is not in choices ['head', 'neck', 'chest']."
}
},
{
"affected area": {
"incorrect_value": None,
"error_message": "Value None is not in choices ['head', 'neck', 'chest']."
}
},
{
"affected area": {
"incorrect_value": None,
"error_message": "Value None is not in choices ['head', 'neck', 'chest']."
}
},
{
"affected area": {
"incorrect_value": None,
"error_message": "Value None is not in choices ['head', 'neck', 'chest']."
}
}
]
}
}
becomes
{
"patient_info": {
"symptoms": [
{
"affected area": "head"
},
{
"affected area": null
},
{
"affected area": null
},
{
"affected area": null
}
]
}
}
from guardrails.
@sophieDataroots thanks for sharing this.
If I understand correctly, in the examples you shared, the The corrected JSON response is:
statement is part of the LLM response, in addition to the original JSON dictionary. And this causes the JSON parsing error.
I believe the reason this is happening is that:
- Since you're testing out the example with a chat model, the way to get correctly structured JSON from the chat model is using a separate system instructions in addition to just the prompt.
- Currently, all of the reasking logic assumes that the end user only a non-chat model, and so the reasking logic only creates a prompt, without creating system instructions to go with the prompt. This causes reasking output to be formatted incorrectly.
This is a pretty interesting bug, and should not be a super hard fix on our end.
from guardrails.
Related Issues (20)
- [feat] Support punkt_tab instead of punkt HOT 1
- [bug] Guards that dont exist in settings.use_remote = True dont error and openAI api error for guard not found not helpful HOT 2
- [bug] ToxicLanguage validator's "fix" behaviour breaks "fix-reask"
- [bug] GuardRunnable may not be reporting tokens appropriately if reasks are allowed HOT 1
- [bug] Verify Filter, Reask, and Refrain ser/de HOT 1
- [feat] Async validate when using multiple validators HOT 1
- [bug] ToxicLanguage seems not to work in french HOT 2
- [documentation] update RAIL docs for hub validators
- [bug] Error after installing SensitiveTopics, error: ModuleNotFoundError: No module named 'guardrails.hub.tryolabs' HOT 3
- [bug] Error: zsh: bus error guardrails start --config config.py HOT 1
- Update hub validators to support remote inference
- [bug] email address [email protected] not receiving emails: "Delivery Status Notification (Failure)" HOT 1
- [docs] request: Add doc examples of how to call the new Guardrail Server endpoints. HOT 1
- [bug] Trouble calling guard without using openai or azure openai APIs HOT 7
- [docs][Bug] HOT 1
- How to use gemini model insted of openai HOT 2
- [feat] Functionality for testing a Guard configured to run on prompt in isolation HOT 1
- [feat] More readable validation failures HOT 2
- [docs] Missing docs for in-code hub installs
- [bug] Toxic language validator returning 500 HOT 10
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from guardrails.