[bug] 'NoneType' object is not subscriptable about guardrails HOT 6 CLOSED

A fix can be to rework the 'post_process' function in run.py:

def post_process(self, output: str) -> str:  
        output = output.strip()  
        # if output.startswith("```"):  
        #     output = output[3:]  
        #     if output.startswith("json"):  
        #         output = output[4:]   
        # if output.endswith("```"):  
        #     output = output[:-3]  
        output = output.replace(" False,","'False'").replace(" True,","'True'").replace(" None,","'None'")  
        output = output.replace("json","JSON").split("JSON")[-1]  
        output = "{" + output.split("{",1)[-1].rsplit("}",1)[0] + "}"   
        return output  

from guardrails.

Changing the high_level_reask_prompt to explicitly mention that it can only return the corrected JSON also improves the behaviour

<high_level_reask_prompt>
I was given the following JSON response, which had problems due to incorrect values.

{previous_response}

Help me correct the incorrect values based on the given error messages. Only return the corrected JSON, don't repeat the given JSON
</high_level_reask_prompt>

(if you add this, you do not need the following line in the post_process function anymore:
output = output.replace("json","JSON").split("JSON")[-1] )

from guardrails.

Hey, thanks for the improvement suggestions.

First off, I'm unable to replicate your example. engine="chatgpt" is invalid. Running it with model="gpt-3.5-turbo" worked for me. Could you double-check how you ran it?

Let me run through your suggestions one by one.

output = output.replace(" False,","'False'").replace(" True,","'True'").replace(" None,","'None'")  

This seems to change the booleans and Nones to string equivalents. Their equivalents should be false, true and null (JSON uses javascript equivalents). This would be a reasonable change, if we can find a way to write this without it affecting the values within strings. Maybe there's a good regex solution?

output = output.replace("json","JSON").split("JSON")[-1]  

If I'm reading this correctly, it assumes that the output will be annotated by a triple-quote json. If it's not, it'll parse the whole thing as it would otherwise. This sounds like a good change, except I'm worried it might hit a "json" substring in the json itself.

output = "{" + output.split("{",1)[-1].rsplit("}",1)[0] + "}"   

I like this one a lot, and it would meaningfully replace the lines you commented out.

 <high_level_reask_prompt>
 I was given the following JSON response, which had problems due to incorrect values.
 
 {previous_response}

-Help me correct the incorrect values based on the given error messages.
+Help me correct the incorrect values based on the given error messages. Only return the corrected JSON, don't repeat the given JSON.
 </high_level_reask_prompt>

This change seems reasonable to me, though I've never personally encountered it repeating the given JSON.

The cases you're proposing to handle are weird, I haven't really seen openai's models make these errors. Could you please:

• provide what model you used (the runtime arguments you provided are unreplicable),
• submit a sample doctors_notes,
• show some examples of the failed generations?

from guardrails.

Okay, let me provide some more context:

• I renamed my "gpt-3.5-turbo" engine to "chatgpt" in the Azure openAI service. So it is the same as the one you used (Sorry I should have clarified this).

• the doctor notes that I used are the same ones as in the getting-started guide:
  doctors_notes = """49 y/o Male with chronic macular rash to face & hair, worse in beard, eyebrows & nares. Itchy, flaky, slightly scaly. Moderate response to OTC steroid cream"""

• An example where guardrails errored because the output from OpenAI was unexpected:

output openAI:

{
    "patient_info": {
        "symptoms": [
            {
                "affected area": {
                    "incorrect_value": "face & hair",
                    "error_message": "Value face & hair is not in choices ['head', 'neck', 'chest']."
                }
            },
            {
                "affected area": {
                    "incorrect_value": null,
                    "error_message": "Value None is not in choices ['head', 'neck', 'chest']."
                }
            },
            {
                "affected area": {
                    "incorrect_value": null,
                    "error_message": "Value None is not in choices ['head', 'neck', 'chest']."
                }
            },
            {
                "affected area": {
                    "incorrect_value": null,
                    "error_message": "Value None is not in choices ['head', 'neck', 'chest']."
                }
            }
        ]
    }
}

The corrected JSON response is:

{
    "patient_info": {
        "symptoms": [
            {
                "affected area": "face & hair"
            },
            {
                "affected area": null
            },
            {
                "affected area": null
            },
            {
                "affected area": null
            }
        ]
    }
}

Which cannot be processed by the JSON decoder and therefore the output_as_dict=None. This leads to errors in the reask functions.

from guardrails.

Another example:
output openAI:

{
    "patient_info": {
        "symptoms": [
            {
                "affected area": {
                    "incorrect_value": "face & hair",
                    "error_message": "Value face & hair is not in choices ['head', 'neck', 'chest']."
                }
            },
            {
                "affected area": {
                    "incorrect_value": None,
                    "error_message": "Value None is not in choices ['head', 'neck', 'chest']."
                }
            },
            {
                "affected area": {
                    "incorrect_value": None,
                    "error_message": "Value None is not in choices ['head', 'neck', 'chest']."
                }
            },
            {
                "affected area": {
                    "incorrect_value": None,
                    "error_message": "Value None is not in choices ['head', 'neck', 'chest']."
                }
            }
        ]
    }
}

becomes

{
    "patient_info": {
        "symptoms": [
            {
                "affected area": "head"
            },
            {
                "affected area": null
            },
            {
                "affected area": null
            },
            {
                "affected area": null
            }
        ]
    }
}

from guardrails.

@sophieDataroots thanks for sharing this.

If I understand correctly, in the examples you shared, the The corrected JSON response is: statement is part of the LLM response, in addition to the original JSON dictionary. And this causes the JSON parsing error.

I believe the reason this is happening is that:

• Since you're testing out the example with a chat model, the way to get correctly structured JSON from the chat model is using a separate system instructions in addition to just the prompt.
• Currently, all of the reasking logic assumes that the end user only a non-chat model, and so the reasking logic only creates a prompt, without creating system instructions to go with the prompt. This causes reasking output to be formatted incorrectly.

This is a pretty interesting bug, and should not be a super hard fix on our end.

from guardrails.