Cannot use JSON prompts containing {{ or --- about promptfoo HOT 3 CLOSED

Sorry, it looks like the issue in model-graded-closedqa is somewhere else. It looks like it might be failing to properly double-escape the original prompt when including it in the LLM message.

I'm getting this:

Error: Chat Completion prompt is not a valid JSON string: SyntaxError: Expected ',' or '}' after property value in JSON at position 751

The snippet around position 751 is:

The merge request is titled \\"add scala

Position 751 is the quote, so there's definitely an issue with not properly double-quoting a quote. I'm guessing it should be

\\\"

from promptfoo.

Hey @iamjackg, thanks for the report. Definitely seems like something is going wrong with escaping.

promptfoo does double-escape the original prompt when including it: https://github.com/promptfoo/promptfoo/blob/main/src/matchers.ts#L311-L313

So I would guess that the string may have already escaped when it got there. As a temporary workaround, you can remove the nested escaped string (or use single quotes).

from promptfoo.

Probable fix landed in #242

from promptfoo.