Add bedrock:completion:anthropic.claude-v2.1 about promptfoo HOT 6 CLOSED

I don't think I have access to this model yet - what is the current behavior when you try to run it with promptfoo?

from promptfoo.

I think it should be added here:

Currently I'm getting "TypeError: Cannot read properties of undefined (reading 'params') TypeError: Cannot read properties of undefined (reading 'params') at AwsBedrockCompletionProvider.callApi"

from promptfoo.

But this'll be more complicatede - claude v2.1 supports system prompts (no "Human:" at the beggining of the prompt).

from promptfoo.

#470 solves the error at least - so you can use claude-2.1 without a system prompt.

As for supporting system prompt, what do you think is the best way? Some thoughts:

• Completely remove the logic that prepends Human: and appends Assistant:. This means that users will have to set custom prompts for Anthropic, but minimizes the amount of magic stuff that's happening
• Support OpenAI-style role, content JSON objects
• Some other approach?

from promptfoo.

Adding Human: Assistant: is similar to langchain - we could try to map langchain solution to promptfoo codebase. From the other side, I agree that this "magic" isn't something I like in langchain. And we have functionality that adds prefix/suffix to the prompt in promptfoo if I remember correctly. So probably good documentation without "magic stuff" sounds reasonable.

from promptfoo.

Closing this out as this format is now legacy with Claude 3

from promptfoo.