Hey all, I'm using promptfoo 0.50.1 and would like to use model-grad

It should be a single object, not an array: <div class="highlight highlight-source

Thanks <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-u

Works like a charm, thanks <a class="user-mention notranslate" data-hovercard-type="us

"[FAIL] Error: Invalid provider definition for output type 'text'" when using model-graded metrics about promptfoo HOT 6 CLOSED

jkng10 commented on June 12, 2024

"[FAIL] Error: Invalid provider definition for output type 'text'" when using model-graded metrics

from promptfoo.

Comments (6)

typpo commented on June 12, 2024 1

Yep, you can put it on the defaultTest object. See here: https://promptfoo.dev/docs/configuration/expected-outputs/model-graded/#multiple-graders

from promptfoo.

typpo commented on June 12, 2024

It should be a single object, not an array:

- type: llm-rubric
      value: is not apologetic
      provider:
        id: azureopenai:embeddings:text-embedding-ada-002
        config:
          temperature: 0
          max_tokens: 128
          #...

from promptfoo.

jkng10 commented on June 12, 2024

Thanks @typpo, appreciate the quick feedback, this error is gone! However now getting the below:

Error: Not implemented
    at AzureOpenAiEmbeddingProvider.callApi (...\npm\node_modules\promptfoo\dist\src\providers\azureopenai.js:76:15)
    at matchesLlmRubric (...\npm\node_modules\promptfoo\dist\src\matchers.js:227:38)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async runAssertion (...\npm\node_modules\promptfoo\dist\src\assertions.js:667:17)
    at async ...\npm\node_modules\promptfoo\dist\src\assertions.js:82:24

Is there any difference in between using

id: azureopenai:embedding:text-embedding-ada-002
as per https://www.promptfoo.dev/docs/configuration/expected-outputs/similar/

id: azureopenai:embeddings:text-embeddings-ada-002
as per #323 (comment)

from promptfoo.

typpo commented on June 12, 2024

Ah, I didn't notice you were using llm-rubric, but that requires a text generation provider (like gpt-4), not an embedding provider. More details here

from promptfoo.

jkng10 commented on June 12, 2024

Thanks @typpo ! It works now!

One last quick question - can provider details/config be also defined more globally on "assert" level?

I'd like to use the gpt-4 provider for all subsequent llm-rubric and context-* types and currently specify same provider details/config (e.g. apiKey and apiHost) again for each "type", which looks redundant.

currently:

assert:
  - type: llm-rubric
    value: does not describe self as an AI or chat assistant
    threshold: 0.85
    provider:
      id: azureopenai:chat:gpt-4
      config:
        apiKey: <key>
        apiHost: <url>
  - type: llm-rubric
    value: is not apologetic
    threshold: 0.85
    provider:
      id: azureopenai:chat:gpt-4
      config:
        apiKey: <key>
        apiHost: <url>
  - type: llm-rubric
    value: is confident and does not guess or speculate
    threshold: 0.85
    provider:
      id: azureopenai:chat:gpt-4
      config:
        apiKey: <key>
        apiHost: <url>

from promptfoo.

jkng10 commented on June 12, 2024

Works like a charm, thanks @typpo!

from promptfoo.

Recommend Projects