Python and Javascript asserts from file about promptfoo HOT 4 CLOSED

Hey @Wehzie,

Just pushed a change 7308df2 that should resolve Promises returned by an external JS file. Can you give it a try?

btw, multiline Python asserts are also supported as of a recent update. You should be able to pass a multiline string. It expected the assertion result to be printed (e.g. print(True), print(0.5), etc), which is a bit weird, but it should work nonetheless.

from promptfoo.

Thanks for the breakdown, @Wehzie. I can probably make this clearer in the documentation, and/or break it out onto its own page.

from promptfoo.

Thank you @typpo.
All works. Issue is correctly closed.

Lessons learned:
Paths to scripts, either .py or .js are provided with

    assert:
      - type: javascript
        value: file://relative/path/to/script.js
      - type: python
        value: file///absolute/path/to/script.py

For Python scripts, since print triggers the return, other prints should be avoided.
For debugging purposes, a simple if DEBUG: print does the job.
Multiline Python works like JavaScript, for example

    assert:
      - type: python
        value: |
          some_var = "hello"
          print(some_var == "hello")

Thank you for documenting this in promptfoo/promptfoo-docs@9b8c04f

For JavaScript files, the to-be-executed function must be exported. For example, main, must be exported with module.exports = main;.
JavaScript async works as expected. For an example of assert.js, see

const VALIDATION_ENDPOINT = "https://example.com/api/validate";

async function evaluate(modelResponse) {
    try {
        const response = await fetch(VALIDATION_ENDPOINT, {
          method: 'POST',
          headers: {
            'Content-Type': 'text/plain'
          },
          body: modelResponse
        });
  
        const data = await response.json();
        return data;
    } catch (error) {
        throw error;
    }
}

async function main (output, context) {
    const success = await evaluate(output);
    console.log(`success: ${testResult}`);
    return success;
  }
  
module.exports = main;

When JavaScript or Python asserts are defined within promptfooconfig.yaml the variable output is injected and can be accessed without declaration.
This variable holds the model response.
When JavaScript or Python asserts are loaded with file://, the output is accessed with positional arguments.
The call to Python is of the form python path/to/assert.py model_output context.
The context variable holds the prompt and the variables of the test case.
The call to JavaScript is of the form main(output, context).
For an example of a assert.py, see

import sys

def main():
    if len(sys.argv) >= 3:
        output = sys.argv[1]
        context = sys.argv[2]
    else:
        raise ValueError("Model output and context are expected from promptfoo.")
    processed = preprocess_output(output)
    success = test_output(processed)
    return success
  
print(main())

I missed most of this from the documentation somehow. Anyway, thanks for implementing all this!

from promptfoo.

This change (promptfoo/promptfoo-docs@69e29d2) improves assertion documentation and adds much of the detail you recorded above.

from promptfoo.