A way to prevent vars array auto-expansion about promptfoo HOT 4 CLOSED

#476 adds an envar that disables var expansion. Unfortunately our options are limited because I don't want to break people who are using this feature. Will merge after I test etc

from promptfoo.

@typpo thanks for the quick response!

What if it was a setting on the testCase itself? i.e. somthing like

tests:
  - disableVarExpansion: true
    vars:
      ingredients:
        - Microwave popcorn
        - Ground beef
        - Pop Rocks

That way you're giving more granular control without breaking any existing setups (assuming disableVarExpansion defaults to false, which it should).

Also, considerting env vars are not a super convenient way to configure things, especially if you have multiple prompfoo files that you're using with different settings, it might be a good idea to just add a new "global settings" section to the promptfoo config definition. Something like that would make it much simpler to "share" tests within teams, because you don't have to configure your run environment before every run, which - for a team - would require documenting the "extra steps" needed to make a test work (i.e. which env vars should be set to what value).

i.e. something like

globalSettings:
  disableVarExpansion: true
  disableTemplating: true

tests:
  - vars:
      ingredients:
        - Microwave popcorn
        - Ground beef
        - Pop Rocks

Another option (which I think isn't as good but could also provide an answer to the sharing problem I discribed) would be adding support for .env files. If promptfoo eval would look for a .env file placed next to the prompfoo configuration file itself, and automatically load it, that would also make it easier to share setups without requiring an additional section in the configuration file.

from promptfoo.

Just updated the PR to include a test option disableVarExpansion, and will land it today!

The globalSettings thing already exists in the form of defaultTest :) It'd look something like this:

defaultTest:
  options:
    disableVarExpansion: true

tests:
  - vars:
      ingredients:
        - Microwave popcorn
        - Ground beef
        - Pop Rocks

from promptfoo.

This is published in 0.42.0

from promptfoo.