[FR] Bring your own evaluation prompt about mlflow HOT 6 OPEN

cc @prithvikannan @dbczumar

from mlflow.

Hi @alena-m, thank you for raising this. We agree that it would be valuable, and we would really appreciate a contribution for it. I'll add the help wanted label for now. Let us know if you'd like to reconsider and take this on.

from mlflow.

cc @sunishsheth2009

from mlflow.

Agreed this would be super useful. The existing components for this are largely already present: make_metric() to create an arbitrary EvaluationMetric and the deployment client.predict() to call LLM. We would need to figure out some approach for users to define their own grading prompt and corresponding parsing logic.

from mlflow.

@mlflow/mlflow-team Please assign a maintainer and start triaging this issue.

from mlflow.

I'd like to work on this issue.

My understanding is that make_genai_metric instantiates a EvaluationModel that has the grading_system_prompt_template hardcoded.
So, the user could provide an optional grading_system_prompt_template argument that simply is used instead of the template in that case, that way the option is provided simply, and it doesn't break the existing approach!

Would that fit the requirements?

from mlflow.