SummarizationMetric is not cached about deepeval HOT 4 CLOSED

Use of evaluate in Monitoring system is an import use-case for us.

from deepeval.

@prescod the summarization metric is the only metric that does not have caching

from deepeval.

@penguine-ip Well then I guess it's bad luck that I used that test randomly to test the caching!

I find it interesting that caching is done at the test layer and not the LLM layer. I would expect you to cache LLM inputs and outputs given that that's where all of the slow stuff happens.

And then it would automatically transfer over to all tests.

from deepeval.

@prescod That's actually a good point, I think the reason why was because this was easier to implement. Also added the docs to let users know about summarizaiton doesn't support cache: https://docs.confident-ai.com/docs/metrics-summarization

from deepeval.