The n_total of n_total result is not 805 about alpaca_eval HOT 6 CLOSED

I also encountered similar problems
I ran the following command multiple times:

alpaca_eval --model_outputs "Mistral-7B+RAHF-DUAL+LoRA.json" --output_path "Mistral-7B+RAHF-DUAL+LoRA_gpt4" ----annotators_config "weighted_alpaca_eval_gpt4_turbo" --reference_outputs "reference_outputs.json" --is_reapply_parsing True

But the result is that only more than 200 examples are evaluated.

from alpaca_eval.

I also encountered similar problems I ran the following command multiple times:

alpaca_eval --model_outputs "Mistral-7B+RAHF-DUAL+LoRA.json" --output_path "Mistral-7B+RAHF-DUAL+LoRA_gpt4" ----annotators_config "weighted_alpaca_eval_gpt4_turbo" --reference_outputs "reference_outputs.json" --is_reapply_parsing True

But the result is that only more than 200 examples are evaluated.

Can you access OpenAI successfully?
Unfortunately, I only have intermittent access to it.
"INFO:root:Sleeping 2 before retrying to call openai API..."
I don't know if this is the reason for the incomplete evaluation.

from alpaca_eval.

Can either of you reproduce this result using OpenAI rather than through a proxy server? In my experience (from previous gh issues) the problem always comes from the proxy server and I can't help for that.

For example, many proxy servers don't return logprobs. Can you share the warnings/errors that you see during annotation?

from alpaca_eval.

E.g. @LiuAmber the fact that you could perform AlpacaEval 1 here: #307, strongly suggests to me that the error comes from the logprobs (necessary for AlapcaEval 2 but not 1) and that this comes from your proxy server.

from alpaca_eval.

I'm assuming that the issue is the fact that your proxy doesn't have access to logprobs. If so try:

alpaca_eval evaluate_from_model --model_configs '...' --annotators_config 'alpaca_eval_clf_gpt4_turbo'

Feel free to reopen if your issue is different

from alpaca_eval.

Can either of you reproduce this result using OpenAI rather than through a proxy server? In my experience (from previous gh issues) the problem always comes from the proxy server and I can't help for that.

For example, many proxy servers don't return logprobs. Can you share the warnings/errors that you see during annotation?

Thank you. I solved this problem. The problem is caused by proxy servers and the setting of annotators_config.

from alpaca_eval.