Question about the GPT-4 API about alpaca_eval HOT 11 CLOSED

That's surprising given that it works for the other examples. It's very hard to help without being able to reproduce.

• Does it always error out on the same examples? if so can you send some to me?
• can you try export OPENAI_MAX_CONCURRENCY=1 and see if the error persits
• can you paste the error/warnings you get during the annotation?

Thanks!

from alpaca_eval.

Thank you for your reply! I will try again and give more information latter.

from alpaca_eval.

I have some questions about the dropping of samples during the evaluation of Alpaca eval. Is it normal to have some dropped samples, or is the normal condition that all 805 samples are correct?

from alpaca_eval.

Usually we don't drop anything, but it may happen that GPT4 API fails and then you drop 2-3 examples, which we don't care much about. But you are dropping 500 examples here, which is definitely not expected and I wouldn't trust the win-rates you are currently getting.

from alpaca_eval.

I think it could be an issue with my API or occurring during the request process. Now I have a unified platform for requesting GPT-4. The process involves uploading a text file with prompts and inputs, and the platform will return the generated results from GPT-4. Then, I utilize the results from GPT-4 to parse and calculate the win rate. Do you have any suggestions on how I should unpack and execute the process you've encapsulated?

from alpaca_eval.

This is part of the process of requesting GPT-4, and no other errors have occurred. However, the following error subsequently appeared.

from alpaca_eval.

Do you have the error if you use the base OpenaAI api? Please try without using lonlie.plus7, I can't help for other APIs

from alpaca_eval.

This is likely your server error not an AlpacaEval error. If you can reproduce the error with the openai base then feel free to reopen.

from alpaca_eval.

Thank you for all your patient responses. I'm currently trying out the openai base API to test the results.

from alpaca_eval.

Have you found a solution?

from alpaca_eval.

Have you found a solution?

I used to use an API provided by my lab, which required setting up an openai_api_base. This was a URL that may not have been the official API. When I switched to the official OPENAI GPT-4 API, the problem was resolved.

from alpaca_eval.