gair-nlp / benbench Goto Github PK

Tidy up and encapsulate the code and open source a small toolkit to help calculate metrics:

• based on the reference data if given its path;
• support multiple metrics, including ppl, ngram-acc, min-k, etc.;
• it could be good if taking the calculation of metrics based on the answer part and full text, respectively.
• ...

orgn-GSM8K-train Average_ppl_accuracy: nan
PPL of xxxx
GSM8K_rewritten-test-1: nan
GSM8K_rewritten-test-2: nan
GSM8K_rewritten-test-3: nan
GSM8K_rewritten-train-1: nan
GSM8K_rewritten-train-2: nan
GSM8K_rewritten-train-3: nan
orgn-GSM8K-test: nan
orgn-GSM8K-train: nan

why this happens?

Model: https://huggingface.co/Qwen/Qwen1.5-110B

Ngram-Acc and PPL:

• GPT-4
• Grok-1

Ngram-ACC:

• Claude 3.0

Stay tuned. Feel free to comment if you are interested in other LLMs.

https://twitter.com/amanrsanger/status/1779620682340704386

Are the ngram and ppl scripts currently capable of leveraging multi-GPU setups for inference? If not, are there any planned updates or workarounds that might support this feature?

Additionally, is it possible to integrate vllm support to accelerate the ngram computations?

Some references:

• https://tatsu-lab.github.io/alpaca_eval/
• https://super.gluebenchmark.com/leaderboard/
• https://crfm.stanford.edu/helm/lite/latest/#/leaderboard
• https://rajpurkar.github.io/SQuAD-explorer/
• https://gair-nlp.github.io/auto-j/
• https://scienceqa.github.io/leaderboard.html
• https://tape-benchmark.com/index.html#leaderboard