Repo for the paper An In-depth Look at Gemini's Language Abilities by CMU, Zeno, and BerriAI LiteLLM

In this paper, we do an in-depth exploration of Google Gemini's language abilities, making two contributions:

• We provide a third-party, objective comparison of the abilities of the OpenAI GPT and Google Gemini models with reproducible code and fully transparent results.
• we take a closer look at the results, identifying areas where one of the two model classes excels.

We perform this analysis over 10 datasets testing a variety of language abilities, including reasoning, answering knowledge-based questions, solving math problems, translating between languages, generating code, and acting as instruction-following agents. From this analysis, we find that (as of this writing on December 18th, 2023):

• Gemini's Pro model achieved comparable but slightly inferior accuracy compared to the current version of OpenAI's GPT 3.5 Turbo for all English tasks, but superior ability to translate into other languages.
• Gemini fails in mathematical reasoning with many digits, and is sensitive to multiple-choice answer ordering, and others.
• Gemini demonstrates comparably high performance in areas such as generation into non-English languages, handling longer and more complex reasoning chains, and word sorting/rearrangement problems.

The overall results table can be found below:

You can find more details on results from each task, and comprehensive analysis at each of the below links:

• Knowledge-based QA (MMLU)
• Reasoning (BIG-Bench Hard)
• Mathematics (GSM8K, SVAMP, ASDIV, MAWPS)
• Code Generation (HumanEval, ODEX)
• Machine Translation (FLORES)
• Web Navigation Agents (WebArena)

• /outputs/{dataset}/{model}: contains the outputs of the systems, separated by dataset and model
• /benchmarking/{dataset}: contains the code for benchmarking, separated by dataset
• /visualization: contains the code for visualization, possibly separated by task type

Create a .env file in the root of the repository with your Zeno API key:

ZENO_API_KEY=your_api_key

This is loaded by dotenv in the visualization files.