The Many-Shot Prediction App is a powerful tool for comparing and analyzing the performance of various AI models. It allows users to run multiple predictions on the same prompt using different models, including those from OpenAI, Anthropic, Gemini, and a local model.

Recent research has shown that running the same prompt multiple times can lead to greater predictability in language model outputs. This app provides an intuitive interface to run a prompt N times and visualize the results, helping you understand the distribution and consistency of model responses.

• Support for multiple AI platforms: OpenAI, Anthropic, Gemini, and a local model
• Customizable number of prediction runs
• Input for questions and multiple answer options
• Visualization of prediction results
• Easy comparison of model performances

1. Clone the repository:

   git clone https://github.com/btahir/many-shot.git
   cd many-shot-predictions

2. Install dependencies:

   npm install
   # or
   yarn install
   # or
   pnpm install

3. Set up environment variables (see below)

4. Run the development server:

   npm run dev
   # or
   yarn dev
   # or
   pnpm dev

5. Open http://localhost:3000 in your browser to use the app.

Create a .env.local file in the root directory and add your API keys:

OPENAI_API_KEY=your_openai_api_key
ANTHROPIC_API_KEY=your_anthropic_api_key
GEMINI_API_KEY=your_gemini_api_key

Note: The app will only work for the platforms whose API keys you have provided. If you want to test it out quickly, you can use the local model option, although it may be slower.

You can try out the demo app here. Please note that the demo version only works with the local model option.

We welcome contributions! Please feel free to submit a Pull Request.

This project is licensed under the MIT License - see the LICENSE file for details.