A gradio web UI for running Large Language Models like LLaMA, llama.cpp, GPT-J, OPT, and GALACTICA.

Its goal is to become the AUTOMATIC1111/stable-diffusion-webui of text generation.

• 3 interface modes: default, notebook, and chat
• Multiple model backends: transformers, llama.cpp, ExLlama, AutoGPTQ, GPTQ-for-LLaMa
• Dropdown menu for quickly switching between different models
• LoRA: load and unload LoRAs on the fly, train a new LoRA
• Precise instruction templates for chat mode, including Llama 2, Alpaca, Vicuna, WizardLM, StableLM, and many others
• Multimodal pipelines, including LLaVA and MiniGPT-4
• 8-bit and 4-bit inference through bitsandbytes
• CPU mode for transformers models
• DeepSpeed ZeRO-3 inference
• Extensions
• Custom chat characters
• Very efficient text streaming
• Markdown output with LaTeX rendering, to use for instance with GALACTICA
• Nice HTML output for GPT-4chan
• API, including endpoints for websocket streaming (see the examples)

To learn how to use the various features, check out the Documentation: https://github.com/oobabooga/text-generation-webui/tree/main/docs

Just download the zip above, extract it, and double-click on "start". The web UI and all its dependencies will be installed in the same folder.

• The source codes are here: https://github.com/oobabooga/one-click-installers
• There is no need to run the installers as admin.
• AMD doesn't work on Windows.
• Huge thanks to @jllllll, @ClayShoaf, and @xNul for their contributions to these installers.

Recommended if you have some experience with the command line.

https://docs.conda.io/en/latest/miniconda.html

On Linux or WSL, it can be automatically installed with these two commands:

curl -sL "https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh" > "Miniconda3.sh"
bash Miniconda3.sh

Source: https://educe-ubc.github.io/conda.html

conda create -n textgen python=3.10.9
conda activate textgen

The up-to-date commands can be found here: https://pytorch.org/get-started/locally/.

• MacOS users: oobabooga#393
• AMD users: https://rentry.org/eq3hg

git clone https://github.com/oobabooga/text-generation-webui
cd text-generation-webui
pip install -r requirements.txt

bitsandbytes >= 0.39 may not work on older NVIDIA GPUs. In that case, to use --load-in-8bit, you may have to downgrade like this:

• Linux: pip install bitsandbytes==0.38.1
• Windows: pip install https://github.com/jllllll/bitsandbytes-windows-webui/raw/main/bitsandbytes-0.38.1-py3-none-any.whl

ln -s docker/{Dockerfile,docker-compose.yml,.dockerignore} .
cp docker/.env.example .env
# Edit .env and set TORCH_CUDA_ARCH_LIST based on your GPU model
docker compose up --build

• You need to have docker compose v2.17 or higher installed. See this guide for instructions.
• For additional docker files, check out this repository.

From time to time, the requirements.txt changes. To update, use this command:

conda activate textgen
cd text-generation-webui
pip install -r requirements.txt --upgrade

Models should be placed inside the models/ folder.

Hugging Face is the main place to download models. These are some examples:

• Pythia
• OPT
• GALACTICA
• GPT-J 6B

You can automatically download a model from HF using the script download-model.py:

python download-model.py organization/model

For example:

python download-model.py facebook/opt-1.3b

To download a protected model, set env vars HF_USER and HF_PASS to your Hugging Face username and password (or User Access Token). The model's terms must first be accepted on the HF website.

You can drop these directly into the models/ folder, making sure that the file name contains ggml somewhere and ends in .bin.

python download-model.py EleutherAI/gpt-j-6B --text-only

conda activate textgen
cd text-generation-webui
python server.py

Then browse to

http://localhost:7860/?__theme=dark

Optionally, you can use the following command-line flags:

⚠️ Requires minimum compute of 7.0 on Windows at the moment.

Inference settings presets can be created under presets/ as yaml files. These files are detected automatically at startup.

The presets that are included by default are the result of a contest that received 7215 votes. More details can be found here.

• Pull requests, suggestions, and issue reports are welcome.
• Make sure to carefully search existing issues before starting a new one.
• If you have some experience with git, testing an open pull request and leaving a comment on whether it works as expected or not is immensely helpful.
• A simple way to contribute, even if you are not a programmer, is to leave a 👍 on an issue or pull request that you find relevant.

• Subreddit: https://www.reddit.com/r/oobaboogazz/
• Discord: https://discord.gg/jwZCF2dPQN