Paper | Blog | Model Checkpoint Download | HuggingFace

This repository contains artifacts for the Meta Chameleon model from FAIR, Meta AI. In this repository is:

• Standalone Inference Code — a fast GPU-based inference implementation for running model checkpoints
• Input-Output Viewing — a harness for richly viewing multimodal model inputs and outputs with a browser-based tool
• Evaluation Prompts — mixed-modal and text-only prompts for human evaluation

Running constituent components for inference and the input-output viewer currently requires a CUDA-capable GPU. If you'd like to run inference on other hardware, other inference implementations, including HuggingFace, are platform agnostic.

First, pip install this repository:

pip install -U git+https://github.com/facebookresearch/chameleon.git

Alternatively, if you want access to the full visualizer, you'll need to clone this repository (instead of installing), then pip install from the repository root:

git clone https://github.com/facebookresearch/chameleon.git
cd chameleon
pip install -e .

Model checkpoints and configs must be downloaded before running inference or the viewer. After requesting model access, run the following script, adding pre-signed download URL you were emailed when prompted:

python -m chameleon.download_data [pre-signed URL]

(you can also paste the command given in the email containing the download link)

The viewer visualizes multi-modal model input and output. It is most easily run with docker-compose. You'll need to clone the repository, not just a pip install.

The following runs both the service and viewer interface.

By default, this runs the 7B parameter model. You can change the model_path variable in ./config/model_viewer.yaml to select another model and alter other configuration:

docker-compose up --build

You can open the viewer at http://localhost:7654/

The miniviewer is a light weight debug visualizer, that can be run with:

python -m chameleon.miniviewer

This runs the 7B parameter model. To run the 30B model, use the following command:

python -m chameleon.miniviewer --model-size 30b

You can open the miniviewer at http://localhost:5000/.

Use of this repository and related resources are governed by the Chameleon Research License and the LICENSE file.

To cite the paper, model, or software, please use the below:

@article{Chameleon_Team_Chameleon_Mixed-Modal_Early-Fusion_2024,
  author = {Chameleon Team},
  doi = {10.48550/arXiv.2405.09818},
  journal = {arXiv preprint arXiv:2405.09818},
  title = {Chameleon: Mixed-Modal Early-Fusion Foundation Models},
  url = {https://github.com/facebookresearch/chameleon},
  year = {2024}
}