PyTorch implementation of Peekaboo, a training-free and zero-latency interactive video generation pipeline

Peekaboo: Interactive Video Generation via Masked-Diffusion
Yash Jain^1*, Anshul Nasery^2*, Vibhav Vineet³, Harkirat Behl^3
1Microsoft, ²University of Washington, ³Microsoft Research
*denotes equal contribution

A Horse galloping through a meadow	A Panda playing Peekaboo	An Eagle flying in the sky

Follow the instructions below to download and run Peekaboo on your own prompts and bbox inputs. These instructions need a GPU with ~40GB VRAM for zeroscope and ~13GB VRAM for modelscope . If you don't have a GPU, you may need to change the default configuration from cuda to cpu or follow intstructions given here.

conda env create -f env.yaml
conda activate peekaboo

python src/generate.py --model zeroscope --prompt "A panda eating bamboo in a lush bamboo forest" --fg_object "panda"

# Optionally, you can specify parameters to tune your result:
# python src/generate.py --model zeroscope --frozen_steps 2 --seed 1234 --num_inference_steps 50 --output_path src/demo --prompt "A panda eating bamboo in a lush bamboo forest" --fg_object "panda"

python src/app_modelscope.py 

(For advice on how to get the best results by tuning parameters, see the Tips section).

We recommend using our interactive Gradio demo to play with different input bbox. Alternatively, editing the bbox_mask variable in generate.py will achieve similar results.

Please follow the README to download the datasets and learn more about Peekaboo's evaluation.

If you're not getting the quality result you want, there may be a few reasons:

1. Is the video not gaining control or degrading? The number of frozen_steps might be insufficient. The number of steps dictate how long the peekaboo attention mask modulation will act during the generation process. Too less will not affect the output video and too much will degrade the video quality. The default frozen_steps is set to 2, but aren't necessarily optimal for each (prompt, bbox) pair. Try:
   • Increasing the frozen_steps, or
   • Decreasing the frozen_steps
2. Poor video quality: A potential reason could be that the base model does not support the prompt well or in other words, generate a poor quality video for the given prompt. Try changing the model or improving the prompt.
3. Try generating results with different random seeds by chaning seed parameter and running generation multiple times.
4. Increasing the number of steps sometimes improves results.

@article{jain2023peekaboo,
  title={PEEKABOO: Interactive Video Generation via Masked-Diffusion},
  author={Jain, Yash and Nasery, Anshul and Vineet, Vibhav and Behl, Harkirat},
  journal={arXiv preprint arXiv:2312.07509},
  year={2023}
}

If you implement Peekaboo in newer text-to-video models, feel free to raise a PR. 😄

This readme is inspired by InstructPix2Pix