AudioStyleNet - Controlling StyleGAN through Audio

This repository contains the code for my master thesis on talking head generation by controlling the latent space of a pretrained StyleGAN model. The work was done at the Visual Computing and Artificial Intelligence Group at the Technical University of Munich under the supervision of Matthias Niessner and Justus Thies.

Video

See the demo video for more details and results.

Set-up

The code uses Python 3.7.5 and it was tested on PyTorch 1.4.0 with cuda 10.1. (This project requires a GPU with cuda support.)

Clone the git project:

$ git clone https://github.com/FeliMe/Emotion-Aware-Facial-Animation.git

Create two virtual environments:

$ conda create -f environment.yml
$ conda create -n deepspeech python=3.6

Install requirements:

$ conda activate audiostylenet
$ pip install -r requirements.txt
$ conda activate deepspeech
$ pip install -r deepspeech/deepspeech_requirements.txt

Install ffmpeg

sudo apt install ffmpeg

Demo

Download the pretrained AudioStyleNet model and the StyleGAN model from Google Drive and place them in the model/ folder.

run

$ python run_audiostylenet.py 

Use your own audio

To test the model with your own audio, first convert your audio to waveform and then run the following:

$ cd deepspeech
$ conda activate deepspeech
$ python run_voca_feature_extraction.py --audiofiles <path to .wav file> --out_dir ../data/audio/
$ conda deactivate

Then run python run_audiostylenet.py with adapted arguments.