PyTorch implementation of Natural TTS Synthesis By Conditioning Wavenet On Mel Spectrogram Predictions.
This implementation includes distributed and automatic mixed precision support and uses the LJSpeech dataset.
Distributed and Automatic Mixed Precision support relies on NVIDIA's Apex and AMP.
Visit our website for audio samples using our published Tacotron 2 and WaveGlow models.
- NVIDIA GPU + CUDA cuDNN
- Download and extract the LJ Speech dataset ver. 1.1
- The extracted directory
LJSpeech-1.1
would have thewavs
directory that contains speech wav files. The path of thewavs
directory will be used in Step 5.
- The extracted directory
- Clone this repo:
git clone https://github.com/sungjae-cho/tacotron2.git
- CD into this repo:
cd tacotron2
- Initialize submodule:
git submodule init; git submodule update
- Update .wav paths:
sed -i -- 's,DUMMY,<ljs_dataset_folder/wavs>,g' filelists/*.txt
<ljs_dataset_folder/wavs>
is the thewavs
directory found in Step 1.- Alternatively, set
load_mel_from_disk=True
inhparams.py
and update mel-spectrogram paths
- Install PyTorch 1.0
- Install Apex
- Install python requirements or build docker image
- Install python requirements:
pip3 install -r requirements.txt
- Install python requirements:
- Login
wandb login
and give API keys given in the WandB website user setting.
If you want to use the second GPU device with device number 1, then enter export CUDA_VISIBLE_DEVICES=1
. If there are 4 GPU devices, then you can use 0, 1, 2, and 3 as a device number.
python3 train.py --output_directory=outdir --log_directory=logdir --run_name=<str_run_name>
- If you get an error message saying, for example,
RuntimeError: CUDA out of memory. Tried to allocate 106.38 MiB (GPU 0; 11.91 GiB total capacity; 10.26 GiB already allocated; 53.06 MiB free; 56.48 MiB cached)
, then reduce the batch size. You can reduce the size on the command in this way:python3 train.py --output_directory=outdir --log_directory=logdir --run_name=<str_run_name> --hparams=batch_size=32
. Or, changebatch_size
inhparams.py
.
- If you get an error message saying, for example,
- (OPTIONAL)
tensorboard --logdir=outdir/logdir
===== What I have executed to here. =====
Training using a pre-trained model can lead to faster convergence
By default, the dataset dependent text embedding layers are ignored
- Download our published Tacotron 2 model
python train.py --output_directory=outdir --log_directory=logdir -c tacotron2_statedict.pt --warm_start
python -m multiproc train.py --output_directory=outdir --log_directory=logdir --hparams=distributed_run=True,fp16_run=True
- Download our published Tacotron 2 model
- Download our published WaveGlow model
wget --no-check-certificate 'https://drive.google.com/uc?id=1WsibBTsuRg_SF2Z6L6NFRTT-NjEy1oTx&export=download' -O waveglow_256channels.ptwget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1WsibBTsuRg_SF2Z6L6NFRTT-NjEy1oTx' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p')&id=1WsibBTsuRg_SF2Z6L6NFRTT-NjEy1oTx" -O waveglow_256channels.pt && rm -rf /tmp/cookies.txt
jupyter notebook --ip=127.0.0.1 --port=31337
- Load inference.ipynb
N.b. When performing Mel-Spectrogram to Audio synthesis, make sure Tacotron 2 and the Mel decoder were trained on the same mel-spectrogram representation.
WaveGlow Faster than real time Flow-based Generative Network for Speech Synthesis
nv-wavenet Faster than real time WaveNet.
This implementation uses code from the following repos: Keith Ito, Prem Seetharaman as described in our code.
We are inspired by Ryuchi Yamamoto's Tacotron PyTorch implementation.
We are thankful to the Tacotron 2 paper authors, specially Jonathan Shen, Yuxuan Wang and Zongheng Yang.