This is an implementation of the following paper.
MnTTS: An Open-Source Mongolian Text-to-Speech Synthesis Dataset and Accompanied Baseline. in Proc. IALP'2022
Yifan Hu, Pengkai Yin, Rui Liu *, Feilong Bao, Guanglai Gao.
This project uses conda
to manage all the dependencies, you should install anaconda if you have not done so.
# Clone the repo
git clone https://github.com/walker-hyf/MnTTS.git
cd $PROJECT_ROOT_DIR
conda env create -f Environment/environment.yaml
conda activate mntts
Prepare our MnTTS dataset in the following format:
|- MnTTS/
| |- metadata.csv
| |- wavs/
| |- file1.wav
| |- ...
Where metadata.csv
has the following format: id|transcription
. This is a ljspeech-like format.
The complete dataset is available from our multilingual corpus website.
The preprocessing has two steps:
- Preprocess audio features
- Convert characters to IDs
- Compute mel spectrograms
- Normalize mel spectrograms to [-1, 1] range
- Split the dataset into train and validation
- Compute the mean and standard deviation of multiple features from the training split
- Standardize mel spectrogram based on computed statistics
To reproduce the steps above:
tensorflow-tts-preprocess --rootdir ./MnTTS --outdir ./dump_mntts --config preprocess/mntts_preprocess.yaml --dataset mntts
tensorflow-tts-normalize --rootdir ./dump_mntts --outdir ./dump_mntts --config preprocess/mntts_preprocess.yaml --dataset mntts
Based on the script train_tacotron2.py
.
This example code show you how to train Tactron-2 from scratch with Tensorflow 2 based on custom training loop and tf.function.
Here is an example command line to training TacoTron2 from scratch:
CUDA_VISIBLE_DEVICES=0 python examples/tacotron2/train_tacotron2.py \
--train-dir ./dump_mntts/train/ \
--dev-dir ./dump_mntts/valid/ \
--outdir ./examples/tacotron2/exp/train.tacotron2.v1/ \
--config ./examples/tacotron2/conf/tacotron2.v1.yaml \
--use-norm 1 \
--mixed_precision 0 \
--resume ""
IF you want to use MultiGPU to training you can replace CUDA_VISIBLE_DEVICES=0
by CUDA_VISIBLE_DEVICES=0,1,2,3
for example. You also need to tune the batch_size
for each GPU (in config file) by yourself to maximize the performance. Note that MultiGPU now support for Training but not yet support for Decode.
In case you want to resume the training progress, please following below example command line:
--resume ./examples/tacotron2/exp/train.tacotron2.v1/checkpoints/ckpt-100000
If you want to finetune a model, use --pretrained
like this with your model filename
--pretrained pretrained.h5
Extract duration from alignments for FastSpeech
You may need to extract durations for student models like fastspeech. Here we use teacher forcing with window masking trick to extract durations from alignment maps:
Extract for valid set:
CUDA_VISIBLE_DEVICES=0 python examples/tacotron2/extract_duration.py \
--rootdir ./dump_mntts/valid/ \
--outdir ./dump_mntts/valid/durations/ \
--checkpoint ./examples/tacotron2/exp/train.tacotron2.v1/checkpoints/model-100000.h5 \
--use-norm 1 \
--config ./examples/tacotron2/conf/tacotron2.v1.yaml \
--batch-size 32
--win-front 3 \
--win-back 3
Extract for training set:
CUDA_VISIBLE_DEVICES=0 python examples/tacotron2/extract_duration.py \
--rootdir ./dump_mntts/train/ \
--outdir ./dump_mntts/train/durations/ \
--checkpoint ./examples/tacotron2/exp/train.tacotron2.v1/checkpoints/model-100000.h5 \
--use-norm 1 \
--config ./examples/tacotron2/conf/tacotron2.v1.yaml \
--batch-size 32
--win-front 3 \
--win-back 3
To extract postnets for training vocoder, follow above steps but with extract_postnets.py
Based on the script train_fastspeech2.py
.
Here is an example command line to training FastSpeech2 from scratch:
CUDA_VISIBLE_DEVICES=0 python examples/fastspeech2/train_fastspeech2.py \
--train-dir ./dump_mntts/train/ \
--dev-dir ./dump_mntts/valid/ \
--outdir ./examples/fastspeech2/exp/train.fastspeech2.v1/ \
--config ./examples/fastspeech2/conf/fastspeech2.v1.yaml \
--use-norm 1 \
--f0-stat ./dump_mntts/stats_f0.npy \
--energy-stat ./dump_mntts/stats_energy.npy \
--mixed_precision 1 \
--resume ""
First, you need training generator with only stft loss:
CUDA_VISIBLE_DEVICES=0 python examples/hifigan/train_hifigan.py \
--train-dir ./dump_mntts/train/ \
--dev-dir ./dump_mntts/valid/ \
--outdir ./examples/hifigan/exp/train.hifigan.v1/ \
--config ./examples/hifigan/conf/hifigan.v1.yaml \
--use-norm 1 \
--generator_mixed_precision 1 \
--resume ""
Then resume and start training generator + discriminator:
CUDA_VISIBLE_DEVICES=0 python examples/hifigan/train_hifigan.py \
--train-dir ./dump_mntts/train/ \
--dev-dir ./dump_mntts/valid/ \
--outdir ./examples/hifigan/exp/train.hifigan.v1/ \
--config ./examples/hifigan/conf/hifigan.v1.yaml \
--use-norm 1 \
--resume ./examples/hifigan/exp/train.hifigan.v1/checkpoints/ckpt-100000
You can follow below example command line to generate synthesized speech for given text in 'dump_mntts/inference.txt' using Griffin-Lim and trained HiFi-GAN vocoder:
CUDA_VISIBLE_DEVICES=0 python examples/fastspeech2/inference_fastspeech2.py \
--outdir prediction/MnTTS_inference \
--infile dump_mntts/inference.txt \
--tts_ckpt examples/fastspeech2/exp/train.fastspeech2.v1/checkpoints/model-200000.h5 \
--vocoder_ckpt examples/hifigan/exp/train.hifigan.v1/checkpoints/generator-200000.h5 \
--stats_path dump_mntts/stats.npy \
--dataset_config preprocess/mntts_preprocess.yaml \
--tts_config examples/fastspeech2/conf/fastspeech2.v1.yaml \
--vocoder_config examples/hifigan/conf/hifigan.v1.yaml \
--lan_json dump_mntts/mntts_mapper.json
You can find pre-trained models in the Links section.
The synthesized speech will save to prediction/MnTTS_inference
folder.
- Pre-trained models: download pre-trained models
- Demo: link to synthesized audio samples
@INPROCEEDINGS{9961271,
author={Hu, Yifan and Yin, Pengkai and Liu, Rui and Bao, Feilong and Gao, Guanglai},
booktitle={2022 International Conference on Asian Language Processing (IALP)},
title={MnTTS: An Open-Source Mongolian Text-to-Speech Synthesis Dataset and Accompanied Baseline},
year={2022},
volume={},
number={},
pages={184-189},
doi={10.1109/IALP57159.2022.9961271}
}
E-mail:[email protected]
Tensorflow-TTS: https://github.com/TensorSpeech/TensorFlowTTS