• Thanks to Shanghai Media Tech(SMT) for providing the data set and rendering service, the copyright of digital human presented in our demo is reserved by SMT.
• We use pre-trained HuBERT model from this repository.
• We use implementation of soft-DTW loss from this repository.

The pre-trained HuBERT model is obtained from this repository.

Please download Chinese HuBERT model and put it on directory ./data/pretrained_models/ by executing the following command:

wget -P ./data/pretrained_models/ https://huggingface.co/TencentGameMate/chinese-hubert-large/resolve/main/chinese-hubert-large-fairseq-ckpt.pt

git clone [email protected]:facebookresearch/fairseq.git
cd fairseq
git checkout acd9a53
pip install --editable ./

python utils/generate_hubert.py --input_dir ./data/wavs/[speaker name] --output_dir ./data/wav_features/[speaker name]

python utils/generate_mfcc.py --input_dir ./data/wavs/[speaker name] --output_dir ./data/wav_features/[speaker name]

run bash train.sh to train

• arch: chinese_hubert_large | mfcc | pgg
• feature_combine: True if you want to use weighted sum of hubert feature
• output_path: "result" if you want to generate output of test set | [other name] if you want to generate other data
• test_input_path: you should explicitly assign path of test_input_path if output_path != "result", test_input_path is the dir of csv files
• test_epoch: do not need to explicitly assign, will find the model with best
• root_dir: dir of dataset root
• feature_dir: hubert_large | wav2vec2_large | mfcc | ppg
• train_speaker_list: assign several speaker names for training
• train_json: used to change data resource, path of json file which includes list of audio name in training set
• freq: 50 if feature is chinese_hubert_large or ppg , 100 if feature is mfcc
• input_dim: 39 for mfcc, 128 for ppg

run bash test.sh to test

run bash validate.sh to pick the best model by validating on validation set of certain speaker, change --val_speaker to decide speaker for validation.