uuembodiedsocialai / facediffuser Goto Github PK

Hi @uuembodiedsocialai ,
thanks a lot for sharing this insightful work !
I wanted to ask if you have plans to release the pretrained weights for the models (particularly the one trained with vocaset). That would be very helpful !

Thanks a lot :)

Thanks for your contribution! Will you give the code for calculate the lip error on vocaset?

when I ran "python predict.py --dataset vocaset ..." .
it reported an error "can't find templates/face_template.obj".
where can I download this .obj file?

print('Diversity: {:.4e}'.format(diversity / num_seq))
ZeroDivisionError: division by zero
why num_seq=0?

Could you please provide some specific code for training and prediction on the BEAT dataset? It seems that this part is not included in the current code.

I tried to modify the code you provided to reproduce the results on the BEAT data set, but the test results were far from those reported in the paper.

It would be appreciated that if you could provide the relevant code and model training weights for the BEAT dataset.

The is a line

listener_path = os.path.join(args.data_path, args.dataset, args.listener_path)

which gives an error because there is no args.listener_path

What is it meant to be? There is nothing in the data that could fit.

Thanks for your interesting work! I want to ask whether the model on BIWI is trained with only e sequences or not-e sequences or both.

In the paper it says only use the emotional sequences, but in the data_loader the training splits is range(1, 33), which IMO is the non-emotional sequences?

Thanks again for the code.

I am one more question about pre-processing the BIWI dataset.
I have requested and obtained the download link of BIWI, which looks like this

It seems like no .flv files are included in the download link, which should be included as suggested in the download page:

• 'videos' contains videos (.flv) of the rendered 3D geometries and original audio (sampling rate: 44.1kHz).

I wonder if i missed something and how the authors handle this problem.
Thanks again if you could kindly reply.

Sorry to disturb.
I see the BIWI dataprepocessing requires only to download the [faces0x.tgz] files, which is different from other preprocessing ways such as CodeTalker that requires other files like [scans0x.tgz].

I wonder if the provided preprocessing code outputs the exactly same precessed files as CodeTalker.

Thanks again for the good work.

mouth_mask = list(range(94, 114)) + list(range(146, 178)) + list(range(183, 192))
upper_mask = [x for x in range(192) if x not in mouth_mask]
In the evalution of the code, I looked for how to calculate the LVE and FDD because the facial vector as a whole, how to know which positional information values represent the lip and the other region.
So how to define the mask?
Thanks.

Thanks so much for the great work!
Could you please let us know the license of this work?

Thanks for the great work and code!
I have one question prior to downloading the Multiface dataset.
Does the training involves using the whole Multiface dataset (which is quite large), or just downloading the mini version suffices?

How long does it take to train on the vocaset dataset?

Overfitting happens when the model is trained on VOCASET. The training loss is descending while the validation loss is rising step by step when training.
So is VOCASET too small? Dose anyone meets the same problem?

1. I guess this should be train+val+test, otherview, I got empty train dataset
   

   FaceDiffuser/data_loader.py

   Line 62 in b1f386f

   all_subjects = args.test_subjects.split() + args.val_subjects.split() + args.test_subjects.split()

2. wrong key building for one hot generation
   

   FaceDiffuser/data_loader.py

   Lines 31 to 33 in b1f386f

   	if self.data_type == "train":
   	subject = file_name.split("_")[0]
   	one_hot = self.one_hot_labels[self.subjects_dict["train"].index(subject)]

Hi

Can you please mention how exactly prediction is done on beat dataset? Please share the command to predict on beat as well.

Also the beat model is not available. Can you please upload the trained model on beat?

Thanks.