afourast / avobjects Goto Github PK

python main.py --resume checkpoints/avobjects_loc_sep.pt --input_video demo.mp4 --output_dir demo_out
Using device: cuda
rm demo_out/* -rf
Checkpoint checkpoints/avobjects_loc_sep.pt loaded!
Resampling media/demo.mp4 to 25 fps
/opt/anaconda3/envs/avobjects/lib/python3.6/site-packages/torchvision/transforms/functional.py:405: UserWarning: Argument interpolation should be of type InterpolationMode instead of int. Please, use InterpolationMode enum.
"Argument interpolation should be of type InterpolationMode instead of int. "
Traceback (most recent call last):
File "main.py", line 44, in
main()
File "main.py", line 41, in main
trainer.eval(dataloader)
File "/data0/audio_project/github/avobjects/trainer.py", line 60, in eval
mel, _, _, _ = wav2filterbanks(audio.to(self.device))
File "/data0/audio_project/github/avobjects/load_audio.py", line 42, in wav2filterbanks
onesided=True) # b x F x T x 2
File "/opt/anaconda3/envs/avobjects/lib/python3.6/site-packages/torch/functional.py", line 573, in stft
normalized, onesided, return_complex)
RuntimeError: stft input and window must be on the same device but got self on cuda:0 and window on cpu

Thank you for sharing such awesome work!
But how to achieve the demo of Active Speaker Detection as the Project page shows?
I noticed the function viz_boxes_with_scores but I didn't find the corresponding scores computing function.

Hi, i want to ask about data for training.
In, paper, LRS2, LRS3 are used for training and validation.
So, I download LRS2, LRS3 dataset.

These datasets consist of videos that has cropped face from original video.
But, in inference, Input videos are original videos that has not cropped face.

So, I want to ask whether you used cropped video or original video when training the model.

Thanks.

Hi, thanks for share your great work.
Could I have 2 quetions for you?
1 > I have just try to run your code, but the visualization looks like there is something wrong in the code? .i.e I guess the bbox should be located in the face areas but the bbox keep moving around
Do you have any idea about that
https://drive.google.com/file/d/1NTx1qkEQxdS6Ezq4mkIWDsX2ODdhy_I4/view?usp=sharing
2> I run on the other type of dataset like LRS2 and choose the n_speaks=1, but it seems the model seems not well-transferable to this type of dataset when most of the image is the face and neck of 1 person? Do have any idea about that
https://drive.google.com/file/d/1vTkbop-xaNHYGPriv50CJ1aGEBzepcro/view?usp=sharing
Thanks

Dear authors:
Thanks for sharing this fancy idea work, it looks really novel and effective! But when i try to test it on my own video, such as several people meeting video in a room, I found it performs badly. I guess it may due to the relatively small face and not-so-clear sound recorded.
But when i try another good-quality video:

It looks still not-so-good, especially compared with the demo video publish by you. I wonder is there any difference to run the code? or any one come across the same situation??? Thank you.

We are passing videos of different durations through the model and we are getting audio embedding of different sizes. Is this normal ?
We are looking forward to use the audio embeddings generated from your model in our network, but since the embedding size is variable, we cant move forward with it. Is there any solution for it?
Thanks in advance.

Thanks for great work.

I want to train this model for my custom data.
Is there any plan to distribute training code?

Thanks.