radekd91 / emoca Goto Github PK

Dear authors,
I am YJHong and thanks for sharing great work!

I am trying to test the provided image-based Resnet50 emotion recognition module from here

• when I download it, there is a file which zipped "checkpoints/deca-epoch=01-val_loss_total/dataloader_idx_0=~~" file. is it right file ?
• If it is a pretrained emotion recognition module, does the pretrained model contain model parameters including emotion prediction head (7 facial expression, valence, arousal) ? When I see the code (gdl/models/EmoCnnModule), it loads checkpoint only Resnet parts.
  • If then, how can I check predicted expression type and valence, arousal ?

Thank you!

Best regards,
YJHong.

I test EMOCA with some wink expression images, but results seems not good :

Do you have some ideas to improve the result ?

When save_codes flag is set to true, it saves the flame codes for shape, pose, expression codes with dimensions 100, 6, 50 respectively. However, the FLAME 2020 version has 300, 18, 100 dimensional shape, pose, expression respectively. Could you please elaborate or give me a pointer on how the emoca inference of FLAME codes correspond to the FLAME 2020 model? Thank you in advance.

Since no. of vertices and faces increases in detailed mesh generated by emoca, the dimensions of faces and uvfaces mismatch, due to which texture couldn't be overlayed over mesh. Would you please help how to rectify this?

Hi, just a small question what is old_Size variable initialise??? What do this mathematical expression return : -
old_size = (right - left + bottom - top) / 2 * 1.1

Hey I have compiled most of the program now faced issues with ffmpeg but fixed it, but it seems there is a missing cfg.yaml file?

Traceback (most recent call last):
File "demos/test_emoca_on_video.py", line 111, in
main()
File "demos/test_emoca_on_video.py", line 75, in main
emoca, conf = load_model(path_to_models, model_name, mode)
File "/home/mil/Desktop/emoca/gdl_apps/EMOCA/utils/load.py", line 167, in load_model
with open(Path(run_path) / "cfg.yaml", "r") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/home/mil/Desktop/emoca/assets/EMOCA/models/EMOCA/cfg.yaml'

Looking forward to your reply

Hey @radekd91 ,
Thank you so much for this amazing work.

I tried the demo code and saved the exp,pose and shape codes.
I have some questions regarding the same.

Since pose is 6-d vector , I'm interpreting first 3 elements as root axis-angle rotation and last 3 elements
as jaw axis angle representation. Am i interpreting correctly?

Are the exp (containing 50 elements) corresponds to the Flame first 50 expressions parameters?
Are the shape (containing 100 elements) corresponds to the Flame first 100 shape parameters?

Hi @radekd91, Thank you for your great work!
I adjusted the code flame extraction from images such that it can extract flame on the fly. However, it is slow and currently can read only 7 fps. I am trying to improve the speed of detection.

1. I profiled my code and found out that _get_face_detected_torch_image take and get_landmarks_from_image are the major contributors to the delay.

Do you have any suggestion to improve the speed of detection ? For example, increasing the number of workers or setting the persistent_worker flag on? Thank you!

I use the default parameters given in test_flame_extraction_images.py

There is a FaceRecognition model being loaded which has a hardcoded path, correction might help a lot of time debugging and also return emotion features along with emotion class so this model can also be used as an emotion feature extraction model. To get the face recognition model we need to run bash get_assests.sh, script from both Emotion recognition and emocap apps, which is unnecessary.

Hello, I would like to know how to restore the corresponding .png and .obj (such as geometry_detail.png, mesh_coarse_detail.obj, etc.) from the generated .npy (such as exp.npy, etc.) files
thanks for your help

Hi, when I do the video reconstruction and run the test_emoca_on_video.py script, there is an error "ffmpeg._run.Error: ffprobe error (see stderr output for detail)", do anyone know how to fix it? Thx.

Hi there!

Thank you so much for your work, it's remarkable!

When I process the video using your test_emoca_on_video.py, I get frames of shape 224x224, I was wondering if there is any chance to get a larger area, let's say 256x256?

On a different note, can your model work on high res videos - let's say 1024x1024 talking face video(like celeb-hq)?

My all best,
Richard

How should I solve this problem? Looking forward to your reply, thank you!

I successfully ran the EMOCA program and get some output, the result is shown below

But can I only get rendered images? Can i get the intermediate results of the program , such as the character model with the current pose and expression(like a .obj file)?

Hi, how to convert emoca expression latent code to AppleARKit 52 blendshapes?

how to get the txt files in emotioNet_challenge_files_server_challenge_1.2_awsis

Hi,

I am following the steps to install this package and run the emoca demo (test_emoca_on_images), but unfortunately I am meeting some package conflicts (resolving them creates other conflicts, and so on). I tested multiple times on a Ubuntu 20.04 machine, as well as a windows machine using WSL and Ubuntu 20.04 as well.

Starting from scratch on Ubuntu 20.04, I first ran "install_38.sh". Running demos/test_emoca_on_images.py I first meet a missing dependency:

  File ".../dev/emoca/gdl/datasets/AffectNetDataModule.py", line 28, in <module>
    import pandas as pd
ModuleNotFoundError: No module named 'pandas'

Its weird since I see this dependency on conda-environment_py38_cu11_ubuntu.yml. Looking at the installation output, mamba lists pandas as something to be installed, but it never installs it. Anyway, I manually installed it with pip install "pandas>=1.4.2"

Re-running the demo I find another issue:

  File ".../miniconda3/envs/work38/lib/python3.8/site-packages/cv2/mat_wrapper/__init__.py", line 33, in <module>
    cv._registerMatType(Mat)
AttributeError: partially initialized module 'cv2' has no attribute '_registerMatType' (most likely due to a circular import)

Again, this is strange since opencv is listed in the requirements file, one of the last things to be installed. Anyway, I re-installed it manually with pip install --ignore-installed "opencv-python~=4.5.5".

Trying again, I am hit with an potential problem with numpy x tensorboard version (looks like it's a part of the pytorch lightining installation):

 File ".../envs/work38/lib/python3.8/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py", line 569, in <module>
    (np.object, string),
  File ".../envs/work38/lib/python3.8/site-packages/numpy/__init__.py", line 305, in __getattr__
    raise AttributeError(__former_attrs__[attr])
AttributeError: module 'numpy' has no attribute 'object'.
`np.object` was a deprecated alias for the builtin `object`

I tried re-installing these components, but I just kept chasing more dependency issues. I was wondering if other people had similar issues during installation, and what can I do to install it. Thanks!

I think maybe you zip a wrong model to ResNet50.zip, cause i downloaded and unzip it and got a SWIN-T.zip

Hi,
I am not sure if inference speed is mentioned anywhere. Couldn't find it in the paper or supplementary material. Could you share that?
Thanks!

When I run demos/test_emoca_on_images.py --input_folder assets/ --output_folder results --model_name EMOCA --save_images True --save_codes True --save_mesh True, it returns coarse and detailed meshes.

In the first image you can see the coarse mesh, it looks really good, but I want to go further and get the detailed mesh. As you can see, it's broken in the second image. Currently, I'm trying to fix that problem, but if you can give me insights about quick solution it would be great!

Can EMOCA V2 export a camera to go along with the .obj? So when you import it into 3D, it matches up perfectly with the plate?

Hey I got your code working and it is an amazing peace of work great job!

I have questions in regards to the output codecs, for the expression I can see the output is 50 different values, but the FLAME face model has 100 expression shape keys, which shapekeys are the 50 values refering too?

The pose output is 6 different values, is this x,y coordinates for the neck, head and jaw or something different?

Hope to hear from you regards this confusen and again great work! ;)

Is expression transfer possible with this repo? I would like to be able to reanimate the expressions on the tracked face using another tracked face while keeping the identity and position.

when I created the conda environment, i chose python=3.6, but in the environment when i run pip install -e . , it reminds "Package 'gdl' requires a different Python: 3.6.13 not in '>=3.8'". This code can't use python 3.6?

Thanks for contributing such great work!!!

I am interested in EMOCA, and I want to reproduce the training of EMOCA.
However, I can NOT access the two files for splitting the validation/test set of AffectNet, as in

Would you please provide me with these two files?
Thanks.

I am trying to setup a training environment for EMOCA, however in emoca.yaml the "emonet_model_path" used for emotion loss is empty. How should we initialize this ?

Hi Radek,

Thanks for your amazing work!
I saw you used DECA training data and affectnet data.
I'm wondering if you could provide how to preprocess the dataset like vggface2? In DECA, they said they are using
FAN to predict 68 2D landmark
face_segmentation to get skin mask.
But many details are not written, such as whether we need to crop and align the image, what size we need to use for getting landmark and skin mask... Would you please provide a way for us to preprocess the training data?

Thank you!

By the way, vggface2 is still available for academics at this link: https://academictorrents.com/details/535113b8395832f09121bc53ac85d7bc8ef6fa5b/tech&filelist=1
But the VoxCeleb2(https://www.robots.ox.ac.uk/~vgg/data/voxceleb/vox2.html) seems not available...

Hey great work, this is exactly what I have been looking for, I'm able to predict smpl-x model poses pretty well today on an actor, but is there a way to combine the output face model here with that of an smpl-x model. For example in blender?

When I executed python demos/test_emotion_recognition_on_images.py --model_type 3dmm --model_name EMOCA-emorec, it showed AttributeError: 'NoneType' object has no attribute 'train'. Then I looked up the reason and found that there was no pre-trained DECA model.

First of all thank you very much for your work.

During conversion of 2D video using EMOCA v1 I noticed there's excessive upper face movements introduced by EMOCA. For example, upper head, eye brows and nose keep moving although original video doesn't have such movements (mainly keeps head straight and talks). So I was wondering if you're aware of it and if you have any recommendation for solving such issue.

Thanks in advance!

Once the 3D mesh is generated of the source actor, can we use it to modify the expressions of the target actor?

Anyone encounters this issue when running test_emoca_on_video.py?

/work36_cu11/lib/python3.6/site-packages/pytorch3d/renderer/mesh/rasterize_meshes.py", line 320, in forward
cull_backfaces,
RuntimeError: Not compiled with GPU support

I searched it looks like the pytorch3d issue but I have pytorch3D 0.6 installed.

When I was running emoca/gdl_apps/EMOCA/demos/test_emoca_on_images.py, it truned out that :

Hi, thanks for your work. Could you please provide a demo of Google Cloud Disk reconstruction?

Thank you for sharing your code and model with the community ! Is it possible to use this repository (with some arguments maybe?) to run only DECA without the emotional improvements as well ?

When I run the demo of gdl_apps/EMOCA/demos/test_emoca_on_images.py, I get the following errors:

Can u help me to solve this problem?
Many thanks!

I tried to run python demos/test_emotion_recognition_on_images.py --model_type 3dmm --model_name EMOCA-emorec
but always get an error that the model checkpoint does not exist - independent which model I choose.

my working directory is gdl_apps/EmotionRecognition

For example:

FileNotFoundError: [Errno 2] No such file or directory: '/is/cluster/work/rdanecek/emoca/finetune_deca/2021_11_09_19-05-01_5101174495546322475_ExpDECA_Affec_clone_NoRing_EmoB_F2_DeSegrend_BlackB_Aug_early/detail/checkpoints/last.ckpt'

Can EMOCA process video clips in real time? Thanks!

Dear Radek Daněček

Good morning, I have a question about the pipeline. I read DECA and EMOCA, but confused on the EMOCA Detail Stage. The psi comes from 'Expression Encoder' in EMOCA different as in DECA. So can you please tell me does the 'Expression Encoder' fineturned in deatil stage in EMOCA?

EMOCA inference on images/videos reconstructs taking into account the head orientation as seen in the images/videos. Is there a way to output the "unposed" (cancelling out the head orientation) vertex data? I checked with both "trans_verts", "verts" in the vals dictionary hoping one of them would be the unposed data but both of them take into account the head orientation.

Or is there a rigid transformation matrix in the prediction "vals" of EMOCA inference that can be used to "unpose" the vertex data?

Hello, great job. Emoca have a dedicated expression encoder. I want to know where I can find the structure of this encoder? I don't seem to find it in the paper and supplementary materials. Thank you for your answers

There are 465 landmarks in mediapipe facemesh, from which only 105 are taken. Can we know which 105 landmarks are seleted? Is there a mapping in the code?

Hi, just wonder, what's the output format of model? it's directly the mesh or blendshapes that can be transfered to any face model?
that would be very useful if the expressions can transfer to any face model.

Thanks for such a good work. I found that the head pose is calculated on each crop image. Is there an easy way to get the global head pose information, including rotation and translation？

In other words, how to combine crop location information and head pose on cropped image to get the overall head pose？

I have recieved Affectnet data from http://mohammadmahoor.com/affectnet/ but not able to preprocess as it does not contain any training.csv files or validation.csv, please help with this issue .

I could change the expressions of 3D mesh but how to do the same for texture of target.