Hello.
Would you like to share the evaluation code on the FreiHAND dataset, if it is convenient for you?
My email is [email protected] .
Thank you！

Hi,
I found there are indeed some keyerror exceptions of smpl_params as you handled. I'd like to realize how it will be processed when we call the function get_smpl_coord, since smpl_params are None. Is it feasible for me to manually populate zeros onto smpl_param? Or just ignore this none smpl_param annotation? Thanks.

Can you provide the inference code for the video?
Like those showing in the demo :

May I know the sources of the three videos?

I wanted to ask if you'd done any experimentation with a more mobile-friendly version of i2l.

I could image a few modifications to try to reduce the number of parameters and speed up network inference

1. Change backbone from Res50 to a smaller version (Res34/18)
2. Use a MobileNetV2/3backbone
3. Directly regress mesh lixel heatmaps rather than first getting pose lixel heatmaps -> feeding into another resnet + image feature

Have you done any of the following? It seems like some of these have been tried by implementations such as Mediapipe face mesh regression method

I also wanted to ask if you had any data on how integral regression and lixel based heatmap regression speed compares to traditional heatmap based methods.

Thanks!

Hi @mks0601, thanks for your code and data! My question is whether you have compared your smpl fitting results with the one from mosh? For example, Kanazawa used to provide SMPL annotations of Human 3.6M which is computing using mosh. Which one is better?

Hi. Thank you for the great work! I have one doubt. While training, in model.py, the following block initializes weights :

if mode == 'train':
pose_backbone.init_weights()
pose_net.apply(init_weights)
pose2feat.apply(init_weights)
mesh_backbone.init_weights()
mesh_net.apply(init_weights)
param_regressor.apply(init_weights)

However, this isn't down during testing. Maybe this is a silly question but could you please explain why this is so? Because this makes quite a bit of difference in performance during testing.

Thanks in advance

I wonder if someone has tested I2L-MeshNet on real time scenario...
I did that and I got poor fps and wrong mesh positioning. I tried it actually in a docker container, I've encountered many docker-related problems.

do you have any insight regarding I2L-MeshNet in real time scenario?

I tested snapshot_8.pth.tar with a half-portrait image which only contains torso and the above parts, it seems that the output_mesh_lixel.obj is also half-portrait and i think it should be correct, but the output_mesh_param.obj is full-portrait, it produces extra hip and legs part. Any suggestion? Thank you!

The following error occurs whatever the 'gpu_ids' is setten '0' or '0-1' in 'demo.py' file.
However, there will be no error if 'model = DataParallel(model).cuda()' is removed and updated model = model.cuda()
It seems that this program can not implemented for multi-GPUs .
>>> Using GPU: 0,1 Load checkpoint from ../snapshot_8.pth.tar Traceback (most recent call last): File "/home/sunjc0306/sunjc0306/I2L-MeshNet_RELEASE/demo/demo.py", line 93, in <module> out = model(inputs, targets, meta_info, 'test') File "/home/sunjc0306/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__ result = self.forward(*input, **kwargs) File "/home/sunjc0306/anaconda3/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 153, in forward return self.gather(outputs, self.output_device) File "/home/sunjc0306/anaconda3/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 165, in gather return gather(outputs, output_device, dim=self.dim) File "/home/sunjc0306/anaconda3/lib/python3.7/site-packages/torch/nn/parallel/scatter_gather.py", line 68, in gather res = gather_map(outputs) File "/home/sunjc0306/anaconda3/lib/python3.7/site-packages/torch/nn/parallel/scatter_gather.py", line 62, in gather_map for k in out)) File "/home/sunjc0306/anaconda3/lib/python3.7/site-packages/torch/nn/parallel/scatter_gather.py", line 62, in <genexpr> for k in out)) File "/home/sunjc0306/anaconda3/lib/python3.7/site-packages/torch/nn/parallel/scatter_gather.py", line 63, in gather_map return type(out)(map(gather_map, zip(*outputs))) TypeError: expected sequence object with len >= 0 or a single integer

Greatest job!
I would like to ask why the SMPL parameter is needed in the training process. The loss function in the paper does not use the SMPL parameter. If I use my own data set, what should I do

@mks0601 Hi, thank you for your great work.
I had a problem while training the model in the first 'lixel' stage.
I care more about the Human Pose and Mesh Estimation performance, and I've downloaded Human3.6M (from your another project 3DMPPE, they are same stuff, right?), MSCOCO2017 and 3DPW datasets with the links you provided and make them meet the requirements as mentioned in README.md. I haven't downloaded MuCo dataset, so I modify main/config.py like this:

## dataset
# MuCo, Human36M, MSCOCO, PW3D, FreiHAND
# trainset_3d = ['Human36M', 'MuCo'] # MuCo, Human36M, FreiHAND
trainset_3d = ['Human36M'] # only Human36M
trainset_2d = ['MSCOCO'] # MSCOCO
testset = 'PW3D' # Human36M, MSCOCO, PW3D, FreiHAND

besides that, I modified the train_batch_size from 16 to 48.
then I try to execute main/train.py with no more modification of the config:
python train.py --gpu 0-2 --stage lixel
it runs on 3 titan rtx, and everything looks fine, but nan loss occurs in the No.0 epoch:

08-25 21:04:53 Epoch 0/13 itr 353/4336: lr: 0.0001 speed: 0.98(0.97s r0.00)s/itr 1.18h/epoch loss_joint_fit: 0.8628 loss_joint_orig: 0.5176 loss_mesh_fit: 1.1137 loss_mesh_joint_orig: 0.5825 loss_mesh_joint_fit: 0.9378 loss_mesh_normal: 0.0103 loss_mesh_edge: 0.0672
08-25 21:04:54 Epoch 0/13 itr 354/4336: lr: 0.0001 speed: 0.98(0.97s r0.00)s/itr 1.18h/epoch loss_joint_fit: 0.6361 loss_joint_orig: 0.5020 loss_mesh_fit: nan loss_mesh_joint_orig: 0.5407 loss_mesh_joint_fit: 0.6459 loss_mesh_normal: nan loss_mesh_edge: nan
08-25 21:04:55 Epoch 0/13 itr 355/4336: lr: 0.0001 speed: 0.98(0.97s r0.00)s/itr 1.18h/epoch loss_joint_fit: nan loss_joint_orig: nan loss_mesh_fit: nan loss_mesh_joint_orig: nan loss_mesh_joint_fit: nan loss_mesh_normal: nan loss_mesh_edge: nan

I modified the train_batch_size from 16 to 48, so the total iters looks quite small. You can see that loss_mesh_fit, loss_mesh_normal and loss_mesh_edge become nan first, and after that all losses become nan.
I debug the program and find out that the weights of relevant layers all become nan when it happens, so it might be due to the occurence of above nan loss, and then spread to the params of all layers through BP.
I've tried several times, with different gpu numbers (0, 1, 2) and different batch size (8, 16, 32, 48), it always happens at a point in the first epoch (0/13). I thought it might be due to some specific imgs, so I record the batch of imgs that seems to trigger the above nan loss several times (simply log out the paths of them), but they don't seems to be intersect in between.
here is 8 imgs from one attempt, recorded when nan loss occurs, with 1 gpu and train_batch_size=8.

../data/MSCOCO/images/train2017/000000524486.jpg
../data/Human36M/images/s_07_act_06_subact_02_ca_04/s_07_act_06_subact_02_ca_04_000176.jpg
../data/Human36M/images/s_01_act_05_subact_02_ca_04/s_01_act_05_subact_02_ca_04_000946.jpg
../data/MSCOCO/images/train2017/000000573349.jpg
../data/MSCOCO/images/train2017/000000396172.jpg
../data/Human36M/images/s_06_act_07_subact_01_ca_02/s_06_act_07_subact_01_ca_02_000956.jpg
../data/Human36M/images/s_07_act_15_subact_02_ca_03/s_07_act_15_subact_02_ca_03_000061.jpg
../data/Human36M/images/s_06_act_03_subact_01_ca_03/s_06_act_03_subact_01_ca_03_000616.jpg

I'm new to pytorch and HPE, and I'd appreciate your suggestion.

Thanks for your great work.

When I tried to run the 'quick demo' program, the rendered result I got looks a little plain/ordinary, like 'assets/video2.gif'. There isn't much texture or light difference in different human parts.
I hope to get the effect like result in 'assets/qualitative results.png', which has rich texture and shadow diversity in the rendered result.

Could you please me how to achieve such effect? I tried to change the 'metallicFactor' and 'color' parameter in 'render_mesh' of 'vis.py', but it doesn't help.

By the way, this is my "plain result", which just looks like a 2D segmentation results instaed of 3D rendered result.

Have you done an estimation of the time it takes for the inference + rendering? Is it possible to use this model in real time applications, for example ~10 FPS? How much will the inference time increase if we have multiple people?

Let's say I am using YOLOv4 as a detection pipeline, so we can ignore the detection overhead.

Hi, Gyeongsik, I read your I2L-MeshNet paper recently. This work is a really great and interesting work, and I hope to follow this work in the future.

However, when I tried to deploy the "pyrender" package in our GPU server, I met a problem. Could you please help me?

When I tried to run an easy commond: import pyrender;pyrender.OffscreenRenderer(1,2,1), I got the error

File "/home/username/anaconda3/lib/python3.7/site-packages/pyrender/offscreen.py", line 149, in _create
self._platform.init_context()
File "/home/username/anaconda3/lib/python3.7/site-packages/pyrender/platforms/pyglet_platform.py", line 45, in init_context
'internal error message was "{}"'.format(e)
ValueError: Failed to initialize Pyglet window with an OpenGL >= 3+ context. If you're logged in via SSH, ensure that you're running your script with vglrun (i.e. VirtualGL). The internal error message was "Cannot connect to "None""

The GPU server is a remote machine and we log on it with SSH usually. I tried to follow the instruction in "pyrender“ to build osmesa from source code but failed.

I'm looking forward to know if you met this problem ever and how did you solve it? Could you please give me some suggestions?

Thanks.

I want to attain I2L rendered meshes for my dataset of images which involve more than 1 person. In your demo code, you have given some values for the bounding box of the input image

I have some images where multi-person are involved like this:

Do I need to train the model again for the multiperson scenario with bounding box coordinates obtained via Fast-RCNN? Can you add the code for Fast-RCNN model? I am unable to find the relevant code in train.py

Hi,
Im trying to download the pretrained models from https://drive.google.com/drive/folders/14SJKZuCudli2xNN6pSh1bJepIZYlBfSE however im getting the error: The number of downloads was exceeded.

Is there any other way to download pretrained models?

Best regards,

I had a question regarding the following ParamRegressor part of the code.

First, why are you concatenating a zero tensor of shape (batch, 3, 1) to the 2nd axis? I see that you're converting from rotation matrix to angle axis, and you're using torch geometry (which is kornia now I think?). Is this just to convert the rotation matrix to a transformation matrix with 0 translation to be compatible with torch geometry?

        pose = torch.cat([pose,torch.zeros((pose.shape[0],3,1)).cuda().float()],2)
        pose = tgm.rotation_matrix_to_angle_axis(pose).reshape(-1,72)

It also looks like the reshape to 72 only works for SMPL estimation and it should be 48 (16*3) for mano?

Thanks for your great work!
I want to modify your demo.py to check FreiHand model. But I found it needs the camera focal parameter and the root_depth. I guess them from the freihand annotations, but I think that is not accurate. Can you give me some help? And I may check your model performance on other hands benchmark.

Could i know why you use meta_info['is_3D']?
In model.py code, the meta_info['is_3D'] is used as input of "self.coord_loss" when GT is targets['orig_joint_img'].
But i think the code "class CoordLoss" seem no difference whether 'is_3D' is None or not None.
Is there anything i know wrong or don't know?

Hello!

I'm trying to calcualte 3D global pose from pseudo gt smpl parameters.
There are pose parameters and trans in annotation file.

As I know, output of SMPL is global 3D world coordinate.(= fitted_3d_pose in annotation file)
So i did this.

vertices, joints = SMPL(pose param, shape param)
absoulte 3D joints = joints+trans

but it's not fitted_3d_pose.
How to calculate this??

I have download the data from h36m website,but the name of the images does not consistent with the annotation provided by this code.Could I acquire how to match the name of the images?In addition, could I get the action and subaction dict for recording which action/subaction has the specific idx?

Got this exeption after running python demo.py --gpu 0 --stage param --test_epoch 8:
Traceback (most recent call last):
File "demo.py", line 72, in
ckpt = torch.load(model_path)
File "/home/justjo/miniconda2/envs/I2L_meshNET/lib/python3.8/site-packages/torch/serialization.py", line 585, in load
return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "/home/justjo/miniconda2/envs/I2L_meshNET/lib/python3.8/site-packages/torch/serialization.py", line 755, in _legacy_load
magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, '<'.

Pytorch 1.6, Python tested both 3.7.3 and 3.8.5.

The original FreiHand dataset only included the camera instrisic parameters (focal, princpt) and the principal point seems to be different from the original (looks like its half the image height/width vs whatever the calibrated parameters where). How did you go about finding the extrinsic parameters (R, t, scale)?

I see that you got the bounding box info from Detectron2, but what training data did you use to create the hand detector? Did you just use the smallest rectangle that encompassed all the projected 2D keypoints? Did you use a pretrained hand detector on some other data?

Thanks!

1. Is the "annotations" file in the MSCOCO folder under the “data” folder the "2017 Train/Val annotations" file or the "2017 Stuff Train/Val annotations" file
   ‘”

2. Where to download the "J_regressor_coco_hip_smpl.npy" file in the MSCOCO folder under the “data“ folder

3. Where should the Yolo.json file in the Humandetection result folder in the link https://drive.google.com/drive/folders/1fWrx0jnWzcudU6FN6QCZWefaOFSAadgR be placed and what does it do?

4. Can you send me a copy of the "images" folder under the Human36M folder and the "augmented_set" and "unaugmented_set" folders under the Muco folder. I can't download it in your shared link. "Download" will appear ”The file has exceeded the download limit, so it cannot be downloaded at this time", but I am really eager to get this data set to carry out my research work. Can you send a copy to my mailbox, [email protected], very thank.

flake8 testing of https://github.com/mks0601/I2L-MeshNet_RELEASE on Python 3.9.0rc1+

$ flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics

./main/test.py:22:23: F821 undefined name 'mem_info'
        gpus[1] = len(mem_info()) if not gpus[1].isdigit() else int(gpus[1]) + 1
                      ^
./main/train.py:20:23: F821 undefined name 'mem_info'
        gpus[1] = len(mem_info()) if not gpus[1].isdigit() else int(gpus[1]) + 1
                      ^
./common/utils/manopth/manopth/tensutils.py:46:5: F821 undefined name 'List'
    # type: (List[int]) -> List[int]
    ^
./common/utils/manopth/manopth/tensutils.py:46:5: F821 undefined name 'List'
    # type: (List[int]) -> List[int]
    ^
./common/utils/manopth/manopth/rodrigues_layer.py:83:31: F821 undefined name 'th_cv2_rod_sub_id'
    test_function = gradcheck(th_cv2_rod_sub_id.apply, (inputs_var, ))
                              ^
./common/utils/manopth/manopth/rodrigues_layer.py:88:25: F821 undefined name 'th_cv2_rod'
    test_th = gradcheck(th_cv2_rod.apply, (inputs_var, ))
                        ^
./common/utils/manopth/mano/webuser/serialization.py:22:1: F822 undefined name 'save_model' in __all__
__all__ = ['load_model', 'save_model']
^
./common/utils/manopth/mano/webuser/serialization.py:38:5: F821 undefined name 'backwards_compatibility_replacements'
    backwards_compatibility_replacements(dd)
    ^
./common/utils/smplpytorch/smplpytorch/pytorch/tensutils.py:52:5: F821 undefined name 'List'
    # type: (List[int]) -> List[int]
    ^
./common/utils/smplpytorch/smplpytorch/pytorch/tensutils.py:52:5: F821 undefined name 'List'
    # type: (List[int]) -> List[int]
    ^
./common/utils/smplpytorch/smplpytorch/pytorch/rodrigues_layer.py:79:31: F821 undefined name 'th_cv2_rod_sub_id'
    test_function = gradcheck(th_cv2_rod_sub_id.apply, (inputs_var, ))
                              ^
./common/utils/smplpytorch/smplpytorch/pytorch/rodrigues_layer.py:84:25: F821 undefined name 'th_cv2_rod'
    test_th = gradcheck(th_cv2_rod.apply, (inputs_var, ))
                        ^
./demo/demo.py:38:23: F821 undefined name 'mem_info'
        gpus[1] = len(mem_info()) if not gpus[1].isdigit() else int(gpus[1]) + 1
                      ^
./data/PW3D/PW3D.py:75:36: F821 undefined name 'joint_cam'
                root_joint_depth = joint_cam[self.root_joint_idx][2]
                                   ^
13    F821 undefined name 'backwards_compatibility_replacements'
1     F822 undefined name 'save_model' in __all__
14

https://flake8.pycqa.org/en/latest/user/error-codes.html

On the flake8 test selection, this PR does not focus on "style violations" (the majority of flake8 error codes that psf/black can autocorrect). Instead, these tests are focus on runtime safety and correctness:

• E9 tests are about Python syntax errors usually raised because flake8 can not build an Abstract Syntax Tree (AST). Often these issues are a sign of unused code or code that has not been ported to Python 3. These would be compile-time errors in a compiled language but in a dynamic language like Python, they result in the script halting/crashing on the user.
• F63 tests are usually about the confusion between identity and equality in Python. Use ==/!= to compare str, bytes, and int literals is the classic case. These are areas where a == b is True but a is b is False (or vice versa). Python >= 3.8 will raise SyntaxWarnings on these instances.
• F7 tests logic errors and syntax errors in type hints
• F82 tests are almost always undefined names which are usually a sign of a typo, missing imports, or code that has not been ported to Python 3. These also would be compile-time errors in a compiled language but in Python, a NameError is raised which will halt/crash the script on the user.

I see there is some numeric variables like 'mesh_param_img' but i'm note sure if this is correct to generate a skeleton representation of the 3D model.

Has anyone had this error when trying to run demo?

Traceback (most recent call last):
File "/home/yzhao/anaconda3/envs/i2l/lib/python3.8/site-packages/pyglet/init.py", line 334, in getattr
return getattr(self._module, name)
AttributeError: 'NoneType' object has no attribute 'Window'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/yzhao/anaconda3/envs/i2l/lib/python3.8/site-packages/pyglet/init.py", line 334, in getattr
return getattr(self._module, name)
AttributeError: 'NoneType' object has no attribute '_create_shadow_window'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "demo.py", line 19, in
from utils.vis import vis_mesh, save_obj, render_mesh
File "/home/yzhao/yzhao/I2L-MeshNet_RELEASE/main/../common/utils/vis.py", line 7, in
import pyrender
File "/home/yzhao/anaconda3/envs/i2l/lib/python3.8/site-packages/pyrender/init.py", line 12, in
from .viewer import Viewer
File "/home/yzhao/anaconda3/envs/i2l/lib/python3.8/site-packages/pyrender/viewer.py", line 36, in
class Viewer(pyglet.window.Window):
File "/home/yzhao/anaconda3/envs/i2l/lib/python3.8/site-packages/pyglet/init.py", line 340, in getattr
import(import_name)
File "/home/yzhao/anaconda3/envs/i2l/lib/python3.8/site-packages/pyglet/window/init.py", line 1897, in
gl._create_shadow_window()
File "/home/yzhao/anaconda3/envs/i2l/lib/python3.8/site-packages/pyglet/init.py", line 340, in getattr
import(import_name)
File "/home/yzhao/anaconda3/envs/i2l/lib/python3.8/site-packages/pyglet/gl/init.py", line 95, in
from pyglet.gl.lib import GLException
File "/home/yzhao/anaconda3/envs/i2l/lib/python3.8/site-packages/pyglet/gl/lib.py", line 149, in
from pyglet.gl.lib_glx import link_GL, link_GLU, link_GLX
File "/home/yzhao/anaconda3/envs/i2l/lib/python3.8/site-packages/pyglet/gl/lib_glx.py", line 46, in
glu_lib = pyglet.lib.load_library('GLU')
File "/home/yzhao/anaconda3/envs/i2l/lib/python3.8/site-packages/pyglet/lib.py", line 164, in load_library
raise ImportError('Library "%s" not found.' % names[0])
ImportError: Library "GLU" not found.

Tested with both python3.7 and python3.8.

Thank you for your awesome work!

I have a question.
I know that the smpl pseudo gt paramter provided in this repository was obtained by fitting it to smplify-x.
Is it reasonable to use the provided pseudo gt in the smpl model?

Hi @mks0601,

Thanks a lot for sharing the Simplify-X fits for H36M. I was checking the quality of fits and it seems quite off when overlayed on the images.

Here is an example:

Is it what you were also getting or am I doing something wrong?

Best,
Umar

Hi, thank you very much for your excellent job!

Is it feasible to convert the MSCOCO 3d meshes to 3d skeleton? Could you please give me advice? Thank you very much for your help!

Thanks for the great paper, I read that your paper suggests 1d heatmap over 3d heatmap since number of mesh vertex is much larger and therefore predicting 3d heatmaps of all mesh vertices become computationally infeasible.
However what if I only need is the PoseNet and not the MeshNet and therefore I'm only predicting relatively less number of joints (such as under 20) is there any further advantage by using 1d heatmap instead of 3d heatmap?

or what if I want to train on 2d human pose with 1d heatmap coordinate representation, can it be better than 2d heatmap in terms of performance or maybe gpu memory?
I would like to know your insights.

I'm trying to modify the demo to checkout the trained Freihand model but get the following error

RuntimeError: Error(s) in loading state_dict for DataParallel:
        Missing key(s) in state_dict: "module.param_regressor.fc.0.weight", "module.param_regressor.fc.0.bias", "module.param_regressor.fc.1.weight", "module.param_regressor.fc.1.bias", "module.param_regressor.fc.1.running_mean", "module.param_regressor.fc.1.running_var", "module.param_regressor.fc.3.weight", "module.param_regressor.fc.3.bias", "module.param_regressor.fc.4.weight", "module.param_regressor.fc.4.bias", "module.param_regressor.fc.4.running_mean", "module.param_regressor.fc.4.running_var", "module.param_regressor.fc_pose.0.weight", "module.param_regressor.fc_pose.0.bias", "module.param_regressor.fc_shape.0.weight", "module.param_regressor.fc_shape.0.bias", "module.human_model_layer.th_betas", "module.human_model_layer.th_shapedirs", "module.human_model_layer.th_posedirs", "module.human_model_layer.th_v_template", "module.human_model_layer.th_J_regressor", "module.human_model_layer.th_weights", "module.human_model_layer.th_faces". 

I downloaded pretrained snapshot_24 and it looks like its missing all of the param regressor weights. Something I'm doing wrong?

Hi @mks0601,

Thank you for the great work, I look forward to experimenting with it !
I want to try your model on some different hand images, and I was therefore wondering if you could share the model of the trained hand RootNet ? (I searched a link to such a pre-trained model both in this repo and https://github.com/mks0601/3DMPPE_ROOTNET_RELEASE)

All the best !

Yana

Looks like whenever the demo was updated to include the mesh rendering visualization, the accompanying render mesh function with utils.viz was not added. Getting the following error

Traceback (most recent call last):
  File "/home/deeplearning/0Dev/repos/I2L-MeshNet_RELEASE/demo/demo.py", line 19, in <module>
    from utils.vis import vis_mesh, save_obj, render_mesh, optimize_camera
ImportError: cannot import name 'render_mesh' from 'utils.vis' (/home/deeplearning/0Dev/repos/I2L-MeshNet_RELEASE/main/../common/utils/vis.py)

Hello,

first of all thanks for the great work! I was able to run the code without any problems and the pose estimation seems to be working fine.

Is it possible to use another mesh for the visualization - for example: a humanoid cartoon character? And what would be the steps to make this possible? I suppose I could still use the human mesh for the estimation and somehow transfer the estimated pose to another mesh via keypoints, but would that be a good idea? I would appreciate some pointers in the right direction!

Trying to run the demo using

python demo.py --gpu 0 --stage param --test_epoch 8

Getting the following error

/home/deeplearning/miniconda3/envs/i2l/bin/python /home/deeplearning/0Dev/repos/I2L-MeshNet_RELEASE/demo/demo.py --gpu 0 --stage param --test_epoch 8
>>> Using GPU: 0
/home/deeplearning/0Dev/repos/I2L-MeshNet_RELEASE/main/../common/utils/smplpytorch/smplpytorch/pytorch/smpl_layer.py:41: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at  /opt/conda/conda-bld/pytorch_1595629395347/work/torch/csrc/utils/tensor_numpy.cpp:141.)
  torch.Tensor(smpl_data['betas'].r).unsqueeze(0))
Load checkpoint from /home/deeplearning/0Dev/repos/I2L-MeshNet_RELEASE/output/model_dump/h36_smpl/snapshot_8.pth.tar
Traceback (most recent call last):
  File "/home/deeplearning/0Dev/repos/I2L-MeshNet_RELEASE/demo/demo.py", line 96, in <module>
    out = model(inputs, targets, meta_info, 'test')
  File "/home/deeplearning/miniconda3/envs/i2l/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/deeplearning/miniconda3/envs/i2l/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 153, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/home/deeplearning/miniconda3/envs/i2l/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "../main/model.py", line 76, in forward
    pose_param, shape_param = self.param_regressor(joint_img_from_mesh.detach())
  File "/home/deeplearning/miniconda3/envs/i2l/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/deeplearning/0Dev/repos/I2L-MeshNet_RELEASE/main/../common/nets/module.py", line 124, in forward
    pose = tgm.rotation_matrix_to_angle_axis(pose).reshape(-1,72)
  File "/home/deeplearning/miniconda3/envs/i2l/lib/python3.8/site-packages/torchgeometry/core/conversions.py", line 233, in rotation_matrix_to_angle_axis
    quaternion = rotation_matrix_to_quaternion(rotation_matrix)
  File "/home/deeplearning/miniconda3/envs/i2l/lib/python3.8/site-packages/torchgeometry/core/conversions.py", line 302, in rotation_matrix_to_quaternion
    mask_c1 = mask_d2 * (1 - mask_d0_d1)
  File "/home/deeplearning/miniconda3/envs/i2l/lib/python3.8/site-packages/torch/tensor.py", line 396, in __rsub__
    return _C._VariableFunctions.rsub(self, other)
RuntimeError: Subtraction, the `-` operator, with a bool tensor is not supported. If you are trying to invert a mask, use the `~` or `logical_not()` operator instead.

Process finished with exit code 1

I thought maybe it had something to do with the models not having write permission, but I made sure to change that and still encounter this. Anyone else seen this?

I am having trouble when running MeshNet + RootNet on hands.
I downloaded the rootnet model, from this repo and cloned rootnet. When I run the rootnet demo with the model provided here, an error is shown
RuntimeError: Error(s) in loading state_dict for DataParallel: Missing key(s) in state_dict: "module.root.deconv_layers.0.weight", "module.root.deconv_layers.1.weight",....
How can I run RootNet with the trained model you provide in this repo?
thank you.

So I wanted to ask how one goes about getting the mano pose/shape parameters from the network output so that we can use outputs for graphics applications for joint angles and to ensure length preservation. As I see it there are two methods I can think of and didn't know if there was something I was missing

1. Training a param regressor head (seems like you've done this already for SMPL parameters but not for MANO). A similar approach to this repo
2. Using inverse kinematics with something like a Levenberg-Marquart solver. Like this or the MEgATrack paper

Is there anything I'm missing? Either way, I appreciate all the help given so far

Hello!
I want to use other datasets to achieve some of my own tasks (not tasks related to hands and bodies). How to use my own dataset? What preparations need to be done? Can you give specific guidance?
Thank you in advance for your reply.

Hi, thanks for your great work.
The paper only says ||P-P*|| (8), but at the code, you used both GT joint coordinate and SMPL joint coordinate for predicting joint. so would you let me know why you use SMPL joint coordinate? Is GT joint an unreliable data?
thanks you.

Hi Gyeongsik,

I am working on training FreiHand and other datasets of hands.
While training on Freihand, I found that the losses (sum(loss[k] for k in loss), and all k in the loss for train.py) is not converging at all while training (even after 10 epoch).
I didn't modify the code except for the self.data_path in FreiHand.py, and I have tried to visualize the image and labels in the dataset, it seems there is no problem with it.
Please help, and thanks for all your contribution :)

Thanks for sharing your great job! I tried to download H3.6M image files from https://drive.google.com/drive/folders/1cx_BuF0tDcNww5OY0pLPn8r8G00NtFag

However, it reports an error:
Download quota exceeds for this file. So you can't download it at this time.

Could you help to fix this problem? Thanks.

Thank you for sharing your code.
Can you also share your rendering script that used for visualization?

Hi, I have one more question.

How did you find the smpl face vertex index such as ears, eyes and noes?

How to Create 3D Human model from 2d Human Image?

Share sample code from your repo?

Hi,

Thank you for the great work. I had one doubt. Some of the bounding box coordinates that I am getting as output from "process_bbox" function are negative. Is this deliberate? One example of such an instance is for the image "s_05_act_04_subact_01_ca_04_003112.jpg". For this image, the bounding box origin coordinates, width and height are :
array([ -6.87618172, 84.10320063, 606.78921781, 606.78921781]).
I found this out because I was trying to save the cropped images according to the bounding box coordinates.

Looking forward to your response