mks0601 / integral-human-pose-regression-for-3d-human-pose-estimation Goto Github PK

I replaced HRNet's heatmaps with this Integral Human Pose Regression. And it can't even reach half of the accuracy of the heatmaps version. I've searched the internet, it seems no one other than the original paper has run this model on the COCO dataset. I'm wondering, is this not fit for the COCO dataset and its OKS evaluation standard？I can reach a rather high accuracy under the MPII's PCKh standard.
Even, in HRNet, under the same accuracy, the heatmaps version has doubled mAP than the Integral version.
Note: accuracy value in HRNet is calculated via PCKh, and mAP during the validation process is calculated via OKS.

thank you very much !
what is the gt_vis and gt_have_depth？

I couldn't see how do you pick the joint score if you only use the integral regression? The peak of the map or some kind of weighted average depends on distance?

hello.
I downloaded your pretrained model and run 'python test.py --gpu 0-1 --test_epoch 16', then I encountered error as

Traceback (most recent call last):
  File "test.py", line 94, in <module>
    main()
  File "test.py", line 46, in main
    tester._make_model()
  File "/home/users/ruishan.liu/codes/github/Integral-Human-Pose-Regression-for-3D-Human-Pose-Estimation/main/../common/base.py", line 153, in _make_model
    model.load_state_dict(ckpt['network'])
  File "/home/users/ruishan.liu/installed/anaconda3/envs/python36/lib/python3.6/site-packages/torch/nn/modules/module.py", line 721, in load_state_dict
    self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for DataParallelModel:
	Unexpected key(s) in state_dict: "module.backbone.bn1.num_batches_tracked", "module.backbone.layer1.0.bn1.num_batches_tracked", "module.backbone.layer1.0.bn2.num_batches_tracked", "module.backbone.layer1.0.bn3.num_batches_tracked", "module.backbone.layer1.0.downsample.1.num_batches_tracked", "module.backbone.layer1.1.bn1.num_batches_tracked", "module.backbone.layer1.1.bn2.num_batches_tracked", "module.backbone.layer1.1.bn3.num_batches_tracked", "module.backbone.layer1.2.bn1.num_batches_tracked", "module.backbone.layer1.2.bn2.num_batches_tracked", "module.backbone.layer1.2.bn3.num_batches_tracked", "module.backbone.layer2.0.bn1.num_batches_tracked", "module.backbone.layer2.0.bn2.num_batches_tracked", "module.backbone.layer2.0.bn3.num_batches_tracked", "module.backbone.layer2.0.downsample.1.num_batches_tracked", "module.backbone.layer2.1.bn1.num_batches_tracked", "module.backbone.layer2.1.bn2.num_batches_tracked", "module.backbone.layer2.1.bn3.num_batches_tracked", "module.backbone.layer2.2.bn1.num_batches_tracked", "module.backbone.layer2.2.bn2.num_batches_tracked", "module.backbone.layer2.2.bn3.num_batches_tracked", "module.backbone.layer2.3.bn1.num_batches_tracked", "module.backbone.layer2.3.bn2.num_batches_tracked", "module.backbone.layer2.3.bn3.num_batches_tracked", "module.backbone.layer3.0.bn1.num_batches_tracked", "module.backbone.layer3.0.bn2.num_batches_tracked", "module.backbone.layer3.0.bn3.num_batches_tracked", "module.backbone.layer3.0.downsample.1.num_batches_tracked", "module.backbone.layer3.1.bn1.num_batches_tracked", "module.backbone.layer3.1.bn2.num_batches_tracked", "module.backbone.layer3.1.bn3.num_batches_tracked", "module.backbone.layer3.2.bn1.num_batches_tracked", "module.backbone.layer3.2.bn2.num_batches_tracked", "module.backbone.layer3.2.bn3.num_batches_tracked", "module.backbone.layer3.3.bn1.num_batches_tracked", "module.backbone.layer3.3.bn2.num_batches_tracked", "module.backbone.layer3.3.bn3.num_batches_tracked", "module.backbone.layer3.4.bn1.num_batches_tracked", "module.backbone.layer3.4.bn2.num_batches_tracked", "module.backbone.layer3.4.bn3.num_batches_tracked", "module.backbone.layer3.5.bn1.num_batches_tracked", "module.backbone.layer3.5.bn2.num_batches_tracked", "module.backbone.layer3.5.bn3.num_batches_tracked", "module.backbone.layer4.0.bn1.num_batches_tracked", "module.backbone.layer4.0.bn2.num_batches_tracked", "module.backbone.layer4.0.bn3.num_batches_tracked", "module.backbone.layer4.0.downsample.1.num_batches_tracked", "module.backbone.layer4.1.bn1.num_batches_tracked", "module.backbone.layer4.1.bn2.num_batches_tracked", "module.backbone.layer4.1.bn3.num_batches_tracked", "module.backbone.layer4.2.bn1.num_batches_tracked", "module.backbone.layer4.2.bn2.num_batches_tracked", "module.backbone.layer4.2.bn3.num_batches_tracked", "module.head.deconv_layers.1.num_batches_tracked", "module.head.deconv_layers.4.num_batches_tracked", "module.head.deconv_layers.7.num_batches_tracked".

I guess maybe it is something wrong with the pytorch version?

hi,
I didn't see where you import mem_info().

args.gpu_ids = str(np.argmin(mem_info()))

thanks for your work.
Is there a demo can be visualized on my image?

Hey, I found the action order in Human36M.py may be wrong.
Orig is
self.action_name = ['Directions', 'Discussion', 'Eating', 'Greeting', 'Phoning', 'Posing', 'Purchases', 'Sitting', 'SittingDown', 'Smoking', 'Photo', 'Waiting', 'Walking', 'WalkDog', 'WalkTogether']
I think the right order should be like this
self.action_name = ['Directions', 'Discussion', 'Eating', 'Greeting', 'Phoning', 'Photo', 'Posing', 'Purchases', 'Sitting', 'SittingDown', 'Smoking', 'Waiting', 'WalkDog', 'Walking', 'WalkTogether']

If you agree with me.

Nice work!
I am also trying to reproduce this paper's result. I sampled data every 10 frames on h36m dataset. The best MPJPE I got is 55mm which is 6mm higher than papers.
Could you please tell your sample strategy?
Thanks so much.

I'm attempting to run the pretrained model #2 on a custom image and running into sizing issues. The image is RGB of size 256x256 (so the tensor is 3x256x256).

How do you edit the config file to enable these images to be run? The model expects 64x3x7x7 tensors. I edited the config to (256x256) images of depth 3, and I'm getting the following error:

File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 719, in load_state_dict
    self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for DataParallelModel:
	size mismatch for module.head.final_layer.weight: copying a param of torch.Size([54, 256, 1, 1]) from checkpoint, where the shape is torch.Size([1152, 256, 1, 1]) in current model.
	size mismatch for module.head.final_layer.bias: copying a param of torch.Size([54]) from checkpoint, where the shape is torch.Size([1152]) in current mod

Excuse me, I am trying to run your code to predict 3D keypoints and then to draw them in the 3D coordinate，but I failed. I don't understand some parameter of several functions such as 'vis_3d_skeleton( )'..，can you please give me a point?

예를 들어 R_Hip이
사진 속 사람 입장에서 오른쪽 hip인가요?!
아니면 제가 사진을 보는 입장에서 제 오른손쪽 방향의 hip을 말하는 것인가요?!

Hi!

Great work!

How do I get mpii annotations in required format? How do I get Human3.6M in required format?

I use the method of Integral-Human-Pose-Regression to train coco dataset, but the performance drops a lot. So I wonder if it is normal? Thank you!

Hi,

I would like to use your model to estimate 3D joints from my images, can you give me some example about how to modify main/test.py?

Thank you!

개인적인 dataset으로 트레이닝 시킨 모델에
human36을 넣어 테스트 시켰을 때 발생한 오류의 원인이 조인트 갯수가 달라서임을 알고
코드를 수정해서 테스트를 시켰고 성공은 했습니다.

그런데 결과가 원했던 것보다 좋지 않은데,

제가 고친 코드들 이외에 더 손을 봐야할 부분이 있는 걸까요..?

위 사진과 같이,
process_world_coordinate 함수에서 9번 인덱스에 해당하는 nose만 빼고
그 다음 ground truth부터 joint_cam에 저장되도록 코드를 고쳤고요,(현재 joint_num = 17)

human36.py의 __init__부분에서도 nose에 대한 정보를 뺐습니다.

개인적으로 갖고 있는 dataset의 image 품질이 그렇게 좋지는 않은데, 그것 때문일까요..?
아니면 코드상 놓친 부분이 있을지 궁금합니다..

hello.
I download code Release-v1.1 from human3.6m dataset. and I put preprocess_h36m.m in that folder and run it.
Then I encountered this error: undefined function ''rotationMatrix''
could you please help me?

개인적인 dataset으로 모델을 트레이닝 시킨 후에
그 모델에 human3.6m으로 테스트시키니까 아래와 같은 오류가 발생했습니다.

가충치와 바이어스 갯수가 안맞는 것 같은데
원인와 해결책에 대한 자문좀 구합니다..

혹시 제가 가진 dataset의 특성과도 관련이 있을까요..?

RuntimeError: Error(s) in loading state_dict for DataParallelModel:
size mismatch for module.head.final_layer.weight: copying a param with shape torch.Size([1088, 256, 1, 1]) from checkpoint, the shape in current model is torch.Size([1152, 256, 1, 1]).
size mismatch for module.head.final_layer.bias: copying a param with shape torch.Size([1088]) from checkpoint, the shape in current model is torch.Size([1152]).

Hi, thank you for sharing.
I followed your readme , put preprocess_h36m.m in the folder Release-v1.1/ and run it. However I got the following error. I wonder if I used the wrong raw dates, or maybe there's other mistakes. Could you help me find it? Thank you very much.

I download the following dates from the official website of Human3.6M.

and the raw dates are recognized as following form:
--raw date
------S1
----------MyPoseFeatures
----------Poses_D2_Positions
----------Poses_D3_Positions_mono
----------Poses_D3_Positions_mono_universal
----------Videos
------S2
----------MyPoseFeatures
----------Poses_D2_Positions
----------Poses_D3_Positions_mono
----------Poses_D3_Positions_mono_universal
----------Videos
……

Thank for your simple implementation of integral human pose, i don't understand why the accu_x/y/z need to subtract num 1, int the code nets/loss.py (28~30 lines)

I found your code that generates 2d bounding box of h36m as common/utils/pose_utils.py#L55-L61. However, there exist some scenes that 2d bbox by this way may not cover person body completely. Can you explain why you choose this way?

I replaced the HRNet's heatmaps supervision with integral regression. But after painstakingly debugging, I only get half of the precision of the heatmaps version. Is it the integral regression problem or my code?

Interestingly, this paper, simple baselines and HRNet are written by the same research group, even have the same author Bin Xiao. I'm wondering, why they didn't use integral regression on simple baselines and HRNet? or they found it was not as good as they thought?

Hi,

This network uses the predicted 2D keypoints, the predicted z-value, and the intrinsic matrix to inversely project the 2D keypoints into 3D camera space.

My question is regarding the effects of certain image augmentations on the final 3D projection.

Assuming we have the intrinsic matrix K of the camera used to take the original (unaugmented) image, and the 2D keypoint predictions of the augmented image (the model output), do the following augmentations to the image affect the final, projected 3D point in camera space (calculated using the original intrinsics K)?:

• Padding (I have an image of size 256x192 and have added black borders as padding on the left hand side of the image to make it 256x256)
• Scaling (I have an image of size 256x192 and have scaled it down to 64x48 or scaled it up to 512x384)
• Rotation (I have an image and rotate it N degrees)
• Center Crop / Non-centered crop (I have an image of size 256x192 and have cropped it to the middle of the image (64x48) / I have an image 256x192 and have cropped it to an arbitrary part of the image (64x48))

Have you tested in the wild videos?

Hi, in main/train.py there is "from base import Trainer", but I can not find the 'base' file.

Integral regression performs much better than heatmap regression, especially at lower resolution heatmaps.

I want the network to output low resolution heatmaps for pose estimation.

I have a question regarding the training of the network:

Would it be possible for me to train my network regularly using the heatmap loss (i.e. taking per-pixel squared error), but then at inference time, use soft-argmax (instead of plain argmax) to accurately compute the joint (x,y) locations?

Would there be a degradation in accuracy if I do this, compared to supervising the training procedure with soft-argmax instead of heatmap loss?

When I run preprocess_h36m.m file to process human36 data error:Undefined function or variable 'H36MSequence'What should I do

Hi , thanks for sharing.

When training , I met the following errors. When testing , I could test with two gpus successfully. I couldn't find out what's wrong. Please advise.

python train.py --gpu 0,1 --continue
python train.py --gpu 0-1 --continue
python test.py --gpu 0,1 --test_epoch 16

Hello, I have trouble with training.
How to set experimental parameters and repeat experiments.
I recently retrained your model, but I can't re-implement your results. Is your training setup the same as config.py file?
I trained 16 epochs to get Protocol #2 error (MPJPE) >> 56.27, I want to ask if there is any detail that I am negligent. Thank you.

I have a quick question. The training source is [h36m, mpii] and the tesing set is only h36m. Is it better off to test in batches of mixed datasets while counting only MPJPE error of h36m?

I think there might be some subtle numerical instability caused by testing on a bunch of h36m images instead of h36m + mpii?

Hello, I have registered on the human3.6 official website, but the account has not been manual reviewed for now, so I cannot get the original dataset.
But ,thanks for XingYi Zhou, I got the pre-processed data provided by https://github.com/xingyizhou/pytorch-pose-hg-3d, I wonder whether this dataset can used for this repo?or could you share your pre-processed data? Any tips will be highly appreciated!
Thank you!

Thanks for your work. Very thanks.

I have generated the H36M data and downloaded your provided trained model. However, the value of MPJPE is very large, using your trained model.

As you said, 'set testing set in config.py'. Is there something modified? I just use your provided config without modification.

Thanks very much! ")

hi,@mks0601. Do you have the code of paper about Integral Human Pose Regresison,

hi there,
I try to train with MPII by modifying "trainset = ['mpii']" in config.py and other related path configs but struggled with the error message shown below.
My environment is Ubuntu 16.04, CUDA 10.0, cuDNN 7.4.1, Python3.6.5, Pytorch1.0.0, 2080Ti. Could you please help me fix it? Thanks a lot.

Using GPU: 1
07-05 19:41:15 Creating dataset...
Traceback (most recent call last):
File "train.py", line 93, in
main()
File "train.py", line 36, in main
trainer._make_batch_generator()
File ".../common/base.py", line 92, in _make_batch_generator
trainset_list.append(eval(self.cfg.trainset[i])("train"))
TypeError: 'module' object is not callable

Hi @mks0601, thanks for sharing your great and organized code.
Actually, I am working on 3D pose estimation method using the Human3.6 dataset and I want to jointly regress 3D and 2D joints. My training pipeline consists in:
input : Center crop of the person, 2D joints (2D heatmap), 3D joints (3D heatmap).

1. Centering crop (2d bbox) the person using the 2D annotation data (I added an offset to get a not too tight crop).
2. Resize and pad the crop (without distorsion) to a fixed training size (368, 368).
3. Adjust the 2D annotation coordinates to fit the resized and mapped crop.
   Now, I have to do the same thing for 3D skeleton (annotation):
   Actually, the 3D annotation data is mapped to the original image size. How to do to map the 3D annotation to the image crop obtained in 3) ?
   I try to get the 3D bbox from the 3D annotation data but I can't figure how to resolve the padding and offset? Any clarification