Can i use python to implement the post processing stage of the pose estimate ? And then use tensorflow to save it to .pb file.And then only use a little interface in java to load the model and give input ,directly get the output Image

Hi, first thanks for your awesome codes to implement such mobile-pose. I just tried the code and run webcam demo. I found that even there is no people appeared in my webcam, the pose also will be estimated and published. I checked the net output, it is a tuple with two parts, the first one is keypoints location, another I didn't figure out (it is a 52*52 data array for each joint I think.). I have no idea how to only display keypoints as long as there is a people there. Thanks.

#28
After completing the steps above, gpu usage is very low
it this a cpu bottleneck?
i want a extreme speed Beyond real-time

on my gpu(1080ti) it's only 45 fps.
gpu usage is 4~5%, vram usage is 500mb

hi, I tried to train your code with few modification

• set resnet input size to 224x224
• set learning_rate to 1e-03 with adam optimizer

and the loss seems reasonable

[epoch 139] train loss: 11.97749722, valid loss: 302.35555648
[epoch 141] train loss: 12.36122017, valid loss: 300.01421413
[epoch 143] train loss: 12.04360196, valid loss: 297.55886086
[epoch 145] train loss: 11.31563594, valid loss: 300.16024249
[epoch 147] train loss: 10.84763370, valid loss: 291.31610589
[epoch 149] train loss: 10.44486987, valid loss: 290.63944685
[epoch 151] train loss: 11.03760799, valid loss: 300.89297566
[epoch 153] train loss: 10.26842959, valid loss: 293.75293954
[epoch 155] train loss: 9.67422970, valid loss: 288.11433823
[epoch 157] train loss: 9.24360814, valid loss: 296.10527976
[epoch 159] train loss: 9.42581674, valid loss: 296.30135754
==> checkpoint model saving to ./models/resnet/final-aug.t7

but when I tried to test, the result is strange


is this reasonable ?

I get an error in the dsntnn library when I try to do inference on a model.

modelname = "mobilenetv2"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

net = CoordRegressionNetwork(n_locations=16, backbone=modelname).to(device)
net(torch.ones([1,3, 224,224], dtype=torch.float, device=device))

File "/Users/Harald/Github/MobilePose-pytorch/inference_test.py", line 16, in
net(torch.ones([1,3, 224,224], dtype=torch.float, device=device))
File "/Users/Harald/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in call
result = self.forward(*input, **kwargs)
File "/Users/Harald/Github/MobilePose-pytorch/network.py", line 55, in forward
coords = dsntnn.dsnt(heatmaps)
File "/Users/Harald/anaconda3/lib/python3.6/site-packages/dsntnn/init.py", line 79, in dsnt
return soft_argmax(heatmaps)
File "/Users/Harald/anaconda3/lib/python3.6/site-packages/dsntnn/init.py", line 67, in soft_argmax
return linear_expectation(heatmaps, values).flip(-1)
RuntimeError: expected flip dims axis >= 0, but got min flip dims=-1

Can I reuse parts of this code?

Can you please share the pretrained model?

i am running run_webcam.py and its giving me correct and accurate output on different images but resulted skeleton is rotated at angle. i think its related to pose_fun lambda function in estimator.py which is running after image inference. I have tried different models but same result. Currently i am using resnet18. Thank you

I tried it on my mac, with my cam,but get nothing , only fixed human cooridnate

I want change heat map range to 0~255 for heatmap visualization.

I guessed heatmap range is 0~1 and tried multiply 255, but it was not.
When multiplied by 255, the heatmap's maximum value was about 30.

pleas let me know about heatmap value's range!

Hello!

I have tried to download the pretrained model and failed. Seems like there is a broken link.
Could you please look into it?

训练过程中不存在图中的关节点（如某部位不在框内或者被遮挡了），热点图不存在该位置响应，进入F运算后（0，0），得到是中心点，显然是不对的，作者有解决方法吗？

Why didn't save the model

I think some of the files are from previous versions of the code base.
When I run the training command I get
"AttributeError: 'CoordRegressionNetwork' object has no attribute 'outsize'"
Would it be possible to push the newest version of the code?
Thanks

mobilenet2?or?

I run a rgb image for test using mobilenetv2 model and get totally incorrect visible result (though evaluation AP is good).
I made some modifications in pose encode and decode parts in order to fit DSNTNN.
Anybody got good visible results?

Traceback (most recent call last):
File "training.py", line 22, in
from dataloader import *
File "/unsullied/sharefs/liangjiajun/workspace/research-2018/MobilePose-pytorch/dataloader.py", line 181, in
import imgaug as ia
ImportError: No module named 'imgaug'

when i train it, i cant find the pretrain model named 'final-aug.t7'

Hi,
when i try run run_webcam.py module raise this exception. I install imgaug for this way:

pip3 install imgaug

pip3 install imgaug
Defaulting to user installation because normal site-packages is not writeable
Collecting imgaug
  Using cached imgaug-0.4.0-py2.py3-none-any.whl (948 kB)
Requirement already satisfied: matplotlib in /home/mpg-agx/.local/lib/python3.6/site-packages (from imgaug) (3.2.1)
Requirement already satisfied: scikit-image>=0.14.2 in /home/mpg-agx/.local/lib/python3.6/site-packages (from imgaug) (0.16.2)
Requirement already satisfied: imageio in /home/mpg-agx/.local/lib/python3.6/site-packages (from imgaug) (2.8.0)
Requirement already satisfied: Pillow in /home/mpg-agx/.local/lib/python3.6/site-packages (from imgaug) (6.2.2)
Requirement already satisfied: scipy in /usr/local/lib/python3.6/dist-packages (from imgaug) (1.4.1)
Requirement already satisfied: numpy>=1.15 in /usr/local/lib/python3.6/dist-packages (from imgaug) (1.18.2)
Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from imgaug) (1.14.0)
Collecting Shapely
  Using cached Shapely-1.7.0.tar.gz (349 kB)
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-ima7o4jb/Shapely/setup.py'"'"'; __file__='"'"'/tmp/pip-install-ima7o4jb/Shapely/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-ima7o4jb/Shapely/pip-egg-info
         cwd: /tmp/pip-install-ima7o4jb/Shapely/
    Complete output (11 lines):
    Failed `CDLL(libgeos_c.so.1)`
    Failed `CDLL(libgeos_c.so)`
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-ima7o4jb/Shapely/setup.py", line 85, in <module>
        from shapely._buildcfg import geos_version_string, geos_version, \
      File "/tmp/pip-install-ima7o4jb/Shapely/shapely/_buildcfg.py", line 170, in <module>
        fallbacks=['libgeos_c.so.1', 'libgeos_c.so'])
      File "/tmp/pip-install-ima7o4jb/Shapely/shapely/_buildcfg.py", line 164, in load_dll
        libname, fallbacks or []))
    OSError: Could not find library geos_c or load any of its variants ['libgeos_c.so.1', 'libgeos_c.so']
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

how to cumpute value of confidence

I changed ".to("cpu")" code to ".to("cuda")."
but it print below error and not work

RuntimeError: Expected object of backend CPU but got backend CUDA for argument #2 'weight'

how i can run this code use gpu?
i really want your advice
thank you!

I saw your data augmentation using Expansion by append two meshgrid channels, what is the purpose ?

Hi~ Recently, I have read a lot papers related to keypoint localization and found out some of them could be categorized into three kinds based on loss ( regerssion directly, dense predicted by heatmap, and one-shot semantic segmentation). I am curious about which way is better.

It seems your work aims to regress 16 point directly. is this possible to get better performance by predicting heatmap densely?

Hi ，Mr. Xiu, I wanna know the epoches u have trained when its backbone was mobilenetv2 model .
. thanks a lot

I would like to convert the model to CoreML to use it on the iPhone.
But the DSNTNN layer is not supported because of its flip and linspace pytroch operands.
Would it be possible to implement those operands in a different way, as such that it does convert?
I am also looking to implement a similar layer to do coordinate and heatmap based regression, as in https://github.com/JimmySuen/integral-human-pose.
These models are compatible with CoreML.

I found that in requirements.txt, the last line of skimage==0.0 will raise an error. It seems to conflict with scikit_image==0.13.1 and should be removed.

hi, yuliang, I test the resnet model, but got very poor accuracy

original input is:

Thanks for the development of this good repo.
Is there any technical report associated with this repo and is there any description (which dataset, validation/test set, pck acc?)of
the resort you show in the table?

the results are which dataset's? MPII? coco?

Hi, @Fangyh09

On the process to deploy this mobilepose for mobile use (raspberry pi), I realized _make.so at MobilePose-pytorch/cocoapi/PythonAPI/pycocotools/ is 64bit file.
I need to get 32bit version to run on it.

So, I tried to create 32bit _mask.so, but when I use it, I get following error.

daisuke@daisuke-Z170X-UD3:~/MobilePose-pytorch$ python eval_psl.py
Traceback (most recent call last):
File "eval_psl.py", line 27, in
from pycocotools.coco import COCO
File "/home/daisuke/MobilePose-pytorch/pycocotools/coco.py", line 55, in
from . import mask as maskUtils
File "/home/daisuke/MobilePose-pytorch/pycocotools/mask.py", line 3, in
import pycocotools._mask as _mask
ImportError: /home/daisuke/MobilePose-pytorch/pycocotools/_mask.so: undefined symbol: rleEncode

Also, I noticed that I cannot reproduce 64bit one either.
My 64 bit _make.so is quite smaller (original is 820kb, but mine is 256kb) and when I use it, I get same error above. (To create 64 bit _make.so, I used following command: gcc -shared -fPIC _mask.c -o _mask.so)

Could you tell me how did you create _make.so?
If you would tell it, I could reproduce 62bit version and prepare 32bit version by myself.

Thank you.

请教，MobileNetV2的训练结果，heat和coor都降不下，即便跑到70epochs

Hi @YuliangXiu , thanks for your kindly sharing. One question: Does this repo support train with coco dataset?

Before i want to transplant the openpose to the mobile used mobileNet-V1 and tensorflow, then i train the model and get the pb file.But only can use the file in android to get the paf and heatmap,so i have to use java to implement it where connects the joint. So if i want to use python, cant you tell some advice?

Thanks for sharing your great work!
I have not found your pre-trained model, can you upload your well-trained model?

Hi,

Could you please elaborate why you need that method?
Thanks!

Hi,
thanks for sharing your repo. And There is a question that confused me a lot, What's the differences between absolute coordinate label and heatmaps label in training process, and which method will generate robust prediction ? I saw many papers transfer the coordinate to heatmaps

model=resnet18 --camera=0
Traceback (most recent call last):
File "run_webcam.py", line 41, in
w, h = model_wh(get_graph_path(args.model))
NameError: name 'model_wh' is not defined
I can't find get_graph_path() either

Then I change the code to be like this:
# load the model
# w, h = model_wh(get_graph_path(args.model)) # w,h not used
modelpath = "./models/resnet18_224_sgd_best.t7" #this is the model I generated using training.py
#e = ResEstimator(get_graph_path(args.model), target_size=(w,h))
e = ResEstimator(modelpath, target_size=(224, 224))

Then I got an error:
Traceback (most recent call last):
File "run_webcam.py", line 44, in
e = ResEstimator(modelpath, target_size=(224, 224))
File "/home/ychen/MobilePose-pytorch-master/estimator.py", line 30, in init
self.net.eval()
AttributeError: 'collections.OrderedDict' object has no attribute 'eval'

The link to the pre-trained model is removed in 73e2b27

Why? Is it still valid?

Hi I am using the default mpii dataset and all the default parameters for training. With mobilenet/resnet, it takes around 8 seconds for each batch (including reading files and bp) with Titan V/1080Ti. Is there anything wrong?
For mobilenet, I already trained 300 epoch, with validation loss around 600. I tested the performance with images from mpii. Almost all the results are in the center of the image like standard position. The same thing for resnet model.
BTW, in the code during inference for resnet, the function "rescale" is slightly different from the one for training, resulting 216, 217 (h!=w) sized image.
The other thing is why use 5 channels in the first layer?

Hi, could you provide an output format like the below one for this so we can see which keypoint location correspondents to which body part (e.g. right knee etc.) ? Thanks

Have you tried combining both dataset LSP and MPII?
LSP has the joint "neck" and MSPII "upper_neck" and "thorax".
from the images it seems neck and upper_neck are semantically the same but I'm not sure.
Do you have any insight in this?