Just curious I have been playing around with your model and trying to figure out exactly how you normalized the input images and whether or not you trained the dataset on BGR or RGB images. From looking through the repo it looks like you only normalized by 255 but did not center the images? Also, it seems from the tests that you probably trained the network on RGB images. Lastly, what joints do the corresponding heat maps relate to?

Hi yuanyuanli85,
I trained the network with my own dataset. I didn't test the network on MPII cause I see someone test it. I meet a confused problem, that's the accuracy during training is not stable, varies from 0.20 to 0.89 at most. I can't find the reason. I used Adam and lr is 5e-4. Maybe it is related to the learning rate? Hope to get your reply! Much Thanks!

thanks for the great repo! I'm pretty new and can't seem to understand how your code is intermediately supervising layers. Would you be able to point that out to me? According to the paper, the are two losses being computed for each block - how do you implement this in keras?

I notice that in bottleneck_block， the structure is Conv2D->ReLU->BN, I wonder weather the network works well because all residual_block I used to see is Conv2D->BN->ReLUs?

I want to use only 6 key points, not a 16 key point pre-trained model.
So I ran the code with scratch.

but I keep getting errors like photos.

Any comments would be greatly appreciated.

Hi @yuanyuanli85 ,

Thanks for sharing your implementation of Stacked hourglass in Keras. I have a very basic question. In hg_blocks.py (line 22), the model is compiled with the following parameters:

model.compile(optimizer=rms, loss=mean_squared_error, metrics=["accuracy"])

May I ask you why is "Accuracy" used as a metric with a regression loss? If the model is performing a regression on the heatmap, why do we need the accuracy here?

Thanks a lot

Best

Hi,

thank you for sharing your work.
I have a question regarding the calculation of the accuracy:
Here you use a norm and threshold to calculate if a keypoint was predicted successfully. What is the use of the norm parameter? Mathematically, both values could be easily summarized to just a single value.

Best
Fabian

I want to train this network on a custom dataset. I have the gt labels and images.

Under the data_gen directory I looked at MPIIDataGen but this class has a lot of custom code specific to MPII. Is there any other place I need to modify apart from this class?

您好，我在按照您文档中提示的命令运行demo时，显示出的照片中没有任何关节点的标记。当我逐步运行demo代码时，发现所有关节点的置信度都小于0.1，请问这是怎么回事呢？

Hi, thank you for your code.
I have a fool question about evaling the dataset. How to generate the preds.mat of MPII dataset or my own dataset?
Thanks !

我打算用这个模型做人脸特征点检测，因此我要修改mpii_annotations.json的内容，请问这几个量如何设置
['objpos', 'scale_provided_other', 'objpos_other', 'annolist_index', 'people_index', 'numOtherPeople']

I tried to train with several stacks and it went well with 2 and 4 stack models, but whenever I try with 1 stack I get these errors :
Epoch 1/150
/usr/local/lib/python2.7/dist-packages/scipy/misc/pilutil.py:482: FutureWarning: Conversion of the second argument of issubdtype from int to np.signedinteger is deprecated. In future, it will be treated as np.int64 == np.dtype(int).type.
if issubdtype(ts, int):
/usr/local/lib/python2.7/dist-packages/scipy/misc/pilutil.py:485: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
elif issubdtype(type(size), float):
1852/1853 [============================>.] - ETA: 0s - loss: 0.0140 - acc: 0.1505('Saving model to ', '../../models/1_stack/weights_epoch0.h5')
Traceback (most recent call last):
File "train.py", line 56, in
xnet.train(epochs=args.epochs, model_path=args.model_path, batch_size=args.batch_size)
File "../net/hourglass.py", line 55, in train
epochs=epochs, callbacks=xcallbacks)
File "/usr/local/lib/python2.7/dist-packages/keras/legacy/interfaces.py", line 87, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 2187, in fit_generator
callbacks.on_epoch_end(epoch, epoch_logs)
File "/usr/local/lib/python2.7/dist-packages/keras/callbacks.py", line 73, in on_epoch_end
callback.on_epoch_end(epoch, logs)
File "../eval/eval_callback.py", line 65, in on_epoch_end
self.run_eval(epoch)
File "../eval/eval_callback.py", line 37, in run_eval
suc, bad = cal_heatmap_acc(out[-1], _meta, threshold)
File "../eval/eval_heatmap.py", line 53, in cal_heatmap_acc
_prehmap = prehmap[i, :, :, :]
IndexError: too many indices for array

作者您好！我想请教一下：
1.在pckh.py文件中所提到的SC_BIAS变量具体是指的什么，把它设成0.6的原因是什么呢。因为我看对于[email protected]的判定标准在pckh.py文件中已经用了threshold这个变量进行设置。所以我不太明白这个SC_BIAS存在什么意义。
2.作者您训练了模型100个epoch后最终验证集收敛值是在0.82左右，但是原模型涉及论文中提到的准确率是在0.883，请问是您在编写代码时对原模型进行了些微改动才造成0.06的准确率下降吗？如果有的话那么具体的改动可否告知在哪里呢。

Hi, thanks for your amazing work. when i inference the pre-trained model on squat image below

the l-hip and r-hip point are missed, because i set the confidence threshold is 0.1,the l-hip and r-hip confidence are below 0.1. Here is the result

what can i do to make hourglass to recognize the hip part or the bottom-torso when the hip part is being shielded, thanks for advance

I've obtained good results on training this network with MPII dataset. Now I want to annotate and train for a new dataset for car pose estimation. I've understood most of the parameters given in the annotation file, but I am confused about 'scale' and how it is calculated from the source image.

Could you please explain how scale is calculated and the changes that needs to be made for training a custom dataset.

没有hip 吗

用您提供的pretrained model以及eval代码跑出来mpii数据集的pckh只有0.79左右，与原文作者所描述的0.91左右差距比较大，是因为提供的模型训练没有收敛么？还是因为其他的原因呢？

When i test on my image, i find that the result is very bad. I notice that it may affect by the image size. But I find that in the code, it has the operation of scaling. So i'm confused that how to get a better result when i use my data.

original image

output

Not too sure what to do here because it's a lot of different methods..any ideas whats wrong?

`File "", line 1, in
runfile('C:/Users/Nabilla Abraham/Documents/Stacked Hourglass/Stacked_Hourglass_Network_Keras-master/src/net/utest_hg.py', wdir='C:/Users/Nabilla Abraham/Documents/Stacked Hourglass/Stacked_Hourglass_Network_Keras-master/src/net')

File "C:\Anaconda\envs\tensorflow\lib\site-packages\spyder\utils\site\sitecustomize.py", line 705, in runfile
execfile(filename, namespace)

File "C:\Anaconda\envs\tensorflow\lib\site-packages\spyder\utils\site\sitecustomize.py", line 102, in execfile
exec(compile(f.read(), filename, 'exec'), namespace)

File "C:/Users/Nabilla Abraham/Documents/Stacked Hourglass/Stacked_Hourglass_Network_Keras-master/src/net/utest_hg.py", line 55, in
main()

File "C:/Users/Nabilla Abraham/Documents/Stacked Hourglass/Stacked_Hourglass_Network_Keras-master/src/net/utest_hg.py", line 11, in main
model = create_hourglass_network(16, 8, (256, 256), (64, 64), bottleneck_mobile)

File "C:\Users\Nabilla Abraham\Documents\Stacked Hourglass\Stacked_Hourglass_Network_Keras-master\src\net\hg_blocks.py", line 11, in create_hourglass_network
front_features = create_front_module(input, bottleneck)

File "C:\Users\Nabilla Abraham\Documents\Stacked Hourglass\Stacked_Hourglass_Network_Keras-master\src\net\hg_blocks.py", line 91, in create_front_module
_x = bottleneck(_x, 128, 'front_residual_x1')

File "C:\Users\Nabilla Abraham\Documents\Stacked Hourglass\Stacked_Hourglass_Network_Keras-master\src\net\hg_blocks.py", line 73, in bottleneck_mobile
_x = SeparableConv2D(num_out_channels/2, kernel_size=(1,1), activation='relu', padding='same', name=block_name+'_conv_1x1_x1') (bottom)

File "C:\Anaconda\envs\tensorflow\lib\site-packages\keras\engine\topology.py", line 592, in call
self.build(input_shapes[0])

File "C:\Anaconda\envs\tensorflow\lib\site-packages\keras\layers\convolutional.py", line 1198, in build
constraint=self.pointwise_constraint)

File "C:\Anaconda\envs\tensorflow\lib\site-packages\keras\legacy\interfaces.py", line 91, in wrapper
return func(*args, **kwargs)

File "C:\Anaconda\envs\tensorflow\lib\site-packages\keras\engine\topology.py", line 413, in add_weight
weight = K.variable(initializer(shape),

File "C:\Anaconda\envs\tensorflow\lib\site-packages\keras\initializers.py", line 217, in call
dtype=dtype, seed=self.seed)

File "C:\Anaconda\envs\tensorflow\lib\site-packages\keras\backend\tensorflow_backend.py", line 3832, in random_uniform
dtype=dtype, seed=seed)

File "C:\Anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\ops\random_ops.py", line 244, in random_uniform
shape, dtype, seed=seed1, seed2=seed2)

File "C:\Anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\ops\gen_random_ops.py", line 473, in _random_uniform
name=name)

File "C:\Anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 609, in _apply_op_helper
param_name=input_name)

File "C:\Anaconda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 60, in _SatisfiesTypeConstraint
", ".join(dtypes.as_dtype(x).name for x in allowed_list)))

TypeError: Value passed to parameter 'shape' has DataType float32 not in list of allowed values: int32, int64`

Hi @yuanyuanli85 ,

I hope you don't mind a question regarding the way your code is normalizing the data. The function "normalize" performs two operations A) Scaling [0-1] and B) mean subtraction.

def get_color_mean(self):
    mean = np.array([0.4404, 0.4440, 0.4327], dtype=np.float)
    return mean

def normalize(imgdata, color_mean):
    '''
    :param imgdata: image in 0 ~ 255
    :return:  image from 0.0 to 1.0
    '''
    imgdata = imgdata / 255.0

    for i in range(imgdata.shape[-1]):
        imgdata[:, :, i] -= color_mean[i]

    return imgdata

Normalizing the input data into 0-1 helps to avoid vanishing / exploding gradients and to improve the speed of training. But why are you shifting your images into the range [-0.44 - 0.56]? Is this something that improves your training regime?

Thanks a lot

Cheers

When I used your code on my device, the CPU consumption more than 2000%(using the $top ) while GPU (Volatile GPU-Util shows )only 20% and it also not always occupied the resources. If you are free, please reply to me. Thanks!!

I'm just trying to understand if I can remove it. I don't see a point of keeping it in regression problem.

I cannot open the links to the pre-trained models, is it possible to load them onto a google drive?

How can i change it?

thank you.

您好，最近我也在可用keras复现HG， 之前的效果一直不好，知道看了您的代码，调整一下参数，有不错的输出；但是把最后的heatmap， print出来，我发现会有负数出现，即使用cv2.imshow画出来的是一个热图相应，但是tensorflow版本的heatmap，print出来是没有负数的。所以我把head_to_loss，那一层由linear激活改成relu激活，想由此避免输出出现负数的情况，但是训练完之后的heatmap，print出来全部都是0，我想问一下，为什么要选用linea激活呢？？

请问得到预测的一堆xy坐标后，怎样和gtound truth的xy 计算pckh0.5呢？？

测试图片就是4通道了，无法使用

Can you explain why you have such a small batch size in the training command please?
thanks

Hey

I have a question that which detector the networks used in this project? The detector provides the proposal to the estimation network, so where can I find the detector network?

Thanks!
Colin Chenug

Dear VictorLi,
First of all, I would like to thank you for providing the Keras translation of the original model.
I have a few doubts/question:

1. I do not see any implementation for the 1st right feature layer
2. Dimension of the 4th left feature is the same as the next three bottleneck layers (just before the 1st right feature layer). As per the hourglass diagram, shouldn't the dimension of this 3 layer be smaller than the 4th left feature layer?

I have modified your implementation a bit to fit my requirement. I have attached the
graph.pdf
legend.pdf
hourglassModule.pdf

Kindly help me to understand these doubts.
Thanks in advance

Hi
Can I use this code to extract pose from still image without any key point? Because I want extract pose from Pascal voc2012 but there aren't any key point in this dataset. can you help me how I can extract pose from this dataset please?