I add 5 refinestage in your net model, and trained it .but the performance is still poor.According to the chapter ,5 refinestage's performance is better than 1refinestage.

thanks

single_heatmap = np.zeros(shape=(heatmap_height, heatmap_width), dtype=np.float32) should in the loop:
for j in range(keypoints.shape[0]):,

at end of file get_paf.py line 99:
mask = np.logical_or.reduce((np.abs(paf_map[0, :, :]) > 0, np.abs(paf_map[0, :, :]) > 0))
is it:
mask = np.logical_or.reduce((np.abs(paf_map[0, :, :]) > 0, np.abs(paf_map[1, :, :]) > 0))

Dear Shiwei Hou，
I download your trained model, and run model_test.py to test the image of ai_challenger dataset. And finally I get the result as follow:




Is the test result of ai_challenger in line with your expectations?

Hi,

May I ask why you choose to set

train_config['input_scale'] = 8 

in "train_config.py"? Especially why 8? Thanks.

Hello
I have 2 problems, but tried but couldn't fix them:
1: my teacher told me that pose estimation model use the CNN to predict keypoints but in source only have convolutions, no put to Neural(dense)
2: then I install and run main.py I keep getting this error in Colab and my laptop:

/content/drive/My Drive/AI data/Animal Classification/lightweight_openpose-tf2.0_version
2020-07-22 15:46:53.617109: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-07-22 15:46:55.521309: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-07-22 15:46:55.534016: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-07-22 15:46:55.534601: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:00:04.0 name: Tesla P100-PCIE-16GB computeCapability: 6.0
coreClock: 1.3285GHz coreCount: 56 deviceMemorySize: 15.90GiB deviceMemoryBandwidth: 681.88GiB/s
2020-07-22 15:46:55.534635: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-07-22 15:46:55.536116: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-07-22 15:46:55.537938: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-07-22 15:46:55.538291: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-07-22 15:46:55.539832: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-07-22 15:46:55.540867: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-07-22 15:46:55.544261: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-07-22 15:46:55.544370: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-07-22 15:46:55.544925: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-07-22 15:46:55.545430: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-07-22 15:46:55.550442: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 2200000000 Hz
2020-07-22 15:46:55.550625: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2dcd100 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-07-22 15:46:55.550653: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-07-22 15:46:55.638071: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-07-22 15:46:55.638803: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2dccf40 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-07-22 15:46:55.638842: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2020-07-22 15:46:55.639055: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-07-22 15:46:55.639654: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:00:04.0 name: Tesla P100-PCIE-16GB computeCapability: 6.0
coreClock: 1.3285GHz coreCount: 56 deviceMemorySize: 15.90GiB deviceMemoryBandwidth: 681.88GiB/s
2020-07-22 15:46:55.639698: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-07-22 15:46:55.639758: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-07-22 15:46:55.639785: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-07-22 15:46:55.639810: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-07-22 15:46:55.639834: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-07-22 15:46:55.639857: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-07-22 15:46:55.639881: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-07-22 15:46:55.639961: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-07-22 15:46:55.640612: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-07-22 15:46:55.641200: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-07-22 15:46:55.641254: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-07-22 15:46:56.141171: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-07-22 15:46:56.141231: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0
2020-07-22 15:46:56.141245: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N
2020-07-22 15:46:56.141457: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-07-22 15:46:56.142097: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-07-22 15:46:56.142604: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.
2020-07-22 15:46:56.142646: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14974 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:04.0, compute capability: 6.0)
2020-07-22 15:46:57.449867: W tensorflow/core/framework/op_kernel.cc:1741] Unknown: AttributeError: 'NoneType' object has no attribute 'shape'
Traceback (most recent call last):

File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/script_ops.py", line 241, in call
return func(device, token, args)

File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/script_ops.py", line 130, in call
ret = self._func(*args)

File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/autograph/impl/api.py", line 309, in wrapper
return func(*args, **kwargs)

File "/content/drive/My Drive/AI data/Animal Classification/lightweight_openpose-tf2.0_version/data/dataset.py", line 53, in paser_func
img_ori, kps = data_aug(img_ori, None, kps)

File "/content/drive/My Drive/AI data/Animal Classification/lightweight_openpose-tf2.0_version/utils/data_aug.py", line 46, in data_aug
kps = ia.KeypointsOnImage([], shape=img.shape)

AttributeError: 'NoneType' object has no attribute 'shape'

2020-07-22 15:46:57.450128: W tensorflow/core/framework/op_kernel.cc:1741] Unknown: AttributeError: 'NoneType' object has no attribute 'shape'
Traceback (most recent call last):

File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/script_ops.py", line 241, in call
return func(device, token, args)

File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/script_ops.py", line 130, in call
ret = self._func(*args)

File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/autograph/impl/api.py", line 309, in wrapper
return func(*args, **kwargs)

File "/content/drive/My Drive/AI data/Animal Classification/lightweight_openpose-tf2.0_version/data/dataset.py", line 53, in paser_func
img_ori, kps = data_aug(img_ori, None, kps)

File "/content/drive/My Drive/AI data/Animal Classification/lightweight_openpose-tf2.0_version/utils/data_aug.py", line 46, in data_aug
kps = ia.KeypointsOnImage([], shape=img.shape)

AttributeError: 'NoneType' object has no attribute 'shape'

Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/context.py", line 1986, in execution_mode
yield
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/iterator_ops.py", line 655, in _next_internal
output_shapes=self._flat_output_shapes)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_dataset_ops.py", line 2363, in iterator_get_next
_ops.raise_from_not_ok_status(e, name)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 6653, in raise_from_not_ok_status
six.raise_from(core._status_to_exception(e.code, message), None)
File "", line 3, in raise_from
tensorflow.python.framework.errors_impl.UnknownError: AttributeError: 'NoneType' object has no attribute 'shape'
Traceback (most recent call last):

File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/script_ops.py", line 241, in call
return func(device, token, args)

File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/script_ops.py", line 130, in call
ret = self._func(*args)

File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/autograph/impl/api.py", line 309, in wrapper
return func(*args, **kwargs)

File "/content/drive/My Drive/AI data/Animal Classification/lightweight_openpose-tf2.0_version/data/dataset.py", line 53, in paser_func
img_ori, kps = data_aug(img_ori, None, kps)

File "/content/drive/My Drive/AI data/Animal Classification/lightweight_openpose-tf2.0_version/utils/data_aug.py", line 46, in data_aug
kps = ia.KeypointsOnImage([], shape=img.shape)

AttributeError: 'NoneType' object has no attribute 'shape'

 [[{{node EagerPyFunc}}]] [Op:IteratorGetNext]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "main.py", line 33, in
2020-07-22 15:46:57.459092: W tensorflow/core/framework/op_kernel.cc:1741] Unknown: AttributeError: 'NoneType' object has no attribute 'shape'
Traceback (most recent call last):

File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/script_ops.py", line 241, in call
return func(device, token, args)

File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/script_ops.py", line 130, in call
ret = self._func(*args)

File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/autograph/impl/api.py", line 309, in wrapper
return func(*args, **kwargs)

File "/content/drive/My Drive/AI data/Animal Classification/lightweight_openpose-tf2.0_version/data/dataset.py", line 53, in paser_func
img_ori, kps = data_aug(img_ori, None, kps)

File "/content/drive/My Drive/AI data/Animal Classification/lightweight_openpose-tf2.0_version/utils/data_aug.py", line 46, in data_aug
kps = ia.KeypointsOnImage([], shape=img.shape)

AttributeError: 'NoneType' object has no attribute 'shape'

train(model, optimizer, dataset, epochs, cur_time, max_keeps=epochs)

File "/content/drive/My Drive/AI data/Animal Classification/lightweight_openpose-tf2.0_version/train/kps_train.py", line 47, in train
2020-07-22 15:46:57.460074: W tensorflow/core/framework/op_kernel.cc:1741] Unknown: AttributeError: 'NoneType' object has no attribute 'shape'
Traceback (most recent call last):

File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/script_ops.py", line 241, in call
return func(device, token, args)

File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/script_ops.py", line 130, in call
ret = self._func(*args)

File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/autograph/impl/api.py", line 309, in wrapper
return func(*args, **kwargs)

File "/content/drive/My Drive/AI data/Animal Classification/lightweight_openpose-tf2.0_version/data/dataset.py", line 53, in paser_func
img_ori, kps = data_aug(img_ori, None, kps)

File "/content/drive/My Drive/AI data/Animal Classification/lightweight_openpose-tf2.0_version/utils/data_aug.py", line 46, in data_aug
kps = ia.KeypointsOnImage([], shape=img.shape)

AttributeError: 'NoneType' object has no attribute 'shape'

for step, (img, heatmap, paf) in enumerate(dataset):

File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/iterator_ops.py", line 631, in next
return self.next()
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/iterator_ops.py", line 670, in next
return self._next_internal()
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/data/ops/iterator_ops.py", line 661, in _next_internal
return structure.from_compatible_tensor_list(self._element_spec, ret)
File "/usr/lib/python3.6/contextlib.py", line 99, in exit
self.gen.throw(type, value, traceback)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/context.py", line 1989, in execution_mode
executor_new.wait()
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/executor.py", line 67, in wait
pywrap_tfe.TFE_ExecutorWaitForAllPendingNodes(self._handle)
tensorflow.python.framework.errors_impl.UnknownError: AttributeError: 'NoneType' object has no attribute 'shape'
Traceback (most recent call last):

File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/script_ops.py", line 241, in call
return func(device, token, args)

File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/script_ops.py", line 130, in call
ret = self._func(*args)

File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/autograph/impl/api.py", line 309, in wrapper
return func(*args, **kwargs)

File "/content/drive/My Drive/AI data/Animal Classification/lightweight_openpose-tf2.0_version/data/dataset.py", line 53, in paser_func
img_ori, kps = data_aug(img_ori, None, kps)

File "/content/drive/My Drive/AI data/Animal Classification/lightweight_openpose-tf2.0_version/utils/data_aug.py", line 46, in data_aug
kps = ia.KeypointsOnImage([], shape=img.shape)

AttributeError: 'NoneType' object has no attribute 'shape'

 [[{{node EagerPyFunc}}]]

I don't know how to run the model to predict result.
Thank for attention.

Hello, bro!
I read the get_paf.py, i have a question in the 99-line:
mask = np.logical_or.reduce((np.abs(paf_map[0, :, :]) > 0, np.abs(paf_map[0, :, :]) > 0))
Maybe I think
mask = np.logical_or.reduce((np.abs(paf_map[0, :, :]) > 0, np.abs(paf_map[1, :, :]) > 0))
is better.
If I was wrong, please tell me how to understand? Thank you!!!

And there is my code to get pb model:

import tensorflow as tf
#from create_tf_record import *
from tensorflow.python.framework import graph_util

def freeze_graph(input_checkpoint,output_graph):
output_node_names = "pre_hmap"
saver = tf.train.import_meta_graph(input_checkpoint + '.meta', clear_devices=True)
graph = tf.get_default_graph() # 获得默认的图
input_graph_def = graph.as_graph_def() # 返回一个序列化的图代表当前的图

with tf.Session() as sess:
    saver.restore(sess, input_checkpoint) #恢复图并得到数据
    output_graph_def = graph_util.convert_variables_to_constants(  # 模型持久化，将变量值固定
        sess=sess,
        input_graph_def=input_graph_def,# 等于:sess.graph_def
        output_node_names=output_node_names.split(','))# 如果有多个输出节点，以逗号隔开

    with tf.gfile.GFile(output_graph, "wb") as f: #保存模型
        f.write(output_graph_def.SerializeToString()) #序列化输出
    print("%d ops in the final graph." % len(output_graph_def.node)) #得到当前图有几个操作节点

input_checkpoint = "./ckpt/model.ckpt-61236"
out_pb_path='pb_pre.pb'
freeze_graph(input_checkpoint, out_pb_path)

As shown in the figure, I visualize the PB model, and the input node name is strange, and there are two inputs. What's wrong when converting ckpt to pb ?

Hi,
we are training the model with COCO dataset, and we found that your code implemented with masks:

mask = labels[..., params['num_kps'] + params['paf']:]
mask = tf.reshape(mask, [-1, params['height']//params['scale'], params['width']//params['scale'], 1])

May I ask why you need the masks here?
Thank you for your time and looking forward to reading your reply.

感觉这个地方不应该用compute_resized_coords里面的resize方法，直接乘个resizeFactor就行，否则会导致检出关键点往右下方偏，改了之后没有这个问题。

when i try to run the train.py that you have updated 2019/3/18,
i recieve error:

File ".\train.py", line 159, in main
ValueError: input_fn should return (features, labels) as a len 2 tuple.

the website always say "incorrect username pwd" even though the setup went well and the change of password does several times also went well. any alternative way to download the dataset?

Hello,

view pairs with max OKSs as match ones, add to oks_all

oks_all = np.concatenate((oks_all, np.max(oks, axis=1)), axis=0)

accumulate total num by max(gtN,pN)

oks_num += np.max(oks.shape)

is there error about oks_num?

I would like to fine-tune starting from MobilenetV1, any guidelines?

Did u download a model from here and directly used it?

hello Murdock:

For the refinement stage "Input", I noticed that you used the concation of heatmaps and pafs without the featuremap(net_cpm).
Is it a bug? or you made this change on purpose?

Thanks
///////////////////////////////////
outs1 = tf.concat([heatmaps1, pafs1], axis=-1)
with tf.variable_scope('RefinementStage', reuse=tf.AUTO_REUSE):
net = RefinementStageBlock(outs1, 128, training=is_training)
//////////////////////////////////////

Hi.. I just tried to build the trained model, It shows sometimes the error

File "/home/ubuntu/openpose-light/lightweight_openpose/src/dataset.py", line 101, in _parse_function
b,g,r = cv2.split(img_data)

and also sometimes it shows "Resource exhausted" error in Tensorflow,
I have changed the ['train_nums'] = 20000/2000 (Limited number of image for testing) and changed the [batch_size] in parameters.py file, but it did not worked. My ai_challenger dataset have 210000 images.

Anyway I am new to Tensorflow. Pls help me in this regard.

Thanks

Hi,

I want to know if you have the pretrained model available with your code?
I already tested the PyTorch version with its pre-trained model and I'm looking for the TensorFlow version of it.

I appreciate it if you help me.
Thanks!

Hi. I am wondering how do you generate paf and heatmap in test/inference time. That is, when your model generate out the heatmap, do you use any kind of non-maximum suppression, as mentioned in the paper, to generate the coordinates of the joints? If so could you kindly point out where are they in the tf 2.0 version of the code?

hi, you have give the deploy.prototxt of caffe ,can you give the train.prototxt of caffe version,thanks

你好, 作者, 就不用英文沟通了, 想请问下, 您训练时候的aichallenger14个点loss是多少, 这是我的loss感觉不太对劲