hrnet / lite-hrnet Goto Github PK

What segmentation head is being used for the mobilenet baselines in Table 8 ?

Hi, i've been following your project and it seems very interesting. Just noticed you updated with the new naive architecture. Could you give me a brief intro on this new approach? Thanks!

This is a great job.
I use LIte-HRNet as the backbone to reproduce HigherHRNet-Human-Pose-Estimation
but the result is not very satisfactory.
I hope you can provide useful suggestions.
Thank you for your work.


Do you have plans to test the bottom-up pose-estimation task?

@ycszen Could you share the Lite-HRNet model pretrianed on ImageNet by Baidu Netdisk? Thank you very much!!

how to convert to onnx, I meet the problem,
assert target.dim() == 4 and target_weight.dim() == 3
AttributeError: 'NoneType' object has no attribute 'dim'

code:
model = build_posenet(cfg.model)
load_checkpoint(model, args.checkpoint, map_location='cpu')
model = model.eval()

x = torch.randn((1,3,384,288))
target = torch.randn((1,4,384,288))
target_weight = torch.randn((1,4,1))
output_path = "onnx/lite_HRNet_960_1280.onnx"
torch.onnx.export(model,               # model being run
                x,                         # model input (or a tuple for multiple inputs)
                output_path,   # where to save the model (can be a file or file-like object)
                export_params=True,        # store the trained parameter weights inside the model file
                opset_version=11,          # the ONNX version to export the model to
                do_constant_folding=True,  # whether to execute constant folding for optimization
                input_names = ['inputx'],   # the model's input names
                output_names = ['outputy'], # the model's output names
                verbose=True,
                )

the error can refer:
https://github.com/HRNet/Lite-HRNet/issues/23
@ycszen

I used it on semantic segmentation and found that the loss convergence speed is very very slow(compare with other backbone)

您好，您仓库的网络和mobilenet v3在分类上相比，哪个效果更好呢？如果优于mobilenet v3在模型大小上，哪个更轻量呢？期待您的回复，谢谢/。

Do we have to use this train func when we train a model？

def train(self, mode=True): 
       """Convert the model into training mode."""
        super().train(mode)
        if mode and self.norm_eval: 
                for m in self.modules():
                        if isinstance(m, _BatchNorm):
                            m.eval()

您好，这个仓库怎么训练自己的数据集啊

Hi there, thanks for this nice work! I am trying to modify the Lite-HRNet model on my segmentation task with my own data, by replacing the head with a seg head to predict segmentation maps, as shown in the picture.

However, the results are pretty bad. I think the possible reasons are 1:) the model cannot be directly utilized for segmentation?If so, can you release the model for seg? 2:) the training is unstable due to the lack of pretrained weights? In this case, is it possible to release the ImgaNet pre-trained weights for Lite-HRNet, like the ones in https://github.com/HRNet/HRNet-Semantic-Segmentation?

Any kind of help or assistance is highly appreciated! Thanks in advance!

@ycszen
非常棒的学术研究！
如果能提供可视化测试例子可能会更好点吧。

The paper uses cross-resolution weighting function and spatial weighting function to replace 1×1 conv.
It works well on shufflenet.
Mobilenetv2 also uses lots of 1×1 conv in the network.
So did you test this method on mobilenetv2? Or would you do this in future?

hi, it seems lite-hrnet semantic segmentation has higher precision than your bisenetv2. will the code relase. wait in hope.

Hi, When I train the model with single GPU, the error raised:

where can I find the config file for the semantic segmentation?

the output of self.fuse_layers[i]j is [1, 40, 256, 455],(i=0, j=1)
but y is [1, 40, 256, 456]

I found this error when training my own datasets. I wonder how can i solve it?

"COCO_val2017_detections_AP_H_56_person.json" is not for training. It is only used for evaluation. The boxes (det boxes) are generated by a detector. person_keypoints_val2017.json also contains bboxes, but boxes (gt boxes) are manually labeled. During inference, one can choose to evaluate by gt or det boxes, by setting use_gt_bbox=True or False (https://github.com/open-mmlab/mmpose/blob/a82dd486853a8a471522ac06b8b9356db61f8547/configs/top_down/resnet/coco/res50_coco_256x192.py#L70)

Paperlink broken

Hi,

I try to use Lite-HRNet on some videos with 480x854 resolution.
But the following part has a size mismatch error (at row ~630).
Following are the code and log of that part, how could I solve this issue?

Thanks in advance.

======================================
if self.with_fuse:
out_fuse = []
for i in range(len(self.fuse_layers)):
y = out[0] if i == 0 else self.fuse_layers[i]0
for j in range(self.num_branches):
if i == j:
y += out[j]
else:
print(i, j, y.shape, self.fuse_layers[i]j.shape)
y += self.fuse_layers[i]j
out_fuse.append(self.relu(y))
out = out_fuse

======================================

0 1 torch.Size([1, 40, 120, 214]) torch.Size([1, 40, 120, 214])
1 0 torch.Size([1, 80, 60, 107]) torch.Size([1, 80, 60, 107])
0 1 torch.Size([1, 40, 120, 214]) torch.Size([1, 40, 120, 214])
1 0 torch.Size([1, 80, 60, 107]) torch.Size([1, 80, 60, 107])
0 1 torch.Size([1, 40, 120, 214]) torch.Size([1, 40, 120, 214])
1 0 torch.Size([1, 80, 60, 107]) torch.Size([1, 80, 60, 107])
0 1 torch.Size([1, 40, 120, 214]) torch.Size([1, 40, 120, 214])
0 2 torch.Size([1, 40, 120, 214]) torch.Size([1, 40, 120, 216])

========================================

when i try to run summary_network through "python tools/summary_network.py configs/top_down/lite_hrnet/coco/litehrnet_18_coco_256x192.py --shape 256 256 --with-head", a error raise as follows:
Traceback (most recent call last):
File "tools/summary_network.py", line 80, in
main()
File "tools/summary_network.py", line 45, in main
model = build_posenet(cfg.model)
File "/usr/local/lib/python3.6/site-packages/mmpose/models/builder.py", line 52, in build_posenet
return build(cfg, POSENETS)
File "/usr/local/lib/python3.6/site-packages/mmpose/models/builder.py", line 27, in build
return build_from_cfg(cfg, registry, default_args)
File "/usr/local/lib/python3.6/site-packages/mmcv/utils/registry.py", line 182, in build_from_cfg
raise type(e)(f'{obj_cls.name}: {e}')
KeyError: "TopDown: 'LiteHRNet is not in the backbone registry'"

but, if i replace "from mmpose.models import build_posenet" with "from models.builder import build_posenet", it raise another error:

Traceback (most recent call last):
File "tools/summary_network.py", line 80, in
main()
File "tools/summary_network.py", line 62, in main
format(model.class.name))
NotImplementedError: FLOPs counter is currently not currently supported with TopDown

So, how can i correctly cal the flops and mac.

besides, i have trans the smallest model to ncnn, but for inference time, it's not faster than mobilenetv2, have you tested the inference time with mobilenetv2?

Hi
This is awesome lightweight networks!!
I want to test it on other mobile device.
Could you provide ONNX format exporting code?
Thanks a lot!!

firstly, this work is great!
and I am trying to using it in other model but an error occurs as below.(i use the pre-trained model ).I wonder how can i solve this problem？

Hi, Thanks your work! When I train with dist_train.sh, the error reported as follows,

Hi,

This is a great job with a light weight model of hrnet. It would be great if you can add any code for inference with images. Or also you can try to upload sample output images to the repository. In the meantime, if you could help us on how to run inference, it would be great. thanks

during training, where to set batch size?

model = dict( type='Topdown', # num_stages=3, pretrained=None, backbone=dict( type='LiteHRNet', in_channels=3, extra=dict( stem=dict( stem_channels=32, out_channels=32, expand_ratio=1), num_stages=3, stages_spec=dict( num_modules=(3, 8, 3), num_branches=(2, 3, 4), num_blocks=(2, 2, 2), module_type=('LITE', 'LITE', 'LITE'), with_fuse=(True, True, True), reduce_ratios=(8, 8, 8), num_channels=( (40, 80), (40, 80, 160), (40, 80, 160, 320), )), with_head=False, )),

any ideas about how to find out "Topdown"?

Test on V100 GPU card, with top-level CPU.
For litehrnet_18_coco_256x192, the inference latency > 200 ms

Thanks a lot for the work and making the code available. It's very easy to get started with it for custom use.
I could easily train and test but I wanted to know the following:

How do I freeze a layer or a stage during training for fine-tuning?
During fine-tuning I don't want to change weights in all the stages/layers that are in the base model.

I was thinking about:

1. freezing everything but the final keypoint detection head
2. freezing everything but the last stage and keypoint detection head.

How do I accomplish this?

Any help or suggestion is greatly appreciated :)

Thanks for your great works. It helps me a lot.
But your project is based on mmcv, how can I get my own dataloader?

how about the performance in classification or deteciton task, this backbone ?

it is amazing lightweight work, would u consider add a script to test a img with model?
as demo.py like HRNet/HigherHRNet-Human-Pose-Estimation@57c1f30

hello:
when I run this command "pip install -r requirements.txt",I encountered the error as bellow:
ERROR: Could not find a version that satisfies the requirement xtcocotools>=1.6 (from -r requirements/runtime.txt (line 11)) (from versions: none)
ERROR: No matching distribution found for xtcocotools>=1.6 (from -r requirements/runtime.txt (line 11)).
I checked that xtcocotools rely on mmpose. can help me how to solve this problem?

Hi, thanks for your great works!
But, could you please tell me how to inference and show result with an input image? I am not familiar with mmpose so I dont know how to do.

run test.py: dataset = build_dataset(cfg.data.test, dict(test_mode=True)) get the error:init() missing 2 required positional arguments: 'doc' and 'pos'

using mmcv, mmpose, while debug difficultly.

batch_size(per_gpu): 4

2021-05-13 10:10:43,596 - mmpose - INFO - Epoch [1][2350/18726] lr: 2.000e-03, eta: 14 days, 15:25:42, time: 0.284, data_time: 0.000, memory: 400, mse_loss: 0.0021, acc_pose: 0.2202, loss: 0.0021
2021-05-13 10:10:57,826 - mmpose - INFO - Epoch [1][2400/18726] lr: 2.000e-03, eta: 14 days, 14:35:49, time: 0.286, data_time: 0.001, memory: 400, mse_loss: 0.0021, acc_pose: 0.2152, loss: 0.0021
2021-05-13 10:11:11,913 - mmpose - INFO - Epoch [1][2450/18726] lr: 2.000e-03, eta: 14 days, 13:42:50, time: 0.282, data_time: 0.000, memory: 400, mse_loss: 0.0020, acc_pose: 0.2176, loss: 0.0020

Connected to pydev debugger (build 202.7319.64)
Use load_from_local loader
The model and loaded state dict do not match exactly

unexpected key in source state_dict: backbone.stage0.2.layers.0.cross_resolution_weighting.conv1.conv.weight, backbone.stage0.2.layers.0.cross_resolution_...

how to load model in right way?

https://github.com/HRNet/HRNet-Semantic-Segmentation/blob/f9fb1ba66ff8aea29d833b885f08df64e62c2b23/lib/models/hrnet.py#L264

https://github.com/HRNet/HRNet-Human-Pose-Estimation/blob/00d7bf72f56382165e504b10ff0dddb82dca6fd2/lib/models/pose_hrnet.py#L258

https://github.com/HRNet/HigherHRNet-Human-Pose-Estimation/blob/aa23881492ff511185acf756a2e14725cc4ab4d7/lib/models/pose_higher_hrnet.py#L236

yours: for j in range(self.num_branches):

others: for j in range(1, self.num_branches):

should the range be (1, self.num_branches) ?

(base) chenxin@chenxin-Nitro-AN515-52:~/disk1/github/Lite-HRNet/mmpose/mmpose/models$ tree .
.
├── backbones
│   ├── alexnet.py
│   ├── base_backbone.py
│   ├── cpm.py
│   ├── hourglass.py
│   ├── hrnet.py
│   ├── init.py
│   ├── mobilenet_v2.py
│   ├── mobilenet_v3.py
│   ├── mspn.py
│   ├── regnet.py
│   ├── resnest.py
│   ├── resnet.py
│   ├── resnext.py
│   ├── rsn.py
│   ├── scnet.py
│   ├── seresnet.py
│   ├── seresnext.py
│   ├── shufflenet_v1.py
│   ├── shufflenet_v2.py
│   ├── tcn.py
│   ├── utils
│   │   ├── channel_shuffle.py
│   │   ├── init.py
│   │   ├── inverted_residual.py
│   │   ├── make_divisible.py
│   │   ├── se_layer.py
│   │   └── utils.py
│   └── vgg.py
├── builder.py
├── detectors
│   ├── associative_embedding.py
│   ├── base.py
│   ├── init.py
│   ├── interhand_3d.py
│   ├── mesh.py
│   ├── multi_task.py
│   ├── pose_lifter.py
│   └── top_down.py
├── heads
│   ├── ae_higher_resolution_head.py
│   ├── ae_simple_head.py
│   ├── deeppose_regression_head.py
│   ├── hmr_head.py
│   ├── init.py
│   ├── interhand_3d_head.py
│   ├── temporal_regression_head.py
│   ├── topdown_heatmap_base_head.py
│   ├── topdown_heatmap_multi_stage_head.py
│   └── topdown_heatmap_simple_head.py
├── init.py
├── losses
│   ├── classfication_loss.py
│   ├── init.py
│   ├── mesh_loss.py
│   ├── mse_loss.py
│   ├── multi_loss_factory.py
│   └── regression_loss.py
├── misc
│   ├── discriminator.py
│   └── init.py
├── necks
│   ├── gap_neck.py
│   └── init.py
└── utils
├── geometry.py
├── init.py
└── ops.py

8 directories, 60 files

安装完成mmpose的代码训练模型时候却如下错误
from mmpose.models.registry import BACKBONES
ModuleNotFoundError: No module named 'mmpose.models.registry'

thank you

你好，请问litehrnet是否只是一个特征提取网络，作为backbone，，是否需要添加其他的网络（人体检测，关键点组合）才能进行关键点检测？

在COCO数据集上的人体姿态估计提供的detector是用的什么检测模型.

1. requirment.txt要求安装numpy=1.19.0，但会报错“ValueError: numpy.ndarray size changed, may indicate binary incompatibility
   ”
   解决方法：版本问题，安装numpy 1.20.0
   pip install numpy==1.20.0

2. 缺少mmpose包，要最近新版的，否则会缺少相应模块，如NECKS
   git clone [email protected]:open-mmlab/mmpose.git
   cd mmpose
   pip install -r requirements.txt
   python setup.py develop

3. ModuleNotFoundError: No module named 'poseval'
   git clone https://github.com/svenkreiss/poseval.git
   cd poseval
   pip install -e .

4. ModuleNotFoundError: No module named 'tensorboard'
   pip install future tensorboard

your code provide batch test, no single image test for drawing pose keypoints.

Have you guys done any testing with real-time inference compared to mobilenetv2/v3? I saw there was a comparison with number of parameters and GFLOPS but didn't see any numbers for ms/fps

hi, @ycszen ,
i test the native-hrnet and lite-hrnet mnn model on my pc,
although theirs flops is 309M vs 203M,but their inference time is almost the same.
i thank is the reason lite-hrnet has more memory read or write than native-hrnet when inference.
is that right?
here is some info:
parameters infer memory(MB) MAdd Flops MemRead(B) MemWrite(B) duration
native-hrnet:

lite-hrnet ：