seanchenxy / handmesh Goto Github PK

请问，有试过Android手机端可以使用MobRecon模型吗，哪种移动端推理框架支持呢

Hi, I have downloaded the FreiHAND dataset. Could you please give a link to the freihand_train_mesh.zip?

Thanks.

File "run.py", line 196, in demo
out = self.model(input)

The type of "input" is ( torch.FloatTensor ) instead of ( torch.cuda.FloatTensor ). Maybe you should use "input.to(self.device)"

hi，
我想尝试跑一下这里human36m的demo和训练，但是都没有成功，训练的时候在这里报错 link，我理解这里的rot应该是一个数，但是在link这里返回的rot是个array，我修改为rot = rot[0]可以正常训练
另外我想跑一下human36m的demo，但是报错如下

刚开始做mesh相关的工作，这里的问题我实在不知道怎么解决了

Can you show your method on a real time video, changing the distance of Z axis for intuitively visualizing the jitter.

Hi, SeanChenxy, thanks for your impressive work.

I have tried the demo according to your instructions, but I can't find the pre-trained models. Does it mean that I need to use the models that are trained by myself ?

陈先生您好，请问公开了complement data吗？可以分享数据下载链接吗？谢谢！

Can you please tell us how did you generate training_mano.json and evaluation_scale.json file?
We need to generate the same for our custom data.
And why are you using training_mano.json, evaluation_scale.json?

How do we prepare test images and get npy files?

您好！demo是输入以手为中心的图片，相机内参根据图片缩放进行变换，如果输入的手在图像中占比很小，不在图像中心，先通过目标检测出手的范围，然后将手的图片传入网络，这样的话相机内参怎么变化呢？

How to train model on FreiHand and other datasets（like RHD） without mesh gt together？

Great thanks for your opensource. Very impressive work.
I tried the demo according to your instructions. And I am a little confused about the xx_K.npy files. I guess these files contains the camera info, but where can I get the corresponding info for my custom data? Could you please help me out, thanks.

Hi, SeanChenxy,

The FreiHAND dataset has no annotation in the evaluation dataset. How to collect the ano folder shown in the above figure? Thanks.

Hi Sean,
I have been trying to train the model on our custom data. I have given around 1000 images as input to the model.
Once these images are loaded it says

Loading of 1000 training samples 
Loaded 4000 training samples

After this i will be getting image not found error. Lets say /training/rgb2/00002782.jpg is not found.

I have observed that here you have multiplied 4.
I wanted to know the reason behind the multiplication by 4.
How should I resolve the above error? , since it's asking the image which is out of our dataset.

Hello Sean,

Thanks for releasing the DenseStack model! I got a quick question regarding the training recipe to achieve the 6.9 PJ shown in Table 5: to achieve such accuracy, do I need to use the complementary data in any stage of the training? I assume that since there is no marker associated with it, so it used the complementary data for pretraining and FreiHand only during fine-tuning. Could you please elaborate a bit on this? that would be super helpful.

Thanks!

Hi, I follow your link to download H36M dataset from I2L repo. But the h36m_mask.zip is not found there.

Could you please provide a link to the h36m_mask.zip?

Thanks

Hello, thank you for your amazing implementation.
Is this HandMesh repo were developed on windows 10 ?

cmr-pg-FreiHAND，在真实场景下测试，mask_pred 的值都是0
mask_pred = (mask_pred[0] > 0.3).cpu().numpy().astype(np.uint8)

这里0.3换为0.1 ，mask_pred依然都是0，奇怪的是手指坐标值都有，是哪里的问题呢？

您好，陈先生，您的mobrecon算法在手部重建方面取得快且准确的效果，于是我想要复现您的工作以便于学习。在算法训练的过程中，遇到了一些问题，希望得到您的指点。通过FreiHand数据集直接训练desestack.path得到checkpoint_best.pt与checkpoint_last.pt与您开源的预训练网络结构有所差异，具体体现在：我用checkpoint_best.pt与checkpoint_last.pt替换您的预训练网络时，model load部分会出现
missing key(s) in state_dict : "backbone.conv.0.weight".
Unexpected key(s) in state_dict : "backboone.reduce.0.weight" , "decoder3d.de_layer_conv.0.weight"
请问这种问题的出现是我在训练过程中有什么没有注意的的错误吗？

Hi Xingyu, I use the CMR-PG model trained on FreiHAND and eval on RHD, but only get 0.54 AUC (The average of pck20, 30, 40 is 0.268 0.466 0.634). The results on FreiHAND without problems. I think this may be caused by the difference in the joint annotations between the two datasets. Can you share the implementation details of the evaluation on RHD?

The 4 figures below show the camera space and root-relative joints prediction, respectively. (The gray dots are the joints annotations of the image: '/RHD_published_v2/evaluation/color/00000.png' root-relative results: pck20:0.238 pck30:0.429 pck40:0.762)


Left: Ground Truth; Right: Prediction

For the code for the eval part, I refer to bihand. And I crop the image based on the hand center and the 2D joint annotation, then scaled it to 224*224 (The left hand has been flipped).

Hi Xingyu, thanks for your great work and releasing the code!
Would you please also release the code for the CMR-PG model used in the paper, if possible? Thanks!

Dear author,

I found a bug in the function drawmesh. The image shape is H x W x 3, but the parameters of 'set_size_inches' is (width, height) according to the matplotlib API. So I suppose that the correct input of 'set_size_inches' should be image.shape[1], image.shape[0]. If I were right, there maybe some same bugs in other functions. Please check them out.

Best wishes,
Jingshuang

Hi @SeanChenxy ,

I am adopting the MobRecon network to train my custom model. As a basic step, I try to train MobRecon without using pre-trained densestack.pth, however the validation pampjpe is higher. I searched the paper but not find too much details about pre-training. Could you give more details about this? e.g., is pre-training a task of 2d keypoints estimation? which loss is used, which datasets are used during it? Thanks ❤️

作者你好

论文中提到额外利用的maya制作数据集
没有考虑使用实际相机拍摄的原因是为什么呢？ 还是因为实际拍摄需要用多视角相机解决视角问题， 成本考虑吗？

如果制作数据集想以实际的照片拍摄的话， RGB采集是不是以多视角采集更好（类似FreiHAND的思路）？
但是三维关键点应该如何进行标注呢？ 找了一轮没有找到类似的标注的软件

又或者是用深度相机转成点云图进行标注？

望作者拨空可以给予建议 谢谢！

Dear author,

Your work is very awesome. I don't know how you build FreiHand mesh data ( freihand_train_mesh.zip)?

作者您好，很棒的工作，请问关于camera space那篇文章的做法，如果我想只估计出root relative 3D pose 而不用得到mesh，有什么比较合理的方法得到更高的精度吗？还有在这种情况下，root depth 怎么获得比较合适？

请教一下作者，mobrecon模型最终输出的21个三维关节点的单位是米（meter）吗

您好！感谢你的工作和贡献 ！

由于对于这个领域的工作不是很熟悉，所以有一些比较多的疑惑，希望您帮助能解答一下。

（1）CS-MPJPE/MPVPE具体是怎样定义和计算的啊，它和MPJPE/MPVPE的关系是怎样的？
我测试了您提供的代码，cmr_pg模型的MPJPE/MPVPE和PA-MPJPE/MPVPE的测试结果都能达到文中的效果，但是对于CS-MPJPE/MPVPE我不太清楚是如何计算的。我尝试测试了没有经过registration的结果，得到了相同的MPJPE/MPVPE和很差的MPJPE/MPVPE，所以我理解文中Adaptive 2D-1D Registration是根据mask等信息对结果进行了一定程度的对齐，从而达到了较好的效果，不知道这样理解是否正确？

（2）我对于文中的camera-space root的坐标我有点疑惑，请问这个初始值（0， 0， 0.6）是如何确定的啊？我一直对数据集中数据的坐标系和坐标比较困惑 ，比如freihand数据集，其中点的坐标都是相对于root的相对坐标系，可以这样理解吗？

（3）请问feature representations是可视化的哪几个特征层啊，我尝试可视化了cmr_pg中的z2，pred4等都没有得到想要的效果，下图是可视化的pred4后的21个关键点map，并没有得到想要的那种比较清晰的关键点，请问这里可视化时有什么细节和技巧吗？

（4）请问MobRecon中的结果最后也都是经过类似Adaptive 2D-1D Registration对齐的吗？

陈先生你好，

您在MobRecon论文中的图13展示了补充数据集的annotations，可是我下载后发现只有mesh，请问能不能提供论文中提到的annotations呢？谢谢。

Hi @SeanChenxy ,

For the implementation of consistency loss, I come up two different approaches but don't know which one is the right one used in the paper, could you give some hints on this? Thanks.

First approach

Use the (image, vertex) from one camera, and then randomly generate a custom T to create paired (image, vertex) as inputs for consistency loss

Second approach

Use two images at the same time from different camera in dataset (e.g., camera 1 and camera 2 in FreiHAND), as the input of consistency loss, i.e., no need to generate novel camera, just use already existing ones

Hi Xingyu,
I want to predict the visibility score of 21 keypoints on image space based on CMR or MobRecon. However I don't get a clean idea about how to creating ground truth label of keypoints visibility on FreiHAND dataset. What I want to do is when some keypoints are occluded by others (like inner keypoints in wist pose) or out of image, then set its score to a low value. Could you give any hints on this? Thanks in advance.

mobrecon可以得到手的绝对三维坐标吗

Thanks for your great work! only detect the right hand now?

hi, thanks for your great work, I'd like to ask that where is the consistency loss?

你好，陈先生，请教两个问题：
第一，请问Freihand数据集没有验证集，验证只能将结果上传服务器得出来，所以你论文中的图中（Camera-Space Hand Mesh Recovery via Semantic Aggregation and Adaptive 2D-1D Registration文章中）的PCK误差是通过Freihand数据集的哪些部分进行evaluation的？
第二，其它方法的AUC是怎么得到的？从这些方法文章中找还是自己做实验跑？以及在RHD上数据集上的PCK误差是只用Freihand数据集训练得到的模型后进行直接在RHD数据集进行evalutation吗？因为你提供了Freihand数据集的Mesh 标签，但是RHD数据集没有。

Hi @SeanChenxy ,
Thanks for sharing this clean and well organized program. Could you share the weights for different loss in MobRecon? I didn't find the detail in the paper. Thanks in advance

Hi. Thank you for sharing your code. Your paper is very impressive.

But I wonder why the code needs the mask of FreiHAND when evaluating the model.

I understood your model only use the mesh in measurement.

Can you explain this problem?

Thank you!

Hi, do you have any experience in converting the model into onnx format? I have encountered some problems in the process of conversion.

Hello, Xingyu, first appreciate your so nice job! As decribed in the title, could you please answer my question if it's convenient for you?

你好，陈先生，您的这个手部Mesh重建的工作我觉得非常棒。但是代码似乎存在一些问题：
第一，您在readme.md文件的预训练模型位置似乎写错了一个。

根据你提供的gooledrive预训练模型下载文件应该是这样的。

第二，我按照你的readme.md步骤，运行./scripts/demo_cmr.sh，会提示smplpytorch无法找到，原因在smplpytorch这个文件应当在HandMesh主文件下，而不是smplpytorch文件下的smplpytorch。下图是我修改后的文件夹形式。

第三，我修改上述文件后，FreiHAND数据集正确放置，但是运行训练代码时：
./scripts/eval_cmr_freihand.sh
出现一下报错：

因此我将/home/su/文档/HandMesh/datasets/FreiHAND/freihand.py", line 125加了contours = list(contours)，将contours改为了list类型，再排完顺序后又转为tuple类型，如下图所示，

然后我修改了fh_utils文件打印了idx，确实有列表存在，这是一种错误，还有一种报错freihand路径没有rgb2文件夹，不知道是不是漏了啥没写；

本人刚学习这方面代码，不知道是不是改错了，如有错误之处请指正。谢谢！

Hi Xingyu, Thanks for your great work. I see in the Experiment part of the paper, you use YouTubeHand as the baseline. I wonder would you release this part of code and model?

I got vertex xyz etc. but how can I get axis-angle or something like rotation matrix?

Hi
Thans for your wonderful work. I have some questions referring to 2D hand pose estimtion followed:
1.You train your model on a 3D benchmark,But Your model can still estimation 2D estimation,How do your model achieve?
2.Dose model can train on a 2D benchmark directly?

Thanks

你好，我打开谷歌网盘链接，提示已停止访问该网页。可以提供一下百度网盘的链接吗，谢谢

rt

这个函数报错：
vertex, align_state = registration(vertex, uv_point_pred[0], self.j_regressor, data['K'][0].cpu().numpy(), args.size, uv_conf=uv_pred_conf[0], poly=poly)

ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 6890 is different from 778)

self.j_regressor 长度是778， vertex长度是6890,这里是36m body的点数？要加个转换为778个hand的vertex吧？
还是argcs里我忽略了某些参数？

Hi Sean, thanks for your wonderful work.
I was trying to train the model with our custom data, and inorder to train the model we require mesh files ( .ply files ).
So, even i also wanted to create mesh files for our dataset.
How can I generate those mesh files.?

Thanks in advance :D

I want to run the demo with my own image, but I have no _K.npy for image