Comments (11)
fuqichen1998
commented on June 21, 2024
I can't reproduce the results of hand pose estimation using the provided checkpoint as well. I got even worse results when trained from scratch. Could @stevenlsw please help explain how to reproduce the performance claimed in the paper? Thanks in advance.
from semi-hand-object.
stevenlsw
commented on June 21, 2024
Hi, @fuqichen1998, can you provide more training and testing logs as well as result details. Thanks!
from semi-hand-object.
fuqichen1998
commented on June 21, 2024
Hi @stevenlsw , thanks for your reply! Below is the score output of the results generated using the checkpoint provided:
Collecting open3d-python
Downloading https://files.pythonhosted.org/packages/5f/5c/a86082dc5efc3d22585e8aa22f9840667d9faa5e727b47c43137090caed4/open3d_python-0.7.0.0-cp27-cp27mu-manylinux1_x86_64.whl (3.7MB)
Requirement already satisfied: numpy in /opt/conda/lib/python2.7/site-packages (from open3d-python)
Requirement already satisfied: notebook in /opt/conda/lib/python2.7/site-packages (from open3d-python)
Collecting widgetsnbextension (from open3d-python)
Downloading https://files.pythonhosted.org/packages/d7/31/7c1107fa30c621cd1d36410e9bbab86f6a518dc208aaec01f02ac6d5c2d2/widgetsnbextension-3.5.2-py2.py3-none-any.whl (1.6MB)
Requirement already satisfied: ipywidgets in /opt/conda/lib/python2.7/site-packages (from open3d-python)
Installing collected packages: widgetsnbextension, open3d-python
Successfully installed open3d-python-0.7.0.0 widgetsnbextension-3.5.2
Loading predictions from /tmp/codalab/tmpyXVmia/run/input/res/pred.json
Evaluation 3D KP results:
auc=0.490, mean_kp3d_avg=3.00 cm
Evaluation 3D KP PROCRUSTES ALIGNED results:
auc=0.797, mean_kp3d_avg=1.02 cm
Evaluation 3D KP SCALE-TRANSLATION ALIGNED results:
auc=0.507, mean_kp3d_avg=2.93 cm
Evaluation 3D MESH results:
auc=0.503, mean_kp3d_avg=2.89 cm
Evaluation 3D MESH ALIGNED results:
auc=0.804, mean_kp3d_avg=0.98 cm
F-scores
[email protected] = 0.232 [email protected] = 0.529
[email protected] = 0.685 [email protected] = 0.950
Scores written to: /tmp/codalab/tmpyXVmia/run/output/scores.txt
Evaluation complete.
and below is the content of the option.txt:
====== Options ======
HO3D_root: /mnt/ssd/qichen/ho3d_v2
blocks: 1
channels: 256
epochs: 60
evaluate: True
host_folder: exp_results
inp_res: 512
lambda_joints2d: 100.0
lambda_objects: 500.0
lr: 0.0001
lr_decay_gamma: 0.7
lr_decay_step: 10
mano_lambda_joints3d: 10000.0
mano_lambda_manopose: 10
mano_lambda_manoshape: 0.1
mano_lambda_regulpose: 1
mano_lambda_regulshape: 100.0
mano_lambda_verts3d: 10000.0
mano_neurons: [1024, 512]
mano_root: assets/mano_models
manual_seed: 0
momentum: 0.9
network: honet_transformer
obj_model_root: assets/object_models
resume: /home/qichen/Semi-Hand-Object/pretrained_models/model.pth.tar
save_results: True
snapshot: 10
stacks: 1
test_batch: 24
test_freq: 10
train_batch: 24
transformer_depth: 1
transformer_head: 1
use_cuda: 1
weight_decay: 0.0005
workers: 16
=====================
launched traineval.py at 2021-12-09 02:21:53.174338
and below is the content of object_results.txt
REP-5
{'021_bleach_cleanser': 0.3959691760521636, '006_mustard_bottle': 0.22134831460674156, '010_potted_meat_can': 0.055685131195335275}
ADD-10
{'021_bleach_cleanser': 0.8847065797273266, '006_mustard_bottle': 0.5696629213483146, '010_potted_meat_can': 0.48892128279883385}
I can also provide the logs about training the model by myself and its score.
from semi-hand-object.
stevenlsw
commented on June 21, 2024
Thanks for the info. I am not fully sure the linked model is the right one. I am in travel now. I upload my local saved checkpoints https://drive.google.com/file/d/1Y4fICIY63MA4J1FiY8QfBAF_AKzMEjDn. Could you please @fuqichen1998 help me to take some evaluation. Thanks in advance.
from semi-hand-object.
fuqichen1998
commented on June 21, 2024
The new checkpoint gives the same results as reported in the paper, thanks @stevenlsw. But after training the model from scratch multiple times, I can't achieve, or even get close to, the performance of the new checkpoint. Is there any difference in the training of the released checkpoint?
from semi-hand-object.
stevenlsw
commented on June 21, 2024
Thanks so much for testing. The only difference in the provided checkpoint is that we are incorporating more pseudo labels from Something-v2 dataset as training data.
from semi-hand-object.
fuqichen1998
commented on June 21, 2024
Got it, thanks for your reply!
from semi-hand-object.
fuqichen1998
commented on June 21, 2024
Hi @stevenlsw, after several rounds of training, I could not reach the performance you reported training solely on the HO3D dataset. Could you please provide the instruction to reproduce the performance reported in the paper?
(Also is this:
Semi-Hand-Object/utils/options.py
Line 36 in 1aaf6ee
from semi-hand-object.
dncfjy
commented on June 21, 2024
Hi @fuqichen1998. Could you share the evaluation of groundtruth json to me? Thanks in advance.
from semi-hand-object.
fuqichen1998
commented on June 21, 2024
@dncfjy we don't have it, it's hidden in the backend of the official HO3D challenge evaluation portal.
from semi-hand-object.
dncfjy
commented on June 21, 2024
@fuqichen1998 Now the channel uploaded for the result prediction I do not see on the web page. Can you share a link with me? Thank you very much.
from semi-hand-object.
Related Issues (16)
- Wonder what GPU is needed for reproducing the results，if 2080ti ok？ HOT 2
- Semi-Supervised Learning and Pseudo Labels HOT 1
- How to draw joint figures in your paper? HOT 3
- obtaining object segmentation
- No such file "ho3d-process" HOT 3
- ho3d-process preprocessing HOT 4
- Evaluating the model on SSV2 HOT 9
- YCB objects HOT 4
- How to get obj p2d?Using 3d point projection? HOT 6
- Some questions about get the results provided by your paper in Table 4(supervised with CR)) HOT 9
- Demo Script HOT 2
- Unable to obtain experimental results in the paper（supervised with CR） HOT 4
- Cannot get reasonable inference result
- Access denied on ho3d-process download link HOT 3
- trained-model HOT 9
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from semi-hand-object.