Comments (8)
The test set is hidden, and we do not provide labels for it. Instead you need to do an inference, and use our website https://bit.ly/4D-OR_evaluator for evaluating scene graph generation results on the test set. Hope this helps. feel free to ask if anything is unclear
from 4d-or.
Based on the above error, I am not able to do the inference on your model. Thus it would not be possible to evaluate further online. I'm just curious why the model needs to use ‘GT_True.npz' in the infer phase?
from 4d-or.
You are right, thanks for bringing this to our attention. Indeed that file is not actually used, but the current implementation required it. I now did a quick fix, can you pull the newest version of the repository and see if it works for you now?
from 4d-or.
It can be run after deleting:
if value in human_name_to_3D_joints[pcd_idx]:
instance_label_to_hand_locations[key] = human_name_to_3D_joints[pcd_idx][value][8:10]
in line.88, 89 of dataet_utils.py. Thanks.
BTW, I'm also wondering if the online evaluator is the same as the ‘classification_report’ used for eval? Because 'classification_report' needs to align the number of 'rel_gts' exactly with the number of 'rel_preds' when evaluation.
from 4d-or.
Thanks for the bug fix suggestion.
Your point regarding evaluation is indeed correct, as we state in our readme:
You can evaluate on the test set as well by using https://bit.ly/4D-OR_evaluator and uploading your inferred predictions. Be aware that compared to the evaluation in the paper, this evaluation does not require human poses to be available, and therefore can slightly overestimate the results. We get a macro 0.76 instead of 0.75 on test set.
The results can deviate, however very slightly. Starting from our most recent work https://arxiv.org/abs/2303.13293, we are also reporting the results received from this tool, so it should be comparable.
from 4d-or.
Thanks for your kindly feedback.
Actually, I want to know if 'https://bit.ly/4D-OR_evaluator' is computed in the same way as the metric used for eval:
cls_report = classification_report(all_rel_gts, all_rel_preds, labels=list(range(len(self.relationNames))),
target_names=self.relationNames)
Because I found that if the model does not use 'gt_true.npz' in "evaluate", it leads to a mismatch between the number of 'all_rel_preds' of eval and 'all_rel_preds' of GT, which results in an error in the calculation of 'classification_report'. But this can be successfully runinng on the 'https://bit.ly/4D-OR_evaluator'.
from 4d-or.
Indeed the evaluator deals with this differently. We have a logic that looks like the following:
- Create a template preds variable, initializing with 'none' with the length of ground truth.
- Given the predictions provided by the user (by uploading), for each individual triplet do one of the following:
- If the subject and object combination (tuple) do not occur in the ground truth, dismiss this prediction
- If the subject and object combination occurs in the ground truth, replace the default "none" value with the predicted relation name.
This way, we guarantee all_rel_gts and all_rel_preds being the same. Essentially those predictions that did not match the subject and object in ground truth are thrown out.
from 4d-or.
I've got it, thank you so much!
from 4d-or.
Related Issues (2)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from 4d-or.