Error: Do not have '2_GT_True.npz' when inferring test sets about 4d-or HOT 8 CLOSED

The test set is hidden, and we do not provide labels for it. Instead you need to do an inference, and use our website https://bit.ly/4D-OR_evaluator for evaluating scene graph generation results on the test set. Hope this helps. feel free to ask if anything is unclear

from 4d-or.

Based on the above error, I am not able to do the inference on your model. Thus it would not be possible to evaluate further online. I'm just curious why the model needs to use ‘GT_True.npz' in the infer phase？

from 4d-or.

You are right, thanks for bringing this to our attention. Indeed that file is not actually used, but the current implementation required it. I now did a quick fix, can you pull the newest version of the repository and see if it works for you now?

from 4d-or.

It can be run after deleting：
if value in human_name_to_3D_joints[pcd_idx]:
instance_label_to_hand_locations[key] = human_name_to_3D_joints[pcd_idx][value][8:10]
in line.88, 89 of dataet_utils.py. Thanks.

BTW, I'm also wondering if the online evaluator is the same as the ‘classification_report’ used for eval? Because 'classification_report' needs to align the number of 'rel_gts' exactly with the number of 'rel_preds' when evaluation.

from 4d-or.

Thanks for the bug fix suggestion.

Your point regarding evaluation is indeed correct, as we state in our readme:

You can evaluate on the test set as well by using https://bit.ly/4D-OR_evaluator and uploading your inferred predictions. Be aware that compared to the evaluation in the paper, this evaluation does not require human poses to be available, and therefore can slightly overestimate the results. We get a macro 0.76 instead of 0.75 on test set.

The results can deviate, however very slightly. Starting from our most recent work https://arxiv.org/abs/2303.13293, we are also reporting the results received from this tool, so it should be comparable.

from 4d-or.

Thanks for your kindly feedback.

Actually, I want to know if 'https://bit.ly/4D-OR_evaluator' is computed in the same way as the metric used for eval:

    cls_report = classification_report(all_rel_gts, all_rel_preds, labels=list(range(len(self.relationNames))),
                                       target_names=self.relationNames)

Because I found that if the model does not use 'gt_true.npz' in "evaluate", it leads to a mismatch between the number of 'all_rel_preds' of eval and 'all_rel_preds' of GT, which results in an error in the calculation of 'classification_report'. But this can be successfully runinng on the 'https://bit.ly/4D-OR_evaluator'.

from 4d-or.

Indeed the evaluator deals with this differently. We have a logic that looks like the following:

1. Create a template preds variable, initializing with 'none' with the length of ground truth.
2. Given the predictions provided by the user (by uploading), for each individual triplet do one of the following:

• If the subject and object combination (tuple) do not occur in the ground truth, dismiss this prediction
• If the subject and object combination occurs in the ground truth, replace the default "none" value with the predicted relation name.

This way, we guarantee all_rel_gts and all_rel_preds being the same. Essentially those predictions that did not match the subject and object in ground truth are thrown out.

from 4d-or.

I've got it, thank you so much!

from 4d-or.