About evaluate_rel_batch() function about reltr HOT 7 OPEN

/home/sherry/anaconda3/envs/graph/lib/python3.8/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or None for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing weights=ResNet50_Weights.IMAGENET1K_V1. You can also use weights=ResNet50_Weights.DEFAULT to get the most up-to-date weights.
warnings.warn(msg)
number of params: 63679528
loading annotations into memory...
Done (t=2.36s)
creating index...
index created!
loading annotations into memory...
Done (t=1.14s)
creating index...
index created!
It is the 149th checkpoint
Traceback (most recent call last):
File "main.py", line 241, in
main(args)
File "main.py", line 183, in main
test_stats, coco_evaluator = evaluate(model, criterion, postprocessors, data_loader_val, base_ds, device, args)
File "/home/sherry/anaconda3/envs/graph/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/sherry/xfn/RelTR/engine.py", line 132, in evaluate
evaluate_rel_batch(outputs, targets, evaluator, evaluator_list)
File "/home/sherry/xfn/RelTR/engine.py", line 219, in evaluate_rel_batch
if evaluator['sgdet'].rel_freq is not None :
AttributeError: 'BasicSceneGraphEvaluator' object has no attribute 'rel_freq'

from reltr.

Hello, I modified the evaluate_rel_batch() function based on your previous suggestions, but the evaluate results are not satisfactory. I would like to understand the possible reasons for this issue. I hope to receive your feedback. Thank you very much. Below are the validation results I obtained：

Test: Total time: 0:58:37 (0.1330 s / it)
======================sgdet============================
R@20: 0.195942
R@50: 0.245618
R@100: 0.270468

relationship: above
======================sgdet============================
R@20: 0.081287
R@50: 0.133741
R@100: 0.145121

relationship: across
======================sgdet============================
R@20: 0.000000
R@50: 0.031746
R@100: 0.031746

relationship: against
======================sgdet============================
R@20: 0.000000
R@50: 0.008065
R@100: 0.024194

relationship: along
======================sgdet============================
R@20: 0.009174
R@50: 0.027523
R@100: 0.027523

relationship: and
======================sgdet============================
R@20: 0.005917
R@50: 0.011834
R@100: 0.017751

relationship: at
======================sgdet============================
R@20: 0.108609
R@50: 0.170085
R@100: 0.184764

relationship: attached to
======================sgdet============================
R@20: 0.022765
R@50: 0.033322
R@100: 0.040987

relationship: behind
======================sgdet============================
R@20: 0.116778
R@50: 0.173084
R@100: 0.220486

relationship: belonging to
======================sgdet============================
R@20: 0.000000
R@50: 0.000000
R@100: 0.000000

relationship: between
======================sgdet============================
R@20: 0.001389
R@50: 0.009722
R@100: 0.014583

relationship: carrying
======================sgdet============================
R@20: 0.139070
R@50: 0.177287
R@100: 0.188527

relationship: covered in
======================sgdet============================
R@20: 0.012500
R@50: 0.014881
R@100: 0.014881

relationship: covering
======================sgdet============================
R@20: 0.029247
R@50: 0.049677
R@100: 0.056935

relationship: eating
======================sgdet============================
R@20: 0.047461
R@50: 0.078366
R@100: 0.090508

relationship: flying in
======================sgdet============================
R@20: 0.000000
R@50: 0.000000
R@100: 0.000000

relationship: for
======================sgdet============================
R@20: 0.038934
R@50: 0.046585
R@100: 0.050683

relationship: from
======================sgdet============================
R@20: 0.000000
R@50: 0.000000
R@100: 0.000000

relationship: growing on
======================sgdet============================
R@20: 0.004310
R@50: 0.004310
R@100: 0.004310

relationship: hanging from
======================sgdet============================
R@20: 0.013114
R@50: 0.033035
R@100: 0.037019

relationship: has
======================sgdet============================
R@20: 0.231985
R@50: 0.287927
R@100: 0.316792

relationship: holding
======================sgdet============================
R@20: 0.195163
R@50: 0.230999
R@100: 0.251340

relationship: in
======================sgdet============================
R@20: 0.101540
R@50: 0.164819
R@100: 0.192017

relationship: in front of
======================sgdet============================
R@20: 0.051342
R@50: 0.096182
R@100: 0.114846

relationship: laying on
======================sgdet============================
R@20: 0.074700
R@50: 0.144520
R@100: 0.151276

relationship: looking at
======================sgdet============================
R@20: 0.021419
R@50: 0.062584
R@100: 0.067938

relationship: lying on
======================sgdet============================
R@20: 0.030612
R@50: 0.051020
R@100: 0.051020

relationship: made of
======================sgdet============================
R@20: 0.000000
R@50: 0.000000
R@100: 0.000000

relationship: mounted on
======================sgdet============================
R@20: 0.000000
R@50: 0.000000
R@100: 0.010417

relationship: near
======================sgdet============================
R@20: 0.091260
R@50: 0.153697
R@100: 0.192226

relationship: of
======================sgdet============================
R@20: 0.194509
R@50: 0.266085
R@100: 0.297834

relationship: on
======================sgdet============================
R@20: 0.219102
R@50: 0.272161
R@100: 0.300958

relationship: on back of
======================sgdet============================
R@20: 0.000000
R@50: 0.021277
R@100: 0.028369

relationship: over
======================sgdet============================
R@20: 0.022333
R@50: 0.038462
R@100: 0.050868

relationship: painted on
======================sgdet============================
R@20: 0.000000
R@50: 0.028736
R@100: 0.028736

relationship: parked on
======================sgdet============================
R@20: 0.138684
R@50: 0.237654
R@100: 0.257504

relationship: part of
======================sgdet============================
R@20: 0.000000
R@50: 0.000000
R@100: 0.000000

relationship: playing
======================sgdet============================
R@20: 0.000000
R@50: 0.000000
R@100: 0.000000

relationship: riding
======================sgdet============================
R@20: 0.287544
R@50: 0.338754
R@100: 0.347416

relationship: says
======================sgdet============================
R@20: 0.000000
R@50: 0.000000
R@100: 0.000000

relationship: sitting on
======================sgdet============================
R@20: 0.117371
R@50: 0.182214
R@100: 0.212227

relationship: standing on
======================sgdet============================
R@20: 0.047672
R@50: 0.097729
R@100: 0.114522

relationship: to
======================sgdet============================
R@20: 0.024590
R@50: 0.028689
R@100: 0.028689

relationship: under
======================sgdet============================
R@20: 0.087786
R@50: 0.125657
R@100: 0.156404

relationship: using
======================sgdet============================
R@20: 0.133542
R@50: 0.177292
R@100: 0.189792

relationship: walking in
======================sgdet============================
R@20: 0.004695
R@50: 0.004695
R@100: 0.018779

relationship: walking on
======================sgdet============================
R@20: 0.046671
R@50: 0.085389
R@100: 0.105228

relationship: watching
======================sgdet============================
R@20: 0.007576
R@50: 0.041021
R@100: 0.065741

relationship: wearing
======================sgdet============================
R@20: 0.342366
R@50: 0.378384
R@100: 0.389855

relationship: wears
======================sgdet============================
R@20: 0.012943
R@50: 0.069027
R@100: 0.073457

relationship: with
======================sgdet============================
R@20: 0.072991
R@50: 0.136298
R@100: 0.148540

======================sgdet mean recall with constraint============================
mR@20: 0.06377901084856602
mR@50: 0.09449064872779793
mR@100: 0.1062562081760442
Averaged stats: class_error: 60.00 sub_error: 50.00 obj_error: 0.00 rel_error: 50.00 loss: 16.0586 (19.7916) loss_ce: 0.3082 (0.4749) loss_bbox: 0.8455 (1.0436) loss_giou: 1.0940 (1.0876) loss_rel: 0.3960 (0.6164) loss_ce_0: 0.3489 (0.5394) loss_bbox_0: 0.8759 (1.1208) loss_giou_0: 1.2037 (1.2298) loss_rel_0: 0.5089 (0.6350) loss_ce_1: 0.3133 (0.5103) loss_bbox_1: 0.8757 (1.0607) loss_giou_1: 1.2299 (1.1330) loss_rel_1: 0.4036 (0.6233) loss_ce_2: 0.2663 (0.4941) loss_bbox_2: 0.8819 (1.0472) loss_giou_2: 1.1390 (1.1086) loss_rel_2: 0.3771 (0.6170) loss_ce_3: 0.2639 (0.4855) loss_bbox_3: 0.8454 (1.0362) loss_giou_3: 1.0848 (1.0923) loss_rel_3: 0.4173 (0.6192) loss_ce_4: 0.2957 (0.4787) loss_bbox_4: 0.8533 (1.0330) loss_giou_4: 1.0675 (1.0837) loss_rel_4: 0.3894 (0.6214) loss_ce_unscaled: 0.3082 (0.4749) class_error_unscaled: 31.5789 (37.4961) sub_error_unscaled: 33.3333 (54.4876) obj_error_unscaled: 33.3333 (47.2545) loss_bbox_unscaled: 0.1691 (0.2087) loss_giou_unscaled: 0.5470 (0.5438) cardinality_error_unscaled: 6.0000 (7.2951) loss_rel_unscaled: 0.3960 (0.6164) rel_error_unscaled: 50.0000 (63.9523) loss_ce_0_unscaled: 0.3489 (0.5394) loss_bbox_0_unscaled: 0.1752 (0.2242) loss_giou_0_unscaled: 0.6018 (0.6149) cardinality_error_0_unscaled: 7.0000 (7.5290) loss_rel_0_unscaled: 0.5089 (0.6350) loss_ce_1_unscaled: 0.3133 (0.5103) loss_bbox_1_unscaled: 0.1751 (0.2121) loss_giou_1_unscaled: 0.6150 (0.5665) cardinality_error_1_unscaled: 7.0000 (7.4123) loss_rel_1_unscaled: 0.4036 (0.6233) loss_ce_2_unscaled: 0.2663 (0.4941) loss_bbox_2_unscaled: 0.1764 (0.2094) loss_giou_2_unscaled: 0.5695 (0.5543) cardinality_error_2_unscaled: 7.0000 (7.5102) loss_rel_2_unscaled: 0.3771 (0.6170) loss_ce_3_unscaled: 0.2639 (0.4855) loss_bbox_3_unscaled: 0.1691 (0.2072) loss_giou_3_unscaled: 0.5424 (0.5461) cardinality_error_3_unscaled: 8.0000 (7.7591) loss_rel_3_unscaled: 0.4173 (0.6192) loss_ce_4_unscaled: 0.2957 (0.4787) loss_bbox_4_unscaled: 0.1707 (0.2066) loss_giou_4_unscaled: 0.5338 (0.5419) cardinality_error_4_unscaled: 6.0000 (6.9804) loss_rel_4_unscaled: 0.3894 (0.6214)
Accumulating evaluation results...
DONE (t=108.10s).
IoU metric: bbox
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.119
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.239
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.103
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.028
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.077
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.163
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.193
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.307
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.314
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.122
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.251
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.374

from reltr.

sorry for the late reply but it seems the image link is invalid.

I would like to ask if these results are based on your reproduced version or our pre-trained checkpoint.
======================sgdet============================
R@20: 0.195942
R@50: 0.245618
R@100: 0.270468
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.239

It seems the results are quite under-scored. If you use our ckpt I remember R@50 should be larger than 25.x and AP50 should be larger than 26.x.

The post-precessing code is not uploaded. With it you will get the scores shown in the PAMI paper. Please refer to this issue #34

from reltr.

抱歉回复了，但图像链接似乎无效。

我想问一下这些结果是基于您的复制版本还是我们预先训练的检查点。 =========================sgdet====== ====================== = R@20: 0.195942 R@50: 0.245618 R@100: 0.270468 平均精度 (AP) @[ IoU=0.50 |面积=全部|最大Dets=100] = 0.239

结果看来是相当低估的。如果您使用我们的ckpt，我记得R@50应大于25.x，AP50应大于26.x。

后续处理代码未上传。有了它，您将获得 PAMI 试卷中显示的分数。请参阅本期#34

Hello, this is the result I obtained by training the VG dataset using the code you provided. I used an RTX 4090 with a batch size of 4 and trained for 150 epochs. For post-processing, I applied the evaluate_rel_batch() function you provided. I will use the provided ckpt for validation. Thank you very much for your response! Have a great day!

from reltr.

I know what the problem might be. In fact, we've found that the larger batch size is not always better. We use bs=2 for the training and use multiple GPUs for acceleration. I think the DETR preprocessing (the padding strategy) is not perfect.

from reltr.

I know what the problem might be. In fact, we've found that the larger batch size is not always better. We use bs=2 for the training and use multiple GPUs for acceleration. I think the DETR preprocessing (the padding strategy) is not perfect.

Thank you very much for your answer!

from reltr.

I would like to understand the possible causes of this issue. I would like to receive your feedback. Thank you. Here are the results of the verification I obtained:
Test: Total Time: 0:58:37 (0.1330 sec/sec)

Can I ask, when you apply evaluate_rel_batch() for post-processing, you don't get this error because rel_freq isn't defined anywhere else? (AttributeError: The 'BasicSceneGraphEvaluator' object doesn't have the property 'rel_freq') Thanks.

Unfortunately, I haven't encountered this issue, so I can't provide an answer for you.

from reltr.