Hello
I'm having issues with PyQt5 on my current system and the push buttons for file selection don't work properly, meaning that the navigation bar doesn't pop up and I cannot load the annotations or gt files.

Could you please add an example on how to use this new version of the library using CLI? Just like in the previous release of this tool.

Big thanks!

Thanks for the great tool! However, when I evaluate on VOC, it includes objects marked as difficult in the evaluation. That leads to an unwanted decrease in mAP. Is there a built-in way to exclude these objects from the ground truth and the predictions?

How to reproduce:
Tick "AP per class" and "mAP" then click results. The "mAP" parameter does not show.
Now click everything under "COCO Average Recall (AR) including "mAP" and in the results "mAP" will show

Expected behavior:
"mAP" should show regardless if the other parameters are ticked or not.

hi, @rafaelpadilla

After I ran this command
conda env create -n env_name --file environment.yml

An Error occured, like this

how can i solve this problem?

Thanks a lot.

Is there any way to produce metrics of each image separately?
Or I should split gt/det here?

Here is the .txt ground truth

0 880 291 74 105
0 1001 288 73 99
0 1108 282 103 84
0 1284 330 108 94
0 1531 466 62 71
0 1337 518 85 91
0 1456 732 81 71
0 1163 664 73 82
0 1030 692 91 88
0 1036 602 91 82
0 1065 426 88 84
0 898 620 64 100
0 839 721 56 81
0 770 694 56 111
0 680 466 68 77
0 414 302 68 93
0 322 382 76 82
0 457 564 79 81
0 362 469 87 71
0 570 568 88 88
0 577 472 90 77

Here is the predicted

0 1 1124 711 1225 807
0 1 892 647 984 769
0 1 492 700 553 751
0 1 974 574 1081 700
0 1 1332 599 1399 678
0 1 631 510 706 607
0 1 977 384 1091 477
0 1 693 307 830 462
0 1 1149 407 1217 458
0 1 1341 323 1460 435
0 1 379 368 437 429
0 1 359 270 450 363
0 1 1463 276 1533 349

Here is with the plot that it shows zero, can someone please suggest why is mAP zero when some are overlapping?

Hi!

I have labels and detections that I am using with YOLO. Once I load all of the annotations for groundtruth and detections to the GUI and click "Run", the program crashes.

My issue might be related to the .txt files that I am using, this is an example of one of my .txt files for one of my images:

Example from a .txt file in the groundtruth folder(generated from LabelImg tool):
2 0.771250 0.220000 0.350000 0.335000

Example from a .txt file in the detection folder(generated from darknet detector):
1 0.2522 0.1515 0.2163 0.3143
0 0.6483 0.6727 0.8530 0.4814

I have a classes.txt file that I have also loaded

This is the output from terminal:

Traceback (most recent call last):
File "/Users/zaraadamou/review_object_detection_metrics/src/ui/run_ui.py", line 312, in btn_run_clicked
det_annotations, passed = self.load_annotations_det()
File "/Users/zaraadamou/review_object_detection_metrics/src/ui/run_ui.py", line 154, in load_annotations_det
ret = converter.text2bb(self.dir_dets,
File "/Users/zaraadamou/review_object_detection_metrics/src/utils/converter.py", line 257, in text2bb
if bb_type == BBType.DETECTED and not validations.is_relative_text_format(
File "/Users/zaraadamou/review_object_detection_metrics/src/utils/validations.py", line 369, in is_relative_text_format
if not is_empty_file(file_path):
File "/Users/zaraadamou/review_object_detection_metrics/src/utils/validations.py", line 542, in is_empty_file
for line in f:
File "/Users/zaraadamou/opt/anaconda3/envs/objmetrics/lib/python3.9/codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc9 in position 99: invalid continuation byte
zsh: abort python run.py

-Zara

As per rafaelpadilla/Object-Detection-Metrics#58, it would be great if this could be published as a pip module on PyPi so that people had a consistent, reliable, trustworthy way to generate metrics for their object detection results. It looks like it is already mostly compatible as there is a skeleton setup.py, so hopefully it isn't too difficult. I don't know what restructuring may be required to avoid naming collisions with other packages.

While the repo is now called "review_object_detection_metrics", I think import object_detection_metrics is still a sensible name for the package.

I don't know whether @andreydung or @malpunek would still be interested in helping with this new version.

I'm using the following bounding box co-ordinates:

class_id, xmin, ymin, xmax, ymax (absolute)
0 288 330 985 694
100000003.txt

When I view this ground truth label in the GUI it's not displayed correctly:

If I use the YOLO format co-ordinates it does display correctly:

class_id, centre X, centre Y, width, height (relative)
0 0.5594664998900839 0.6356930616425305 0.6354730515252976 0.4891887385670732

Here's my image:

Please can you help me to understand why this is occurring and advise how I can fix this? (if it's a bug)

When displaying the ground truth statistics would it be possible to display the class name with each bounding box?
This would greatly help with training set verification. Thanks!

When I launch the GUI I'm unable to resize it to fit my screen resolution. I only have the capability to minimize or maximise the window. What this means is I'm unable to see the lower section of the window containing the detail needed for metrics.
Please can this issue be fixed somehow? (without having to change the screen resolution settings on my laptop)

I tested it with 6500 images, it works fine.
But when the number of images comes to more than ten thousnads, the program crash!

The default format for ground truth absolute values is XYWH (ret = converter.text2bb(self.dir_annotations_gt, bb_type=BBType.GROUND_TRUTH)).

Since this is not documented anywhere I had to look at the code to try and find why my GT bounding boxes were not matching.

Ideally there should be two format options for Absolute Values, XYWH and XYX2Y2.

I've managed to successfully display the ground truth statistics but when I try to run the metrics I receive the following error:

My files containing the classes looks like this:

hrsc2016_class_lev2.txt

I'm using YOLO annotation and trying to run this on my training set.

The results format as specified by COCO cannot be parsed.
[{
"image_id": int, "category_id": int, "bbox": [x,y,width,height], "score": float,
}]

Throws error:
(vendor-eval) C:\Vendor evaluation2\review_object_detection_metrics>python run.py
Traceback (most recent call last):
File "C:\Vendor evaluation2\review_object_detection_metrics\src\ui\run_ui.py", line 258, in btn_statistics_det_clicked
det_annotations, passed = self.load_annotations_det()
File "C:\Vendor evaluation2\review_object_detection_metrics\src\ui\run_ui.py", line 115, in load_annotations_det
ret = converter.coco2bb(self.dir_dets, bb_type=BBType.DETECTED)
File "C:\Vendor evaluation2\review_object_detection_metrics\src\utils\converter.py", line 31, in coco2bb
if not validations.is_coco_format(file_path):
File "C:\Vendor evaluation2\review_object_detection_metrics\src\utils\validations.py", line 302, in is_coco_format
return is_json(file_path) and json_contains_tags(file_path, [
File "C:\Vendor evaluation2\review_object_detection_metrics\src\utils\validations.py", line 115, in json_contains_tags
for key, item in json_object.items():
AttributeError: 'list' object has no attribute 'items'

Hi. Thanks for the clean repository!

I am slightly confused in this part of the code: https://github.com/rafaelpadilla/review_object_detection_metrics/blob/main/src/evaluators/tube_evaluator.py#L217

Shouldn't it be iou = inter / union ?

The coordinates format is ok but why it showing?

Hi all, thank you for this great tool.

I am using the code here from Python (need to test and compare a model trained in various scenarios) and would like to see some improvements around the usage from code.
I can do the code changes around most of this but wanted to make sure it is welcome here in the current community, as they might change a lot of existing code.

The changes that jumped into mind:

• Renaming the main package name from src to some more useful name
• Publishing as a pip package
• Building a documentation site from sources
• Not opening a plotly window right on import (from src.evaluators.coco_evaluator import get_coco_summary makes a window to show)

Is there any interest besides me in such king of changes?

I understand how it is possible to add the class name to the detection box, but is it possible to add both the class name and confidence score?
Even better would be a colour-coded detection box to distinguish between a True Positive and a False Positive.

Hi @rafaelpadilla Sir,
Hope you are fine. Sir i would like to know you that i am using your project and i am facing trouble with the your projects UI interface in windows 10 (64 bit), as i am not not able to resize, close or minimize the UI interface; its freezing as it is appearing on my windows screen. And i am assuming some related warning showing up on command prompt which i tried to solve but failed.
Here i am giving you some screenshot of my command prompt and windows screen.
Hoping to hear you soon...
Thanks in advance!

Command Prompt ::

Windows Screen ::

Hi,

I have compared the pycocotools PythonAPI and this tool. Following are the results I found for my object detection model

Pycocotools

Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=0.00s).
Accumulating evaluation results...
DONE (t=0.00s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.640
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.880
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.880
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.692
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.690
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.690
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.690
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.690
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000

This Tool

COCO METRICS:
AP: 0.6927034910485187
AP50: 0.9877452861565228
AP75: 0.9499113594946688
APsmall: nan
APmedium: nan
APlarge: 0.6927034910485187
AR1: 0.7631782945736434
AR10: 0.7631782945736434
AR100: 0.7631782945736434
ARsmall: nan
ARmedium: nan
ARlarge: 0.7631782945736434

PASCAL METRIC (AP per class)
MRZ: 0.9950323498187128

PASCAL METRIC (mAP)
mAP: 0.9950323498187128

Has anyone else encountered this? or care to comment.

Is there any way to print tables with columns for image_frame, TP, FP for all classes in the ground truth like a summary report? I need to make a comparison based on the number of TP and FP detections for all the classes and need to know in which image_frame the FP occurred.

Is it possible to run it from command line, in a similar way as https://github.com/rafaelpadilla/Object-Detection-Metrics?

The error message is

(metrics) maged-ubuntu-20@magedubuntu20-virtual-machine:~/Desktop/PyCharm_Projects/16_metrics_measurement$ python run.py
/home/maged-ubuntu-20/anaconda3/envs/metrics/lib/python3.9/site-packages/numpy/core/fromnumeric.py:3372: RuntimeWarning: Mean of empty slice.
  return _methods._mean(a, axis=axis, dtype=dtype,
/home/maged-ubuntu-20/anaconda3/envs/metrics/lib/python3.9/site-packages/numpy/core/_methods.py:170: RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)
/home/maged-ubuntu-20/anaconda3/envs/metrics/lib/python3.9/site-packages/numpy/core/fromnumeric.py:3372: RuntimeWarning: Mean of empty slice.
  return _methods._mean(a, axis=axis, dtype=dtype,
/home/maged-ubuntu-20/anaconda3/envs/metrics/lib/python3.9/site-packages/numpy/core/_methods.py:170: RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)
/home/maged-ubuntu-20/anaconda3/envs/metrics/lib/python3.9/site-packages/numpy/core/fromnumeric.py:3372: RuntimeWarning: Mean of empty slice.
  return _methods._mean(a, axis=axis, dtype=dtype,
/home/maged-ubuntu-20/anaconda3/envs/metrics/lib/python3.9/site-packages/numpy/core/_methods.py:170: RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)

Here is my ground truth file

0 412 130 95 104
0 521 82 126 106
0 521 314 226 155
0 820 372 123 101
0 473 616 101 120
0 956 350 98 105
0 177 581 110 88
0 680 638 125 120
0 958 611 135 113

Here is my predicted truth file

0 0.99 699 1012 778 1079
0 0.99 529 1014 591 1079
0 0.99 0 997 77 1079
0 0.99 77 1003 251 1079
0 0.99 367 1027 454 1079
0 0.99 1458 975 1579 1077
0 0.99 1274 882 1425 1038
0 0.99 952 925 1056 1033
0 0.99 500 978 554 1026
0 0.99 1788 952 1865 1021
0 0.99 733 913 877 1004
0 0.99 196 901 305 998
0 0.99 1840 870 1919 965

As you can see all accuracy score is zero

Can someone please help with how to fix this issue?

Hi, thanks for the tool!

I think the condition to set the coordinates for relative bounding boxes are swaped:

from src.bounding_box import BoundingBox

# a GT bbox with relative coordinates in the XYWH format
bbox = BoundingBox(
    "img_filename",
    "category_class",
    (
        0.04287003610108303,
        0.04801829268292683,
        0.22021660649819497,
        0.11178861788617887,
    ),
    CoordinatesType.RELATIVE,
    (540, 960),
    None,
    BBType.GROUND_TRUTH,
    BBFormat.XYWH,
)

print(bbox)

>>> image name: img_filename
class: category_class
bb (XYWH): (-36.0, -8.0, 119.0, 108.0)
bb (X1Y1X2Y2): (-36.0, -8.0, 83.0, 100.0)
area: 13080.0
bb_type: None

If I change it to the XYX2Y2 format:

# a GT bbox with relative coordinates in the XYWH format
bbox = BoundingBox(
    "img_filename",
    "category_class",
    (
        0.04287003610108303,
        0.04801829268292683,
        0.22021660649819497,
        0.11178861788617887,
    ),
    CoordinatesType.RELATIVE,
    (540, 960),
    None,
    BBType.GROUND_TRUTH,
    BBFormat.XYX2Y2,
)

print(bbox)

>>> image name: img_filename
class: category_class
bb (XYWH): (23.0, 46.0, 96.0, 61.0)
bb (X1Y1X2Y2): (23.0, 46.0, 119.0, 107.0)
area: 6014.0
bb_type: None

I can successfully generate 1 detection box for 1 image but when I try to add another image to my detections folder which contains 2 detections in the image the GUI crashes and closes.
Is it because of the FP picked up in the 2nd image?

Here are my images:

ground truth:
100000003.txt
100000005.txt
(2nd image has 1 labelled object)
detections:
100000003.txt
100000005.txt
(2nd image has 2 detections, one with very low confidence.)

class names:
hrsc2016_class_lev2.txt

Once the ground-truth or detection statistics viewer is displayed I'm unable to resize it either by maximizing or dragging with the cursor. The image window is quite small which makes things difficult to see when objects, bounding boxes and labels are small or clustered together.
Please would it be possible to add the capability to resize this window and collapse the statistics box?

Can't not choose coco annotation file, because coco annotations just one file. Your tool just can choose a directory.

what's wrong with my operations?

Hi Devs!

I think there is an issue when validating that coordinates are absolute. Currently when having absolute but not integer coordinates the validation fails.

The issue seems to happen here. Coordinates are checked to be integers instead of absolute values. Changing int to abs should fix this.

Getting this error when prediction.json is used in detection section

predication.json file format

Is there a possibility to add OKS metrics to this? I can write the code and submit a PR is that's acceptable

Hi there, would you be able to help with how to get the False Negative number? Thanks.

Hi

I have loaded ground truth labels as pascal VOC (xml) and their images and it successfully plotted them (on show ground truth statics). I also made my detection labels as the toyexample for classid confidence left top height width (absolute) as below:
0 0.99 37 143 98 206
and I just simply gave the directory of all detections. however, the GUI give the error of "For the selected annotation type, it is necessary to inform a directory with the dataset images. Directory is empty or does not have valid images."

So I just copied all the images (as the ones I gave in first part) to the detection labels directory. But it still give me the mentioned error. My images are 300*300 png frames.

Thanks for support.

It would be really nice to calculate 'segm' metric. Any plans to add it soon?

If I choose class name confidence left top right bottom (ABSOLUTE) pruducing error as coordinates format for Detections and I want to show the detection statistics, I always get the UI error: "No file was found for the seleted detection format in the annotations directory" and python warning: "Image not found in the directory None. It is required to get its dimensions"

Does anyone know how to fix this issue?

Hi,

Would it be possible to also plot a PR curve with all classes in it?

Thanks,
Eduardo

Hi,

I found an error here:

I think it should be convert.yolo2bb.

I got also a problem with my own data.
When using 7 0.94 0.475215 0.524656 0.884800 0.873787 from one of your toyexamples for relativ coordinates the detection gets counted. But if i use my own data which looks like this 0 0.74 0.271635 0.153846 0.687500 0.427885 there are no detections at all. The groundtruth looks like this: 0 0.1466346153846154 0.4230769230769231 0.09134615384615384 0.08173076923076923 and gets recognized perfectly. Is there a reason for this?

I have all my .txt annotations in the directory file, with the correct format (the 1st one in the list), where does this error comes from then ?

Hii, Thank you so much for this, saved me a ton of time!

I have an issue with the Pascal VOC implementation:
I was reading the Pascal VOC 2012 doc which defines AP like,

I was wondering why the outputs are not monotonically decreasing when I run this code?

The green box is the ground truth and the red the predicted box. If I set the iou threshold to 0.5 I would consider this prediction as a bad prediction, so it would be a FP. But as the ground truth is not detected would it count as well as a FN?

I mean does this count as FP, or as a FP and FN?

Thank you.

I try to use the program while having YOLO-like text annotations per image in the following format.

Ground truth annotations:

6 0.4435201401050788 0.6020274689339438 0.059544658493870466 0.14061478090255072
8 0.36646234676007006 0.18999345977763243 0.05341506129597193 0.13276651406147807
10 0.36274080560420313 0.289404839764552 0.05647985989492113 0.17593198168737742
7 0.1725043782837128 0.36527141922825374 0.08231173380035027 0.20078482668410727
11 0.2539404553415061 0.7432962720732506 0.06917688266199651 0.1366906474820143
11 0.20556042031523644 0.05493786788750817 0.06348511383537656 0.10987573577501634

Predictions:

6.0 0.9641061425209045 0.7914067506790161 0.36787542700767517 0.07676965743303299 0.20017971098423004
2.0 0.9538318514823914 0.4433167576789856 0.3955462872982025 0.0672389417886734 0.14360184967517853
3.0 0.9517371654510498 0.6636454463005066 0.14527581632137299 0.050998225808143616 0.11069779098033905
8.0 0.9489786028862 0.5190893411636353 0.3004150092601776 0.049637917429208755 0.10972640663385391
3.0 0.9478994607925415 0.6771969199180603 0.2292448729276657 0.05333113670349121 0.10690070688724518
3.0 0.9314520955085754 0.699550211429596 0.40322744846343994 0.05508119612932205 0.11604183167219162

when having the file structure:

images
   a.jpg
   b.jpg
   c.jpg
annotations
   a.txt
   b.txt
   c.txt
predicted_labels
   a.txt
   b.txt
   c.txt

However, I have the following error: No file was found for the selected detection format in the annotations directory.
What goes wrong?

Coordinates format for Gt: (*)YOLO (.txt)
Coordinates format for Detections: <class_id> <x_center> <y_center> (REL)

(By the way, why images are expected for evaluating labels only?)

Hello @rafaelpadilla,

first thanks for this repo, it's been very useful! However, for my work I needed to compute confusion matrices to understand better why my model was not performing better on some classes. I saw that on the "old repo", there were two issues related to that (https://github.com/rafaelpadilla/Object-Detection-Metrics/issues?q=confusion+matrix) but no implementation has been done.
Given that I implemented a function to compute confusion matrix based on your repo, would you be interested if I implement this as a new feature in this repo?
I based the logic of my function on that article https://towardsdatascience.com/confusion-matrix-in-object-detection-with-tensorflow-b9640a927285

See below the logic of the algorithm

Let me know,

Benoit

Hi @rafaelpadilla . I used mmdetection toolbox for my project. After detection, I can get detection results in JSON format. But I couldn't use this file with your toolbox. How can I solve that? I read the same issue but I don't know that how can I convert my son to your JSON format? You can reach my JSON with the help of the URL that followed;

https://drive.google.com/file/d/19zrA94O1CjP9H-ogkuy5Tid0XKDXFamx/view?usp=sharing

Issue:
When I provided a folder path which has absolute annotations files (.txt), tool also trying to read metadata file ".DS_Store" (macOS) present in the same folder. This crashing the GUI tool.

Suggestion for fixing: Validate the file extension before accessing/reading.

Hi.
I've been trying to find the mAP score for my dataset using this tool, but it can't find the detection annotations. I'm following the format specified:
<class_name> <confidence> <left> <top> <right> <bottom>

My Dataset has two classes, 'person' and 'vehicle' and I've listed them in the aider-classes.txt as per VOC format.

Here is a sample file in my detections annotation directory:

person 0.9996482 224 69 248 116 
person 0.9983903 212 160 231 202 
person 0.7466929 241 89 252 117 
vehicle 0.9925392 103 133 163 236 

The environment setup is all as per the environment.yml file provided. It loads the ground truths correctly but fails to identify the detections annotations.

Could you please help me investigate this?

5.4.0-72-generic #80~18.04.1-Ubuntu

I've managed to calculate the F1-score here:
https://github.com/rafaelpadilla/review_object_detection_metrics/blob/main/src/evaluators/pascal_voc_evaluator.py

But I'm having difficulty in understanding how this metric can be added to the results output window.
Please can you give some guidance on which parts of the code need to be edited? thanks!

Hi,
is there any way to make the plots axis limited to a fixed value? (e.g. always show axis from 0 to 1)
This would make comparing different results with each other easier.
Thanks for any help in advance.

Hi
I found no result output after pressing the "run" button.
In the terminal it outputs "[QNSApplication macMinorVersion]: unrecognized selector sent to instance 0x7f8f8579a940" .
I' m running this application on Mac os.
Hope get your help, thanks!