writer.CocoAnnotationsWriter creating annotations with empty segmentations and bboxes with 0 area about blenderproc HOT 8 CLOSED

@travis575757 thank you, I believe you are correct. I don't know why this check present in the original pycococreatortools was taken out during refactoring.. Will be reintroduced.

Independent of the Coco Writer, the single pixel segmentations should be gone in newer BlenderProc versions (was related to a denoiser).

from blenderproc.

So we will probably change the "iscrowd" parameter to 0.

This change is now active in version 1.8.2 of BlenderProc.

I assume this issue can then be closed again.

from blenderproc.

@travis575757 Hey, if I understood you correctly, the bbox area of 0 was/is an issue of the original coco code (due to they way it is calculated) thus since we strictly use the original code, this problem was always present in BlenderProc (object being visible by just one pixel in the image leads to the wrong segmentation and bounding box).
For cases such as this one, we have a script that can format the generated annotations by dropping the faulty annotations. I hope this helps.

from blenderproc.

The scripts help. However I've generated 50000+ images before and never had this issue. Looking at src/utility/Coco/pycococreatortools.create_annotation_info I found this.

    @staticmethod
    def create_annotation_info(annotation_id, image_id, category_info, binary_mask, 
                            image_size=None, tolerance=2, bounding_box=None):
        """Creates info section of coco annotation
        Args:
            annotation_id: integer to uniquly identify the annotation
            image_id: integer to uniquly identify image
            category_info: dict which contains category info and a boolean attribute "crowd" which tells if the annotation is a crowd of a class 
        """
        if image_size is not None:
            binary_mask = PycocoCreatorTools.resize_binary_mask(binary_mask, image_size)

        if bounding_box is None:
            bounding_box = PycocoCreatorTools.bbox(binary_mask)
            area = bounding_box[2] * bounding_box[3]

        if category_info["is_crowd"]:
            is_crowd = 1
            segmentation = PycocoCreatorTools.binary_mask_to_rle(binary_mask)
        else :
            is_crowd = 0
            segmentation = PycocoCreatorTools.binary_mask_to_polygon(binary_mask, tolerance)
            if not segmentation:
                return None

        annotation_info = {
            "id": annotation_id,
            "image_id": image_id,
            "category_id": int(category_info["id"]),
            "iscrowd": is_crowd,
            "area": [area],
            "bbox": bounding_box,
            "segmentation": segmentation,
            "width": binary_mask.shape[1],
            "height": binary_mask.shape[0],
        }
        return annotation_info

So based on my problem above I would not experience an empty segmentation list based on this section of code. It seems like recently this file has been deleted and merged with src.utility.CocoUtililty.py which contains this.

    @staticmethod
    def create_annotation_info(annotation_id, image_id, object_id, binary_mask, tolerance=2):
        """Creates info section of coco annotation
        :param annotation_id: integer to uniquly identify the annotation
        :param image_id: integer to uniquly identify image
        :param object_id: The object id, should match with the object's category id
        :param binary_mask: A binary image mask of the object with the shape [H, W].
        :param tolerance: The tolerance for fitting polygons to the objects mask.
        """
        bounding_box = CocoUtility.bbox_from_binary_mask(binary_mask)
        area = bounding_box[2] * bounding_box[3]

        segmentation = CocoUtility.binary_mask_to_polygon(binary_mask, tolerance)

        annotation_info = {
            "id": annotation_id,
            "image_id": image_id,
            "category_id": object_id,
            "iscrowd": 0,
            "area": [area],
            "bbox": bounding_box,
            "segmentation": segmentation,
            "width": binary_mask.shape[1],
            "height": binary_mask.shape[0],
        }
        return annotation_info

Unlike the old code there is no empty list check so you can now receive empty lists. Am I getting something wrong here? Otherwise it seems like BlenderProc did once have this functionality.
Link to the old code I'm referencing

from blenderproc.

I have stumbled over the same issue today, when trying to load the generated data into Detectron2.
Is it still planned to fix this?

from blenderproc.

Hi,

Thanks. Your issue also relates to the polygon mask encoding format, right?

We thought, that we caught the bug by excluding empty binary_masks.

But apparently for some special cases binary_mask_to_polygon can return empty segmentation lists for area>1.

So for the polygon format we will also introduce again

            if not segmentation:
                return None

That said, since this issue we have also integrated the RLE mask encoding format (as default) into the CocoAnnotationsWriter which should be preferred because it can represent masks with holes. The RLE format should not suffer from this issue in newer BlenderProc versions, please let us know if your experience is different :)

from blenderproc.

Hey, thanks for your quick reply.
Indeed, the RLE encoding does not seem to suffer from this bug. At least none of my ~4000 generated annotations do not contain empty segmentation fields. Also, masks with holes are nice :)

The only issue I experienced is that the default iscrowd=1 for RLE leads to Detectron2 ignoring the masks.However, I guess that this should be another issue.

from blenderproc.

Yes, Detectron2 really seems to ignore iscrowd=1 annotations. That is interesting because it is used for RLE annotations in the pycococreatortools:

https://github.com/waspinator/pycococreator/blob/207b4fa8bbaae22ebcdeb3bbf00b724498e026a7/pycococreatortools/pycococreatortools.py#L94-L101

But I also found an issue in their repo which supports the claim that is_crowd should be left out.

waspinator/pycococreator#10

So we will probably change the "iscrowd" parameter to 0.

from blenderproc.