jsbroks / imantics Goto Github PK

Was wondering what does imantics do with occlusions?

I've written a custom library that resolves occlusions and when it generates polygon metadata, it actually has a height property represent their z-index layering.

But basically if an object occludes another object. There are 2 ways to go about this.

The background object can substract the foreground object, thus the background polygon is like having a chunk ripped out of it.

The background object includes the foreground object. But only when the background object is completely surrounding the foreground object.

For the purposes of object detection, I did both. The former for partial occlusions. The latter for complete surroundings.

When exporting image to coco format some information goes both as metadata and exported information, which can be seen here:

'license': self.metadata.get('license'),
'fickr_url': self.metadata.get('flicker_url'),
'coco_url': self.metadata.get('coco_url'),
'date_captured': self.metadata.get('date_captured'),
'metadata': self.metadata

resulting in duplicate information (like coco_url is both inside metadata and on its own key). The keys already exported outside metadata should be removed from this dictionary before exporting.

Following up from details mentioned in #24

When adding an image to a dataset, The dataset index methods calls image.index(self) , here: https://github.com/jsbroks/imantics/blob/master/imantics/dataset.py#L161

The method def index(self, dataset) of image looks ok and makes call to
annotation.index(dataset) for each annotation in the image to index itself into the dataset.

However, the index method of annotation here is

def index(self, image)

Is that correct, shouldnt it be def index(self, dataset)?

Edit - Just noticed, When adding an annotation to an image, annotation gets indexed to the image as well, so def index(self, image) is called here too with image as argument.

The images and annotations were defined as the class variables, which polluted the following instants of the Dataset class. Is this an intended behavior?

Hi, Thanks for your work

How can I get polygon and points from an image? I use this but not working

Mask example:

import numpy as np
from imantics import Polygons, Mask
from PIL import Image
instance_path = r"test.png"
instance_image = Image.open(instance_path)
instance_array = np.array(instance_image)
polygons = Mask(instance_array).polygons()

print(polygons.points)
print(polygons.segmentation)

Hi.

Given an image like the following.

imantics.Mask(img).polygons() generates the resulting 7 polygons. However, is there a way to also denote which class the polygons belong to? That is, if the border of the polygon goes from black to white, let the class be 0, if the border goes from white to black, let the class be 1, or something along those lines.

Best regards.

When creating an annotation, there is an assert that might give the wrong result when one of the asserted objects is an np.array.

    assert bbox != None or mask != None or polygons != None, "you must provide a mask, bbox or polygon"
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

Suggestion: replace by

assert bbox is not None or mask is not None or polygons is not None, "you must provide a mask, bbox or polygon"

I annotated images using PixelAnnotationTool provided here: https://github.com/abreheret/PixelAnnotationTool and using the provided dictionary:
{ "labels": { "unlabeled": { "categorie": "void", "color": [ 0, 0, 0 ], "id": 0, "id_categorie": 0, "name": "unlabeled" }, "bicycle_motorcycle": { "categorie": "bicycle_motorcycle", "color": [ 119, 11, 32 ], "id": 1, "id_categorie": 1, "name": "bicycle_motorcycle" }, "bus": { "categorie": "bus", "color": [ 102, 51, 0 ], "id": 2, "id_categorie": 2, "name": "bus" }, "truck": { "categorie": "truck", "color": [ 0, 0, 70 ], "id": 3, "id_categorie": 3, "name": "truck" }, "pedestrian": { "categorie": "pedestrian", "color": [ 0, 2, 180 ], "id": 4, "id_categorie": 4, "name": "pedestrian" }, "car": { "categorie": "car", "color": [ 0, 180, 0 ], "id": 5, "id_categorie": 5, "name": "car" }, "out_of_roi": { "categorie": "out_of_roi", "color": [ 0, 0, 0 ], "id": 6, "id_categorie": 6, "name": "out_of_roi" } } }

I want to convert these RGB masks into json polygon format so that I can use them in Mask R-CNN: I posted this question in stackoverflow here: https://stackoverflow.com/questions/54802089/convert-an-rgb-mask-image-to-coco-json-polygon-format

Hello,

Thanks for creating this handy package! I have a suggestion. Is it a good idea to have a base_dir argument to the class Dataset, so that Dataset.add(str) can handle the relative path given by os.path.join(base_dir, rel_path). When load from an external file, the image_file paths will be concatenated by the base_dir to find the correct path.

I think this would be useful if one wants to create a self-contained dataset with relative paths.

Thanks.

Is it possible to convert 8bit PNG with multiple classes to json coco annotations with multiple categories?

The function `Dataset.split doesn't return anything, but rather it prints the ratio results.

filename="instances_train2017.json"
with open(filename) as f:
    data = json.load(f)
    dataset = imantics.Dataset.from_coco(data)

    for annotation in dataset.annotations:
        print (annotation)

The above code does not work. The init() in the Image class is overwriting the self. Subsequent initialization in the code are not reflected in caller function, as caller gets different self.

If I comment out the following, then I am able to load the json file.

        if len(path) != 0 and image_array is None:
            pass
            #self = Image.from_path(path)

the algorithm that is used to create the polygon notation from a mask sometimes creates single-point polygons.
When using pycocotools' annToMask, this throws an exception:

import imantics as imt
import numpy as np
from pycocotools.coco import COCO
import json

mask = [[0, 0, 0, 0, 0, 0], [0, 1, 1, 0, 0, 0], [0, 1, 1, 1, 0, 0], [1, 1, 1, 0, 0, 0], [1, 1, 0, 0, 1, 0], [1, 0, 0, 0, 0, 0]]
mask = np.array(mask, dtype='uint8')

dataset = imt.Dataset('dataset')
catt = imt.Category('dummy', id=1)

maskk = imt.Mask(mask)

img = np.reshape(np.array([mask, mask, mask]), [6, 6, 3])
imgg = imt.Image(img, path='hello/world.png')
dataset.add(imgg)

ann = imt.Annotation.from_mask(maskk, image=imgg, category=catt)

dataset.add(ann)
with open('dummy.json', 'w') as f:
f.write(json.dumps(dataset.coco()))
coco = COCO('dummy.json')
new_mask = coco.annToMask(coco.anns[1])

by rejecting those single(or two)-point polygons, this can be solved:
just edit the polygon- method in the Mask-Class and replace:
polygons = [polygon.flatten() for polygon in polygons]
with
polygons = [polygon.flatten() for polygon in polygons if polygon.shape[0] > 2]

Hello,

I recently faced an issue with the tool while converting a very large dataset. The problem is that in the dataset object the image is loaded and save, for drawing purposes. But for very large dataset it means blowing away the RAM...

I don't know if having an option to disable the drawing for large dataset is something we want or not... What do you think ?

Does imantics provide any built-in methods for removing entries in polygons = Mask(maskMorph).polygons()?

I would like to remove identified polygons that are smaller than a given size (say len(polygons.segmentation[l]) < 80), but cannot figure out how to do that with a imantics annotation polygons class.

I'm generating coco jsons from binary masks and correcting them with the COCO annotator tool. However, sometimes, too many control points are generated, which increases the size of the jsons, complicates the correction and results in unnatural masks. I was wondering if it is possible to smooth the generated polygons reducing the number of generated control points

Thanks in advance

When annotation['segmentation'] contains a single polygon list that is not provided as a list of lists but as a single list of coordinates, the logic in https://github.com/jsbroks/imantics/blob/master/imantics/annotation.py#L286 removes the entire segmentation.

I do not get any warnings or errors in my pseudo code below when I start creating a dataset:

                        # Add image to coco database
                        ds_im = Image.from_path(im_path)
                        ds_poly = Polygons(json.loads(im_polygon_path))
                        ds_im.add(ds_poly, category=Category(polygon_label))
                        img_set.append(ds_im)

Perhaps a quick fix for you @george-gca in #58

BBox has 4 properties:

• top_right
• top_left
• bottom_right
• bottom_left

These are wrongly set, i.e.

• top_right must return self._xmax, self._ymax and not self._xmax, self._ymin
• top_left must return self._xmax, self._ymin and not self._xmin, self._ymin
• bottom_right must return self._xmin, self._ymax and not self._xmax, self._ymax
• bottom_left must return self._xmin, self._ymin and not self._xmin, self._ymax

Hi, it's really great to find the project, I have being searching this for several days and almost give up. I sincerely thank you developing this awesome tool, but i meet with a problem using it to build my dataset.

below is my code, basically i loop twice and for each time I read a image and mask, then create annoation, then use Dataset.add method adding them to the dataset

data = imantics.Dataset(name='allen_342')`
for i in tqdm(range(2)):`
    image = imantics.Image.from_path('/mydir/image_{}.png'.format(i))
    mask_array =cv2.imread(/mydir/mask_{}.png'.format(i),cv2.IMREAD_GRAYSCALE)
    mask=imantics.Mask(mask_array)
    ann=(imantics.Annotation.from_mask(mask,image,imantics.Category('cell')))
    data.add(image)
    data.add(ann)
    out = data.coco()

the problem is , i got very strange output which is
for i==0

{'annotations': [{'area': 2915,
                  'bbox': (201, 33, 271, 444),
                  'category_id': 0,
                  'color': '#214596',
                  'height': 512,
                  'id': 1,
                  'image_id': 0,
                  'isbbox': False,
                  'iscrowd': 0,
                  'metadata': {},
                  'width': 512}],
 'categories': [{'color': '#c81940',
                 'id': 0,
                 'metadata': {},
                 'name': 'cell',
                 'supercategory': None}],
 'images': [{'coco_url': None,
             'date_captured': None,
             'fickr_url': None,
             'file_name': 'image_951980471.png',
             'height': 512,
             'id': 0,
             'license': None,
             'metadata': {},
             'path': '/home/seeker/Swin-Transformer-Object-Detection/image/image_951980471.png',
             'width': 512}],
 'info': {}}

for i==1

{'annotations': [{'area': 2915,
                  'bbox': (201, 33, 271, 444),
                  'category_id': 0,
                  'color': '#214596',
                  'height': 512,
                  'id': 1,
                  'image_id': 0,
                  'isbbox': False,
                  'iscrowd': 0,
                  'metadata': {},
                  'width': 512},
                 {'area': 1583,
                  'bbox': (95, 52, 324, 446),
                  'category_id': 0,
                  'color': '#3d1ea0',
                  'height': 512,
                  'id': 2,
                  'image_id': 0,
                  'isbbox': False,
                  'iscrowd': 0,
                  'metadata': {},
                  'width': 512}],
 'categories': [{'color': '#c81940',
                 'id': 0,
                 'metadata': {},
                 'name': 'cell',
                 'supercategory': None}],
 'images': [{'coco_url': None,
             'date_captured': None,
             'fickr_url': None,
             'file_name': 'image_951980484.png',
             'height': 512,
             'id': 0,
             'license': None,
             'metadata': {},
             'path': '/home/seeker/Swin-Transformer-Object-Detection/image/image_951980484.png',
             'width': 512}],
 'info': {}}

allow me to point out the problems.

1. The image do no add, there are still one image
2. the annotations add, but there are only one annotation for each loop, the origin mask have multiple area for each image.
3. the annotation['image_id'] and the images['id'] always 0, however, the file_name did change

I read about the docus, maybe the Dataset is not a "dataset" but a handle to iterator the image and annos? but even though this doesn't make sense anyway. For,example, the new add anno has the same image_id with the previous one.

How can I make ONE json file with images contains multiple object in each one for future train ? Thank you,

Hi,

I have installed imantics-0.1.12 using below command. Installation is successfull.
pip3 install imantics==0.1.12
Output:

Collecting imantics==0.1.12
  Downloading imantics-0.1.12.tar.gz (13 kB)
Requirement already satisfied: numpy in /usr/local/lib/python3.5/dist-packages (from imantics==0.1.12) (1.18.1)
Requirement already satisfied: opencv-python>=3 in /usr/local/lib/python3.5/dist-packages (from imantics==0.1.12) (4.2.0.32)
Collecting lxml
  Downloading lxml-4.5.0-cp35-cp35m-manylinux1_x86_64.whl (5.7 MB)
     |################################| 5.7 MB 5.0 MB/s 
Collecting xmljson
  Downloading xmljson-0.2.0-py2.py3-none-any.whl (14 kB)
Building wheels for collected packages: imantics
  Building wheel for imantics (setup.py) ... done
  Created wheel for imantics: filename=imantics-0.1.12-py3-none-any.whl size=17859 sha256=ed65a9e5f6c947e5dfab558d6ae3cd33f13c8b44aca188aa5b1f496d57e591ec
  Stored in directory: /root/.cache/pip/wheels/a7/25/c6/b6bf5764b15519ac1047c2c0aab8f4255778eb01a7d502ce6a
Successfully built imantics

Getting syntax error while importing Mask, using below command in python3 console.
>>> from imantics import Mask

Getting below error.

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.5/dist-packages/imantics/__init__.py", line 3, in <module>
    from .dataset import *
  File "/usr/local/lib/python3.5/dist-packages/imantics/dataset.py", line 29
    xml_list += list(xml_folder.glob(f"*.{ext}"))
                                              ^
SyntaxError: invalid syntax

Please help me resolve this issue, Thanks in advance.

Let me know if you need more information.

Regards

The notebook does not show the segmentations.

Example usage of the library would be very helpful for users to get started without the need to delve into the code.

When creating a category from coco, for example, it gets 'parent': coco.get('supercategory'),, which is a string.
When exporting to coco format, it does 'supercategory': self.parent.name if self.parent else None,, which fails when parent is a string, not a category.

Traceback (most recent call last):
  File "new.py", line 7, in <module>
    polygons = imantics.Polygons.from_mask(array)
  File "/home/pcfic/anaconda3/lib/python3.6/site-packages/imantics/annotation.py", line 567, in from_mask
    return mask.polygons()
AttributeError: 'numpy.ndarray' object has no attribute 'polygons'

It is showing the error when I try to get the polygon points/segmentation

When I try to export a dataset to COCO (ds.export(style='coco')) with 75k images (200-800kb), each containing a single segmentation, the export function uses around 128GB RAM and 60GB swap.

I assume that images are loaded but not released from memory when coco annotations are generated.

I am using python 3.8.18
opencv-python==4.5.3.56
imantics 3cd6526

Greetings,
I'm trying to convert a labelled dataset consisting of binary masks into coco format, how can I achieve it using imantics?
Thanks!

Split dataset function into x sub-datasets

dataset.split([0.2, 0.2, 0.4])

Hello,

I'm wondering if VOC format is really supported as told iin the description. I don't find any from_voc method. I'm currently writing one that I can pull in the future if you want.

Best

Hello everyone,

I'm converting binary masks created from exported COCO Annotator JSON files. From a JSON file, I generate binary masks using cv2.fillPoly and process them. This results in more masks that are usually next to each other (their pixels touch each other). However, after I generate the annotation segmention (using [list(map(round, polygon)) for polygon in mask.polygons()]) and importing the generated JSON to COCO Annotator, I see that these masks do not touch each other. There is a thin space between them. That's why I wonder if in the imantics conversion some pixels are missed. Another possibility is a bug or a different convention in cv2.fillPoly. I'll investigate the problem, but any help here is welcome.

I really appreciate any help you can provide.

Such that this mypy error dissapears:

main.py:13: error: Skipping analyzing "imantics": module is installed, but missing library stubs or py.typed marker  [import]

this library will likely be used in environments where from PIL import Image is also used, so consider renaming the current Image class to something like AnnotatedImage to avoid namespace collisions.

FIrst of all, thanks a lot for this useful library :)

@jsbroks there are some important fixes committed after v0.1.12 release. When are you planning to publish the v0.1.13 release to pypi?

When Dataset.add(str) is called, the new image does not have a distinct id. It will be overwritten after calling Dataset.add(str) again due to the duplicated id.

I'm working on DeepLabV3 for semantic segmentation on models using cityscapes datasets. I want to extract the polygon coordinates for all segmented objects in an image. Is this possible with imantics? If yes, where can I find relevant example?

imantics 0.1.12 is bugged, prefer to install older version or master branch (that has nearly all fixes).

pip install imantics==0.1.9
or 
pip install git+https://github.com/jsbroks/imantics.git

Note that master branch also has a bug, that invert color order from RGB to BGR.
To use a fixed version:

pip install git+https://github.com/SixK/imantics.git

I am running issues when using the example jupyter notebook:

TypeError Traceback (most recent call last)
in
----> 1 dataset = Dataset.from_xml(Path("data/xml_example"))

~/anaconda3/lib/python3.6/site-packages/imantics/dataset.py in from_xml(cls, xml_folder, name)
27 xml = bf.data(fromstring(open(imgp.with_suffix(".xml"),"r").read()))
28 for ann in xml["annotation"]["object"]:
---> 29 cat = ann["name"]["$"]
30 categories.append(cat)
31 categories = list(set(categories))

TypeError: string indices must be integers

I get this error when try to add images to a dataset. Does this error happen in python3?

    dataset.add(ann_img)
  File "/home/robonaut/.local/lib/python2.7/site-packages/imantics/dataset.py", line 98, in add
    image.index(self)
  File "/home/robonaut/.local/lib/python2.7/site-packages/imantics/image.py", line 156, in index
    annotation.index(dataset)
  File "/home/robonaut/.local/lib/python2.7/site-packages/imantics/annotation.py", line 192, in index
    annotation_index.id += 1
AttributeError: 'dict' object has no attribute 'id'

Both in example.json and image.py there is a typo: it is written fickr_url instead of flickr_url.