kevinsbarnard / m3-download Goto Github PK

I would like to add the ability to convert localizations to the YOLO format, including the names file.

These might be helpful resources:
https://github.com/ultralytics/JSON2YOLO
https://github.com/ssaru/convert2Yolo
https://github.com/ultralytics/yolov5

Thanks!

Please prepend images and all subsequent data files (i.e., voc xml, for example) with the imaged_moment_uuid. I think for long term tracking we'll need that information, in case we need to get back any additional metadata in VARS.

When doing a search for a large class and descendants, for example Actiniaria with the flags -a -d, the generate digest job fails without much indication as to what's going on. 1) I would like to be able to do this on larger, higher level classes like Actiniaria and 2) if it fails it should tell me why. Any chance of making it so we can get at larger sets like this?

Running the following command results in the error below. I presume this is because the query is too big?
Can we investigate this?

python generate_digest.py -a 'Nanomia bijuga'

Getting observations for Nanomia bijuga...
Traceback (most recent call last):
File "/Users/lonny/Documents/Code/Python/4_m3-download/generate_digest.py", line 125, in
main(args.concept, args.config, args.descendants, args.all)
File "/Users/lonny/Documents/Code/Python/4_m3-download/generate_digest.py", line 41, in main
json_data = get_fast_concept_images(config, concept)
File "/Users/lonny/Documents/Code/Python/4_m3-download/lib/m3_requests.py", line 11, in get_fast_concept_images
return requests.get(config('m3', 'fastconceptimages') + '/' + concept).json()
File "/Users/lonny/.pyenv/versions/3.9.5/lib/python3.9/site-packages/requests/models.py", line 900, in json
return complexjson.loads(self.text, **kwargs)
File "/Users/lonny/.pyenv/versions/3.9.5/lib/python3.9/site-packages/simplejson/init.py", line 525, in loads
return _default_decoder.decode(s)
File "/Users/lonny/.pyenv/versions/3.9.5/lib/python3.9/site-packages/simplejson/decoder.py", line 370, in decode
obj, end = self.raw_decode(s)
File "/Users/lonny/.pyenv/versions/3.9.5/lib/python3.9/site-packages/simplejson/decoder.py", line 400, in raw_decode
return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Kevin,

I've started trying to work with the VOC xml from m3-download and found some issues which make it unusable.

1. The files have png.xml as the extension and therefore, they don't get matched up with the image because the names are different. This is an issue for RectLabel and othe ml frameworks as well.
   00_00_24_10.xml
   00_00_24_10.png.xml

2. The bounding box tags are named incorrectly. bnxnox in your files should be bndbox:
   
   946
   512
   1625
   1080
   
   should be:
   
   946
   513
   1625
   1080
   

3. The image size field is coming up as 0:
   
   0
   0
   3
   

should be:

1920
1080
3

I've attached the actual files (zipped) so you can see how they differ.
Thanks,
LL

xml_files.zip

I often get errors like this when I convert to VOC. The current solution is to delete all localizations from the localizations.json file associated with the offending image reference uuid. The image actually exists so I'm not sure what the problem is. This is an example is from Psolus squamatus with the -d and -a flags, in case you want to reproduce the error.

Traceback (most recent call last):
File "/Users/lonny/Documents/Code/Python/4_m3-download/reformat.py", line 103, in
main(_args.localizations, _output, _args.format.upper(), _args.voc_images)
File "/Users/lonny/Documents/Code/Python/4_m3-download/reformat.py", line 71, in main
annotation_record.add_annotation(iruuid, locs, voc_images)
File "/Users/lonny/Documents/Code/Python/4_m3-download/lib/localization.py", line 278, in add_annotation
raise ValueError(f'No image found for image reference UUID {image_reference_uuid}')
ValueError: No image found for image reference UUID be75ebb2-0027-4ccb-8027-3b7bf33a0a54