unrecognized arguments and info = coco['info'] about cocosplit HOT 19 OPEN

@minooei Thanks, would you have any python courses that you could refer me to?
see this list:
https://github.com/quobit/awesome-python-in-education#courses-and-lessons

from cocosplit.

@minooei Thanks. I ran and installed it on the terminal: pip install funcy
However when I ran again hoping to generate the three .json, respectively for train, test and val.
(1) This error appeared:

(base) C:\Users\ribei\OneDrive\Documentos\fish_segmentation_detectron2\fish_segmentation_instance\fish_segmentation_instance\Dataset-COCO-Split>python coco_split_train_test_val.py -s 0.6 coco_annotations.json train.json test.json val.json
Traceback (most recent call last):
  File "coco_split_train_test_val.py", line 59, in <module>
    main(args)
  File "coco_split_train_test_val.py", line 32, in main
    info = coco['info']
KeyError: 'info'

you have had this problem one month ago and I answered back then.

from cocosplit.

Please see my https://github.com/e1-io/echo1-coco-split library, based on this library. I've added support for train, valid, test, and a seed parameter for generation. @minooei , I've also fixed the key errors. This is published to pypi.

from cocosplit.

Your coco dataset probably doesn't have info. Remove all instances of info in cocosplit.py, you're trying to reference something that isn't in your initial JSON data.

from cocosplit.

@nofficer, @favervn and @akarazniewicz I ran coco_split via Colab for my custom dataset:

1. !git clone https://github.com/akarazniewicz/cocosplit.git
2. !ls
   Output: cocosplit sample_data
3. %cd cocosplit/
   Output: /content/cocosplit
4. !ls
   Output: cocosplit.py README.md requirements.txt
5. !pip install -r requirements.txt
6. I connected to the drive to access the Data along with the json file.

from google.colab import drive
drive.mount('/content/drive') 

7. There was a syntax error, but I generated a json file with my complete Dataset after making the annotations via Labelme.

python cocosplit.py --having-annotations -s 0.8 /content/drive/MyDrive/Dataset-COCO-Split/coco_annotations.json train.json test.json`

8. Error:

File "<ipython-input-10-a30723afe766>", line 1
    python cocosplit.py --having-annotations -s 0.8 /content/drive/MyDrive/Dataset-COCO-Split/coco_annotations.json train.json test.json
                   ^
SyntaxError: invalid syntax

@nofficer, @favervn and @akarazniewicz Would you help me?

from cocosplit.

@nofficer, @favervn and @akarazniewicz I ran coco_split via Colab for my custom dataset:

1. !git clone https://github.com/akarazniewicz/cocosplit.git
2. !ls
   Output: cocosplit sample_data
3. %cd cocosplit/
   Output: /content/cocosplit
4. !ls
   Output: cocosplit.py README.md requirements.txt
5. !pip install -r requirements.txt
6. I connected to the drive to access the Data along with the json file.

from google.colab import drive
drive.mount('/content/drive') 

1. There was a syntax error, but I generated a json file with my complete Dataset after making the annotations via Labelme.

python cocosplit.py --having-annotations -s 0.8 /content/drive/MyDrive/Dataset-COCO-Split/coco_annotations.json train.json test.json`

1. Error:

File "<ipython-input-10-a30723afe766>", line 1
    python cocosplit.py --having-annotations -s 0.8 /content/drive/MyDrive/Dataset-COCO-Split/coco_annotations.json train.json test.json
                   ^
SyntaxError: invalid syntax

@nofficer, @favervn and @akarazniewicz Would you help me?

it seems you forgot to put an exclamation mark (!) before python command in google colab.
alternatively, you can copy-paste python codes from cocosplit.py in a colab cell.

from cocosplit.

@minooei I tested it by placing (!) Before the python command in Google Colab. However, this error appeared below:

Traceback (most recent call last):
  File "cocosplit.py", line 52, in <module>
    main(args)
  File "cocosplit.py", line 30, in main
    info = coco['info']
KeyError: 'info'

Any suggestions @minooei?

from cocosplit.

@minooei I tried the alternative of copying the code from the cocosplit.py file in a cell and the output was this:

usage: ipykernel_launcher.py [-h] -s SPLIT [--having-annotations]
                             coco_annotations train test
ipykernel_launcher.py: error: the following arguments are required: train, test, -s
An exception has occurred, use %tb to see the full traceback.

SystemExit: 2
/usr/local/lib/python3.6/dist-packages/IPython/core/interactiveshell.py:2890: UserWarning: To exit: use 'exit', 'quit', or Ctrl-D.
  warn("To exit: use 'exit', 'quit', or Ctrl-D.", stacklevel=1)

from cocosplit.

@minooei I tested it by placing (!) Before the python command in Google Colab. However, this error appeared below:

Traceback (most recent call last):
  File "cocosplit.py", line 52, in <module>
    main(args)
  File "cocosplit.py", line 30, in main
    info = coco['info']
KeyError: 'info'

Any suggestions @minooei?

just simply delete lines 30,31 and change line 20:

        json.dump({ 'images': images, 
            'annotations': annotations, 'categories': categories}, coco, indent=2, sort_keys=True)

you need some basic python knowledge.

from cocosplit.

@minooei In the case where there is the term (info) should I delete it?
Also in this line:

save_coco(args.train, info, licenses, x, filter_annotations(annotations, x), categories)
 save_coco(args.test, info, licenses, y, filter_annotations(annotations, y), categories)

from cocosplit.

@minooei In the case where there is the term (info) should I delete it?
Also in this line:

save_coco(args.train, info, licenses, x, filter_annotations(annotations, x), categories)
 save_coco(args.test, info, licenses, y, filter_annotations(annotations, y), categories)

yes, since your dataset does not have any info you can delete or just replace with empty dict : {}
save_coco(args.train, {}, {}, x, filter_annotations(annotations, x), categories)

from cocosplit.

@minooei Thanks, would you have any python courses that you could refer me to?

from cocosplit.

@minooei Do you know if it is possible to change the ratio of the Script to 70 (training) / 15 (test) / and 15 (validation)?

from cocosplit.

@minooei Do you know if it is possible to change the ratio of the Script to 70 (training) / 15 (test) / and 15 (validation)?

you can split the test set in two half and use it for validation

        x, y = train_test_split(images, train_size=args.split)
        z, y = train_test_split(y, train_size=0.5)

        save_coco(args.train, info, licenses, x, filter_annotations(annotations, x), categories)
        save_coco(args.test, info, licenses, y, filter_annotations(annotations, y), categories)
        save_coco('val.json', info, licenses, z, filter_annotations(annotations, z), categories)

from cocosplit.

from cocosplit.

Can you explain to me how the ratio is by placing this line with train_size = 0.5, how much is left for the other train and test? z, y = train_test_split (y, train_size = 0.5) Em seg, 18 de jan de 2021 18:42, mohammad minooei [email protected] escreveu:
…
@minooei https://github.com/minooei Do you know if it is possible to change the ratio of the Script to 70 (training) / 15 (test) / and 15 (validation)? you can split the test set in two half and use it for validation x, y = train_test_split(images, train_size=args.split) z, y = train_test_split(y, train_size=0.5) save_coco(args.train, info, licenses, x, filter_annotations(annotations, x), categories) save_coco(args.test, info, licenses, y, filter_annotations(annotations, y), categories) save_coco('val.json', info, licenses, z, filter_annotations(annotations, z), categories) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AK3B5AR6LYK5UIVCD7MV533S2STMXANCNFSM4NBKBJFQ .

don't let the names of the function and argument makes you confused. train_test_split is just a function that gets an array and split it by a ratio. you can call this function multiple times without affecting previous splits.
https://datascience.stackexchange.com/questions/15135/train-test-validation-set-splitting-in-sklearn

from cocosplit.

from cocosplit.

@minooei Thanks for the explanation.

(1) I formatted the code like this:

import json
import argparse
import funcy
from sklearn.model_selection import train_test_split

parser = argparse.ArgumentParser(description='Splits COCO annotations file into training, test and validation sets.')
parser.add_argument('annotations', metavar='coco_annotations', type=str,
                    help='Path to COCO annotations file.')
parser.add_argument('train', type=str, help='Where to store COCO training annotations')
parser.add_argument('test', type=str, help='Where to store COCO test annotations')
parser.add_argument('val', type=str, help='Where to store COCO val annotations')
parser.add_argument('-s', dest='split', type=float, required=True,
                    help="A percentage of a split; a number in (0, 1)")
parser.add_argument('--having-annotations', dest='having_annotations', action='store_true',
                    help='Ignore all images without annotations. Keep only these with at least one annotation')


args = parser.parse_args()

def save_coco(file, info, licenses, images, annotations, categories):
    with open(file, 'wt', encoding='UTF-8') as coco:
        json.dump({ 'info': info, 'licenses': licenses, 'images': images, 
            'annotations': annotations, 'categories': categories}, coco, indent=2, sort_keys=True)

def filter_annotations(annotations, images):
    image_ids = funcy.lmap(lambda i: int(i['id']), images)
    return funcy.lfilter(lambda a: int(a['image_id']) in image_ids, annotations)

def main(args):
    with open(args.annotations, 'rt', encoding='UTF-8') as annotations:
        coco = json.load(annotations)
        info = coco['info']
        licenses = coco['licenses']
        images = coco['images']
        annotations = coco['annotations']
        categories = coco['categories']

        number_of_images = len(images)

        images_with_annotations = funcy.lmap(lambda a: int(a['image_id']), annotations)

        if args.having_annotations:
            images = funcy.lremove(lambda i: i['id'] not in images_with_annotations, images)

        train_X, test_Y = train_test_split(images, train_size=args.split) # 60% train - 30% teste
        train_X, val_Z = train_test_split(test_Y, train_size=0.50)  # 30%/2 = 0.15 e 0.15

        save_coco(args.train, info, licenses, train_X, filter_annotations(annotations, train_X), categories)
        save_coco(args.test, info, licenses, test_Y, filter_annotations(annotations, test_Y), categories)
        save_coco(args.val, info, licenses, val_Z, filter_annotations(annotations, val_Z), categories)

        print("Saved {} entries in {} and {} in {}".format(len(train_X), args.train, len(test_Y), args.test, len(val_Z), args.val))
        print("Number of images in the train set:", len(train_X))
        print("Number of images in the test set:", len(test_Y))
        print("Number of images in the validation set:", len(val_Z))
        

if __name__ == "__main__":
    main(args)

(2) Then at the terminal I tried to execute the following command:
python coco_split_train_test_val.py -s 0.6 coco_annotations.json train.json test.json val.json

(3) However, the following error occurred, would you have any suggestions:

Dataset-COCO-Split>python coco_split_train_test_val.py -s 0.6 coco_annotations.json train.json test.json val.json
Traceback (most recent call last):
  File "coco_split_train_test_val.py", line 3, in <module>
    import funcy
ModuleNotFoundError: No module named 'funcy'

Would you help me?

from cocosplit.

    train_X, val_Z = train_test_split(test_Y, train_size=0.50)  # 30%/2 = 0.15 e 0.15

simply install funcy by: pip install funcy

and result of the second split should be saved in test_Y , val_Z like this:

        test_Y, val_Z = train_test_split(test_Y, train_size=0.50)  # 30%/2 = 0.15 e 0.15

this means that the test_Y array will be split into two half and be saved in two new variables named test_Y, val_Z.

from cocosplit.