Network is not learning. mAP stays the same and Total Loss does not go below 6 about handtracking HOT 7 CLOSED

Hi @gustavz

• Zero mAP
  This is quite unusual, and I agree with you that an mAP of 0 suggests there is something wrong with the labelling. Given that the training is initialized from pretrained weights, its unlikely that its predictions are wrong all the time.

• Batch Size
  This was selected only because of memory issues.

• Original CSV files
  I am attaching my initial train/test label csv files, which you can compare to what is generated by the egohands script in the repo. I did some code cleanup after I built the original model so there is a slight possibility something changed slightly.

csvs.zip

Other Thoughts

• Something you could do is verifying if training proceeds reasonably using only the tf records before combining it with the oxford dataset.
• Also see the note here tensorflow/models#1863

Let me know how you resolve this.
P.S. I also wanted to combine both datasets but just havent had time to get around to it! Looking forward to learning more on your findings.

Thanks.
-V.

from handtracking.

from handtracking.

I compared your csv files with mine and they are totally different.
Yours are perfectly ordered by name and mine are all shuffled.
Furthermore the values are not the same and the sets are of unequal size.

Here is an example for unequal values (see rest attached):
csvs_gustav.tar.gz
YOURS:

filename,width,height,class,xmin,ymin,xmax,ymax
CARDS_COURTYARD_B_T_frame_0011.jpg,1280,720,hand,647,453,824,551
CARDS_COURTYARD_B_T_frame_0011.jpg,1280,720,hand,515,431,622,543

MINE:

CARDS_COURTYARD_B_T_frame_0011.jpg,1280,720,hand,803,623,981,718
CARDS_COURTYARD_B_T_frame_0011.jpg,1280,720,hand,679,340,856,464
CARDS_COURTYARD_B_T_frame_0011.jpg,1280,720,hand,511,299,676,428

But the thing is, i did not create the csv files by my own, i used your script to create them from the default dataset.

How did you modify your code after the commit zu github? Could you update it?

Would be very thankfull!

from handtracking.

Hmmm ... I actually did a similar check on Friday before I sent you my original csv's. I generated train and test csv's using the script and compared them and there were similar values for the same images. Can you retry the process? I'll suggest you pull a fresh copy of the repo and start from scratch (i.e in a new folder ... where the dataset gets downloaded from scratch).
Let me know if you still get different values.

-V.

from handtracking.

I did a complete delete and re-clone of your repo.
But the results are still the same, the csv are still in a shuffled order and train and test sets constist of different images with different bboxes than your csv files.

What could be the reason? I am using python 2.7 and openCV 3.3.1 if this could be important.

from handtracking.

Hey Victor,
i still have not found the reason why my machine produces completely different csv files with the same code you are using.
Have you ever experienced something similar with creating csv files?
Could you maybe share your train/test directories including images and the final csv files?

EDIT: maybe it is something with the random 10% test dataset creation?
Could you come up with a version of your function def split_data_test_eval_train(image_dir):
that is not using random but maybe a fixed number?

EDIT2: Ok the test and train datasets created by your dataset can't be equal each time you run the code as you dont use random.seed()
But eitherways the code will work differently on python 2.x and 3.x because of this:
https://stackoverflow.com/questions/11929701/why-is-seeding-the-random-generator-not-stable-between-versions-of-python

from handtracking.

Are you sure you convert the bounding boxes correctly?
Where do you set your coordinate center in the image? Top left or Bottom left?

Also i saw that You got some unused variables in your get_bbox_visualize() function:

index = 0
height = width = 0

idk if this was on purpose or forgotten for some value convertation?

furthermore in your readme you write that you split the data in three sets, but actually you only split in 2two, that's a bit confusing.

CLOSE REASON: sorting the image array solved all problems

from handtracking.