Inconsistent predictons using refinedet model about deepdetect HOT 12 CLOSED

Hi @YaYaB here is a fix that makes your example work here #1210
...
shortest patches sometimes are the longest to produce

from deepdetect.

Hi, have you tried with / without CUDNN ? CUDNN non deterministically chooses internal algorithms depending on batch size. The non deterministic component comes from rolling dice on some decisions, AFAIK.

The fact that there's no issue with classification models may rule this out, but it is still worth testing.

@fantes if not CUDNN, it may be in one of the bbox output layers ? (e.g. DetectionOutput ?).

from deepdetect.

I tried with a cpu image but it can not even load the model because of a caffe error (refinedet can not be used with cpu only?). I am building a version without cudnn to see if that works but I am not sure if I will be able to create the service before trying predictions.

[2021-02-24 15:01:41.628] [caffe] [error] Layer conv1_1 has unknown engine.
[2021-02-24 15:01:41.638] [face_512] [error] Error creating network
[2021-02-24 15:01:41.640] [face_512] [error] service creation call failed: Dynamic exception type: CaffeErrorException
std::exception::what: /opt/deepdetect/build/caffe_dd/src/caffe_dd/include/caffe/llogging.h:194 / Fatal Caffe error

from deepdetect.

maybe you built w/o cpu but with USE_CUDNN=ON , in case of refinedet and CUDNN set to on at compile time, we force cudnn engine, but if it is not built into caffe it will fail. Could you double check that ?

from deepdetect.

The caffe gpu build without CUDNN just finished and I get tthe same error when trying to create the service. I've check that I specified DUSE_CUDNN=OFF that was also printed when building the image.

[2021-02-24 17:09:07.103] [caffe] [error] Layer conv1_1 has unknown engine.
[2021-02-24 17:09:07.104] [face_512] [error] Error creating network
[2021-02-24 17:09:07.104] [face_512] [error] service creation call failed: Dynamic exception type: CaffeErrorException
std::exception::what: /opt/deepdetect/build/caffe_dd/src/caffe_dd/include/caffe/llogging.h:194 / Fatal Caffe error

from deepdetect.

Hi @YaYaB
did you clean up your repo between your tests w/o cudnn ? maybe your first tests added cudnn engine param in deploy.prototxt (you can search "engine" in deploy.prototxt to check out) and subsequent tests do not support it (because of no cudnn at compile time in dede/caffe ). When cudnn is not compiled in dede, this keyword should not be added (but it is not removed if present, we did not think about this case) . (I'd like to solve this engine problem before looking at the main issue)

from deepdetect.

@fantes Ah indeed, the cudnn engine is indicated in the deploy.prototxt, my bad I should have checked this. I've removed it and it runs correctly. However a more verbose error indicated that the deploy is set for cudnn could be helpful to debug :)

Then, without cudnn I still obtain differences.

Number of documents: v1(98), v2(98)
Number of predictions: 98
1_diff_mean	 	0.0004954249122921301
1_diff_std	 	0.004904456410707221 
1_diff_max	 	0.048551641404628754 
1_diff_min	 	0.0 

from deepdetect.

ok, thx, I am starting investigations

from deepdetect.

Nice, It happens to everyone then ^^
I'll try it first thing tomorrow, thanks @fantes

from deepdetect.

Congrats to both of you @YaYaB and @fantes for catching this one! Good detailed report + careful inspection are rewarded!

from deepdetect.

from deepdetect.

Sorry for the late answer. It seems to work correctly on my side, cheers @fantes @beniz

from deepdetect.