derenlei / unity_detection2ar Goto Github PK

Hi again,

I wanted to open up a new issue about this. I trained a custom model using Darknet. It is trained to detect only one class, a pillbox. And when tested in Darknet, it did. I trained the model for 4000 iterations.
Then I converted it to Tensorflow, froze the graph and converted it to ONNX successfully. Then I was able to import it into Unity.
I adjusted the anchor values to my own and filled out the Detector in the Inspector. But, unfortunately, nothing is detected when I build the app on Android. But there are also no errors.
What could the mistake be? Could it have something to do with the input shape?

Any help is very apprechiated :)

Greeting again,

Currently when I testing the app, the app will start freeze when the app detect more than 2 objects.
I just wonder that is it possible to allow the app to stop scanning after 2 objects is detected?

by the way, this is the Unity Version I'm using,
Unity3D 2020.3.16f1
and here are the phone that I'm testing with,
Samsung s10 Plus

Edit: I also using YOLO 3 Tiny to test to scan everyday item.

Greetings!

I tried running this project with my custom model and it seems to work somewhat fine, it detects the objects (2 classes) but it does not show bounding boxes on my mobile.

The reason is that it gives irrealistic width, height values for the object which cannot be seen on the mobile screen(2220x1080) as seen in the picture below:

The original model seems to work fine:

This is the part where I added the debug messages in PhoneARCamera:

Another thing that bugs me is that the position values are negative with my custom model but it still places the anchor to the object in real life, so it can certainly detect it, but something is off. Any idea why this might be?

Is there a way to estimate a detected object's rotation(custom dataset)? I know that it is not a 3D object detection project.
For example with a detected object's key features, like a human's eyes but with an object's typical part, so something that is clearly associated with the front. I don't have 3D CAD Models.

Dear, I'm trying to use a YOLO-v2 tiny from onnx model zoo and I change the name of input and output in the Detector script bu the model does not work.
Analyzing the model I see there are some differences, like constants and number of weights. Can you share your yolo configuration to obtain the same model with my train data?

when start and some bounding box appear, device stay caton status and program should be restarted and again caton...

Hello, do you think barracuda can support Yolov7?

could you please provide some docs on how to integrate the yolov3 detector into our own project ? i've reverse engineered 4 proyects this week already and would appreciate a million if you could provide some insight into how to instantiate the detector, what to pass (and what format) and the outputs (what and what format).

Hi @derenlei,

thank you very much for creating this repository. I got everything set up, but, I am struggeling to import the .onnx model into Unity. It isn't recognized.

I used this repository https://github.com/hunglc007/tensorflow-yolov4-tflite to convert the .weights file to .pb.
Then I used this repository https://github.com/onnx/tensorflow-onnx to convert .pb to .onnx, I used opset 10 because it would fail with anything lower than that.

Which Tools did you use?

Thank you in advance!

I know you mentioned you would look into fixing this already, just recording it here

On Android with ARCore (using Sony Xperia XZ3 for my tests), the anchors placed on detection are always at least 30cm out from the detected object when the anchor is placed. This behaviour is consistent on android in my tests with different dishes.

Hello Sir, do you mby know solution to these warning ?
Initialize XR on Startup is checked, but there is no ARCamera option in Plug-in providers, if enabling it should be done there.

Thanks for your time and sharing your project ;D

I have an onnx file trained with grayscale images and want to run the Detector Yolo3 script for it. But, its predictions are incorrect. So, I modified the script as follows, but unable to resolve all the bugs in the PhoneARCamera script.
Please help in this regard.
I have also changed the texture as TextureFormat.Alpha8.

We already check for them in the first place(PhoneARCamera.cs):

And then we check again:

Or am I missing something? By deleting the latter it works just as fine.

Hi, thank you for this brilliant demo.
Maybe add an instruction to set “Project Settings > Player > Others Settings > Minimum API Level” to “Android 7.0 'Nougat' (API Level 24)“. I got an error on build and changed it, but it may not be obvious for some users. Cheers.

Hi. Thanks for your project.

my model is trained from darkflow.
I'm trying to convert my .pb model to onnx model.
But when I'm use below command I got error,
python3 -m tf2onnx.convert --graphdef .yolov2-tiny-voc-2c.pb --output yolov2-tiny.onnx --inputs yolov2-tiny/net1:0 --outputs yolov2-tiny/convolutional9/BiasAdd:0

the error is
AssertionError: yolov2-tiny/convolutional9/BiasAdd is not in graph

Is there any solution?

here is my model link.
https://drive.google.com/drive/folders/1q-NnsnUMUpNy9m7IenWP2x_sVCOcfvOg?usp=sharing

All of this was done by colab.

This is the file we used.
https://drive.google.com/drive/folders/1CyP5i498dxbd5XLoM5qNS6nRNd6eZWHc?usp=sharing

Our ultimate goal is to get onnx for use in Barracuda.

Our weights file and cfg file are these.

With this, we did a darknet to keras/keras to onnx

The version we used is as follows.

keras==2.4.3
tensorflow==2.3.0
To match the onnxopset version to 8,
onnx==1.3.0

When we convert darknet to keras,
!conda create -n yad2k python=3.6.0
!pip install protobuf==3.6.1

You set up your virtual environment using the code above, and made other efforts.

As a result of running darknet to keras,
I was able to get an h5 file like this.
!python yad2k.py off-yolov2-tiny.cfg off-yolov2-tiny.weights yolov2.h5

2022-08-25 15:05:29.819139: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
Loading weights.
Weights Header: [ 0 2 0 26316800]
Parsing Darknet config.
Creating Keras model.
Parsing section net_0
Parsing section convolutional_0
conv2d bn leaky (3, 3, 3, 16)
2022-08-25 15:05:31.036783: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2022-08-25 15:05:31.045699: E tensorflow/stream_executor/cuda/cuda_driver.cc:314] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2022-08-25 15:05:31.045752: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (065e3bdef5e0): /proc/driver/nvidia/version does not exist
2022-08-25 15:05:31.046078: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-08-25 15:05:31.053293: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2199995000 Hz
2022-08-25 15:05:31.053580: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x564118db8840 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2022-08-25 15:05:31.053621: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
Parsing section maxpool_0
Parsing section convolutional_1
conv2d bn leaky (3, 3, 16, 32)
Parsing section maxpool_1
Parsing section convolutional_2
conv2d bn leaky (3, 3, 32, 64)
Parsing section maxpool_2
Parsing section convolutional_3
conv2d bn leaky (3, 3, 64, 128)
Parsing section maxpool_3
Parsing section convolutional_4
conv2d bn leaky (3, 3, 128, 256)
Parsing section maxpool_4
Parsing section convolutional_5
conv2d bn leaky (3, 3, 256, 512)
Parsing section maxpool_5
Parsing section convolutional_6
conv2d bn leaky (3, 3, 512, 1024)
Parsing section convolutional_7
conv2d bn leaky (3, 3, 1024, 512)
Parsing section convolutional_8
conv2d linear (1, 1, 512, 425)
Parsing section region_0
Model: "functional_1"

Layer (type) Output Shape Param #

input_1 (InputLayer) [(None, 416, 416, 3)] 0

conv2d (Conv2D) (None, 416, 416, 16) 432

batch_normalization (BatchNo (None, 416, 416, 16) 64

leaky_re_lu (LeakyReLU) (None, 416, 416, 16) 0

max_pooling2d (MaxPooling2D) (None, 208, 208, 16) 0

conv2d_1 (Conv2D) (None, 208, 208, 32) 4608

batch_normalization_1 (Batch (None, 208, 208, 32) 128

leaky_re_lu_1 (LeakyReLU) (None, 208, 208, 32) 0

max_pooling2d_1 (MaxPooling2 (None, 104, 104, 32) 0

conv2d_2 (Conv2D) (None, 104, 104, 64) 18432

batch_normalization_2 (Batch (None, 104, 104, 64) 256

leaky_re_lu_2 (LeakyReLU) (None, 104, 104, 64) 0

max_pooling2d_2 (MaxPooling2 (None, 52, 52, 64) 0

conv2d_3 (Conv2D) (None, 52, 52, 128) 73728

batch_normalization_3 (Batch (None, 52, 52, 128) 512

leaky_re_lu_3 (LeakyReLU) (None, 52, 52, 128) 0

max_pooling2d_3 (MaxPooling2 (None, 26, 26, 128) 0

conv2d_4 (Conv2D) (None, 26, 26, 256) 294912

batch_normalization_4 (Batch (None, 26, 26, 256) 1024

leaky_re_lu_4 (LeakyReLU) (None, 26, 26, 256) 0

max_pooling2d_4 (MaxPooling2 (None, 13, 13, 256) 0

conv2d_5 (Conv2D) (None, 13, 13, 512) 1179648

batch_normalization_5 (Batch (None, 13, 13, 512) 2048

leaky_re_lu_5 (LeakyReLU) (None, 13, 13, 512) 0

max_pooling2d_5 (MaxPooling2 (None, 13, 13, 512) 0

conv2d_6 (Conv2D) (None, 13, 13, 1024) 4718592

batch_normalization_6 (Batch (None, 13, 13, 1024) 4096

leaky_re_lu_6 (LeakyReLU) (None, 13, 13, 1024) 0

conv2d_7 (Conv2D) (None, 13, 13, 512) 4718592

batch_normalization_7 (Batch (None, 13, 13, 512) 2048

leaky_re_lu_7 (LeakyReLU) (None, 13, 13, 512) 0

conv2d_8 (Conv2D) (None, 13, 13, 425) 218025

Total params: 11,237,145
Trainable params: 11,232,057
Non-trainable params: 5,088

None
Saved Keras model to yolov2.h5
Read 11237145 of 11237146.0 from Darknet weights.
Warning: 1.0 unused weights

!pip install keras2onnx

As a result of the installation of this method and the execution of Keras to onnx,
I was able to get an onnx file like this.
from tensorflow.python.keras import backend as K
from tensorflow.python.keras.models import load_model
import onnx
import keras2onnx

onnx_model_name = 'yolov2-tiny_825.onnx'

model = load_model('yolov2-tiny_825.h5')
onnx_model = keras2onnx.convert_keras(model, model.name)
onnx.save_model(onnx_model, onnx_model_name)

WARNING:tensorflow:No training configuration found in the save file, so the model was not compiled. Compile it manually.
tf executing eager_mode: True
tf.keras model eager_mode: False
/usr/local/lib/python3.7/site-packages/keras2onnx/ke2onnx/batch_norm.py:45: RuntimeWarning: invalid value encountered in sqrt
gamma = params[0] / np.sqrt(params[3] + op.epsilon)
/usr/local/lib/python3.7/site-packages/keras2onnx/ke2onnx/batch_norm.py:46: RuntimeWarning: invalid value encountered in sqrt
beta = params[1] - params[0] * params[2] / np.sqrt(params[3] + op.epsilon)
The ONNX operator number change on the optimization: 96 -> 33

After inserting the onnx file into the barracuda in unity,

Put this txt file in
Conversion of the number of classes /
There is an error that is not recognized properly when setting yolov2-tiny in camera image.

I want to know why h5 and onnx files are not recognized when they are obtained normally.
also, Errors do not appear.

Not really an issue, I am just investigating open-source object detection methods and wondered if this can re-localise once an object has moved or localise in a more 'real time' way if the object is moving?

Hi,

Can someone give me instructions how convert my custom darknet model (yolov2-tiny) to ONNX? I know I should convert my darknet model to tensorflow, then to onnx, but I struggle with this for several days.

What I've tried:
Convert darknet model (weights + cfg) to tensorflow using darkflow -> I got .pb file (SUCCESS)

Then I wanted to convert pb file to onnx:

1. Tried to convert pb file to onnx using mmconvert:
   mmconvert -sf tensorflow -iw path_to_my_pb_file -df onnx -om yolotfonnx --inNodeName input --dstNodeName output
   error:
   TypeError: can only concatenate list (not "NoneType") to list
2. Tried to convert using tf2onnx:
   python3 -m tf2onnx.convert --graphdef path_to_my_pb_file --output yolov2-tiny.onnx --inputs input:0 --outputs output:0
   error:
   AssertionError: output is not in graph

this also doesn't work:
python3 -m tf2onnx.convert --saved-model path_to_my_pb_file --output yolov2-tiny.onnx
error:

RuntimeError: MetaGraphDef associated with tags 'serve' could not be found in SavedModel. To inspect available tag-sets in the SavedModel, please use the SavedModel CLI: `saved_model_cli`
available_tags: []

3. Tried to convert using onnx-tf:
   onnx-tf convert -i path_to_my_pb_file -o path_for_output_onnx_file
   error:
   ValidationError: The model does not have an ir_version set properly.

I think darkflow can be not compatible with yolov2-tiny. I also tried to convert original darknet yolov2 model and have same problems after conversion with darkflow.

Hello, Thank you for your amazing work.
I'm a beginner in Unity and was working always with darknet YOLO bevor. Now I'd like to test also my custom detector in unity but at first I'd like just start with your version of YOLO in Unity. BUT as beginner in unity I have several issues. If I'm looking in "Usage":

1. my version is 2020.2.1f Personal
2. "Edit -> Player Settings -> Other XR Plug-in Management" - doesn't exist in my version. I've found under file->build settings->player settings->XR Plug-in Management
   

3)From Inspector Scene: Detect -> Game Object: Camera Image -> Script: Phone AR Camera, choose Selected_detector to either Yolo2_tiny or Yolo3_tiny(default).
-> doesn't exist if I open downloaded "SampleScence"

Next steps are unavailable for me. I did probably something wrong as beginner but I think that many people would have same issues and if I start with it, it could probably help most of them.

Thank you!
Best regards!

I am using yolov2tiny and showing 3d model at center of camera but when i start the app from left or right position and detect object then it show the 3d model on left or right position but i want to change the 3d model position to show on the same place where bounding box is visible. How can i get bounding box position ? I can change it using transform.position but i want to know how to get bounding box position?

Thank you

So I used AlexeyAB's Darknet fork to train a custom model using Yolo-v3-tiny making my own weights from scratch to make a detector for apples/oranges as a quick test.

I used this guide to convert my model to onnx
https://github.com/jkjung-avt/tensorrt_demos#demo-5-yolov4

Apparently there is an issue with yolov3 tiny import in Unity Barracuda 1.2.0 preview and above
Unity-Technologies/barracuda-release#122

I can't seem to get the model to work in your project, using Barracuda preview 1.2.0 and using my names file + modifying Detector.cs to read the input and one of the outputs.

Here is the link to my onnx file, some test images and the names file https://drive.google.com/file/d/1uY81Gh2Edh63R7G_6i5KI_oVLdMCuiem/view?usp=sharing

Any help or direction on why the Yolo v3 onnx model doesn't work would be great, I'm still investigating.

Detection will stop after the first set of anchor placement. Find a way to refresh the states and trigger detection again.

Hi,
thanks for this neat App! When running the v3 model I get mostly confidence scores of 100% for every object, even if the prediction is wrong.
Has anyone else had this issue and knows how to solve it?

Is there anything in the project which is omittable? I am still trying to fully understand the hierarchy and the project's workflow, but the main issue is with the template is that the FPS becomes incredibly low while detecting something. Would performance increase with custom model and lower number of object to detect?(Like 3-4)

BTW, a big round of applause for this project, it is very useful, you are very skilled.

Also, I got these warnings, what could be the culprit?

I do not understand how the output is parsed from the onnx model.

I have set-up a Unity project similar to this that uses YOLOv5m as its model and the output does not make sense. I would like to understand how this repository has achieved the transformation of the output layers into something useable.

Can anybody provide some insight?
Thanks

Got the demo working, fantastic work and thankyou for sharing!

Once I have detected and localized one dish, like noodles or rice, how do I detect and localize more dishes during the session without closing/reopening the app?

In your example gif on the repo, you detect 2 dishes at once, are you able to detect more dishes after that or do they all have to be in the camera view when you detect + localize for the first time during the session?

I am using yolov2tiny my own model and i want to show 3d model when it detects specific class. Could you describe where to put if condition. My model is train on 3 classes i need to show 3 different 3d models when it detect each class. For example when it detect class 1 i need to show class 1 3d model, when it detect class 2 then class 2 3d model and same case for class 3.

Thank you

Is there a way to make the Anchors face the camera all the time?
I don't seem to find the way to do it.

After I build app in Unity 2020.3.16f1 and run with my Galaxy S20 application crash immediately. Does anyone know why is it happening and how can problem be solved?

Hi,

I have 2 models : 1) yolo v2 tiny , 2) yolov3 tiny
https://drive.google.com/drive/folders/1gwM091F_euu-Zzqz2orOL50DqI26vaue?usp=sharing
Above is the link to the models.

After importing in Unity project and make related changes in the detector script, the models didn't seem to generate bounding boxes nor labels.

Could anybody help to fix this issue?

Thanks,
Surabhi

I am using yolov2tiny my own model i want to detect object from far and near distance but current project is not detecting from far distance and also it need to hold the phone for few seconds to detect from near distance. I want instance detection could you please guide which script do i need to modify. My same model is detecting well on tfunity below project it is detecting from far and near distance instantly with bounding box i need same behavior in this project.

https://github.com/shaqian/TF-Unity-ARFoundation
Thank you.

Greeting,

I downloaded this project, and tested with this project original Unity version and the latest Unity version.

All compiled app crashes after the splash screen. And I did follow the instructions that was given.

Anyone faced this issue before?