Edge TPU module about mlapi HOT 18 CLOSED

The describe() function is essentially a simple way for external programs to get access to self.description, especially if you instantiate a plugin but don't extend it. However, this is a very minor thing. We can keep extending the base later as needed (and what is logical). Yes, let's discuss on code. Note that when you extend, if there are similar functions in mlapi and ES, the ES always has the latest code (for example, I recently added a lot of error handling). So you may want to use the ES functions where applicable. (And hopefully, once we make these all common modules that can be installed with pip3, I won't have to keep patching both)

from mlapi.

Very cool. Yes, PRs welcome. I have a google coral and it will be useful to me too.

from mlapi.

That's great :-). Do you already have a plan how to implement different modules? I think it would be best if a module could be a pluggable module/package which includes settings and an object detection class. It should be possible to use this package directly in zmes en mlapi. What do you think?

from mlapi.

Closed by error

from mlapi.

No, not really. I agree - I've long wanted to break up all the modules so that it is reusable by both mlapi and local installs. I'm generally not a good coder (not my regular profession), so I do stuff and then think "ah, I should have done it better" and by then it's too late. Feel free to modularize as fit!

from mlapi.

I understand, I'm not a coder by profession myself. The first thing I would like to do is to add the sections. This way the following structure can be setup:

[yolo]
enabled=true
config=./models/yolov3/yolov3.cfg
weights=./models/yolov3/yolov3.weights
labels=./models/yolov3/yolov3.labels

[yolo_tiny]
enabled=false
config=./models/tinyyolo/yolov3-tiny.cfg
weights=./models/tinyyolo/yolov3-tiny.weights
labels=./models/tinyyolo/yolov3-tiny.labels

[edgetpu_mobilenet_ssd_v1]
enabled=false
min_confidence=0.4
model=./models/mobilenet-ssd-edgetpu/ssd_mobilenet_v1_coco_quant_postprocess_edgetpu.tflite
labels=./models/mobilenet-ssd-edgetpu/coco_labels.txt

[edgetpu_mobilenet_ssd_v2]
enabled=false
min_confidence=0.4
model=./models/mobilenet-ssd-edgetpu/ssd_mobilenet_v2_coco_quant_postprocess_edgetpu.tflite
labels=./models/mobilenet-ssd-edgetpu/coco_labels.txt

I also thought that it would be nice to expose every object dectection model to a different endpoint. Maybe we could define it here as well? (So the integration with zmes keeps working :-)) So:

[yolo]
enabled=true
config=./models/yolov3/yolov3.cfg
weights=./models/yolov3/yolov3.weights
labels=./models/yolov3/yolov3.labels
endpoint=/object    # so it's still compatible with zmes and other services

I think in a first modular version we should still just have all variables in one file to not make to too complication. We can split it up later. Also the mlapi.py file needs to support every module. However i would suggest that we move the model download logic to the specific module files.

What do you think about this idea?

Edit; I found some instructions on using plugin modules. I'll see if I can structure the current modules (opencv face detection, opencv object detection) into some kind of plugin. - https://packaging.python.org/guides/creating-and-discovering-plugins/

from mlapi.

I like the idea of keeping a single config file and if the models can have their own sections it doesn't break zmes too.

I did not understand endpoint=/object -> what does that mean?

Finally, if you find a good method of doing a plugin system, I am open to doing a breaking change if we agree its a good/flexible way for the future. My goal is to also implement a real time (per frame) analysis system in the future where we should be able to reuse our modularization.

from mlapi.

Regarding the endpoints I think I didn't understand it correctly. When looking at the code now i see there's one endpoint available (http://localhost:5000/api/v1/detect/object) and face detection is an option (http://localhost:5000/api/v1/detect/object?type=face).

If it would be possible to define a type for every model, in the config file (or use the section name...) You could instruct the API as below and the API would instruct the plugin to process the image.
http://localhost:5000/api/v1/detect/object?type=face
http://localhost:5000/api/v1/detect/object?type=yolo
http://localhost:5000/api/v1/detect/object?type=yolo_tiny
http://localhost:5000/api/v1/detect/object?type=edgetpu_mobilenet_ssd_v1
http://localhost:5000/api/v1/detect/object?type=edgetpu_mobilenet_ssd_v2
http://localhost:5000/api/v1/detect/object?type=edgetpu_face (or something)

With endpoint I meant the type of the object, however i think it would be best to use the name of the plugin there?

For the plugin framework I will do a bit of searching and will let you know. Lets first decide on the framework before any real change is made. This page looks like a good start: https://alysivji.github.io/simple-plugin-system.html

The realtime analysis system sounds good. Thats really what i'm missing at the moment, but implementing the detecton methods as plugins would be a nice start 👍

from mlapi.

It might be better to do this:
http://server/api/v1/detect/<plugin>?<parameters>

Where <plugin> = face, object,gait,img2txt,...
And <parameters> could be anything specific to that plugin including, say, model=yolo&type=tiny or model=ssd_net&whatever

Thoughts?

from mlapi.

Thats even better:-) it would also be nice to define default options for each endpoint, to ensure backwards compatibility?

from mlapi.

Sure. My thought is let’s do the right approach and get to backward compatibility later. I’m ok with the interface breaking as it’s still beta.

from mlapi.

Check. I'll try to make a poc, let's discuss further based on that code.

from mlapi.

I've been playing around with the instructions on https://www.guidodiepen.nl/2019/02/implementing-a-simple-plugin-framework-in-python/ and came up the following. This code automatically registers a class with the following name: plugins.<subfolder name>.<file name>,<class name>. As you can see below the classes register themselves and the init section of the script is loaded.

Looking for plugins under package plugins
    Found plugin class: plugins.identity.Identity_test
Initializing log
    Found plugin class: plugins.object_detection.yolo3.Object
    Found plugin class: plugins.face_recognition.face.cnn
DEBUG: Initializing face recognition with model:cnn upsample:1, jitters:0

There's a global class definition where the main class is defined. I've changed the example a bit to match the current class definition:

class Plugin(object):
    """Base class that each plugin must inherit from. within this class
    you must define the methods that all of your plugins must implement
    """

    def __init__(self):
        self.description = 'UNKNOWN'

    # def perform_operation(self, argument):
    def detect(self, argument):
        """The method that we expect all plugins to implement. This is the
        method that our framework will call
        """
        raise NotImplementedError

Now I would like to discuss what methods there need to be in Plugin class. On a functional level, I'm thinking of:

• init: loads the model into memory, make sure there's a minimal load time when the first image is processed.
• parameters: defines the parameters bound to this plugin (currently in common_params.py file.
• models: defines the download method for the models
• detect: runs detection of an image.

is there anymore you can think of?

from mlapi.

From a method perspective:

• init : any init work
• detect: actual detection
• destroy: any cleanup
• describe: just a function that returns a text string about the plugin. Might be useful to show

Do we really need models/parameters? Should they not be implicitly invoked inside detect (or init)? To keep parameters simple, we can pass a params object to detect. Each plugin can mandate fields within that or not.

from mlapi.

Yes you are right, I don't think we need to expose the models and parameters.
Is describe required as there's already a way to describe the plugin?

self.description = 'UNKNOWN'

I'll try to convert your current mlapi code into a plugin based part based on your method perspective.

from mlapi.

@Rick-Jongbloed how are things going?

from mlapi.

Hi, I've been busy with other stuff due to the lockdown... I'll try to spend a weekend in may on this!

from mlapi.

closing this as EdgeTPU is done and moved to pyzm

from mlapi.