Track objects detected by YOLO and publish that info to MQTT

Yolo2Mqtt is a collection of python scripts for running image recognition on RTSP camera streams using a Yolov5 model, detecting when two objects interact, saving a video of the event, using MQTT as a communications mechanism.

This program was written to track cat feeding/drinking in a multi-cat home and to graph provide that information to Home Assistant via MQTT. Tracking both the cats and the food/water bowls allowed the bowls to be moved about the room, rather than designating fixed windows to trigger evnets.

It could be used for pretty much anything in which you want to run YOLO and detect if two objects are overlapping.

Docker is the easiest way to run this. An MQTT broker is required as well. The sample docker-compose will run an MQTT broker, but for Home Assistant integration the same broker must be used by HA and Yolo2Mqtt

The minimal commands to get running would are:

$ git clone --recursive https://github.com/cobryan05/Yolo2Mqtt.git
$ cd Yolo2Mqtt
Yolo2Mqtt$ docker-compose build
Yolo2Mqtt$ docker-compose up

This will build the docker container and run the default configuration.

The default configuration will use YoloV8 and auto-download a sample model then run sample recognitions on a static image. If all goes well, you should be able to connect to the mqtt broker at <hostIp>:1833. Detection information should be published to the topic /myhome/ObjectTrackers/.

With the default sample config you should soon see topics identifying a dog riding a motorcycle.

Configuration is done by editing /config/config.yml. A default config.yml is available as config.defaults.yml. You should configure the volumes in your docker-compose.yml so that so that your custom config.yml is available to it at /config/config.yml.

All settings and default values (if any) are commented in config.defaults.yml.

Note: This is meant to be used with a custom-trained YOLOv5 model. The default configuration will download the base yolov5 model, and the example interactions are some nonsense made with default labels. Any useful interactions will require a custom model.

To enable CUDA acceleration (assuming CUDA is already working on your system): * At the top of the Dockerfile change the FROM: image to the -cuda variant (then rebuild the image) * In the docker-compose.yml uncomment the 'deploy' block to pass the GPU to Docker * In the config.yml switch the 'yolo: device:' setting to 'cuda'

The Docker container internally runs an RTSP Proxy on port 8554. If a port is forwarded to 8554 on Docker container, then any RTSP camera in the config.yml should be proxied at, eg <hostIp>:8554/<cameraName>

There are two ways in which Yolo2Mqtt can be 'integrated' with Home Assistant.

• Yolo2Mqtt's 'mediaRoot' folder can be set as a 'media_dir' in Home Assistant. This will allow browsing and playback of Yolo2Mqtt recordings via the Home Assistant 'media' browser. (See https://www.home-assistant.io/integrations/media_source/#using-custom-or-additional-media-folders)

• When the homeAssistant 'discoveryEnabled' setting is true in config.yml then event interactions will post to the Home Assistant MQTT Discovery topic (https://www.home-assistant.io/docs/mqtt/discovery/) for autoconfiguration. Events will create new binary_sensors in Home Assistant with names such as
  
  binary_sensor.[entityPrefix]_[eventName]_[cameraName]_[object1]_[object2]
  
  

In Home Assistant you can create 'aggregation sensors' to help process further process these events. For example, adding this template sensor to Home Assistant:

binary_sensor:
 - platform: template
   sensors:
     cattracker_summary_sammy_eating_food:
       friendly_name: "Sammy Eating Food"
       unique_id: "cattracker_summary_sammy_food"
       value_template: >-
         {% set action='cateatingfood' %}
         {% set actor='sammy' %}

         {% set re=action + '[a-z_]+' + actor %}
         {% set ns = namespace(state='off') %}

         {% for entity in states.binary_sensor if entity|regex_search( re ) and entity.state=='on'%}
           {% set ns.state = 'on' %}
         {% endfor %}
         {{ ns.state }}

will create a new binary_sensor.cattracker_summary_sammy_eating_food which will be 'true' if any sensor matches the regex 'sammy[a-z]+cateatingfood'. Multiple yolo2mqtt 'events' can be combined into one logical Home Assistant event using tricks like this.