This is a crude demo project made to mimic the supposed live video ingestion capabilities of Google's multimodal Gemini LLM, but made with the GPT-4 Vision API.

Demo: https://youtu.be/UxQb88gENeg

$ pip install -r requirements.txt
$ export OPENAI_API_KEY=YOUR_OPENAI_API_KEY

To run the voice commanded terminal version, run the voice.py script.

$ python3 voice.py VIDEO_STREAM_URL

The assistant only reacts to voice commands.

To run the motion detecting version, run the motion.py script.

$ python3 motion.py VIDEO_STREAM_URL

The assistants reacts every time motion is detected in the video. A tripod is recommended.

To run the automatic version that detects both voice commands and motion in the video, run the auto.py script.

$ python3 auto.py VIDEO_STREAM_URL

The assistants reacts every time motion is detected in the video or a voice command is given. A tripod is recommended.

There is also a version with a "UI" made with CV2 (it sucks but kinda works). It both listens to voice commands and detects motion in the video and automatically sends both to the GPT4V API.

$ python3 auto_with_ui.py VIDEO_STREAM_URL

In my testing, I have used my phone camera as the video stream. For this, I used the IP Webcam app on Play Store. I set the camera to 10 fps at 640x480 resolution.

The VIDEO_STREAM_URL is passed directly into cv2.VideoCapture(), so I guess you should be able to pass in a video file too, or any kind of video stream.

There is a config.py file where you can tweak some settings if you are having trouble with the motion detection or speech detection.

• GPT-4V API is often slow
• Sometimes the assistant response is detected as a user message
• The CV2 UI sucks and should be made with another way
• The CV2 UI can only be closed by hittin Ctrl+C in the terminal