luxonis / depthai-hardware Goto Github PK
View Code? Open in Web Editor NEWAltium Designs for DepthAI Carrier Boards
License: MIT License
Altium Designs for DepthAI Carrier Boards
License: MIT License
I know that it might not be on the list now but Is OAK D able to be used with android devices ?
Or is there some plans to add android support for oakd?
why
:As Katherine_Scott (of ROS) mentioned here there is a huge diversity of M12 lenses available to suit all sorts of problems. These also come in options for various filtering of specific spectra (opening up things like NDVI and others).
So having an M12-supporting version of of the IMX477 (low-CRA version of IMX378 on megaAI and DepthAI would open up all sorts of applications.
how
:ArduCam has offered to make an IMX477 M12-compatible camera module for megaAI and DepthAI ecosystem use. Our existing driver can be easily adapted to use this version.
what
:Support the ArduCam IMX477 M12-compatible camera module.
why
:Spatial AI is starting to be possible with embedded platforms. Spatial AI provides information on what objects are, and where they are in physical space. So this information can then be transduced into other formats (text, audio, haptic/vibration, tactile display, etc.) which can help folks with visual impairments.
For example, such a spatial AI solution could provide this sort of insight when walking in a park:
“There is a park bench 12 feet in front of you at 1 o’clock and it has empty seats”
Or it could find text on a page or a sign and offer to the user to read it, automatically. It could also provide insight as to where the user physically is in proximity to other objects like vehicles, people, bikes, etc., including even warning when a vehicle collision is imminent (e.g. here). Or feedback on where someone is on a path (like here).
This current effort started when Marx Melencio (who is visually impaired) reached out here, showcasing the system he has already made, and with interest in using DepthAI to actually productize.
how
:DepthAI is an Embedded Spatial AI platform (perhaps the only one as of this writing?) which provides neural inference (e.g. object detection) in combination with disparity depth to give object localization (i.e. what an object is and where in physical space).
And it can run series/parallel networks as well (a coming feature, see first version here), which allows a pipeline of networks to provide additional information based on the context (automatically or manually). For example, when outside/walking in a downtown area (the context) the system could automatically detect road signs/stop signs/stop lights and tell the user the state of them (which would be a cascade of a find-signs network followed by a ‘read the signs’ and/or ‘state of digital sign’ network).
So we plan to use DepthAI to make an easy-to-use visual assistance system, taking advantage of the fact that DepthAI is embedded (i.e. doesn’t require an OS or anything), low-power (i.e. can be battery powered over the course of a normal day) and is small enough to be body-worn w/out encumbrance.
There are a variety of ways we could attack such a device (a variety of potential ‘how’), which fall into two categories:
Piecing together existing hardware (largely for prototyping)
Building custom hardware (for a path to productization and selling as a standard product)
There exist a couple variants of DepthAI which could be used for such an application, and fortunately DepthAI is open-source (MIT-licensed, here), so these designs can be modified into something that is more applicable for this visual-assistance device. Below are the applicable designs and how we’ve thought to use them:
DepthAI Raspberry Pi Compute Module Edition (BW1097), here -
a. This has a whole computer built in.
b. It could be used with a GoPro adapter mount (here) and this GoPro mounting kit (here) to allow mounting practically anywhere on the body (head, chest, wrist, etc.).
c. This has all the processing and communication all built-in (it’s running Raspbian), but does not have a solution for battery, so that would need to be worked out - mainly in terms of how to mount and communicate with the battery.
d. One option would be to make a new mount that has the battery built in.
DepthAI Onboard Cameras Edition (BW1098OBC) here.
a. This also could be used with a GoPro adapter mount (here) and then connected (and powered) over USB to some other on-person computer/power source (say a Pi w/ a Pi HAT LiPo power source).
b. So this solution would have 1x USB cable going from the perception device (DepthAI BW1098OBC) to the host processor device (say a Pi with a battery).
DepthAI Modular Cameras Edition (BW1098FFC) here
a. This would allow ‘hacking together’ a prototype of smart glasses.
b. For example the main board could be on the nape of the back of the neck, connected via FFC to the cameras on smart glasses.
c. The trouble is it’s a lot of flexible flat cables (FFC) cables and these cables are relatively fragile, so it’s not ideal.
Making actual spatial-AI glasses where everything is integrated. This picture here summarizes it. A battery would likely be integrated directly in the the frame (with the ESP32) or on a lanyard which attaches to the back of the frames.
a. The disadvantage of this is that it is specifically designed for wearing on the head… and it may be nice to for example have a head-mounted unit and a wrist-mounted device (for example the head-mounted gives situational awareness, and the wrist-mounted let’s you explore around you (e.g. read a piece of paper) without having to move your head all over
b. This is also a more complex custom design w/ some technical risk and user-experience risk including us as a team not really knowing how to make comfortable glasses/etc.
Make a small fully-integrated Spatial AI box w/ GoPro mount so it can mounted to wrist, chest, or head (using a GoPro adapter set like here) which has WiFi and BT interface.
a. This is the simplest/fastest approach, having the lowest technical risk and user experience risk as it can just be a self-contained, small system which uses field-proven GoPro mounts for attachment to head, chest, or wrist.
b. It also allows using multiple devices on a person. So for example one on the back, one on the head, one on the wrist, one on the chest. And they connect over BT or WiFi to say an on-person Raspberry Pi which handles prioritizing which data comes back based on the person interacting w/ the onboard software.
c. It is a not-huge amount of work to re-used the design for the BW1098OBC (here) while adding an ESP32 and a battery w/ charge/protection circuitry.
d. Probably worth reducing the stereo baseline from the 7.5cm there to something like 4cm or so, as it will still provide plenty of distance vision while allowing a closer-in min stereo-disparity depth-perception distance (see calculation here) of 0.367 meters (for 4cm baseline) instead of the 0.689 (for 7.5cm baseline).
So after exploring the options above, we decided that ‘2’ from custom hardware section seems like the way to go. So below in ‘what’ we describe what this device we are building:
what
:So out of wanting to design this, we realized we should do it in stages, so the first part of this effort will be a DepthAI ESP32 Reference Design / Embedded DepthAI Reference Design as in #10, which will be re-usable as the core of the implementation needed below:
I tried different 5V USB batteries (LiIon based) that output 2A, 2.4A and 3A and seems to be fairly stable on DM1097 CM4 models, the most power hungry.
I will run some endurance tests and provide more details once I mount them on my robots.
Let's collect here what are your recommendations to power by battery the DepthAI modules.
Also please provide the min/max voltage that can be used and what temperatures we should expect when the voltage is on lower end, for each model.
I just noticed that the Myriad heat-sink temp is about 57 degrees Celsius, while running the demo (on API version 1.0) and using a DeWALT 20V 2Ah battery using a 18V convertor that has a USB port that outputs 5V at 2A max.
The RPI CM4 temp is about 62 degrees Celsius:
pi@raspberrypi:~ $ vcgencmd measure_temp
No load:
temp=62.8'C
Under load:
temp=68.6'C
Here is the adapter, I assume the voltage is a bit lower than 5 under use (need to measure it):
https://www.amazon.ca/TENMOER-DCA1820-Battery-Adapter-Compatible/dp/B08HMVYYG5
why
:We’ve now gotten a bunch of reach-outs about something like OAK-D, but with 150 DFOV, IR-laser-dot-projection depth, and matching resolution/FOV/global shutter for the color camera.
This is similar to this idea, #21, which we already had on the roadmap, but with the change that apparently a lot of applications want this and wide FOV, which is separately #15.
Many folks want to use such a product on small aerial vehicles. And as such size and weight are important.
how
:ArduCam already implemented #15 , and now produces this 150DFOV OV9282 module as a standard product for DepthAI customers, here. So far though, this has only been used in custom products built with OAK-SOM/etc.
We should work with ArduCam to (1) update this OV9282 to the new/standard ArduCam connector format here and (2) make a OV9782 variant with the same connector/format but that blocks IR (for good color representation). Likely prototyping first with the existing format/connector
what
:Make a Wide FOV version of OAK-D Series 2 (#115)
OAK-D W (W = Wide FOV)
why
:While adding pattern projectors like in #20 affords disparity depth to work in absence of any ambient lighting, neural inference likely will not (as there will be little dots of illumination, not overall illumination).
So to allow 3D object localization (i.e. an object detector like YOLO or MobileNet-SSD fused with spatial information), having overall illumination (and not just pattern projection) is required.
how
:So there are 3 options for supporting the IR-capable camera modules:
We have tested both 1
and 2
with DepthAI (example images below) and found that 1
produces much sharper images when IR illumination is being used (this IR flashlight was used). See the quick test results below:
We have not yet tested option 3
, but it could be an interesting solution - but with the cost that usually such mechanical moving parts are a point of failure.
what
:Support IR illumination for object detection and disparity depth in total (ambient) darkness conditions.
why
:As it stands now DepthAI is optimized for real-time spatial AI, so where objects are in physical space and attributes about them. So for this, narrower field of view is desirable.
But for applications where it is desirable to self-orient and self-locate (i.e. simultaneous location and mapping - SLAM, e.g. discussion in ROS discourse here), wider field of view is of interest. So having support for a wide field of view OV9282 camera module is of interest.
And it just so happened that Arducam reached out offering cooperation, and has a module that is ideal in terms of FOV:
how
:We are working with ArduCam to integrate this OV9282 fisheye version in with DepthAI.
what
:Support ArduCam 155-degree FishEye OV9282 Camera
Hi,
We are working on the mechanical design which has OAK-D as a component. However, we could not open the 3D model file or the mechanical file. We have tried several different software, even online tools, but still cannot open them. What is the exact software that we should use? Thanks.
I was not able to boot this board
https://github.com/luxonis/depthai-hardware/tree/master/DM1097_DepthAI_Compute_Module_4
when using I connected to the monitor thorough a 4 port HDMI switch (a few years old model).
I had problems with mouse, left/right click was not working, when was connected through a USB switch, this model:
https://www.amazon.ca/dp/B081V977MX/
Also the Myriad X stopped working after a few seconds when running the automated demo (the NN FPS was 0 and not more detections), and also the whole system will lock (requiring reset) after a minute or two, when the USB switch was connected.
Without the USB switch everything works fine and it is stable, but the no HDMI boot issue is still there.
why
:The OAK-SOM-IoT (1099EMB) has a reference design for embedded use cases (actually it has 2, the OAK-D-IoT-40 and OAK-D-IoT-75, but the OAK-SOM-PRO (2099) does not yet have such a reference design.
And in many cases, the OAK-SOM-PRO may be more appropriate for such embedded applications - including cases like the CommuteGuardian, where it may be desirable to store video to onboard eMMC or SD-Card.
And also in many applications having onboard microphones is quite helpful or an absolute requirement (which the OAK-SOM-PRO support).
Another related update (which we should propagate back to the IoT-40 and IoT-75) is that the ESP32 programmer microUSB connector can easily break off (as it's surface mount, and microUSB is just generally too fragile), and also it’s a bit annoying to have to have 2x USB cables plugged into the board.
We also realized that we could put a USB2 hub on board, and allow the USB3 to go straight to the MX, allowing both the ESP32 programmer (microUSB in the current 1092 design) and the Myriad X USB2 interface could be combined into the single USB3C connector. This will make the development experience easier (and make building an enclosure easier).
how
:Using the same idea as the OAK-D-IoT-40, make an equivalent using the OAK-SOM-PRO SOM.
We should also change the CCMs with the new/better design from Arducam, as it allows better hardware-level sync, takes less board space, and is more resilient mechanically (both for production and field robustness).
what
:why
:OAK-D-PoE was our “make it exist” product. It served as a pathfinding design to see what is the community's reaction to it and what could be improved. It has been quite well received. With specific feedback on what would be nice in a Pro version.
The OAK-D-PRO-POE will be the next step and an upgrade to the original OAK-D-PoE, implementing user feedback and improvements through experience gained from previous design.
Overall folks are happy with OAK-D-PoE. But from the requests we do get, they are nearly all about illumination. And specifically the requests ask for:
Everyone loved that OAK-D-PoE is IP67 sealed (so we're keeping that) but many desired an M12 connector instead of the RJ45 IP67 gland, which is a bit large - and more importantly is the reason OAK-D-PoE is a bit bigger and heavier than some desired.
And the final common request was some form of external IO for being able to trigger or be-triggered by external equipment:
how
:Just like the OAK-D-POE design, the OAK-D-PRO-POE will have and ethernet connection for data and PoE for power. But that is where similarities end, as the OAK-D-PRO-POE will feature a more robust and industrial connector type called M12.
To afford external IO connectivity, we will implement an M8 or similar connector. Likely with USB host-support from the Myriad X so that external USB devices (like thermal cameras, or IO expanders) can be used.
Form factor will be heavily based on the OAK-D-PRO design (#114) with it's sleek small case. Afforded by the M12 connector, it is now possible to make this design a lot smaller. And the design will still have the same IP67 rating as was with OAK-D-PoE.
Implementing IR led and dot projector will be the same as in the OAK-D-PRO model (details in #114), as it is already in the testing phase and shows a lot of promise.
what
:A small, lightweight, cost-reduced version of OAK-D-POE that is still IP67 sealed, that has IR laser dot projector and IR illumination LED on-board (Pro version), and IP67-sealed IO connectivity.
why
:The mechanical design of OAK-D is limiting, with the following draw-backs:
OAK-D was architected for applications where passive depth performs well (and can often outperform active depth; as IR from the sun can be blocked by the optics when purely-passive disparity depth is used).
There are many applications however where active depth is absolutely necessary (operation in no light, with big/blank surfaces, etc.). And there are many OAK-D customers who would like to use the power of OAK-D in these scenarios where purely-passive depth is prohibitive.
how
:Architect the IR laser and IR LED such that all these modes of operation are supported:
The idea is that they'd be used in one of these permutations:
It is likely that mode 1 and 5 will be used the most. But enabling all the permutations above allow maximum flexibility for adapting illumination to an application (including dynamically). For example Mode 6 will likely rarely be used, but there are certain cases where having both on may be beneficial.
what
:Same image sensors and FOV of OAK-D:
why
:TLDR: Get DepthAI to the masses.
The existing DepthAI line has proven useful to many, but so many have lost out on learning about its capability and being able to build with it because it is too expensive for them to justify purchasing it without first knowing for sure if they can pull off what they want with it. At $199 for the lowest-price DepthAI, that's a big ask for a lot of people. Particularly people who don't immediately know if it works for their application.
This problem is confounded by the fact that we're not good at marketing or sales. So we have not done a good job showing people that it is good for their application. So we can solve the marketing problem to solve this, or we can make a device that is low-cost enough that people can buy it even if they don't yet know if it will work for them.
So if we make this low-cost variant, and price it low enough, people will buy it not yet knowing if it solves their problem.
This enables folks to then buy it and use if for things for which we did not even know it would be useful. If it's low enough cost, it enables all sorts of engineers to take the plunge and then educate us about how the thing can be used. Folks will find all sorts of new uses for it.
And importantly, it gets it into the hands of folks who have discretionary income for a $99 device, but not for a $199 device. Which is likely a LOT more people.
how
:So how in the world can we make a $99 DepthAI? Before, there was no way, we couldn't even produce the device for that - mainly because we didn't have the necessary degrees of freedom on camera modules.
But with ArduCam now working closely with us, we can use slightly-lower-spec image sensors that are approximately an order-of magnitude less expensive, which then allows us to hopefully build a complete, 3-camera DepthAI that is less than $99 MSRP. Which is huge in terms of folks being able to buy it at risk compared to $199 MSRP.
And this might allow us to hit $75 KickStarter Early-Bird for example. But even at $99, this is allows so many more folks to buy and experiment with it.
And pivotally, many applications, maybe something like 90% of them, can deal with these lower-spec modules without a problem.
what
:The OAK-D-LITE, with the following specs:
Simplify and cost-down the design where possible, perhaps not listed above.
When running a demo/script on 1092, a host can't find a device or can't enable it. Following error messages appear:
Cannot enable. Maybe the USB cable is bad?
attempt power cycle
or
RuntimeError: Failed to find device after booting, error message: X_LINK_DEVICE_NOT_FOUND
why
:In some applications, exact, pixel-by-pixel alignment of the color camera and the grayscale/depth cameras is desirable. To support this, having the color camera have the exact same parameters (global shutter, image sensor pixel size, chief ray angle (for lighting consistency), field of view, focal length, capability to hardware-level sync, etc.) is extremely beneficial.
Because of such a use-case, OmniVision actually makes the equivalent sensor to the grayscale OV9282, the OV9782.
It is made for this exact sort of application.
how
:ArduCam is building OV9282 and OV9782 module for DepthAI that match in all the above ways, and we are integrating them into the open source ecosystem (and ArduCam will likely release their own DepthAI variants with them).
Below is initial testing of the OV9782 (left) and the OV9282 (right) from ArduCam. These variants use the M12-mount lenses for flexibility. But there will be fixed-lens (i.e. smaller) modules as well.
what
:Support OV9782 (the color version of the OV9282) global shutter as the color camera in place of the IMX378.
This is about this model:
https://github.com/luxonis/depthai-hardware/tree/master/DM1097_DepthAI_Compute_Module_4
I set this option, but still no boot when HDMI is not plugged:
https://www.raspberrypi.org/documentation/configuration/config-txt/video.md
hdmi_force_hotplug=1
Are there other settings required?
Is this a know issues or is just my board problem?
Why
when building muti-imager setups, sometimes we need to design with clear understanding of the imager FOVs to determine imager/lens type quantity and placement. and in general there doesn't seem to be a single place where this information is publishing alongside other detailed information.
What/Where
this is a request to include similar information to Intel included in their device documentation ( example below). Ideally this would include H/V/Diag FOVs, sensor size and perhaps information on depthai supported modes.
From researching it looks like there is some (scattered) information for this, but seemingly incomplete having canonical info in this repo seems to make sense and would be an excellent reference.
The readme doc for each module folder could included information like the above.
why
:From the O.G. OAK-D KickStarter we learned that aint nobody want an OAK-D w/out an enclosure. "To add insult to poorly planned injury" was the most memorable quote about OAK-D-IoT-75 not including an enclosure.
So, we're making an enclosure for it.
how
:It's got WiFi. That's why we didn't make an enclosure at first. As it was a time-crunched KickStarter delivery. And we didn't want to be late.
So we took the time to iterate/test and design in plastic openings that are sufficiently wide for the WiFi to escape.
what
:And enclosure that has all the things we learned folks want:
Note that this is an older prototype, so the mounting on the back is now M4 and 7.5cm spacing, instead of whatever is shown there (likely M2 at like 43.5cm).
why
:There are many applications for which thermal information is incredibly useful, particularly when paired with high-res RGB and depth information. Previously-unsolvable industrial, agricultural, and medical problems are much easier solved with this pairing. Examples include easier understanding machine state (i.e. does that pipe have high pressure/heat behind it... should an alarm go off if a worker approaches it).
So adding something like the MLX90640 from the Open Thermal Camera to #10 could be incredibly useful. The pairing of the high-res RGB and depth allows augmenting the lower-resolution of the Melexis in interesting ways, particularly using ML techniques to aid in accuracy/etc.
how
:Evaluate if the board size of #10 needs to be increased to add the MLX sensor. Connect the MLX into the Myriad directly so that ML/CV techniques can be applied directly to this sensors output (and/or fused w/ the other sensor data).
what
:Make a version of #10 with integrated thermal sensor.
why
:It's time. We've implemented all the basis functions of the platform (https://github.com/orgs/luxonis/projects/2), all of which were guided by the North Star of keeping people who ride bikes safe (here).
Namely, we now have all the pieces that go into building this below, in hardware, firmware, and software:
And likely a bunch of others.
how
:For the first version:
what
:Make the first version of Commute Guardian. Probably something that looks like this:
DepthAI DEMO is not working on OAK-D Lite, and here is the log:
(.venv) PS D:\_depthai\depthai> python .\depthai_demo.py
Using depthai module from: D:\_depthai\.venv\lib\site-packages\depthai.cp39-win_amd64.pyd
Depthai version installed: 2.11.1.0.dev+dfd52ac01c3b7514e85cb894b9d5381e999859df
Depthai development version found, skipping check.
Setting up demo...
Available devices:
[0] 1844301061514DF500 [X_LINK_UNBOOTED]
USB Connection speed: UsbSpeed.HIGH
Disabling depth...
Disabling depth preview...
Disabling depthRaw preview...
Disabling left preview...
Disabling rectifiedLeft preview...
Disabling right preview...
Disabling rectifiedRight preview...
Enabling low-bandwidth mode due to low USB speed... (speed: UsbSpeed.HIGH)
Creating MJPEG link for ColorCamera node and color xlink stream...
[1844301061514DF500] [8.400] [system] [error] Attempted to start Color camera - NOT detected!
Stopping demo...
=== TOTAL FPS ===
File "D:\_depthai\depthai\depthai_demo.py", line 536, in run
self.instance.run_all(self.conf)
File "D:\_depthai\depthai\depthai_demo.py", line 59, in run_all
self.run()
File "D:\_depthai\depthai\depthai_demo.py", line 245, in run
self.loop()
File "D:\_depthai\depthai\depthai_demo.py", line 272, in loop
self._pv.prepareFrames(callback=self.onNewFrame)
File "d:\_depthai\depthai\depthai_sdk\src\depthai_sdk\managers\preview_manager.py", line 113, in prepareFrames
packet = queue.tryGet()
Communication exception - possible device error/misconfiguration. Original message 'Couldn't read data from stream: 'nnInput' (X_LINK_ERROR)'
file:///D:/_depthai/depthai/gui/views/CameraPreview.qml:57:13: TypeError: Cannot read property 'height' of null
file:///D:/_depthai/depthai/gui/views/CameraPreview.qml:56:13: TypeError: Cannot read property 'width' of null
(.venv) PS D:\_depthai\depthai>
The error is at the line
[1844301061514DF500] [8.400] [system] [error] Attempted to start Color camera - NOT detected!
why
:The ArduCam OV9281 and IMX477 and other modules that offer M12, wide FOV, C-mount, etc. are extremely useful for making depthai applicable for varieties of use-cases. So it would be beneficial to have direct support for them one the BW1098FFC without any adapters.
Also, in many applications, the IMU present on the OAK-D, BW1092, etc. would be beneficial, so when modifying this board, it would make sense to add the IMU.
how
:Pull the IMU from the BW1098OAK and change the FFC adapter connectors to be what ArduCam uses on their FFC cameras.
what
:Pinout definition for 4-lane (RGB) and 2-lane (stereo left, stereo right) examples are below:
why
:There is a lot of interest in the community to have a decent Linux host integrated with DepthAI. Having all enclosed (possibly IP rated) and PoE powered would give a lot of options to use the device for final integration. The intention is to use PoE as a power source for the device.
how
:The same PoE design can be used, same as for OAK-D-POE and OAK-1-POE meaning by default only Ethernet port would be accessible when the device is in the enclosure. Connectors like micro-HDMI, micro-USB should be also added for development and debugging purposes.
Due to having a full-blown Linux alongside the depthAI it would be good to have SoM connected over some fast speed interface so we were thinking of using PCIe to USB3 bridge as it is used on RPi 4B+ and connect depthAI SoM over USB3 interface.
Later we came to the idea that it would be the best to have CM4 and SoM connected directly over PCIe and thus making the most of it and reducing the BOM. At the moment we could not find relevant info about if RPi CM4 supports needed plug-and-play PCIe so we will have to test this out with the first prototypes. We could also do some workarounds in FW if needed to get it working, only if PnP is not supported on CM4.
As a workaround (if PCIe fails to be a good idea) USB2 interface will still connect CM4 with SoM as on previous versions of 1097 devices. USB connection is also great to have for flashing NOR on DepthAI SoM if needed for recovery mode.
As for some use cases, it is great having the IMU/magnetometer on the baseboard as it brings extra functionalities to the device.
The device should use a bare minimum of components needed to implement all of the above features.
what
:Make a design that can be easily placed in an enclosure try to make it as small as possible.
Add PoE functionality as a power source, remove all unnecessary connectors/interfaces like CSI, DSI, barrel jack power, audio,... Use smaller connectors where possible. Add IMU and other features which are possible to fit onboard to make first prototypes better for evaluation and testing of as many features as possible. Some connectors/circuits can be populated only on a few devices for easier bring up and later removed in production.
Hi, are the schematics and PCB for this project also open-source as many of the other projects. I can only find the NG2094_OAK-D-PRO-W-DEV repository, but sadly it is not a single-board assembly as the OAK-D-PRO product is.
I assume this is to protect the IP in the SoMs?
Hello there, I build a simple files for a version of BW1098OBC for people who just want a model with real dimensions and no details, maybe the Depth AI can add to documentation to support community
Dimensions according to https://github.com/luxonis/depthai-hardware/tree/master/BW1098OBC_DepthAI_USB3C
Public link of the files https://grabcad.com/library/bw1098obc-for-oak-d-stereo-camera-1
Also, for BW1093
Public link of the files https://grabcad.com/library/bw1093-for-oak-1-color-camera-opencv-1
why
:One of the core value-adds of DepthAI is that it runs everything on-board. So when used with a host computer over USB, this offloads the host from having to do any of these computer vision or AI work-loads.
This can actually be even more valuable in embedded applications - where the host
is a microcontroller communicated with over SPI, UART, or I2C and is either running no operating system at all or some very-light-weight RTOS like freeRTOS - so running any CV on the host is challenging or outright intractable.
DepthAI allows such use-cases as it convert high-bandwidth video (up to 12MP) into extremely low-bandwidth structured data (in the low-kbps range; e.g. person at 3 meters away, 0.7 meters to the right).
So this allows tiny microcontrollers to easily parse this output data and take action on it. For example an ATTiny8 microcontroller is thousands of times faster than what is required to parse and take action off of this metadata, and it's a TINY microcontroller.
So for applications which already have microcontrollers, where full operating-system capable 'edge' systems have disadvantages or are intractable (like size, weight, power, boot-time, cost, etc.), or where such systems are just not overkill, being able to cleanly/easily use DepthAI with embedded systems is extremely valuable.
So we plan to release an API/spec for embedded communication with DepthAI over SPI, I2C, and UART to support this.
However, a customer having to write against this spec from scratch is a pain, particularly if hardware connections/prototyping need to be done to physically connect DepthAI to the embedded target of interest. And having to do these steps -before- being able to do a quick prototype is annoying and sometimes a show-stopper.
So the goal of this design is to allow an all-in-one reference design for embedded application of DepthAI which someone can purchase, get up and running on, and then leverage for their own designs/etc. after having done a proof-of-concept and shown what they want works with a representative design.
So to have something embedded working right away, which can then serve as a reference in terms of hardware, firmware, and software.
The next decision is what platform to build this reference design around. STM32? MSP430? ESP32? ATMega/Arduino? TI cc31xx/32xx?
We are choosing the ESP32 as it is very popular, provides nice IoT connectivity to Amazon/etc. (e.g. AWS IoT here) and includes Bluetooth, Bluetooth Low Energy (BTLE), and WiFi, and comes in a convenient FCC/CE-certified module.
And it also has a bunch of examples for communicating over WiFi, communicating with smartphone apps over BT and/or WiFi, etc.
And since we want users to be able to take the hardware design and modify it to their needs we plan to:
Often embedded applications are space-constrained, so making this reference design as compact as reasonable (while still using the modular approach above) is important.
Another why
is this will serve as the basis for the Visually-Impaired assistance device (#9).
how
:Modify the BW1098OBC to add an ESP32 System on Module, likely the ESP32-WROOM-32.
To keep the design small, it may be necessary to increase the number of layers on the BW1098OBC (from the 4 currently) to 6 or perhaps 8 and/or use HDI techniques such that the ESP32 module can be on the opposite side of the boards (but overall in the same physical location) as the BW1099 module, and perhaps the cameras above or below this ESP32 module.
In this system DepthAI SoM will be the SPI peripheral
and the microcontroller (ESP32 in this case) will be the SPI controller
.
Have a GPIO line from the ESP32 go to the DepthAI SoM, for DepthAI to be able to poke the ESP32 when it has new data.
That way we can support interrupt-driven data transfer on the ESP32.
So the flow would be:
Note that this would also still allow the ESP32 to do polling instead.
what
:Here's a super-initial idea on what this could look like, based on setting cameras and a sheet of paper the size of the ESP32 module above (18mm x 25.5mm) on an old BW1098EMB design, which is 60x40x20.5mm including heatsink and 60x40x7mm without heatsink (which is likely how it would be used in a product... simply heatsink to the product enclosure).
I'm looking at using one of my spare OAK-D-Lites to 'extend' my pipeline and hopefully increase the FPS, but think it's a bit of a waste of resources when I'm not using the cameras on it. As I understand it, the NCS isn't compatible with DepthAI (not sure if it could be made to be) but a cameraless OAK with USB interface (stick form-factor) would be nice.
Thanks to Eric Schleicher for making this AWESOME case.
The design files are here:
https://drive.google.com/file/d/1WM5IjW0DGskhd5H5P6Wyq4jQzqwWz4to/view?usp=sharing
I'm having issues communicating with the NOR flash on the BW099EMB through the DepthAI Embedded Development Kit.
In the BW1099EMB datasheet, it states there are 1K Pull-Up resistors on the SPI0_SIO2 and SPI0_SIO3 signals, which are hardwired to the NOR flash ~WP and ~HOLD signals. The dev kit doesn't have the ~WP and ~HOLD signals routed to the Molex connectors where I can control them externally, so I've probed the MT25Q flash physically on pin 7 and 3, and both were 0V.
I'm assuming the issue is the ~HOLD signal being asserted which is preventing proper SPI communications. Is there a way to verify the pull-up resistors on the SoM, or maybe something is pulling ~WP and ~HOLD down? I've performed combinations of holding the DepthAI in reset and not in reset, and both cases ~HOLD was asserted to ground.
Thanks guys.
Add product weights to the documentation.
why
:For low-light color and long-range high-quality digital zoom applications, it is beneficial to have a large image sensor with large pixels. The IMX283 provides both, with 20MP and a 15.86 mm optical diagonal sensor size (quite large) with 2.4um square pixels.
how
:ArduCam is helping us to make IMX283 sensor boards that work with DepthAI. We will then be writing a driver for this module, and see how much tuning is required.
what
:Support the IMX283 on DepthAI.
why
:For high-granularity depth and for support of low-visual-interest objects, having a dot projector to augment these surfaces with visual interest is extremely beneficial.
how
:Leverage IR-capable camera modules (either visible-light + IR or IR-bandpass, such as the upcoming ArduCam OV9282 modules) and an IR dot projector pair (and timing/strobe circuitry for more efficient power use) to make an active-illumination variant of DepthAI.
what
:Support active illumination on DepthAI.
I've found these two positions of OAK-D,
https://eu.mouser.com/ProductDetail/Luxonis/OAK-D-AB?qs=DPoM0jnrROWvGlNkjV8g6g%3D%3D
and
https://eu.mouser.com/ProductDetail/Luxonis/OAK-D-INTL?qs=DPoM0jnrROXjQVWBpbQqSw%3D%3D
What is the actual difference between those two ?
Comparision shows different "adaptor" but what does it means? Is it about different adopters or an actual interface to some other part?
why
:Although not intended for V-SLAM, DepthAI could provide decent V-SLAM in parallel with other functions (through depth + feature-tracking (VTRACK)) stream outputs.
To do so, having an integrated IMU would enable this additional/parallel use-case.
how
:We also have initially prototyped support for the BMI160 and BMM150 and have the IO for it brought out through the 100-pin connector of the BW1099 100-pin interface.
We've also prototyped the BMX055/BMI055, which are other options. And discussing offline, it seems the BNO085 may be an excellent choice, as it provides an absolute orientation result in one package.
We could alternatively add these directly to the BW1099 module, but it probably makes sense to leave the choice as to whether they are included (and where
) to the designer of the baseboard... as IMU location can be of critical important depending on the constraints on the design and colocation of image sensors/IMU/etc. to pertinent center locations on the design. So having the IMU/magnetometer on the baseboard affords these degrees of freedoms.
WRT the USB transfer of IMU data, XLink won't be a good option if we want low latency, as large transfers (from video streams) would delay the transmission.
We could add an interrupt endpoint to the USB device (could use HID class for example). XLink currently uses a single endpoint, bulk. And we can have on DepthAI a total of 3 different endpoints for output to host.
what
:Add the IMU & Magnetometer to the following baseboard designs, in this priority:
Implement an interrupt endpoint to the USB device for super-low-latency 500 or 1,000 Hz update to the host from the IMU/Mag (comb).
Hello,
Have you thought about - how to handle headlight glare and WLC and BLC (which are usually present in CCTV)
regards
why
:We’ve now gotten a bunch of reachouts about something like OAK-D, but with 150 DFOV, IR-laser-dot-projection depth, and matching resolution/FOV/global shutter for the color camera.
This is similar to this idea, #21, which we already had on the roadmap, but with the change that apparently a lot of applications want this and wide FOV, which is separately #15.
how
:ArduCam already implemented #15 , and now produces this IR-capable OV9282 150DFOV IR-capable module as a standard product for DepthAI customers, here. So far though, this has only been used in custom products built with OAK-SOM/etc.
We should work with ArduCam to (1) update this OV9282 to the new/standard ArduCam connector format here and (2) make a OV9782 variant with the same connector/format but that blocks IR (for good color representation). Likely prototyping first with the existing format/connector
what
:Wide FOV version of OAK-D-PRO (#114)
OAK-D-PRO-W (W = WideFOV)
The current iteration of hardware does not support the dot-emitters. The main reason is that the laser certification was taking longer than expected and the chip shortage for the projector prevented us from making enough boards. As we got a lot of requests for WFOV cameras only, we instead shifted the focus on this aspect of the design to have it out as soon as possible. The flood LED is still implemented on the design and can be used for testing in low light situations.
why
:For the 1099 SOM, we have the 1090FFC board, which allows folks to quickly prototype with various cameras (e.g. from ArduCam here), stereo baselines, filters (for hyperspectral systems), etc. quickly. We are currently lacking such a board for the 2099 SOM.
The 2099 SOM supports several additional things that the 1099 SOM does not, including:
So we need a board that allows such prototyping, while also enabling the additional functionality of the 2099 SOM above.
On the 1090FFC board, we tried to make it as small as possible so that when folks prototype, their overall prototype is not huge (as we have found that most of our customers need to use even their prototypes in space-constrained environments). We have actually found that even though we tried to make it small, it is still borderline "too big" for many customers, or outright is too big.
So for the 2090FFC variant, it is key to try to keep it as small, or close to as small, as the 1090FFC.
As this 4-camera board may be used with a variety of cameras, including those with hardware frame-sync, there should be a system onboard for allowing up to all 4 cameras to be synchronized.
how
:Increase layers as necessary from the 1090FFC, and spend additional routing/placement time/effort to make the 2090FFC as compact as reasonably possible. And we can also change/remove connectors/etc. to do so.
Likely use slots for the mounting holes, like below, to maximize the usable board area while minimizing the actual size of the device:
what
:A compact, 4-camera-capable FFC board with onboard SD-Card and IMU with onboard hardware allowing all 4 cameras to be hardware synced (FSIN).
why
:Improve the mechanical of OAK-D, as being done in #114 but without adding the cost of active depth (both in terms of $, and in terms of worse bright outdoor depth performance).
how
:Reuse all of OAK-D-PRO but no-populate the active illumination components, black out the glass where these would be, and use the existing OAK-D CCM.
what
:Make an existing passive-only OAK-D successor with the mechanical advantages of #114. And offer fixed-focus RGB like OAK-D-Pro as well for better vibration handling.
This will most likely still be called OAK-D
but will be Series 2
or S2
for short.
Hello,
I got to know of luxonis products yesterday while searching for depth camera, I saw real sense then I saw luxonis.
Can you please elaborate - how these products are different (intel realsense d4xx series + movidius)
Hi DepthAI team,
I see the 3D models of the carrier boards have the heatsink model integrated, but I'm not seeing the heatsink in the standalone SoM models. Is there a 3D STEP available for the BW1099EMB with the heatsink mounted, and if not is there a STEP just for the heatsink available? Thank you!
why
:Meanwhile, DepthAI and CEP (https://github.com/cortictechnology/cep#cortic-edge-computing-platform-cep) are AWESOME on the Pi, once you're done with Annoying^3 power above.
And for example, installing CEP alone takes 30 minutes (after you've cloned the repo). And cloning the repo itself may take a very long time, depending on the internet connection. And the installation process can sometimes kill SD cards (because of the compilation/etc. uses SWAP aggressively).
Given that we have thousands of customers using DepthAI and CEP on Raspberry Pi, that's a TON of time wasted by customers - waiting on things to download/compile/install. Best-case, it's 30 minutes if folks want CEP in addition to DepthAI.
Worse, the whole purpose of CEP is to be easy. Look at that drag-and-drop awesome:
It's such a buzz-kill to fight CLI stuff and wait for 30+ minutes of "Is this really going to work?" to just try it out.
So it would be cool if there were a way to prevent that buzz-kill (the annoyance^3). And save at least 30,000-minutes of folks time (1,000 folks, bare-minimum will use this, and it's 30 minutes for each of them). That's 500 hours of wasted time.
Worse, often customer uSD cards are too-small to fit all the nifty CEP software (which takes 10GB or so). So someone with an 8GB or 16GB card will discover 30+ minutes into this that it's not going to work, at all.
And even worse, the process of trying to install actually just corrupted their already-working 8GB or 16GB uSD card. And that's a really painful way to experience a new product, with a typical end-result of these customers being:
Using a 32GB uSD card prevents this.
how
:One idea is to do all this installation once to a "Golden 32GB" uSD. And then copy it 32GB uSD cards at scale, and sell these pre-flashed uSD cards. There are huge advantages of this:
(Ignore that the GIF is of a 16GB uSD card; I couldn't find a good GIF of a 32GB one. Rest assured; we'll be doing 32GB.)
what
:Make 32GB pre-flashed uSD card that has everything pre-installed. Is there some step that we could do so a customer doesn't have to? Then we'll do that.
why
:As in #16 there is a huge diversity of M12 lenses available to suit all sorts of problems. These also come in options for various filtering of specific spectra (opening up things like NDVI and others). Since the OV9281 is grayscale covering UV-A and IR spectra, this also opens up options for various filtering of specific spectra (opening up things like NDVI and others).
how
:ArduCam has an existing OV9281 module with M12 lens mount (see below for it used on a stereo camera) which we can leverage on DepthAI (as the OV9282 and OV9281 drivers are effectively the same).
what
:Support the M-12 Mount OV9282 (grayscale) and OV9782 (color global shutter equivalent of same specs) camera module from ArduCam. Prototype samples of both OV9282 and OV9782 from ArduCam are shown below working with depthai. The Debayering of the OV9782 is not implemented yet, but the sensor is returning data.
why
:In many cases it's advantageous to have DepthAI deployed remotely and to be able to selectibly (based on what it sees, or just settings) return metadata, images, and/or video - without need for hard wiring or Ethernet. If no video (and perhaps no images) are needed, then perhaps LoRa WAN could be used instead (and we're planning on supporting that separately.
If in some cases, having remote deployment with the capability to send video is needed in locations for which WiFi is intractable. So in this case, cell is the most valuable solution.
how
:We are evaluating various cell modules which we can integrate onboard. We will likely downselect one primarily on its flexibility (e.g. capability to be used in the widest geographical areas).
what
:Make a cell-module version of DepthAI.
why
:Higher resolution global shutter color can be necessary in conditions where the neural model or CV pipeline needs to be able to perceive high-speed moving objects and color with high(-ish) resolution. Global shutter gets quite pricey above ~2MP, so the 2.3MP AR0234 is a "sweet spot" sensor in terms of allowing high resolution (for global shutter) while also color and global shutter performance.
So for these application that require higher resolution than the OV9782 (which DepthAI already supports, see #17 and #21) on fast-moving objects, color, and global shutter, it would be beneficial to support the AR0234, which is 2.25 the resolution of the OV9782.
how
:Work with ArduCam to build the AR0234 modules as needed, and leverage our Gen2 pipeline builder to integrate a color camera node that supports AR0234.
what
:Support global shutter 1920x1200 60FPS AR0234 in the DepthAI ecosystem.
why
:We've gotten a lot of requests to make an OAK-1-Lite
in addition to OAK-D-Lite (here).
So we're going to make it.
how
:The OAK-1 enclosure is fine except that it does not have a dual-fastener system. So we're going to add dual M4 holes to the OAK-1 enclosure. And use this for all OAK-1-Lite, and all future OAK-1 enclosures (as a roll-in). This changes only the rear of the OAK-1 enclosure, as rendered below:
This will address the main (only?) complaint with the current OAK-1. Folks already love that it's small (unlike the O.G. OAK-D; which was too big).
what
:why
:Cortic Technology, here has been using OAK-D for K-12 education. He is on a mission to get this tech to K-12. He’s done a lot of these education series and has discovered what is actually needed in the market to have this sort of technology accessible by K-12 students. And what are the big pain-points and blockers.
One of the key needed bits is easy software. Cortic has been working on the software to make this applicable to K-12 (e.g. here), but there are hardware parts that are really really painful still. And if those are solved, he thinks this could scalably be applied across education.
And it looks great and will work fantastically for this. I’ve (Brandon has) done 1st-grade Hour of Code with such interfaces before.
In K-12 education settings there is a need for a “just works” spatial AI system where everything is included. Since it is kids working with it, it needs to be all enclosed, simple, and boot up and just work.
The use-case is on a robot like here:
https://twitter.com/CorticTechnolo1/status/1386761565475545092?s=20
And the robotics portions of these robots are actually controlled over Bluetooth or WiFi. So no electrical connection is needed.
And in fact, it is preferred to have the whole perception/host-compute/etc. battery powered so that it can be built into the lego/etc. anywhere on the bot - without wiring concerns/mess/etc. Ad wireless control of the bot allows.
One of the most painful parts of using such a system is getting the headless
going - meaning getting remote desktop connection going. So the feedback from Ye Lu is that having an enclosed product with an integrated battery, a single USB port (for charging), a built-in screen, and WiFi/BT for communication would be super helpful.
This would allow the product to boot up w/out any cables, show the demo right away on the screen so that students can see how the computer is seeing. And then from there, configure the device to do things and see how the computer sees things.
And having IMU, microphones, and speaker on the device will allow students to explore other aspects of interaction all on-device. Motion, audio input, audio output for feedback and or allowing students to construct interactive games with the robot.
Since this device will be used in education, it would be beneficial for it to be able to mount directly to legos.
how
:Leverage the CM4 designs we have (OAK-CM4-POE) to make a DepthAI + CM4 integrated product, leveraging the battery charge/discharge we have from the experimental 1092+LiPo work we did.
Use the CM4 with built-in eMMC so students can store video/etc. to the device for as long as the battery lasts (does not need to be any longer than that).
Use the CM4’s capability to drive a display to have an integrated display on the back.
Design an aluminum enclosure to help dissipate heat and make this easy/rugged for users and keep it cool enough.
Leverage the microphone design (likely using just a stereo pair of mics - that’s enough probably) and speaker design from the DM2092.
what
:All-in-one enclosed product with:
USB3C port:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.