jasonmatos23-git / depthperceptionhapticsystem Goto Github PK

Make image from the camera available to other software components via the input component.

Place integer quantized model.

Bug has to do with #27 and its associated PR. The bus is closed before the pwm script can write 0 to all outputs, which results in the LEDs remaining on after program termination. A quick fix is to write 0 to all outputs before closing the smbus object, but this may not be an ideal solution.

The essential components need to be fully integrated to produce a minimum, working, and error-free product. In particular, the following must be connected:

• API: Camera module, Forward LiDAR, Input, Output, Speaker, PWM Motor control
• Services and Routines: DepthPerception, EmergencyDetection
• Modes: General, Low Power, and (for demonstration) LiDAR/CV only
• Scheduler, System (mode/state), Hardware Interrupts, and mode indications for callback functions

Note: The intention is for this, and the representative value issue #31 , and potentially #14 be the last PRs for dev/v1-0-0. Tuning, optimizations, error checking/handling should be covered in a future dev/v1-1-0 or dev/v2-0-0

Enable a low power mode on detection of no movement for some amount of time.

When canceling execution of Output.py, there is no function to delete the PWM instance (which enables sleep mode). Additionally, flickering occurs if chip is not disconnected from power between runs.

Modules included in the API do not properly disable/re-enable/destroy. Must implement better resource management to prevent errors such as #33 . Additional optimizations may be included in this issue.

Use GPIO interrupts to change the system state/mode on button push.

Implement curb detection using LiDAR.

Setup DPHS, SystemContainer, Executable, SystemService, and Service components. Basically, setting up the framework for the core functionality of the project.

Methods and variables that should not be exposed to other components should be preceded by a '_'. For example, "self.input_" should become "self._input_".

Set peripheral devices (e.g. LiDAR, camera, motor controller) to their low power modes. Mode may be revisited in the future for improved performance.

Find out how to connect software to PCA9685 (PWM LED/motor control chip). Additionally set output unit to handle changing duty cycles.

Noticing redundant creation of SMBus objects. Will be more effective to create one in system.py and pass it into the input and output components. Additionally consider using with _ as bus and implementing __enter__ and __exit__. Former likely in DPHS.py, the latter within system.py. Frankly creating multiple of these objects may not even work, so this is important to address.

Tie general mode, low power mode, and outdoor mode together. This is likely to go in scheduler.py. Scheduler should poll the mode and run/toggle services based on that mode.

Current model (Intel Midas/v2_1_small) runs at approximately 900 ms without optimizations on the Raspberry Pi 4 using TFLite with Python. Should find either a new model which has a lower latency, modify a model, or create a model based on existing architectures.

Must transform data from depth inference to 9 duty cycle values. Current approach is bilinear resize output to 32x32, apply Gaussian kernels to different points of image and find a maximum value, feed the maximums into a function for intensity of estimated depth, and output the results over PWM.

Create a TensorFlow Lite model and apply it to images to produce a depth map. Based on Zhang, Ziyu, et al. “A Simple Baseline for Fast and Accurate Depth Estimation on Mobile Devices.” 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2021.

implement outdoor mode with curb detection

Get accelerometer data and make it available to services.

Errors and exceptions should be handled to prevent the system from crashing or stalling out across all components.

Use hardware PWM to connect to and play audio through the speaker. Figure out what kind of audio can be played, how to store and play audio clips, and consider how routines will request audio to be played.

Components should be as independent as possible to allow for faster testing and demo development. For example, services should be able to be run without creating a RoutineContainer object.

Special signal for front-facing LiDAR detecting threat.

Must set pins to their appropriate ALTs based on @ChadPauley 's GPIO assignment document. GPIO0/1 may not work for the lidar. In that case use any other unoccupied SDA/SCL.

Introduction of PWM and LiDAR modules to the API folder has made things slightly disorganized. Adding an Input/ and Output/ directory to place these modules in (or a modules/ directory) is desirable. Additionally, separating the Camera code from the Input.py file is desirable for organization, clarity, ease-of-updating, and ensuring proper resource management.

Connect to LiDAR and make data available to services.