The idea behind this work is to setup appropriate framework for a camera-lidar based collision detection/avoidance system. For this purpose, a camera-based 2D feature tracker is built here and various detector / descriptor combinations are tested, to see which ones perform best. There are four-parts to the project:

• loading images, setting up data structures and putting everything into a ring buffer to optimize memory load.
• integrate several keypoint detectors such as HARRIS, FAST, BRISK and SIFT and compare them with regard to number of keypoints and speed.
• then focus on descriptor extraction and matching using brute force and also the FLANN approach.
• once the code framework is complete, test various algorithms in different combinations and compare them with regard to some performance measures.

• cmake >= 2.8
  • All OSes: click here for installation instructions
• make >= 4.1 (Linux, Mac), 3.81 (Windows)
  • Linux: make is installed by default on most Linux distros
  • Mac: install Xcode command line tools to get make
  • Windows: Click here for installation instructions
• OpenCV >= 4.1
  • This must be compiled from source using the -D OPENCV_ENABLE_NONFREE=ON cmake flag for testing the SIFT and SURF detectors.
  • The OpenCV 4.1.0 source code can be found here
• gcc/g++ >= 5.4
  • Linux: gcc / g++ is installed by default on most Linux distros
  • Mac: same deal as make - install Xcode command line tools
  • Windows: recommend using MinGW

1. Clone this repo.
2. Make a build directory in the top level directory: mkdir build && cd build
3. Compile: cmake .. && make
4. Run it: ./2D_feature_tracking.

After implementation of keypoint-detection, descriptor extraction and keypoint-matching in consecutive images, the results obtained are presented in the following sections. Here are some notes on the results:

• AKAZE detector works only with AKAZE descriptor
• For matching with SIFT descriptor we need to use FLANN. SIFT does not work with BF
• SIFT detector and ORB descriptor do not work together
• for SHITOMASI blockSize of 4 has been used
• for HARRIS blockSize of 2 has been used

Total number of keypoints detected by each method for every image are listed in the table bellow:

Corresponding time taken in detection is shown in the plot here along with a sample output for AKAZE descriptor:

Associated number of keypoints detected on preceding vehicle in the images are:

As far as distribution of neighborhood size of keypoints detected on preceding vehicle is concerned, some of the detector methods have fixed neighborhood sizes, namely:

• SHITOMASI -- 4
• HARRIS -- 6
• FAST -- 7

while others have a distribution, among which ORB seems to have large spread. The distribution is shown in following four figures and standard deviation is tabulated here:

A mean of the number of matched keypoints for all 10 images using all possible combinations of detectors and descriptors is shown here.

The details for every single combination of detectors and descriptors with corresponding matched points are also provided in the following table for reference

Total number of matched keypoints for each combination:

A sample matched-pair with ORB-FREAK combination is shown here:

The mean of time taken for keypoint detection and descriptor extraction for all combinations is listed below.

These results and all others presented above are derived from data that can be found in the results excel-sheet

In a dynamic scenario like autonomous driving, the speed and accuracy of detection are of utmost importance. Hence, to arrive at recommendation, I computed the ratio of mean-matched-keypoints to mean-detection-extraction-time, defining a score parameter as:

score = 1000 * (mean-matched-keypoints / mean-detection-extraction-time)

The corresponding scores are shown in image below, with highest three highlighted in different shades of green.

Based on these scores my recommendations are: