Code to reproduce the results of the paper ELF: Embedded Localisation of Features in Pre-Trained CNN ICCV19

If you use this code, please cite the following paper:

Assia Benbihi, Matthieu Geist, Cédric Pradalier "ELF: Embedded Localisation of Features in Pre-Trained CNN" Proc. ICCV (2019)

@InProceedings{Benbihi_2019_ICCV,
  author={Benbihi, Assia and Geist, Matthieu and Pradalier, C{\'e}dric},
  title={ELF: Embedded Localisation of Features in Pre-Trained CNN},
  booktitle = {The IEEE International Conference on Computer Vision (ICCV)},
  month = {June},
  year = {2019}
}

• datasets: datasets used in the paper

• docker: tools to build the docker img

• meta: network weights, misc. files

• methods: feature extraction methods

  • cv: opencv implemented ones
  • elf_superpoint: superpoint variant of elf
  • elf_vgg: vgg variant of elf
  • superpoint: integration of code released by superpoint authors
  • tilde: integration of c++ code of the tilde authors

• plots: scripts to plot the paper figures

• scripts: examples on how to run the experiments

• tools: metrics code, misc. functions used by elf

    git clone --recursive https://github.com/abenbihi/elf.git
  

We recommend using docker to have an image with all the dependencies. All the python dependencies are specified in requirements.txt.

Currently, the online display of the image matches is not supported inside docker (because of the issue with X server). If you run the code inside docker, we recommend writing the images and their matches to disk to see them.

We require the specified OpenCV versions as other versions may lead to error messages regarding the patented methods (e.g. sift).

This code requires a GPU. We are not aware of hardware models requirements. This code is tested on the following models: QuadroM2200, K20 and GT1080Ti.

Install the dependencies specified in requirements.txt.

sudo pip3 install -r requirements.txt

Warning: if opencv complains about patented algoritms (cf error message at the bottom of the page), one solution is to install these two specific versions:

sudo pip3 install opencv-python==3.4.0.14
sudo pip3 install opencv-contrib-python==3.4.2.17

If you have issues with installing tensorflow, run this command instead:

sudo pip3 install --upgrade https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow_gpu-1.4.0-cp35-cp35m-linux_x86_64.whl

Requires nvidia-docker. Installation instructions can be found at nvidia-docker. Edit the file docker/Makefile to the path where you clone the data.

EXTERNAL_PATH_TO_WS=/path/to/elf

Build the image

cd docker
make root

This generates the docker image and run it once the compilation is done. It mounts the elf directory onto the docker image workspace directory.

It can take some time, so while it builds, get the datasets and the weights.

Once built, you can run the image with

nvidia-docker run --volume="/path/to/elf":"/home/ws" -it  -u root elf bash

Download all the datasets used in the paper with the provided script:

cd datasets
chmod +x get_datasets.sh
./get_datasets.sh

Download the pre-trained weights of the networks used in paper with the provided scripts:

cd meta/weights/
chmod +x get_weights.sh
./get_weights.sh

Set the working space path in tools/cst.py. If you cloned you directory in /path/to/ws then:

WS_DIR='/path/to/ws/elf/'
WS_DIR='/home/ws/' # for docker

Set the dataset you want to use in tools/cst.py':

DATA = data_name

with data_name in:

• 'hpatches': for the general performances
• 'hpatches_rot': for the multi-angle version of hpatches
• 'hpatches_s': for the multi-scale version of hpatches
• 'webcam': for the webcam dataset
• 'strecha': for the strecha dataset

Set DEBUG to true if you want to see the matches on the images online. Only available if you run the code outside of docker or if you can connect the right docker port that I did not find (sorry).

DEBUG = (1==1)

To evaluate an opencv method on experiment number <trials>:

./scripts/bench_cv.sh <method> <data> <trials>
# e.g.
./scripts/bench_cv.sh surf hpatch 0
./scripts/bench_cv.sh sift strecha 1

with:

• <method> in sift, surf, orb, kaze, mser.
• data in hpatch (for hpatches, hpatches_s, hpatches_rot), webcam, strecha.

The results are stored in the file res/<method>/<trials>/log.txt with the following format

*** i_ajuntament *** 0:00
** 2 **
rep:0.568 - N1:500 - N2:500 - M:284
ms:0.332 - N1:500 - N2:500 - M:284 - M_d:500 - inter:166
** 3 **
rep:0.281 - N1:500 - N2:417 - M:117
ms:0.065 - N1:500 - N2:417 - M:117 - M_d:417 - inter:27

The first line holds the scene name. It is followed by the image id being matches with the first image of the scene. The two following lines holds the metrics.

• rep: repeatability.
• N1/N2: number of keypoints detected in image1/image2.
• M: number of matching detected keypoints according to their geometric distance.
• M_d: number of matching detected keypoints according to their descriptor distance.
• inter: intersection of the keypoints matched based on their geometric distances and the keypoints matched based on their descriptor distance.

To compute the average repeatability and matching score for that expriments:

python3 tools/display_res.py --method <method> --trials <trials>
# e.g.
python3 tools/display_res.py --method sift --trials 0
python3 tools/display_res.py --method sift --trials 1
>>>> rep:    0.51191
>>>> ms:     0.24603

Open the script scripts/bench_cv.sh and set ok=1 to activate the script you want to run:

• The script suffixed with des_proxy runs the OpenCV detector with the proxy descriptor.
• The script suffixed with det_elf runs the elf detector with the OpenCV descriptor.

See OpenCV methods section to set the dataset to play with.

Results are written to res/elf/.

./scripts/bench_elf.sh <trials> <data>
python3 tools/display_res.py --method <method> --trials <trials>

# e.g.
./scripts/bench_elf.sh 0 hpatch
# set DATA='strecha' in tools/cst.py
./scripts/bench_elf.sh 1 strecha
# set DATA='webcam' in tools/cst.py
./scripts/bench_elf.sh 2 webcam
python3 tools/display_res.py --method elf --trials 2

Results are written to res/superpoint/.

./scripts/bench_elf_superpoint.sh <trials> <data>

# e.g.
./scripts/bench_elf_superpoint.sh 0 hpatch
python3 tools/display_res.py --method superpoint --trials 0

Choose the script to run in scripts/bench_superpoint.sh by setting ok=1 before the script. This shell scripts allows you to run the following evaluations.

To run experiment number <trials>:

./scripts/bench_superpoints.sh <data> <trials>
# e.g.
./scripts/bench_superpoint hpatch 0

The script in 2 steps. The first step runs superpoint detection and saves the keypoint in txt format in res/superpoint/<kp_dir_id>. The second step runs the proxy descriptor and stores the evaluation in res/superpoint/<trials>.

./scripts/bench_superpoints.sh <data> <trials> <kp_dir_id>
# e.g.
./scripts/bench_superpoint.sh 3 hpatch 2
# writes superpoint detector in res/superpoint/2
# writes evaluation results in res/superpoint/3

The script in 2 steps. The first step runs elf detection and saves the keypoint in txt format in res/elf/<kp_dir_id>. The second step runs the superpoint descriptor and stores the evaluation in res/superpoint/<trials>.

./scripts/bench_superpoints.sh <data> <trials> <kp_dir_id>
./scripts/bench_superpoint.sh 4 hpatch 1
# writes superpoint detector in res/elf/1
# writes evaluation results in res/superpoint/4

You need to compile the C++ version of TILDE.

cd methods/tilde/c++
mkdir build
cd build
cmake ..
make

./scripts/bench_tilde.sh <trials> <data>
python3 tools/display_res.py --method <method> --trials <trials>

# e.g.
./scripts/bench_tilde.sh 0 hpatch

Choose the script to run in scripts/bench_lift.sh by setting ok=1 before the script. This shell sctrips allows you to run the following evaluations.

Two models are provided: either the rotation-augmented model or the non augmented one. The script runs in 2 steps. First generate the lift keypoints and descriptors and saves them file with evaluation/gen_lift.sh: the outputs are stored in res/lift/<lift_data_id>. Then compute the metrics with scripts/bench_lfnet.sh.

./evaluation/gen_lift.sh <lift_data_id>
./scripts/bench_lift.sh <lift_data_id> <data> <trials>
python3 evaluation/display_res.py --method lift --trials <trials>

# e.g.
# generate lift keypoint and descriptor in res/lift/0
./evaluation/gen_lift.sh 0

# evaluate lift output in res/lift/0 and writes evaluation in res/lift/1
./scripts/bench_lift.sh 0 hpatch 1
python3 evaluation/display_res.py --method lift --trials 1

First generate the lift keypoints (see previous section): the outputs are stored in res/lift/<lift_data_id>. Then run the proxy descriptor and stores the evaluation in res/elf/<trials>.

./scripts/gen_lift.sh <lift_data_id>
./scripts/bench_lift.sh <lift_data_id> <data> <trials> 

# e.g.
# generate lift keypoint and descriptor in res/lift/0
./sriptsn/gen_lift.sh 0

# evaluates proxy desriptor with lift kp stored res/lift/0 
# writes evaluation in res/lift/1
./scripts/bench_lift.sh 0 hpatch 1
python3 tools/display_res.py --method lift --trials 1

The script in 3 steps.

• The first step runs elf detection and saves the keypoint in txt format in res/elf/<kp_dir_id>.

• The second step generates the lift descriptors and stores the output in /res/lift/<lift_data_id>.

• The final step computes the metrics and stores the evaluation in res/superpoint/<trials>.

    ./scripts/bench_lift.sh <lift_data_id> <data> <trials> <kp_dir_id>
    
    # e.g.
    ./scripts/bench_lift.sh 2 hpatch 3 0
    # writes elf keypoints in res/elf/0
    # writes lift descriptor in res/lift/2
    # write evaluation in res/lift/3
    python3 tools/display_res.py --method lift --trials 3
  

Choose the script to run in scripts/bench_lfnet.sh by setting ok=1 before the script. This shell sctrips allows you to run the following evaluations.

With either the indoor or the outdoor model.

./scripts/bench_lfnet.sh <trials> <data> <model>
python3 tools/display_res.py --method <method> --trials <trials>

# e.g.
./scripts/bench_lfnet.sh 0 hpatch indoor
./scripts/bench_lfnet.sh 1 hpatch outdoor
python3 tools/display_res.py --method lnet --trials 0

The script in 2 steps. The first step runs lfnet detection and saves the keypoint in txt format in res/lfnet/<kp_dir_id>. The second step runs the proxy descriptor and stores the evaluation in res/elf/<trials>.

./scripts/bench_lfnet.sh <data> <trials> <kp_dir_id>
./scripts/bench_lfnet.sh 3 hpatch indoor 2
# writes lfnet detector in res/lfnet/2
# writes evaluation results in res/lfnet/3

The script in 2 steps. The first step runs elf detection and saves the keypoint in txt format in res/elf/<kp_dir_id>. The second step runs the lfnet descriptor and stores the evaluation in res/lfnet/<trials>.

./scripts/bench_lfnet.sh <data> <trials> <kp_dir_id>
./scripts/bench_lfnet.sh 4 hpatch indoor 1
# writes lfnet detector in res/elf/1
# writes evaluation results in res/lfnet/4

./scripts/bench_elf_lfnet.sh <trials>

python3 tools/display_res.py --method elf-lfnet --trials <trials>
python3 tools/display_res.py --method elf-lfnet --trials 0

Each script plot is named after the paper figure number.

cd plots
python3 fig5_hpatch.py 

• Pb: OpenCV complains that sift is in the non-free package of opencv

    C:\projects\opencv-python\opencv_contrib\modules\xfeatures2d\src\sift.cpp:1207:
    error: (-213:The function/feature is not implemented) 
    This algorithm is patented and is excluded in this configuration; Set
    OPENCV_ENABLE_NONFREE CMake 
    option and rebuild the library in function
    'cv::xfeatures2d::SIFT::create'
  

• Sol: Please, install the specific required versions above as there are some versions imcompability issues. I don't know why, nor how to solve it so the simple solution is to use the given versions that are tested and work.