bluedream1121 / key.net_pytorch Goto Github PK

[Reproduce] KeyNet (Barroso-Laguna et al. ICCV 2019) implementation using PyTorch. (Unofficial)

Python 100.00%

key.net_pytorch's Introduction

Key.Net: Keypoint Detection by Handcrafted and Learned CNN Filters (PyTorch)

PyTorch reproduce source code for the ICCV19 paper:

"Key.Net: Keypoint Detection by Handcrafted and Learned CNN Filters".
Axel Barroso-Laguna, Edgar Riba, Daniel Ponsa, Krystian Mikolajczyk. ICCV 2019.

[Paper on arxiv]

Most of the source codes are replaced by the original Key.Net Tensorflow source code.

Prerequisite

conda create --name keyNet 
conda activate keyNet 
conda install pytorch==1.8.0 torchvision==0.9.0 cudatoolkit=11.1 -c pytorch -c conda-forge
pip install opencv-python tqdm scikit-image pandas
pip install torchgeometry

Training Key.Net

Before training Key.Net a synthetic dataset must be generated. In the original paper, we downloaded ImageNet and used it to generate synthetic pairs of images, however, any other dataset would work if it is big enough. Therefore, the first time you run the train.py script, .npy files for synthetic pairs will be generated at datasets folder, one for training and another for validation. This is only done when the code couldn't find them, thus, the next runs of the script will skip this part.

python train.py --data-dir /path/to/ImageNet

Check the arguments to customize your training, some parameters you might want to change are:

Dataset parameters:
- max-angle: The max angle value for generating a synthetic view to train Key.Net.
- max-scale: The max scale value for generating a synthetic view to train Key.Net.
- max-shearing: The max shearing value for generating a synthetic view to train Key.Net.
Network Architecture:
- num-filters: The number of filters in each learnable block.
- num-learnable-blocks: The number of learnable blocks after handcrafted block.
- num-levels-within-net: The number of pyramid levels inside the architecture.
- factor-scaling-pyramid: The scale factor between the multi-scale pyramid levels in the architecture.
- conv-kernel-size: The size of the convolutional filters in each of the learnable blocks.

Feature Extraction

extract_multiscale_features.py can be used to extract Key.Net features for a given list of images. The list of images must contain the full path to them, if they do not exist, an error will raise.

The script generates two numpy files, one '.kpt' for keypoints, and a '.dsc' for descriptors. The descriptor used together with Key.Net is HardNet. The output format of the keypoints is as follow:

keypoints [N x 4] array containing the positions of keypoints x, y, scales s and their scores sc.

Arguments:

list-images: File containing the image paths for extracting features.
results-dir: The output path to save the extracted features.
checkpoint-det-dir: The path to the checkpoint file to load the detector weights. Default: Pretrained Key.Net.
checkpoint-desc-dir: The path to the checkpoint file to load the HardNet descriptor weights.
num-points: The number of desired features to extract. Default: 1500.
extract-MS: Set to True if you want to extract multi-scale features. Default: True.

Run the following script to generate the keypoint and descriptor numpy files from the image allocated in test_im directory.

python extract_multiscale_features.py --list-images test_im/image.txt --results-dir test_im/

HSequences Benchmark

We also provide the benchmark to compute HSequences repeatability (single- and multi-scale), and MMA metrics. To do so, first download full images (HSequences) from HPatches repository. Once downloaded, place it on the root directory of the project. We provide a file HSequences_bench/HPatches_images.txt containing the list of images inside HSequences.

Run the next script to compute the features from HSequences:

python extract_multiscale_features.py --list-images HSequences_bench/HPatches_images.txt --checkpoint-det-dir keyNet/pretrained_nets/keyNet.pt

Once all features have been extracted, to compute repeatability and MMA metrics run:

python hsequences_bench.py --results-dir extracted_features --results-bench-dir HSequences_bench/results --split full

Use arguments to set different options:

results-bench-dir: The output path to save the results in a pickle file.
results-dir: The output path to load the extracted features.
split: The name of the HPatches (HSequences) split. Use full, view or illum.
top-k-points: The number of top points to use for evaluation. Set to None to use all points.
pixel-threshold: The distance of pixels for a matching correspondence to be considered correct.
overlap: The overlap threshold for a correspondence to be considered correct.
detector-name: Set the name of the detector for which you desire to compute the benchmark (and features have been already extracted).

BibTeX

If you use this code in your research, please cite the original authors' paper:

@InProceedings{Barroso-Laguna2019ICCV,
    author = {Barroso-Laguna, Axel and Riba, Edgar and Ponsa, Daniel and Mikolajczyk, Krystian},
    title = {{Key.Net: Keypoint Detection by Handcrafted and Learned CNN Filters}},
    booktitle = {Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision},
    year = {2019},
}

key.net_pytorch's People

Contributors

Stargazers

Watchers

Forkers

shangyongbin lianhui1993 js228 baorunhan noticeable

key.net_pytorch's Issues

FileNotFoundError: [Errno 2] No such file or directory: 'datasets/synth/'

How to fix this error? The input command -------- "python train.py --data-dir /path/to/ImageNet"
I downloaded the data set, but I still don't know how to solve it? I sent an email earlier. Sorry to interrupt.

model cannot convergence？

Start training Key.Net Architecture
Total images in directory at " /second_disk/data/imagenet/ILSVRC/Data/CLS-LOC/val/ " is : 50000
Generating Synthetic pairs . . .
8%|█████████████▎ | 4205/50000 [07:33<1:22:15, 9.28it/s]
Generating Synthetic pairs . . .
3%|████▊ | 1487/50000 [02:12<32:51, 24.61it/s]
28%|███████████████████████████████████████████▉ | 13913/50000 [29:19<1:16:02, 7.91it/s]

of Training / validation : 4001 1501

mode : train the number of examples : 4001
mode : val the number of examples : 1501
MSIP hyperparameters : [8, 16, 24, 32, 40] [256.0, 64.0, 16.0, 4.0, 1.0]
================ List of Learnable model parameters ================
learner.conv_0.weight torch.Size([8, 10, 5, 5])
learner.conv_0.bias torch.Size([8])
learner.bn_0.weight torch.Size([8])
learner.bn_0.bias torch.Size([8])
learner.conv_1.weight torch.Size([8, 8, 5, 5])
learner.conv_1.bias torch.Size([8])
learner.bn_1.weight torch.Size([8])
learner.bn_1.bias torch.Size([8])
learner.conv_2.weight torch.Size([8, 8, 5, 5])
learner.conv_2.bias torch.Size([8])
learner.bn_2.weight torch.Size([8])
learner.bn_2.bias torch.Size([8])
last_layer_learner.bn_last.weight torch.Size([24])
last_layer_learner.bn_last.bias torch.Size([24])
last_layer_learner.conv_last.weight torch.Size([1, 24, 1, 1])
last_layer_learner.conv_last.bias torch.Size([1])
The number of learnable parameters : 5345

Key.Net Validation time 12.386 33.382 9.088 9.071 4.530: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 1501/1501 [01:09<00:00, 21.48it/s]

Epoch -1 : Repeatability Validation: 53.282.
Key.Net Training: 0%| | 0/251 [00:00<?, ?it/s]/home/ly/anaconda3/lib/python3.8/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:2157.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/home/ly/anaconda3/lib/python3.8/site-packages/torch/nn/functional.py:4003: UserWarning: Default grid_sample and affine_grid behavior has changed to align_corners=False since 1.3.0. Please specify align_corners=True if the old behavior is desired. See the documentation of grid_sample for details.
warnings.warn(
current loss : 15453.7783, avg loss : 982.9105: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 251/251 [00:50<00:00, 4.99it/s]
Epoch 0 (Training). Loss: 982.9105. Time per epoch: 50.2772
Key.Net Validation time 12.015 31.985 9.023 8.860 4.428: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 1501/1501 [01:07<00:00, 22.17it/s]
Epoch 0 (Validation) : Repeatability (rep_s): 53.282.
rep_m : 31.083, error_overlap_s : 0.171, error_overlap_m : 0.143, possible_matches : 5.171.

current loss : 14464.5400, avg loss : 979.1736: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 251/251 [00:49<00:00, 5.11it/s]
Epoch 1 (Training). Loss: 979.1736. Time per epoch: 49.0984
Key.Net Validation time 12.016 31.967 9.051 9.002 4.479: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 1501/1501 [01:07<00:00, 22.13it/s]
Epoch 1 (Validation) : Repeatability (rep_s): 53.282.
rep_m : 31.083, error_overlap_s : 0.171, error_overlap_m : 0.143, possible_matches : 5.171.

current loss : 14928.6025, avg loss : 981.0668: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 251/251 [00:49<00:00, 5.09it/s]
Epoch 2 (Training). Loss: 981.0668. Time per epoch: 49.3484
Key.Net Validation time 12.197 32.535 9.064 8.994 4.454: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 1501/1501 [01:08<00:00, 21.86it/s]
Epoch 2 (Validation) : Repeatability (rep_s): 53.282.
rep_m : 31.083, error_overlap_s : 0.171, error_overlap_m : 0.143, possible_matches : 5.171.

current loss : 13211.3369, avg loss : 974.2202: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 251/251 [00:51<00:00, 4.87it/s]
Epoch 3 (Training). Loss: 974.2202. Time per epoch: 51.5639
Key.Net Validation time 12.180 32.873 9.017 8.950 4.423: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 1501/1501 [01:08<00:00, 21.78it/s]
Epoch 3 (Validation) : Repeatability (rep_s): 53.282.
rep_m : 31.083, error_overlap_s : 0.171, error_overlap_m : 0.143, possible_matches : 5.171.

current loss : 14682.5088, avg loss : 979.8560: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 251/251 [00:53<00:00, 4.72it/s]
Epoch 4 (Training). Loss: 979.8560. Time per epoch: 53.1963
Key.Net Validation time 12.050 31.874 9.081 8.963 4.463: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 1501/1501 [01:07<00:00, 22.15it/s]
Epoch 4 (Validation) : Repeatability (rep_s): 53.282.
rep_m : 31.083, error_overlap_s : 0.171, error_overlap_m : 0.143, possible_matches : 5.171.

current loss : 14804.9980, avg loss : 980.3887: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 251/251 [00:43<00:00, 5.80it/s]
Epoch 5 (Training). Loss: 980.3887. Time per epoch: 43.3099
Key.Net Validation time 13.058 35.885 9.078 8.997 4.478: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 1501/1501 [01:13<00:00, 20.52it/s]
Epoch 5 (Validation) : Repeatability (rep_s): 53.282.
rep_m : 31.083, error_overlap_s : 0.171, error_overlap_m : 0.143, possible_matches : 5.171.

current loss : 14994.8564, avg loss : 980.9094: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 251/251 [00:45<00:00, 5.46it/s]
Epoch 6 (Training). Loss: 980.9094. Time per epoch: 45.965
Key.Net Validation time 12.277 32.585 9.041 8.973 4.475: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 1501/1501 [01:08<00:00, 21.80it/s]
Epoch 6 (Validation) : Repeatability (rep_s): 53.282.
rep_m : 31.083, error_overlap_s : 0.171, error_overlap_m : 0.143, possible_matches : 5.171.

current loss : 14354.8457, avg loss : 978.5595: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 251/251 [00:46<00:00, 5.43it/s]
Epoch 7 (Training). Loss: 978.5595. Time per epoch: 46.2289
Key.Net Validation time 12.470 33.744 9.028 8.973 4.456: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 1501/1501 [01:10<00:00, 21.39it/s]
Epoch 7 (Validation) : Repeatability (rep_s): 53.282.
rep_m : 31.083, error_overlap_s : 0.171, error_overlap_m : 0.143, possible_matches : 5.171.

current loss : 14517.4482, avg loss : 979.5426: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 251/251 [00:45<00:00, 5.51it/s]
Epoch 8 (Training). Loss: 979.5426. Time per epoch: 45.5276
Key.Net Validation time 12.770 34.819 9.045 8.939 4.442: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 1501/1501 [01:11<00:00, 21.00it/s]
Epoch 8 (Validation) : Repeatability (rep_s): 53.282.
rep_m : 31.083, error_overlap_s : 0.171, error_overlap_m : 0.143, possible_matches : 5.171.

current loss : 15131.8047, avg loss : 981.9330: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 251/251 [00:44<00:00, 5.62it/s]
Epoch 9 (Training). Loss: 981.9330. Time per epoch: 44.6557
Key.Net Validation time 12.918 35.169 9.100 9.042 4.506: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 1501/1501 [01:12<00:00, 20.76it/s]
Epoch 9 (Validation) : Repeatability (rep_s): 53.282.
rep_m : 31.083, error_overlap_s : 0.171, error_overlap_m : 0.143, possible_matches : 5.171.

current loss : 15062.2842, avg loss : 981.2635: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 251/251 [00:53<00:00, 4.74it/s]
Epoch 10 (Training). Loss: 981.2635. Time per epoch: 53.0044
Key.Net Validation time 13.150 36.047 9.059 8.988 4.465: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 1501/1501 [01:13<00:00, 20.51it/s]
Epoch 10 (Validation) : Repeatability (rep_s): 53.282.
rep_m : 31.083, error_overlap_s : 0.171, error_overlap_m : 0.143, possible_matches : 5.171.

current loss : 17635.6484, avg loss : 991.6248: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 251/251 [00:54<00:00, 4.61it/s]
Epoch 11 (Training). Loss: 991.6248. Time per epoch: 54.4852
Key.Net Validation time 12.225 33.178 9.016 8.902 4.428: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 1501/1501 [01:09<00:00, 21.71it/s]
Epoch 11 (Validation) : Repeatability (rep_s): 53.282.
rep_m : 31.083, error_overlap_s : 0.171, error_overlap_m : 0.143, possible_matches : 5.171.

current loss : 16682.6250, avg loss : 987.4334: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 251/251 [00:55<00:00, 4.50it/s]
Epoch 12 (Training). Loss: 987.4334. Time per epoch: 55.8171
Key.Net Validation time 12.089 32.511 9.026 8.965 4.445: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 1501/1501 [01:08<00:00, 21.91it/s]
Epoch 12 (Validation) : Repeatability (rep_s): 53.282.
rep_m : 31.083, error_overlap_s : 0.171, error_overlap_m : 0.143, possible_matches : 5.171.

current loss : 13877.7773, avg loss : 976.8004: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 251/251 [00:47<00:00, 5.33it/s]
Epoch 13 (Training). Loss: 976.8004. Time per epoch: 47.106
Key.Net Validation time 12.397 33.194 9.130 9.062 4.467: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 1501/1501 [01:09<00:00, 21.49it/s]
Epoch 13 (Validation) : Repeatability (rep_s): 53.282.
rep_m : 31.083, error_overlap_s : 0.171, error_overlap_m : 0.143, possible_matches : 5.171.

current loss : 15138.8750, avg loss : 981.9058: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 251/251 [00:59<00:00, 4.23it/s]
Epoch 14 (Training). Loss: 981.9058. Time per epoch: 59.2877
Key.Net Validation time 13.117 35.595 9.129 9.060 4.500: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 1501/1501 [01:13<00:00, 20.53it/s]
Epoch 14 (Validation) : Repeatability (rep_s): 53.282.
rep_m : 31.083, error_overlap_s : 0.171, error_overlap_m : 0.143, possible_matches : 5.171.

current loss : 14566.9287, avg loss : 979.8281: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 251/251 [01:02<00:00, 4.04it/s]
Epoch 15 (Training). Loss: 979.8281. Time per epoch: 62.071
Key.Net Validation time 12.306 32.738 9.103 9.067 4.471: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 1501/1501 [01:09<00:00, 21.72it/s]
Epoch 15 (Validation) : Repeatability (rep_s): 53.282.
rep_m : 31.083, error_overlap_s : 0.171, error_overlap_m : 0.143, possible_matches : 5.171.

current loss : 14144.3525, avg loss : 977.8405: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 251/251 [00:48<00:00, 5.17it/s]
Epoch 16 (Training). Loss: 977.8405. Time per epoch: 48.5055
Key.Net Validation time 13.120 36.314 9.062 8.990 4.468: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 1501/1501 [01:13<00:00, 20.43it/s]
Epoch 16 (Validation) : Repeatability (rep_s): 53.282.
rep_m : 31.083, error_overlap_s : 0.171, error_overlap_m : 0.143, possible_matches : 5.171.

datasets train

Hello, I would like to ask you some questions about the KeyNet-pytorch project. We downloaded the ImageNet ILSVRC2012 data set, which contains the training set ILSVRC2012_img_train.tar, ILSVRC2012_img_train_t3.tar, the verification set ILSVRC2012_img_val.tar folder, we are in ILSVRC2012_img_train_t3 for the first time. Tar selected 9000 images, and selected 3000 images in ILSVRC2012_img_val.tar for training, but the training effect is not good, so I want to ask how your training data set is selected? Looking forward to your reply!

Data set on training

I was wondering where the training data set was downloaded from? I didn't understand the training... How are synthetic datasets generated??

datasets train

0it [00:00, ?it/s]
0it [00:00, ?it/s]
mode : train the number of examples : 0
mode : val the number of examples : 0
Start training Key.Net Architecture
SAVEPAIR datasets/synth/
Loading Synthetic pairs . . .
Loading Synthetic pairs . . .

of Training / validation : 0 0

Hello，I didn't even read his data when I tried to train.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.