Code Monkey home page Code Monkey logo

shape_based_matching's Introduction

shape_based_matching

update:
fusion implementation to run faster!
icp is also refined to be faster and easier to use

Transforms in shape-based matching
pose refine with icp branch, 0.1-0.5 degree accuracy
icp + subpixel branch, < 0.1 degree accuracy
icp + subpixel + sim3(previous is so3) branch, deal with scale error

try to implement halcon shape based matching, refer to machine vision algorithms and applications, page 317 3.11.5, written by halcon engineers
We find that shape based matching is the same as linemod. linemod pdf

halcon match solution guide for how to select matching methods(halcon documentation):
match

steps

  1. change test.cpp line 9 prefix to top level folder

  2. in cmakeList line 23, change /opt/ros/kinetic to somewhere opencv3 can be found(if opencv3 is installed in default env then don't need to)

  3. cmake make & run. To learn usage, see different tests in test.cpp. Particularly, scale_test are fully commented.

NOTE: On windows, it's confirmed that visual studio 17 works fine, but there are some problems with MIPP in vs13. You may want old codes without MIPP: old commit

thoughts about the method

The key of shape based matching, or linemod, is using gradient orientation only. Though both edge and orientation are resistant to disturbance, edge have only 1bit info(there is an edge or not), so it's hard to dig wanted shapes out if there are too many edges, but we have to have as many edges as possible if we want to find all the target shapes. It's quite a dilemma.

However, gradient orientation has much more info than edge, so we can easily match shape orientation in the overwhelming img orientation by template matching across the img.

Speed is also important. Thanks to the speeding up magic in linemod, we can handle 1000 templates in 20ms or so.

Chinese blog about the thoughts

improvment

Comparing to opencv linemod src, we improve from 6 aspects:

  1. delete depth modality so we don't need virtual func, this may speed up

  2. opencv linemod can't use more than 63 features. Now wo can have up to 8191

  3. simple codes for rotating and scaling img for training. see test.cpp for examples

  4. nms for accurate edge selection

  5. one channel orientation extraction to save time, slightly faster for gray img

  6. use MIPP for multiple platforms SIMD, for example, x86 SSE AVX, arm neon. To have better performance, we have extended MIPP to uint8_t for some instructions.(Otherwise we can only use half feature points to avoid int8_t overflow)

  7. rotate features directly to speed up template extractions; selectScatteredFeatures more evenly; exautive select all features if not enough rather than abort templates(but features <= 4 will abort)

some test

Example for circle shape

You can imagine how many circles we will find if use edges

circle1 circle1

Not that circular

circle2 circle2

Blur

circle3 circle3

circle template before and after nms

before nms

before

after nms

after

Simple example for arbitary shape

Well, the example is too simple to show the robustness
running time: 1024x1024, 60ms to construct response map, 7ms for 360 templates

test img & templ features
test
templ

noise test

test2

some issues you may want to know

Well, issues are not clearly classified and many questions are discussed in one issue sometimes. For better reference, some typical discussions are pasted here.

object too small?
failure case?
how to run even faster?

shape_based_matching's People

Contributors

meiqua avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

shape_based_matching's Issues

Scene_edge的query函数的idx有越界危险

连续检测的程序,碰到idx出现了几次越界,然后程序就直接崩了……

还没查明白为什么会出现idx越界,
不过在计算idx后,加上越界检测,测试基本不再会出现程序崩溃了。

 __device__ __host__
    void query(const Vec2f& src_pcd, Vec2f& dst_pcd, Vec2f& dst_normal, bool& valid) const {
        size_t idx = size_t(src_pcd.x + 0.5f) + size_t(src_pcd.y + 0.5f) * width;
		if (idx >= width * height)
		{
			valid = false;
		}
		else
		{
			if(pcd_ptr[idx].x >= 0){
				dst_pcd = pcd_ptr[idx];
				idx = size_t(dst_pcd.x) + size_t(dst_pcd.y) * width;
				dst_normal = normal_ptr[idx];
				valid = true;
			}
			else
			{
				valid = false;
			}
		}
        return;
    }

運行速度問題

您好,我有設置O2,也有開啟openmp,100萬像素的圖像匹配速度在一百毫秒以上,請問問題出在哪裡呢,謝謝。
construct response map:
elasped time:0.217464s
templ match:
elasped time:0.131106s
total time:
elasped time:0.351647s
matches size: 83

windows下openmp报错

image
如图所示:Windows下VS打开openmp编译报以下错误,关闭OMP则可以用。这是啥情况?

How to speed up

dear meiqua
My Computer is windows10 x64, Visual Studio 2017
I tested the basic example rotation finder.
It took about 5 minutes in train mode and about 3 minutes in test mode.
In the description, it is said to be less than 100ms.
I am curious about the difference.

Below is test logs Information..
////////////////////////////////////////
test img size: 2355200
construct response map
elasped time:13.9197s
templ matche
lasped time:58.167s
elasped time:72.092s
matches.size(): 9
match.template_id: 340
match.similarity: 99.4141
////////////////////////////////////////

Thank you all the time.

关于RegistrationResult

关于新的带icp的分支里,请问RegistrationResult的结果为什么是3x3不是4x4的呢?(平移的误差部分应该如何计算?)
现在的亚像素只是指angle可以用icp修正到亚像素的理解对吗?
谢谢。

模版生成问题

楼主,我想问一下对于论文里面的模版生成是2000张,他这个生成是怎样的形式啊?以竖直向上为轴旋转360度,一个斜坡的90度,再加上平面上的两个旋转,还有尺度的变换,感觉生成的数量远比2000张多啊~;楼主的例子里面好像都是对于一张图片而言的,如果我想获取一个三维模型的物体的位姿,应该怎么去弄啊?有点浑~想来请教一下

训练报错问题

我在尝试自己训练时,在模板保存时发生错误。查看opencv FileStorage类也找不到具体原因。`void Detector::writeClass(const std::string &class_id, FileStorage &fs) const
{
TemplatesMap::const_iterator it = class_templates.find(class_id);
CV_Assert(it != class_templates.end());
const std::vector &tps = it->second;

fs << "class_id" << it->first;
fs << "pyramid_levels" << pyramid_levels;
fs << "template_pyramids"
   << "[";
for (size_t i = 0; i < tps.size(); ++i)
{
    const TemplatePyramid &tp = tps[i];
    fs << "{";
    fs << "template_id" << int(i); //TODO is this cast correct? won't be good if rolls over...
    fs << "templates"`

运行到fs << "templates"这里就出现错误:0x00007FF80450A839 处(位于 test2.exe 中)有未经处理的异常: Microsoft C++ 异常: cv::Exception,位于内存位置 0x0000005FA06FE320 处。

请问是什么原因导致的呢?

My trainning issue

我训练了一下这个图片
cue1
测试结果是这样的:
shot1
请问有办法改进吗?

How to set feature numbers?

Sometimes feature numbers are too small for big objects, so templates are not so discriminative. But if set feature numbers too big, training will fail because there are no enough edges.
(asked by others elsewhere, record it here)

模板匹配问题请教

你好,麻烦请教个问题,《machine vision algorithms and applications》中的这一段,方向向量的变换是怎么推理的?

image

arm板(ti5728)匹配速度问题

系统:Linux version 4.14.79-rt47-ga72bf1418c ,平台:arm板(ti5728),图片尺寸:1280*1024灰度图,生成模板数为360个,单纯匹配花费时间3~4秒,若将模板数减少为120个,匹配时间并未明显减少,大概减少了零点几秒。生成模板是在window7做的,360个模板需要花费几十秒。请问是哪里设置的不对吗?arm板 MIPP test结果:
1568095038(1)
cmake设置情况:
360桌面截图20190910141118

如何结合尺度和旋转不变

我看到程式码有scale_test()和angle_test(),
请问这两个要如何结合使用呢?
我想要Match同时有尺度和旋转不变,谢谢!

VS2012編譯問題

HI meiqua

由於專案關係,用到VS2012

照著VS2017的方法,將line2Dup.h .cpp以及icp內的檔案加入工程,配置都相同

出現這些錯誤,有解嗎?

image

使用VS2017重配一次,編譯正常Orz

开启加速措施后,运行速度较慢

vs2019,开启/O2和/AVX2,测试结果如下(未开启openmp):

MIPP tests

Instr. type: AVX
Instr. full type: AVX2
Instr. version: 2
Instr. size: 256 bits
Instr. lanes: 2
64-bit support: yes
Byte/word support: yes

test img size: 2355200

construct response map
elasped time:1.76129s

templ match
elasped time:0.0258888s

elasped time:1.79249s

matches.size(): 9

match.template_id: 340
match.similarity: 99.4141

运行正常,但速度较慢.
但是开启openmp后,运行报错:

错误 C3001 “declare”: 应为 OpenMP 指令名称 Project1 D:\Project1\Project1\shape_based_matching\line2Dup.cpp 1118
错误 C3036 “omp_insert”: OpenMP“reduction”子句中的运算符标记无效 Project1 D:\Project1\Project1\shape_based_matching\line2Dup.cpp 1121

请问一下是否抗缩放呢?

目的:想识别很小图的轮廓;
但是直接拿很小图的训练,好像很难拿到足够的特征;
直接使用大图训练可以拿到足够多的特征,但是不能直接使用训练的结果去识别小图,我该怎么做呢?

How to set the parameters?

In case0,only find out 4 circles in case0/2.jpg by built-in case0/circle_templ.yaml without any modification,but the scores is large than in case0/result/2.png?

关于参数T的选择

实例中使用的是4,8。有一个地方不明白,就是最底层金子塔(最小图像),其图像尺寸比上一层小,为什么T的值比上一层还要大?

無尺度(Scale)變化時訓練樣板加速的計算問題

meiqua您好:

目前為了加速訓練過程,要把訓練時,每個角度都旋轉一張影像改為 : 旋轉0度的Feature

問題如下
計算完0度角樣板每一層金字塔的 Feature (x, y, quantized)後,
下一個角度旋轉Feature的 (x, y) -->這一步應該沒問題,蒐集所有feature的x, y後,用cv::transform

但feature的quantized參數如何以旋轉去做?
可以把 (0-8)先正規化到(0-360),接著直接加上角度嗎? (不好意思,角度的quantized那段原理不是很懂Orz)

謝謝

好像模板必须要是3通道的?

试了下1通道的gray好像不行,id一直是-1了。
比如把test的132行改为
Mat img = imread(prefix+"case1/train.png", 0);
读取为灰图的话,就id一直是-1了。

看了代码的quantizedOrientations函数部分,发现比起opencv的原版已经加了单通道的对应。

if (src.channels() == 1) {
		Mat sobel_dx, sobel_dy, magnitude, sobel_ag;
		Sobel(smoothed, sobel_dx, CV_32F, 1, 0, 3, 1.0, 0.0, BORDER_REPLICATE);
		Sobel(smoothed, sobel_dy, CV_32F, 0, 1, 3, 1.0, 0.0, BORDER_REPLICATE);
		magnitude = sobel_dx.mul(sobel_dx) + sobel_dy.mul(sobel_dy);
		phase(sobel_dx, sobel_dy, sobel_ag, true);
		hysteresisGradient(magnitude, angle, sobel_ag, threshold * threshold);
}

但处理的结果看来不对。

尺度和旋转不变如何解决?

非常好的技术文章和实现!感谢!请教要做到尺度和旋转不变是否只能通过每一个尺度都建立不同旋转角度的模板?有没有更好的方法呢?再次感谢!

icp优化问题

我尝试用较大的anglestep,例如:3度,预期是:在匹配时,先找到一个大致的范围,然后期望icp优化能够获得较优的解,但icp并没办法优化这3度的误差,多次匹配相同图像的结果会有3度左右的一个偏差,是否icp陷入了一个 local minimum导致优化不成功。

图像padding的大小的决定因素

meiqua:
再咨询一下,看了你的程序,无论在训练过程还是检测过程中,输入图片的四周padding的非常的大,在一定程度上影响了时间开销。按照我的理解,训练过程图片padding的大小只是由训练图片大小和旋转角度决定,保证训练图片经过旋转后,能够在padding范围之内。在检测过程中,只需要保证输入图片padding不小于训练的padding大小,另外为了计算上的考量,padding后的图片大小为16字节对齐即可,以此为依据设置训练和检测过程的padding大小。
不知这样来设计padding大小的想法对不对?

相似性的计算原理?

"Gradient Response Maps for Real-Time Detection of Texture-Less Objects"这篇文章里面的公式2怎么理解?
里面r表示特征点的位置,c表示特征点r的平移量?这里的c是怎么确定的?大括号里面是取c+r的T邻域内的相似性最大,这里的T邻域的中心点为什么是c+r?

请问这个在ubuntu16.04系统能跑吗?

Usage

cmake [options]
cmake [options]

Specify a source directory to (re-)generate a build system for it in the
current working directory. Specify an existing build directory to
re-generate its build system.
出现这样错误?请问如何解决?

VS17+OpenCV3.4.2编译错误

你好,我在VS17+OpenCV3.4.2编译报错:"line2Dup::Detector"中没有找到重载的成员函数。以及:没有与指定类型匹配的重载函数“line2Dup::Detector”实例。
谢谢!

代码测试问题

首先,非常感谢你的分享!有一个问题想要请教您,问题如下:
我在测试没有 MIPP版本的代码时,直接运行test.cpp中的angle_test()函数,最终匹配出来的结果和实际的结果相差挺大的,代码没有做任何的改动,以下为测试结果:
default
以下为匹配模板:
default

问题复现:直接运行angle_test()函数即可,模板图片和roi都保持不变。

Cmake + OpenCV3.0無法通過

您好,我使用Cmake-gui嘗試建置專案。
我的OpenCV安裝路徑 : C:/OpenCV3.0

CMakeLists.txt中 :

...

opencv

set(CMAKE_PREFIX_PATH ${CMAKE_PREFIX_PATH} C:/OpenCV3.0/opencv/build)
find_package(OpenCV 3 REQUIRED)
include_directories(${OpenCV_INCLUDE_DIRS})
...

Configure第一次結果 :

OpenCV ARCH: x86
OpenCV RUNTIME:
OpenCV STATIC: ON
CMake Warning at C:/OpenCV3.0/opencv/build/OpenCVConfig.cmake:166 (message):
Found OpenCV Windows Pack but it has no binaries compatible with your
configuration.

You should manually point CMake variable OpenCV_DIR to your build of OpenCV
library.
Call Stack (most recent call first):
CMakeLists.txt:32 (find_package)

CMake Error at CMakeLists.txt:32 (find_package):
Found package configuration file:

C:/OpenCV3.0/opencv/build/OpenCVConfig.cmake

but it set OpenCV_FOUND to FALSE so package "OpenCV" is considered to be
NOT FOUND.

能否給點建議?感謝

匹配精度

感谢楼主的分享,我在实际测试中将两张一模一样的图片进行匹配,最终的结果有误差,请问这是正常现象吗?谢谢

关于detector 的初始参数的疑惑

对detector 初始化时两个参数的作用有些疑惑
第一个参数理解的是特征点数量, 默认128个特征点,想问以128个特征点初始化后,如果后续加载的yaml模板特征点数量高于或者低于128个特征点会有什么影响,能不能同时加载几个不同特征点数目的yaml模板。

第二个参数是滑动步长,但是有些疑惑这个{x, y}的步长对应的是论文中哪一步分的内容,是指fig4中提到的梯度方向扩散(某像素的梯度方向向+-x, +-y内扩散)?,还是用来计算fig5中线性内存参数T的(相当于滑窗搜索的滑动步长)?或是其他?
现在做实验几个不同物体的模板在相同阈值的情况下,有的第二个参数设得小才能检测出来,有的第二个参数设得大才能检测出来。比较疑惑的就是如果是梯度扩散那参数设得越大所有的物体都应该越容易检测,如果是滑窗步长那参数设得越小所有的物体都应该越容易检测

对应点

请问怎找test图像 与 train图像的对应点呢?然后进行配准呢? 不知道问题是否合适?

關於明暗反轉

現在的算法對明暗反轉不敏感,如果我不想要明暗反轉的結果,請問需要修改哪裡呢?

How to speed up the creation of responsemap?

Hi Meiqua,
测试中发现整个时间,大部分用于了创建responsemap, 请问这个过程可以有办法加速么?例如我有张130万像素的图片,用了150ms创建的,有办法加快么?谢谢!

匹配准确

匹配的时候,匹配结果不对。

问题

左上角的图片是初始模板的特征点的位置;最右边为模板训练,角度从负45°到45°,笔尖的位置用圆画出来了;中间图片为匹配的结果,结果模板为0度的模板,但是整体的特征点左移动了,针尖的位置也相差很大。也尝试移动其他,换成其他角度,匹配的结果更差,不知道什么原因。希望可以指点一下。

提高金字塔層數,計算響應圖時Assert

Hi meiqua,又是我

這次的問題是這樣的 :

Source size : 2592 x 1944 Template size : 1170 x 696
經過padding後,傳入detector.match ()時,cols = 2768、rows = 2608
這樣在金字塔第三層 (cols = 696、rows = 652)就會有(cols*rows) % mipp:N<uint8_t> () != 0的情況產生
有什麼辦法改嗎?

line2Dup.h的transform的中心是否该用float型?

 static cv::Mat transform(cv::Mat src, float angle, float scale){
        cv::Mat dst;

        cv::Point center(src.cols/2, src.rows/2);
        cv::Mat rot_mat = cv::getRotationMatrix2D(center, angle, scale);
        cv::warpAffine(src, dst, rot_mat, src.size());

        return dst;
    }

这里边的center是否该用
cv::Point2f center((float)src.cols / 2, (float)src.rows / 2);

只有angle_test可以运行

测试了下代码,只有angle_test可以正常运行,运行scale_test和noise_test的时候都是直接出train,代码没有修改,问下什么原因?

运行速度问题

你好, 我用的你的master版本进行了测试。VS2015 + opencv330, VS开了02和AVX2, 现在map和match的运行时间是0.4s和0.1s左右。后来发现test里面的的padding值设了500,后改回100.。现在匹配时间是20ms左右。但是map时间还是在350ms左右。‪
image
还有一个奇怪的问题, 第一次便宜生成response map的时间要比较长,之后生成的时间就维持在350ms左右。当没有图片的时候,response map运行时间也要200ms左右。
与之前你提到的几十ms 的map时间,差的有点大, 还有哪里需要注意的地方吗?

代码bug

windows平台,opencv249
angle_test();
getLabel(int quantized) //quantized == 0
CV_Error(CV_StsBadArg, "Invalid value of quantized parameter");

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.