t-taniai / localexpstereo Goto Github PK

dear author:
there is still one question after i've read the paper <WHAT ENERGY FUNCTIONS CAN BE MINIMIZED VIA GRAPH CUTS?>.The paper tell us that how to decompose the edges.the paper has a table to show ABCD variables.the table 3 in paper showing :A=Eij(0,0)=V(fp,fq); B=Eij(0,1)=V(fp,α);C=Eij(1,0)=V(α,fq);D=Eij(1,1)=V(α,α).
but the code give me a different definition about the variable D.
the function computeSmoothnessTermsExpansion in class StereoEnergy have calculated the cost00,cost01 and cost10:

void computeSmoothnessTermsExpansion(const cv::Mat& labeling0_m, Plane label1, cv::Rect region, std::vectorcv::Mat& cost00, std::vectorcv::Mat& cost01, std::vectorcv::Mat& cost10, bool onlyForward = false, int mode = 0) const
{
cv::Rect rect_ee = cv::Rect(M + region.x, M + region.y, region.width, region.height);
cv::Mat label0_ee = labeling0_m(rect_ee);
cv::Mat coord_ee = coordinates_m(rect_ee);
cv::Scalar sc = label1.toScalar();
cv::Mat disp0_of_ee_at_ee = cvutils::channelDot(label0_ee, coord_ee);
cv::Mat disp1_at_ee = cvutils::channelSum(coord_ee.mul(sc));
//cv::Mat disp1_at_ee = label1.toDispMap(region); // This changes results due to small numerical differences.

	if (disp0_of_ee_at_ee.depth() != CV_32F){
		disp0_of_ee_at_ee.convertTo(disp0_of_ee_at_ee, CV_32F);
		disp1_at_ee.convertTo(disp1_at_ee, CV_32F);
	}
	cost00 = std::vector<cv::Mat>(neighbors.size());
	cost01 = std::vector<cv::Mat>(neighbors.size());
	cost10 = std::vector<cv::Mat>(neighbors.size());
	for (int i = 0; i < neighbors.size(); i++){
		if (onlyForward && (neighbors[i].y * width + neighbors[i].x <= 0))
			continue;
		cv::Rect rect_le = rect_ee + neighbors[i];
		cv::Mat label0_le = labeling0_m(rect_le);
		cv::Mat coord_le = coordinates_m(rect_le);
		std::cout<<label0_le.at<cv::Vec4f>(0,0)<<std::endl;
		cv::Mat disp0_of_le_at_ee = cvutils::channelDot(label0_le, coord_ee);
		cv::Mat disp0_of_ee_at_le = cvutils::channelDot(label0_ee, coord_le);
		cv::Mat disp0_of_le_at_le = cvutils::channelDot(label0_le, coord_le);
		cv::Mat disp1_at_le = cvutils::channelSum(coord_le.mul(sc));
		if (disp0_of_le_at_ee.depth() != CV_32F){
			disp0_of_le_at_ee.convertTo(disp0_of_le_at_ee, CV_32F);
			disp0_of_ee_at_le.convertTo(disp0_of_ee_at_le, CV_32F);
			disp0_of_le_at_le.convertTo(disp0_of_le_at_le, CV_32F);
			disp1_at_le.convertTo(disp1_at_le, CV_32F);
		}
		cv::Mat smoothnessCoeffL_nb = smoothnessCoeff[mode][i](rect_ee);
	**//cost00:|dp(fp)−dp(fq)| + |dq(fq)−dq(fp)|<=>V(fp,fq)=A**
		cost00[i] = cv::abs(disp0_of_ee_at_ee - disp0_of_le_at_ee) + cv::abs(disp0_of_ee_at_le - disp0_of_le_at_le);
		cv::threshold(cost00[i], cost00[i], params.th_smooth, 0, cv::THRESH_TRUNC);
		cost00[i] = cost00[i].mul(smoothnessCoeffL_nb, params.lambda);
        **//cost01:|dp(fp)−dp(fα)| + |dq(fp)−dq(fα)|<=>V(fp,α)=B**
		cost01[i] = cv::abs(disp0_of_ee_at_ee - disp1_at_ee) + cv::abs(disp0_of_ee_at_le - disp1_at_le);
		cv::threshold(cost01[i], cost01[i], params.th_smooth, 0, cv::THRESH_TRUNC);
		cost01[i] = cost01[i].mul(smoothnessCoeffL_nb, params.lambda);
      **//cost10:|dp(fα)−dp(fq)| + |dq(fq)−dq(fα)|<=>V(fq,α)=C**
		cost10[i] = cv::abs(disp1_at_ee - disp0_of_le_at_ee) + cv::abs(disp1_at_le - disp0_of_le_at_le);
		cv::threshold(cost10[i], cost10[i], params.th_smooth, 0, cv::THRESH_TRUNC);
		cost10[i] = cost10[i].mul(smoothnessCoeffL_nb, params.lambda);
	}

The definition of the cost00 means A in table,cost01 means B,cost10 means C in table.but the code use cost00 as D in table as following in function expandtionMoveBK :
// ee <-> gg
// ***
// ***
// **@
for (int y = 0; y < region.height - 1; y++){
for (int x = 0; x < region.width - 1; x++){
int i = y*region.width + x;
int j = (y + 1)*region.width + x + 1;
float B = cost10[StereoEnergy::NB_GG].at(y, x);
float C = cost01[StereoEnergy::NB_GG].at(y, x);
float D = cost00[StereoEnergy::NB_GG].at(y, x);
graph.add_edge(i, j, std::max(0.f, B + C - D), 0);
graph.add_tweights(i, C, 0);
graph.add_tweights(j, D - C, 0);
}
}
}
The variable D should connect with α :D=V(α,α),while the D in code is uncorrelated with α. the cost01 and cost10 are comply with paper.could you please explain the definition of D in code?

this is to say,for exam，the output image is a chair，and its four legs is on the top of the image

Hi, When I compiled and run in debug mode, it give the following error. what might be the reason? -thanks,

----------- parameter settings -----------
mode : MiddV2
outputDir : ../results
targetDir : ../data/MiddV2/cones
threadNum : -1
doDual : 0
pmIterations : 2
iterations : 5
ndisp : 100
filterRadious : 20
smooth_weight : 1.000000
mc_threshold : 0.500000

Running by Middlebury V2 mode.
ndisp = 100
0 0.0 14625435998 14625397810 38188 98.62 98.63
OpenCV Error: Assertion failed (elemSize() == (((((DataType<_Tp>::type) & ((512 - 1) << 3)) >> 3) + 1) << ((((sizeof(size_t)/4+1)*16384|0x3a50) >> ((DataType<_Tp>::type) & ((1 << 3) - 1))*2) & 3))) in cv::Mat::at, file c:\users\shufei.fan\workspace\projects\localexpstereo\packages\opencvcontrib.3.1.0\build\native\include\opencv2\core\mat.inl.hpp, line 962
Op

Dear Professor,
I am a Undergraduate, i cannot understand as follows (in file ’StereoEnergy.h'):

1.Which words are called 'ee' for short.
So that I cannot understand these objects: disp0_of_le_at_ee, disp0_of_ee_at_le, disp0_of_le_at_le ...

2.why the object 'cost' be computed at 4 times as 'cost00', 'cost01', 'cost10' and ’cost11‘ ?

Thanks for your times!
Wei Xue

Plane(float a, float b, float c, float y) : a(a), b(b), c(c), v(v){}

Above is the code in Plane.h line 12.
Maybe the float y should be float v.

Fixed by adding:
template class Graph<float,float,double>;
To instances.inc in maxflow project.
I think it should be added to the "How to Run?" section

Hello, here is an error in

Hi

I was giving this project a go and tried evaluating it against the Bicycle2 sample. I'm using the matching costs provided in the Adirondack link. However, the output I get is the following:

I'm already using the corresponding calib.txt:

cam0=[1948.17 0 532.418; 0 1948.17 488.228; 0 0 1]
cam1=[1948.17 0 614.35; 0 1948.17 488.228; 0 0 1]
doffs=81.931
baseline=173.557
width=1426
height=976
ndisp=125

The options I use are:

"%bin%" -targetDir "%datasetroot%\test2" -outputDir "%resultsroot%\test2" -mode MiddV3 -smooth_weight 0.5

This is the debug output:

Time	Eng	Data	Smooth	all	nonocc
0.000000	-nan(ind)	-nan(ind)	138868.715077	-nan(ind)	99.667810
34.431000	-nan(ind)	-nan(ind)	211043.893761	-nan(ind)	99.978780
68.153000	-nan(ind)	-nan(ind)	268027.314525	-nan(ind)	99.978780
129.158000	-nan(ind)	-nan(ind)	3541.673733	-nan(ind)	99.913212
189.342000	-nan(ind)	-nan(ind)	4004.365720	-nan(ind)	90.420596
250.601000	-nan(ind)	-nan(ind)	4623.280927	-nan(ind)	99.924280
311.898000	-nan(ind)	-nan(ind)	2288.273245	-nan(ind)	100.000000
373.796000	-nan(ind)	-nan(ind)	1892.510750	-nan(ind)	100.000000

Adirondack works fine though. I haven't tried every sample, but I get similar results from Classroom2 and Jadeplant as well. Does anyone know why this might be the case? Am I missing something?

Thanks

Dear author:
The function of expansionMoveBK in FastGCStereo.h used the function of add_tweights and add_edge in Graph Class.In my opioin, the add_edge is the way to set a t_link to the graph and the add_tweights is the way to set a n_link to the graph.So it's make me feel confused that the code mixed two functions in function expansionMoveBK.The paper didn't explain how to build the graph,so i can't read the code. My specific question have written in the annotating codes below.

The part of source code of expansionMoveBK:
double expansionMoveBK(updateMask, Plane label1, region, proposalCosts, mode = 0):
~~
~~
for (int y = 0; y < region.height; y++){
for (int x = 0; x < region.width; x++){
int s = yregion.width + x; //I think it's building two links from node s to the source and the terminal.
graph.add_tweights(s, subCurrent.at(y, x), proposalCosts.at(y, x));
~~
~~
for (int y = 0; y < region.height; y++){
for (int x = 0; x < region.width; x++){
int s = yregion.width + x;
graph.add_tweights(s, subCurrent.at(y, x), proposalCosts.at(y, x));

			bool x0 = x == 0;
			bool x1 = x == region.width - 1;
			bool y0 = y == 0;
			bool y1 = y == region.height - 1;

			if (x0 || x1 || y0 || y1)
			{
				cv::Point ps = cv::Point(x, y) + region.tl();
				for (int k = 0; k < stereoEnergy->neighbors.size(); k++)
				{
					cv::Point pt = ps + stereoEnergy->neighbors[k];
					if (region.contains(pt))
						continue;
					if (imageDomain.contains(pt) == false)
						continue;

					// pt is always label0;**why the pt is always label0?**
					float _cost00 = stereoEnergy->computeSmoothnessTerm(currentLabeling.at<Plane>(ps), currentLabeling.at<Plane>(pt), ps, k, mode);
					float _cost10 = stereoEnergy->computeSmoothnessTerm(label1, currentLabeling.at<Plane>(pt), ps, k, mode);

					graph.add_tweights(s, _cost00, _cost10);// **i don't understand why we have to rebuild two t-links**  
				}
			}
		}
	}

	// ee <-> ge
	// ***
	// **@
	// ***
	for (int y = 0; y < region.height; y++){
		for (int x = 0; x < region.width - 1; x++){
			int i = y*region.width + x;
			int j = y*region.width + x + 1;
			float B = cost10[StereoEnergy::NB_GE].at<float>(y, x);
			float C = cost01[StereoEnergy::NB_GE].at<float>(y, x);
			float D = cost00[StereoEnergy::NB_GE].at<float>(y, x);
			graph.add_edge(i, j, std::max(0.f, B + C - D), 0); **// i think it's building the n_links ,but don't know why we set the value at  std::max(0.f, B + C - D)**
			graph.add_tweights(i, C, 0);**// the old question why we rebulid the t_links and don' know how to set the value.** 
			graph.add_tweights(j, D - C, 0);
		}
	}

~~
~~

It crashed at ransac proposer at the function, RANSACPlane(cv::Mat pts, cv::Mat disp, float threshold).

I put the link for the stereo input images used for the testing.

https://photos.google.com/share/AF1QipNMo4mbDj9MjwB2y27NkE6TfgkEiQHo9IZDoOkKm_-DETH7WfPtPY-SGJL_2kKqnA?key=Rkc0Y1MxRVN0Z3Y0el9GQmR6SmV3Ym05STBOWDZ3

Dear Professor,

I have also a question:
How can the algorithm know that the plane which is represented by the label is correct?
Because labels after the first correct one can be estimated by using Energy Function, but the first label has no Reference.

I have read your code, but I can not understand the part of it very well. OTZ

Thanks for your reply!
Wei Xue

I downloaded the source code, set up maxflow source, opencv, but the system showed that <time.h> could not be recognized, and the SDK version was incorrect, so I upgraded the SDK tool, and finally LNK2001 appeared, using the version of Windows 10, vs2019, What is the problem and how to solve it, thank you very much

Hi，
I want to use my own images（left image and right） to generate disparity image by your code（MidV2）， what parameters should I modify in the info.txt? Thank you very much!@t-taniai

what do we have to build in release mode I mean which files
as if I use main.cpp it says error as it is not getting LocalExpansionStereo.exe file

Hi,
I tried to reobtain middlebury evaluation results but I got slightly different result than middlebury results. Did you fine tune the parameters or other things that not mentioned.
Edit : I used your costs provided in drive.
My Result

Expected Result

After fresh clone and build.