Comments (9)
Hi,
could you specify a bit better the problem? I didn't really understand :D
from libfreenect2pclgrabber.
Hello Again! :)
The problem is as follows:
- I want to use a K4W2 to create a textured mesh.
- I am able to generate the mesh via the pointcloud proved by libfreenect2pclgrabber. I also am able to write a JPG file which contains the RGB view of the camera.
- I would like to use this class:
http://docs.pointclouds.org/trunk/classpcl_1_1_texture_mapping.html to texture the generated mesh. However, I need to provide it camera information in order for it to work.
http://docs.pointclouds.org/trunk/structpcl_1_1texture__mapping_1_1_camera.html
So the question is:
- What is the appropriate camera pose value? "Eigen::Affine3f pose" I tried of depth2world_ and world2rgb_ and their inverse values but to no avail.
- From research it appears that the camera value 'focal_length' is about 1090f, is that correct?
Thank you for any insight or thoughts you may have. Also, hello from Seattle! :)
from libfreenect2pclgrabber.
Hi there!
I have been learning a bit more about the problem. I think it can be expressed most simply as follows:
How do I calculate the 4x4 projection matrix that represents the location and rotation of the kinect's RGB camera from the perspective of the depth camera?
Thanks for any help you may have!
from libfreenect2pclgrabber.
Hi,
to get from one camera to the other you need the parameters of a stereo calibrate, which are a rotation and a translation. These parameters are already present in my code and if you calibrated your camera with it then you can just read the matrix.
from libfreenect2pclgrabber.
That was my first thought as well. However, after a bit of investigation and trial-and-error it appears that the matrix acquired from stereo calibrate does not actually give the XYZ & rotation from RGB to depth but rather it is a projection matrix that can be used to map projected points between the two camera spaces.
I examined the code and saw:
Eigen::Matrix3d rgb2depth = (((orgb_matrix * rototranslation * depth2world)).block<3,3>(0,0)).inverse();
If I wanted to get an 4x4 transformation matrix relative to the depth camera could I do something like:
Eigen::Matrix4d depth2rgb = ( depth2world * rototranslation * orgb_matrix).inverse();
?
from libfreenect2pclgrabber.
yes if you want to go from rgb to depth. I created also a function to register color on depth, but it was not fully working so I did not include it:
void
createFullCloudFromRgb(const cv::Mat & depth, const cv::Mat & color, typename pcl::PointCloud<PointT>::Ptr & cloud) const
{
Eigen::Map<Eigen::Matrix<double, 3, 3, Eigen::RowMajor> > erotation((double*)(rotation_.data));
Eigen::Matrix3d rotation = erotation;
Eigen::Map<Eigen::Vector3d> etranslation ((double*)(translation_.data));
Eigen::Vector3d translation = etranslation;
Eigen::Map<Eigen::Matrix<double, 3, 3, Eigen::RowMajor> > edepth_matrix((double*)(depth_camera_matrix_.data));
Eigen::Matrix3d depth_matrix = edepth_matrix;
Eigen::Map<Eigen::Matrix<double, 3, 3, Eigen::RowMajor> > ergb_matrix((double*)(rgb_camera_matrix_.data));
Eigen::Matrix3d rgb_matrix = ergb_matrix;
Eigen::Matrix4d rototranslation = Eigen::Matrix4d::Zero();
Eigen::Matrix4d odepth_matrix = Eigen::Matrix4d::Zero();
Eigen::Matrix4d orgb_matrix = Eigen::Matrix4d::Zero();
rototranslation.block<3,3>(0,0) = rotation;
rototranslation.block<3,1>(0,3) = translation;
rototranslation(3,3) = 1;
odepth_matrix.block<3,3>(0,0) = depth_matrix;
odepth_matrix(3,3) = 1;
orgb_matrix.block<3,3>(0,0) = rgb_matrix;
orgb_matrix(3,3) = 1;
std::cout << rototranslation << std::endl;
Eigen::Matrix4d depth2world = odepth_matrix.inverse();
Eigen::Matrix3d rgb2depth = (((orgb_matrix * rototranslation * depth2world)).block<3,3>(0,0)).inverse();
for(int i =0; i <cloud->points.size(); i++)
{
cloud->points[i].x = std::numeric_limits<float>::quiet_NaN();
cloud->points[i].y = std::numeric_limits<float>::quiet_NaN();
cloud->points[i].z = std::numeric_limits<float>::quiet_NaN();
cloud->points[i].rgba = 0;
}
int invalid = 0;
int outside = 0;
int replaced = 0;
int newone = 0;
// #pragma omp parallel for
for(int y = 0; y < color.rows; ++y)
{
for(size_t x = 0; x < (size_t)color.cols; ++x)
{
const cv::Vec3b ptcolor = color.at<cv::Vec3b>(y, x);
Eigen::Vector3d rgb_i(x,y,1); // 2D homo
Eigen::Vector3d depth_sub_i = rgb2depth * rgb_i; // to 2D homo
PointT *itP = &cloud->points[(int)depth_sub_i.x() + (int)(depth_sub_i.y()) * color.cols];
double newdepth = std::numeric_limits<float>::quiet_NaN();
if(depth_sub_i.x() >= 0 && depth_sub_i.x() < depth.cols && depth_sub_i.y() >= 0 && depth_sub_i.y() < depth.rows )
{
newdepth = (depth.at<uint16_t>((int)depth_sub_i.y(),(int)depth_sub_i.x()))/1000.0;
if(isnan(newdepth))
{
invalid++;
}
}
if(!isnan(newdepth) && (isnan(itP->z) || newdepth < itP->z) )
{
if(isnan(itP->z))
newone++;
else
replaced++;
Eigen::Vector4d psd(depth_sub_i.x(),depth_sub_i.y(), 1.0, 1.0/newdepth);
Eigen::Vector4d psddiv = psd * newdepth; // (x*newdepth,y*newdepth,newdepth,1)
Eigen::Vector4d pworld = depth2world * psddiv;
itP->x = pworld.x() ;
itP->y = pworld.y() ;
itP->z = newdepth;
itP->b = ptcolor.val[0];
itP->g = ptcolor.val[1];
itP->r = ptcolor.val[2];
}
}
}
std::cout << "total: " << color.rows*color.cols << " nan: "<<invalid << " outside:"<< outside << " AND replaced: " << replaced << " newone:" << newone << std::endl;
}
from libfreenect2pclgrabber.
Why is:
Eigen::Matrix3d rgb2depth = (((orgb_matrix * rototranslation * depth2world)).block<3,3>(0,0)).inverse();
abbridged into a 3x3 matrix?
Also, if I wanted instead depth2rgb as a 4x4 transformation matrix would I do:
Eigen::Matrix4d depth2rgb = (((odepth_matrix * rototranslation.inverse() * rgb2world))).inverse();
Background:
I am trying to create a system where multiple kinects work together to build a large scene. I have kinect 2 kinect point mapping down. From the joined point clouds I want to mesh them and then project a texture onto the resulting mesh. However, I need the exact XYZ and rotation of the RGB camera relative to its depth camera.
from libfreenect2pclgrabber.
I get a 3x3 beacuse to go from rgb to depth you need just the upper 3x3
matrix.
Second question yes. Right now I saw that they changed libfreenect2 for
automatic calibration and I will try it out in the next days. I have also
changed a bit the grabber and I will
release it soon with the new method.
2015-05-04 12:26 GMT+02:00 lolz0r [email protected]:
Why is:
Eigen::Matrix3d rgb2depth = (((orgb_matrix * rototranslation *
depth2world)).block<3,3>(0,0)).inverse(); abbridged into a 3x3 matrix?Also, if I wanted instead depth2rgb as a 4x4 transformation matrix would I
do:
Eigen::Matrix4d depth2rgb = (((odepth_matrix * rototranslation.inverse() *
rgb2world))).inverse();Background:
I am trying to create a system where multiple kinects work together to
build a large scene. I have kinect 2 kinect point mapping down. From the
joined point clouds I want to mesh them and then project a texture onto the
resulting mesh. However, I need the exact XYZ and rotation of the RGB
camera relative to its depth camera.—
Reply to this email directly or view it on GitHub
#1 (comment)
.
G. Dabisias, PhD Student
PERCRO, (Laboratory of Perceptual Robotics)
Scuola Superiore Sant'Anna
via Luigi Alamanni 13D, San Giuliano Terme 56010 (PI), Italy
mob.: +39 3480839095
from libfreenect2pclgrabber.
Great! Thanks for all your help! I can't wait to check out the newest commit ;)
from libfreenect2pclgrabber.
Related Issues (20)
- Writing cloud to PLY HOT 6
- 'libfreenect2/config.h' file not found HOT 3
- Is it possible to apply to multi Kinects ? HOT 50
- Packet stream parse issues HOT 9
- CUDA error: K2G unable to locate libfreenect2 HOT 3
- MultiKinect2Grabber packets lost without displaying point cloud? HOT 32
- libfreenect2pclgrabber not working HOT 29
- cmake error HOT 8
- pure virtual method called terminate called without an active exception HOT 5
- k2g library error HOT 1
- Undefined reference to libfreenect2 HOT 1
- 2
- Increasing maximum depth of pointcloud HOT 1
- Is it possible to build libfreenect2pclgrabber on a system without OpenGL? HOT 4
- error: expected type-specifier on make
- make error HOT 2
- pcl_viewer visualizes everything upsidedown
- ERROR BUILDING HOT 2
- Usage of iai registration functions HOT 2
- Error while making the file ( error: no viable overloaded '=' cloud = k2g.getCloud(); ) HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from libfreenect2pclgrabber.