guanyingc / tom-net_rendering Goto Github PK

I am surely doing something wrong here, any advice or help would be highly appreciated.

GOAL:
I want to transform the object mesh (.inc) in the same pose as the object in the captured image and then render it on the object in the captured image.

that is, I would like this rendered object by pov-ray to alight exactly with the object in the captured image, so that i can also get the refractive flow and attenuation mask in the same pose, using the provided rendering script

DESCRIPTION:
I get my object mesh from the keypose/ BOP dataset, with its corresponding ground-truth object pose in the form of a Transformation matrix(see image below). This is in the Right Hand Coordinate system.

Since POV-ray operates in the Left-hand coordinate system I tried to convert the transformation matrix ( the ground-truth pose provided by the dataset) to the right hand by following suggestions:

https://stackoverflow.com/questions/1263072/changing-a-matrix-from-right-handed-to-left-handed-coordinate-system

That is from the original transformation matrix:

{ rx, ry, rz, 0 }
{ ux, uy, uz, 0 }
{ lx, ly, lz, 0 }
{ px, py, pz, 1 }

To change it from left to right or right to left, flip it like this:

{ rx, rz, ry, 0 }
{ lx, lz, ly, 0 }
{ ux, uz, uy, 0 }
{ px, pz, py, 1 }

But this leads to the following result (see image below).

Since this throws the object out of the image, i change the "CamLoc" from "<0,0,0>" to "<0,0,-2>" in the script below (see image below).

THE SCRIPT I AM USING IS TOM-Net Rendering script:

#version 3.7;
global_settings {max_trace_level 40 }

#include "colors.inc"
#include "shapes.inc"

#ifdef (cl_x) #declare CamLoc=<0,0,0>; #end // Camera Location
#ifdef (lk_x) #declare LookAt=<0,0,1>; #end // Look at
#ifdef (cs_x) #declare CamSky=<0.0, 1, 0>; #end // Camera Sky
#declare CamDir = vnormalize(LookAt - CamLoc);
#declare CamR = vnormalize(vcross(CamSky,CamDir));
#declare CamUp = vnormalize(vcross(CamDir,CamR));

#ifdef (cam_a) #declare CamA = cam_a; #else #declare CamA = 1; #end // Aspect ratio
#ifdef (cam_z) #declare CamZ = cam_z; #else #declare CamZ = 1.5; #end // Zoom
#ifdef (bg_sc) #declare BgSc = bg_sc; #else #declare BgSc = 3; #end // Background scale

#ifndef (IOR) #declare IOR = 1.5; #end // Index of refraction
#ifndef (Trans) #declare Trans = 1; #end // Transmit
#ifndef (Dim) #declare Dim = 1; #end // Ambient intensity

camera {
location CamLoc
direction CamDir * CamZ
angle 86.9
right CamR * CamA
up CamUp
}

#macro OrientZ(p1,p2,cs)
#local nz = vnormalize(p2-p1);
#local nx = vnormalize(vcross(cs,nz));
#local ny = vcross(nz,nx);
matrix <nx.x, nx.y, nx.z,
ny.x, ny.y, ny.z,
nz.x, nz.y, nz.z,
p1.x, p1.y, p1.z>
#end

#ifdef (Calib) // Render objects with graycode patterns
#declare pigmentStructure = pigment {
#if (clock <= 2)
#if (clock = 1) #declare mask = 1; #end color White
#else
image_map { png concat(${ImageName}, str(clock-2,1,0), ".png") map_type 0 interpolate 2 }
#end
}
#else
#declare pigmentImage = pigment {
#ifdef (mask)
color White
#else
#ifdef (rho) color White #else image_map { jpeg ${ImageName} map_type 0 interpolate 2 } #end
#end
}
#end

box { <0,0,0> <1,1,0.01>
#ifdef (Calib)
pigment { pigmentStructure }
finish { ambient 1 }
#else
pigment { pigmentImage }
finish { ambient Dim }
#end
translate <-0.5,-0.5,0>
scale <CamA, 1, 1>
translate <0, 0, CamZ>
scale BgSc
OrientZ(CamLoc,LookAt,CamSky)
}

#ifndef (Empty)
object {
shape
#ifdef (mask)
pigment {color Black}
#else
texture {
pigment{
color filter 1
#ifdef (Calib) transmit 1 #else transmit Trans #end
}
}
interior {
ior IOR
#ifndef (Calib)
#ifdef (FadeD) fade_distance FadeD fade_power FadeP #end
#end
}
#end

matrix <0.9985836 , -0.03011384, -0.04386286,
    -0.05320521, -0.56519264, -0.82324145,
    0.        ,  0.82440915, -0.56599431,
     -0.03461049,  0.7220574 ,  0.08761759>

}
#end

PROBLEM:

As you can see this particular transform throws the object out of the image.

Ideally, i would like this rendered object by pov-ray to alight exactly with the object in the image, so that i can also get the refractive flow, mask and attenuation in the same pose.

I have also tried a few other transforms but they also do not seem to work.

for reference, i also add the result of the pov-ray rendered image using the original transform given by the dataset (attached image below )

Any help would be greatly appreciated!

First of all Amazing work!

I am also working with transparent objects and came across this amazing work.

I was wondering, how can I generate more pov-ray objects, like you all did like the "object.inc" file?

Some insight would be appreciable.

Best Regards,
Hrishikesh Gupta