Comments (6)
I think I figured out the problem. The camera poses must be centered and placed with the "center of attention" within the [-1,1] cube before writing to the json.
from blendernerf.
Hi @advaiths12, thanks for your comment!
The scene in Blender that you are trying to capture should ideally always be bounded by the aabb
volume from Instant NGP. Since this cube is centred around the origin, centring your scene might be necessary in order to align the cameras from Blender with your scene in the NGP framework.
On the other hand, increasing aabb
should allow you to capture a larger scene, thus circumventing the need to rescale your scene between two bounds.
Glad you managed to figure this issue out by yourself and feel free to ask more questions if in doubt :)
from blendernerf.
I think I figured out the problem. The camera poses must be centered and placed with the "center of attention" within the [-1,1] cube before writing to the json.
hi I met the same problem, do i need to change the code of BlenderNeRF or need to do something in blender?
from blendernerf.
Hi,
I took some code from the original NeRF repo to center the camera poses within the [-1, 1] cube.
I've attached an example of it in cos_operator.py below:
import os
import shutil
import bpy
from . import helper, blender_nerf_operator
import numpy as np
import mathutils
from mathutils import Matrix, Quaternion, Vector
# global addon script variables
EMPTY_NAME = 'BlenderNeRF Sphere'
CAMERA_NAME = 'BlenderNeRF Camera'
#taken from the original NeRF repo
def closest_point_2_lines(oa, da, ob, db): # returns point closest to both rays of form o+t*d, and a weight factor that goes to 0 if the lines are parallel
da = da / np.linalg.norm(da)
db = db / np.linalg.norm(db)
c = np.cross(da, db)
denom = np.linalg.norm(c)**2
t = ob - oa
ta = np.linalg.det([t, db, c]) / (denom + 1e-10)
tb = np.linalg.det([t, da, c]) / (denom + 1e-10)
if ta > 0:
ta = 0
if tb > 0:
tb = 0
return (oa+ta*da+ob+tb*db) * 0.5, denom
# camera on sphere operator class
class CameraOnSphere(blender_nerf_operator.BlenderNeRF_Operator):
'''Camera on Sphere Operator'''
bl_idname = 'object.camera_on_sphere'
bl_label = 'Camera on Sphere COS'
def execute(self, context):
scene = context.scene
camera = scene.camera
# check if camera is selected : next errors depend on an existing camera
if camera == None:
self.report({'ERROR'}, 'Be sure to have a selected camera!')
return {'FINISHED'}
# if there is an error, print first error message
error_messages = self.asserts(scene, method='COS')
if len(error_messages) > 0:
self.report({'ERROR'}, error_messages[0])
return {'FINISHED'}
output_data = self.get_camera_intrinsics(scene, camera)
# clean directory name (unsupported characters replaced) and output path
output_dir = bpy.path.clean_name(scene.cos_dataset_name)
output_path = os.path.join(scene.save_path, output_dir)
os.makedirs(output_path, exist_ok=True)
if scene.logs: self.save_log_file(scene, output_path, method='COS')
# initial property might have changed since set_init_props update
scene.init_output_path = scene.render.filepath
# other intial properties
scene.init_sphere_exists = scene.show_sphere
scene.init_camera_exists = scene.show_camera
scene.init_frame_end = scene.frame_end
scene.init_active_camera = camera
if scene.test_data:
# testing transforms
output_data['frames'] = self.get_camera_extrinsics(scene, camera, mode='TEST', method='COS')
print("computing center of attention...")
totw = 0.0
totp = np.array([0.0, 0.0, 0.0])
for f in output_data["frames"]:
mf = f["transform_matrix"][0:3,:]
for g in output_data["frames"]:
mg = g["transform_matrix"][0:3,:]
p, w = closest_point_2_lines(mf[:,3], mf[:,2], mg[:,3], mg[:,2])
if w > 0.00001:
totp += p*w
totw += w
if totw > 0.0:
totp /= totw
print(totp) # the cameras are looking at totp
nframes = 0
for f in output_data['frames']:
f["transform_matrix"][0:3,3] -= totp
nframes += 1
avglen = 0.
for f in output_data['frames']:
avglen += np.linalg.norm(f["transform_matrix"][0:3,3])
avglen /= nframes
print("avg camera distance from origin", avglen)
for f in output_data['frames']:
f["transform_matrix"][0:3,3] *= 4.0 / avglen # scale to "nerf sized"
for f in output_data['frames']:
f["transform_matrix"] = f["transform_matrix"].tolist()
self.save_json(output_path, 'transforms_test.json', output_data)
if scene.train_data:
if not scene.show_camera: scene.show_camera = True
# train camera on sphere
sphere_camera = scene.objects[CAMERA_NAME]
sphere_output_data = self.get_camera_intrinsics(scene, sphere_camera)
scene.camera = sphere_camera
# training transforms
sphere_output_data['frames'] = self.get_camera_extrinsics(scene, sphere_camera, mode='TRAIN', method='COS')
print("computing center of attention...")
totw = 0.0
totp = np.array([0.0, 0.0, 0.0])
for f in sphere_output_data['frames']:
mf = f["transform_matrix"][0:3,:]
for g in sphere_output_data['frames']:
mg = g["transform_matrix"][0:3,:]
p, w = closest_point_2_lines(mf[:,3], mf[:,2], mg[:,3], mg[:,2])
if w > 0.00001:
totp += p*w
totw += w
if totw > 0.0:
totp /= totw
print(totp) # the cameras are looking at totp
for f in sphere_output_data['frames']:
f["transform_matrix"][0:3,3] -= totp
avglen = 0.
nframes = 0
for f in sphere_output_data['frames']:
avglen += np.linalg.norm(f["transform_matrix"][0:3,3])
nframes += 1
avglen /= nframes
print("avg camera distance from origin", avglen)
for f in sphere_output_data['frames']:
f["transform_matrix"][0:3,3] *= 4.0 / avglen # scale to "nerf sized"
for f in sphere_output_data['frames']:
f["transform_matrix"] = f["transform_matrix"].tolist()
self.save_json(output_path, 'transforms_train.json', sphere_output_data)
# rendering
if scene.render_frames:
output_train = os.path.join(output_path, 'train')
os.makedirs(output_train, exist_ok=True)
scene.rendering = (False, False, True)
scene.frame_end = scene.frame_start + scene.nb_frames - 1 # update end frame
scene.render.filepath = os.path.join(output_train, '') # training frames path
bpy.ops.render.render('INVOKE_DEFAULT', animation=True, write_still=True) # render scene
scene.frame_set(scene.frame_start)
# if frames are rendered, the below code is executed by the handler function
if not any(scene.rendering):
# reset camera settings
if not scene.init_camera_exists: helper.delete_camera(scene, CAMERA_NAME)
if not scene.init_sphere_exists:
objects = bpy.data.objects
objects.remove(objects[EMPTY_NAME], do_unlink=True)
scene.show_sphere = False
scene.sphere_exists = False
scene.camera = scene.init_active_camera
# compress dataset and remove folder (only keep zip)
shutil.make_archive(output_path, 'zip', output_path) # output filename = output_path
shutil.rmtree(output_path)
return {'FINISHED'}
from blendernerf.
from blendernerf.
Hi @Quyans,
As far as I know, all you need is to have your camera path and scene of interest within the AABB
bounding volume from Instant NGP. For that, you can either increase AABB
such that your whole scene fits within thad bound, or you can center your whole scene (technically just the exported camera locations) and hopefully get most of what you need within that bound.
You can either do this via code by somehow normalising the camera location (see the code snippet provided by @advaiths12), or directly manually in Blender by approximately centering your scene and camera around the origin.
from blendernerf.
Related Issues (20)
- DroNeRF: new paper on determining ideal capture (render) locations: HOT 1
- The output images didn't get whole scene of a model HOT 2
- I export a data set from blender but camera doesnt match in instant NGP HOT 11
- Question regarding GT HOT 6
- Request for Feature Addition: Simultaneous Rendering of Depth Maps with Optical Images and JSON Integration HOT 1
- How to automatic the process without using blender HOT 3
- how to start, please give more detailed documentations? HOT 5
- Is this tool only generate training image directory? HOT 2
- How to calculate transform_matrix? HOT 3
- Call in a for loop HOT 3
- Scene is shifting? HOT 6
- BlenderNeRF Camera not found HOT 5
- Can I use this tool to generate 3D Gaussian splatting datasets? HOT 7
- The color of the object in the generated image is different. HOT 4
- generating test renderings. HOT 5
- TypeError: empty_fn() missing 1 required positional argument: 'context' HOT 2
- glass ceiling on radius parameter HOT 1
- Failed to reconstruct with SPEED+ dataset HOT 1
- Request: Support creation of 3DGS datasets HOT 6
- Just a question really HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from blendernerf.