salykovaa / instant-dexnerf Goto Github PK

Instant NeRF for depth/3d shape estimation of transparent objects

License: Other

CMake 2.11% C++ 12.86% Cuda 73.49% C 1.44% Python 10.11%

dex-net instant-ngp nerf neural-radiance-fields dexnerf depth-estimation shape-estimation transparent-objects transparent

instant-dexnerf's Introduction

Instant DexNerf

Depth/3D Shape estimation of transparent objects using multiview posed RGB images. The project was inspired by Dex-NeRF: Using a Neural Radiance field to Grasp Transparent Objects and Instant Neural Graphics Primitives. Combination of two methods provides both fast training/rendering speed and accurate depth map estimation.

How to install

For installation steps please refer to Instant NGP.

How to run

There is an example of a scene with transparent object. Run from the command line

./build/testbed --scene data/nerf/canister/transforms.json

In the GUI you can adjust sigma parameter and switch between normal and Dex depth rendering. By default, sigma = 15.

How to create depth maps of captured scenes

In /scripts folder there is a main.py script for depth map generation.

First, fill the scene_dirs list with paths to your folders with rgb images. These folders must have the following structure. In /data/nerf/canister you can find examples of groundtruth_handeye.txt and intrinsics.txt

├── scene_folder
│   ├── img_dir (folder with rgb images. default "rgb", but you can change the name in main.py) 
│   ├── groundtruth_handeye.txt (contains c2w extrinsics as quaternions for each image. x,y,z,w format)
|   ├── intrinsics.txt (w h fx 0 cx 0 fy cy 0 0 1 [360 or 180]). 180 for forward-facing scene, 360 for 360° scene.

Set parameters for training and rendering: depth_dir, sigma_thrsh, aabb_scale, train_steps
Run main.py Rendered depth maps are found in scene_folder/depth_dir folder.

Note: if you use world coordinate system different from ours, please adapt transform_matrix in ours2nerf.py. c2w matrices are multiplied by transform_matrix before they written to the transforms.json file. Otherwise, poses are expected to be in OPENCV format.

Depending on your scene geometry, you may need to tune scale, offset parameters !!!

Results

Example 1

Example 2

Depth error

Example 1

Example 2

Citation

Kudos to the authors for their great work

@inproceedings{IchnowskiAvigal2021DexNeRF,
  title={{Dex-NeRF}: Using a Neural Radiance field to Grasp Transparent Objects},
  author={Ichnowski*, Jeffrey and Avigal*, Yahav and Kerr, Justin and Goldberg, Ken},
  booktitle={Conference on Robot Learning (CoRL)},
  year={2020}
}

@article{mueller2022instant,
    author = {Thomas M\"uller and Alex Evans and Christoph Schied and Alexander Keller},
    title = {Instant Neural Graphics Primitives with a Multiresolution Hash Encoding},
    journal = {ACM Trans. Graph.},
    issue_date = {July 2022},
    volume = {41},
    number = {4},
    month = jul,
    year = {2022},
    pages = {102:1--102:15},
    articleno = {102},
    numpages = {15},
    url = {https://doi.org/10.1145/3528223.3530127},
    doi = {10.1145/3528223.3530127},
    publisher = {ACM},
    address = {New York, NY, USA},
}

instant-dexnerf's People

Contributors

Stargazers

Watchers

Forkers

melhashash wyngjf kevinchan04 sungminlee0810 lifeixiao liuyiyang01

instant-dexnerf's Issues

Question about camera pose in groundtruth_handeye.txt

Hi,

I have a question about the convention used in groundtruth_handeye.txt. How to generate it given the pose in COLMAP format?

I tried the following code to generate groundtruth_handeye.txt and then use it with instrinsics.txt to generated the json file using ours2nerf.py but the generated camera poses are incorrect, as shown below (they should be 360)

This code reads images.txt file in COLMAP format

	for line in f:
		line = line.strip()
		if line[0] == "#":
			continue
		i = i + 1
		if i < SKIP_EARLY*2:
			continue
		if  i % 2 == 1:
			elems=line.split(" ") # 1-4 is quat, 5-7 is trans, 9ff is filename (9, if filename contains no spaces)
			image_rel = os.path.relpath(IMAGE_FOLDER)
			name = str(f"./{image_rel}/{'_'.join(elems[9:])}")	
			image_id = int(elems[0])
			qvec = np.array(tuple(map(float, elems[1:5])))
			tvec = np.array(tuple(map(float, elems[5:8])))
			R = qvec2rotmat(qvec)
			t = tvec.reshape([3,1])
			m = np.concatenate([np.concatenate([R, t], 1), bottom], 0)
			c2w = np.linalg.inv(m)
			R = c2w[:3,:3]
			t = c2w[:3,3]
			q = rotm2quat(R)
			f_c2w_pose_file.write(f"{counter} {q[0]} {q[1]} {q[2]} {q[3]} {t[0]} {t[1]} {t[2]}\n")
			counter = counter + 1

How to evaluate the depth map error?

Hi, thank you for your code! May I ask how to evaluate the depth map error in your case?

How to get the right groundtruth_handeye.txt

May I ask how the file groundtruth_handeye.txt is obtained, I use the colmap software to get an error result

Reading wrong rotation

Hi and Thanks for your work!

I am trying to get the repo running with my own data. However, the extrinsics get messed up when loaded (they should point inward, not outward the circle):

I want to mention that the poses were generated with COLMAP in a transforms.json file. They are displayed correctly by instant-ngp and produce good results.

Also, I am getting this error when I try to start the training, which is probably because of it not being able to build the rays from the positions.

WARNING  Nerf training generated 0 samples. Aborting training.

Could not match the transform.json provided in instant-DexNerf/data/nerf/canister

I run the colmap2nerf.py for canister dataset to obtain a new transform.json. However, it seems different from the one provided here(https://github.com/salykovaa/instant-DexNerf/blob/main/data/nerf/canister/transforms.json). I cannot sure if sth wrong? Thanks a lot!

run error

The error is "colmap fail for some images". How to resolve it?

salykovaa / instant-dexnerf Goto Github PK

instant-dexnerf's Introduction

Instant DexNerf

How to install

How to run

How to create depth maps of captured scenes

Results

Example 1

Example 2

Depth error

Example 1

Example 2

Citation

instant-dexnerf's People

Contributors

Stargazers

Watchers

Forkers

instant-dexnerf's Issues

Recommend Projects

Recommend Topics

Recommend Org