Hi! Thank you for making this project available! Amazing work.

Would it be possible to extract a 360 degree image from the result? Eg, an environment HDR?

Thanks!

Thanks for your contribution! For evaluation purposes, it would help a lot if you could add a small example on how to run launch_inference.sh on new images. I tried to run zeronvs on the following image (resized to 256x256 PNG image):

But the output is completely wrong and the program eventually crashes because no mesh could be reconstructed.

I launched launch.py in the following way (NOTE I replaced --train with --export):

python launch.py --config configs/zero123_scene.yaml --export --gpu 0 \
    system.guidance.cond_image_path=/home/user/Documents/projects/zeronvs/test_data/2008_001546_256x256.png \
    data.image_path=/home/user/Documents/projects/zeronvs/test_data/2008_001546_256x256.png \
    system.guidance.pretrained_model_name_or_path=zeronvs.ckpt \
    system.guidance.pretrained_config=zeronvs_config.yaml \
    data.view_synthesis=null \
    data.default_elevation_deg=0 \
    data.default_fovy_deg=52.55 \
    data.random_camera.fovy_range="[52.55,52.55]" \
    data.random_camera.eval_fovy_deg=52.55 \
    system.loss.lambda_opaque=0.0 \
    system.background.color='[0.5,0.5,0.5]' \
    system.background.random_aug=true \
    system.background.random_aug_prob=1.0 \
    system.guidance.guidance_scale=9.5 \
    system.renderer.near_plane=0.5 \
    system.renderer.far_plane=1000.0 \
    system.guidance.precomputed_scale=0.7 \
    system.guidance.use_anisotropic_schedule=true \
    system.guidance.anisotropic_offset=1000

Let me know if you need further information. Thanks in advance!

Thanks for your open source and that's a really great work!

I evaluate zeonvs.ckpt using the parameters that you provide on my desktop. Many of them show unsatisfied results as below. I would like to know whether something I get wrong when running. All the settings are the same as yours. Both linux and windows cannot show satisfied results.

Are this method unstable or something haven't released yet?

Thanks for your assistance in advance.

Hi, I am a bit stuck on RealEstate10k's scale representation, is this being compared to some Multi-View Synthesis pipelines like COLMAP? Are the scale of the depth estimations matching that of sun a representation for the ZeroNVS?

I realize that SDS anchoring is turned off by default.
Is there any way to turn on it by simply controling the configuration?

code updated to the latest. When tested on the image "motorcycle.png", no vision was shown. no errors or warnings were present except resource limitation.

Hi, do u have scripts for evaluation on CO3D dataset?

I want to run the model on my 3090 device, but there is not enough memory. How can I adjust the relevant parameters to make it work

Hi! I found you trained zeronvs_diffusion model on RealEstate10k with colmap depth, could you tell me how you get the RealEstate 10k dataset with colmap depth?

Thanks for your great work! I encountered a problem when I was trying to reproduce the result of motorcycle.png. I used all the default hyper-parameters but I got the following outputs:

Unfortunately, I'm a freshman in diffusion model. Could you give me some suggestions? Thanks a lot!

Hi! I try to re-training zeronvs_diffusion model with zeronvs_config.yaml on my own dataset. Should the value of quantile_scale_blend be True in model.params.conditioning_config?

When I pip from conda , ModuleNotFoundError: No module named 'tinycudann'

I want to run zeronvs_diffusion on my own customed dataset. I wonder the actual value of compute_nearplane_quantile in zeronvs_diffusion when you trained on CO3D and Re10K?

I don't see this variable in config files.

Hi! Sorry to bother you. Could you provide the reference or the tutorial about how you compute the focus point? I still cannot understand the code here:

def focus_point_fn(poses: np.ndarray) -> np.ndarray:
    directions, origins = poses[:, :3, 2:3], poses[:, :3, 3:4]
    m = np.eye(3) - directions * np.transpose(directions, [0, 2, 1])
    mt_m = np.transpose(m, [0, 2, 1]) @ m
    focus_pt = np.linalg.inv(mt_m.mean(0)) @ (mt_m @ origins).mean(0)[:, 0]
    return focus_pt

Hi,

Is SDS anchoring by default turned off? I saw n_aux_c2w = 0 in zero123_scene.yaml, which I guess means no anchor cameras. So if I run launch_eval_mipnerf360.sh directly, it's standard SDS without anchoring?

Thanks for the great work!

Did the clip loss based on SDS anchoring improve with your results? If it does, which scale did you use?
Thanks:)

Hello, my device is 3090, I tried to run the inference on my own image and cuda was out of memory. I checked the passed issue but the solution doesn't work. Is there a way to solve this issue? Thanks!

Hi Kyle,

Thanks for your fantastic work! I'd like to know how to generate novel views with your model trained with $\mathbf{M}_{\mathrm{6DoF+1,~viewer}}$ parameterization. I tried to use the code in the "zeronvs_diffusion" submodule but found it only uses 3dof parameterization. Could you please provide any instructions?

Thanks in advance,
Rundong

          I ran the prediction with launch_inference.sh in test mode and  the model was  released by your pre-trained model here. Still can't get the test result

Originally posted by @ezone123 in #7 (comment)