rehg-lab / 3dshapegen Goto Github PK

Hi

I was wondering if you could enable link sharing so that the data can be downloaded from a Linux console with wget?

Thank you

Hi,

Thanks a lot for releasing the amazing codes. Everything works well except for this small rendering issue.

We would like to make use of your rendering codes. However, I cannot get exactly the same images when I run wrapper.py with the default data_generation_parameters.json (my blender version is 2.79). The scale of the object is smaller than the provided rendered images. Here is an example. The first image is what I obtained by running wrapper.py and the second image is provided in your repo (ShapeNet55_3DOF-VC_LRBg). In addition, current codes do not seem to support blending the background. Can you also released the background part? Thank you very much!

Hi,
thanks for providing the code to your paper.
I wanted to generate meshes from the sdf values using your pipeline eval.py but run into the error:

(sdf_net)` user@host SDFNet: python eval.py            
Loading model from 3DShapeGen/out/train_overfit_test/best_model_iou.pth.tar
Loading data...
  0%|                                                                  | 0/1 [00:00<?, ?it/s]
./isosurface/computeMarchingCubes: error while loading shared libraries: libpng12.so.0: cannot open shared object file: No such file or directory

in the environment.yaml another libpng version is specified: - libpng=1.6.37=hed695b0_0
I tried to downgrade the libpng version to 12 in the conda environment. This did not work.
Could you please provide (or point me to) the codebase s.t. I can build MarchingCubes myself with libpng16 or could you update this repo with a new build that uses the version from the environment?

Thanks a lot in advance.

Hi all,

Thank you for releasing the code for the paper! Could you please add the license information?

Don't know if you can choose, but it would be great if the license was MIT.

Thanks for sharing the code and data.

The SDFNet README listed some data. I got a "permission denied" message when accessing "3-DOF VC D+N Pred" at this link:
https://www.dropbox.com/s/qz227pr2vbrzu9l/ShapeNet55_HVC_LR_PRED.tar

Other data links seem to work fine.

BTW, is "3-DOF VC D+N Pred" the data output from the Stage 1 (2D -> 2.5D) network after processing the ground truth data in the link "3-DOF viewpoint LRBg ShapeNetCore.v2 renders"?

Thanks for your great work! I have a question: I use "best_model_img_3DOF.pth.tar" to generate 3D shape from an image, and how to preprocess the image? I tried image=image/255.0, the result seems not correct. Can you help me?

Thanks with regards.

The result is:

Hi,

Thanks again for this work and the awesome code / data release! We will be trying to compare with your tables so here a few queries to make sure we are doing things properly.

Our main source of hesitations comes from :

• Does SDFNet denotes a method taking estimated depth as input or GT?
• Throughout the paper, which version of Table's 6 background is used?

Here is my current understanding please correct me if you spot something wrong :

• Fig 1 : SDFNet trained with estimated depth

• Fig 2 : SDFNet trained with estimated depth

• Fig 3 : SDFNet trained with ground-truth depth

• Table 2 : GenRe data. 3 categories for training / 10 for testing , using estimated depth to train/test

• Table 3 : ShapeNet data. 13 categories for training / 42 for testing , renderings with 2 DoF, using estimated depth to train/test

• Table 4 : ShapeNet data. 13 categories for training / 42 for testing , renderings with 3 DoF, using estimated depth to train/test

• Table 5 : ShapeNet data. 13 categories for training / 42 for testing , using ground-truth depth to train/test

• Table 6 : ShapeNet data. 13 categories for training and testing , using estimated and ground-truth depth to train/test

Some questions to make sure I understand :

• Why doesn't SDFNet Orcl results in Table 6 appear in Table 5?
• Why isn't SDFNet Est results in Table 6 the same as SDFNet Est - Seen Classes in Table 5?

Thank you so much for your precious help,
Thibault

Hi, thanks for your release. I tried to get 2-DOF_VC_with predicted_Depth+Normals by their link when I found on READEME.md, but I don't have access. Can you help me with this?

Hi, there.
Thanks for your release. I'm interested in your work and trying to implement it. I can get reasonable results directly using images without depth and normal estimators. But it doesn't work when depth and normal estimators are also implemented. I want to make sure if the two parts are trained totally separated? And do you add additional operations on the depth maps, like the sigmoid or clamp function? Because I also found some values less than 0.

This is my code of depth and normal estimators:

class Net(nn.Module):
    def __init__(self, out_planes, layer_names, input_planes=3):
        super().__init__()
        # Encoder
        module_list = list()
        resnet = resnet18(pretrained=True)
        in_conv = nn.Conv2d(input_planes, 64, kernel_size=7, stride=2, padding=3, bias=False)
        module_list.append(
            nn.Sequential(
                resnet.conv1 if input_planes == 3 else in_conv,
                resnet.bn1,
                resnet.relu,
                resnet.maxpool
            )
        )
        module_list.append(resnet.layer1)
        module_list.append(resnet.layer2)
        module_list.append(resnet.layer3)
        module_list.append(resnet.layer4)
        self.encoder = nn.ModuleList(module_list)

        # Decoder
        for out_plane, layer_name in zip(out_planes, layer_names):
            module_list = list()
            revresnet = revuresnet18(out_planes=out_plane)
            module_list.append(revresnet.layer1)
            module_list.append(revresnet.layer2)
            module_list.append(revresnet.layer3)
            module_list.append(revresnet.layer4)
            module_list.append(
                nn.Sequential(
                    revresnet.deconv1,
                    revresnet.bn1,
                    revresnet.relu,
                    revresnet.deconv2
                )
            )
            module_list = nn.ModuleList(module_list).cuda()
            setattr(self, 'decoder_' + layer_name, module_list)

    def forward(self, im):
        # Encode
        feat = im
        feat_maps = list()
        for f in self.encoder:
            feat = f(feat)
            feat_maps.append(feat)
        x = feat_maps[-1]
        for idx, f in enumerate(self.decoder_depth):
            x = f(x)
            if idx < len(self.decoder_depth) - 1:
                feat_map = feat_maps[-(idx + 2)]
                assert feat_map.shape[2:4] == x.shape[2:4]
                x = torch.cat((x, feat_map), dim=1)
        depth_output = x

        x = feat_maps[-1]
        for idx, f in enumerate(self.decoder_mask):
            x = f(x)
            if idx < len(self.decoder_depth) - 1:
                feat_map = feat_maps[-(idx + 2)]
                assert feat_map.shape[2:4] == x.shape[2:4]
                x = torch.cat((x, feat_map), dim=1)
        mask_output = x

        x = feat_maps[-1]
        for idx, f in enumerate(self.decoder_normal):
            x = f(x)
            if idx < len(self.decoder_depth) - 1:
                feat_map = feat_maps[-(idx + 2)]
                assert feat_map.shape[2:4] == x.shape[2:4]
                x = torch.cat((x, feat_map), dim=1)
        normal_output = x

        return depth_output, mask_output, normal_output

For inference:

depth, mask, normal = depth_normal_model(img_input)
depth = depth*mask
input_2 = torch.cat([depth, normal], dim=1)
sdf = model(points_input, input_2)

Hi,

Thanks for your great work and for sharing all the code to generate the data.

I am trying to get the rotation matrix from the metadata. I've been struggling a bit with this so I tried reusing the rendering script you provide i.e generate.py to render a shapenet object with simple parameters. The script gives several errors on Mac, hence my question :

• did you render shapenet on Linux or Mac?

These errors are for instance :

  File "utils/render_utils.py", line 66, in make_area_lamp
    nodes = lamp.node_tree.nodes
AttributeError: 'NoneType' object has no attribute 'nodes'

This can be solved by setting :
lamp.use_nodes = True but there are other errors after this.

If you have any pointers to go from the metadata files to the rotation matrix, that would be great. I tried inversing the sequence of Euler-angle transformations in generate.py but the resulting matrix is not consistent with the renderings so far.
Cheers,
Thibault

Where can I find the data split and rendering meta data of ABC dataset?

Hi,

Thank you for this work. I am interested in using the image only version as a baseline. Before I try training it myself, I was wondering if you happened to have the pre-trained model (image only, 3DOF, VC) hanging around somewhere. It would be very much appreciated!

Thank you,
Marissa

Thank you for sharing the code! I have an issue with the depth map generation. It seems that the depth map rendered with cycles engine in blender 2.79 is not the z-value, which is the orthogonal distance. https://blender.stackexchange.com/questions/87504/blender-cycles-z-coordinate-vs-camera-centered-z-coordinate-orthogonal-distan

Will this depth map behavior affects the GenRe results? I think that the depth map used by GenRe rendered by Mitsuba is keeping the z-value.

It seems that the depth estimation part is missing in the repo. Is it possible to open-source depth estimation codes as well? Thanks!

Hi,

I am trying to train and evaluate the network for a small scale experiment. I have managed to get everything working up to training— setting up environment, rendering input image data, changing configurations, managing dependencies issues...

At the start of training in train.py I get the following error:

RuntimeError: cuda runtime error (8) : invalid device function at /opt/conda/conda-bld/pytorch_1544174967633/work/aten/src/THC/THCGeneral.cpp:405

I am running the training on a single NVIDIA GeForce GPU with 25GB memory usage. Driver Version: 510.60.02 and CUDA Version: 11.6. I suspect that this error comes from some incompatibility between my GPU CUDA version and the PyTorch and other packages versions set up by the environment provided on environment.yml

I was wondering if anyone or the authors have run into a similar issue or have any suggestions on how to manage this possible incompatibility.

I tried to install higher versions of PyTorch, torchvision, and install cudatoolkit on the environment but this lead to incompatibilities with other packages' versions specified in environment.yml. conda update --all also created dependency issues

Thank you very much for releasing the nice codes. I would like to reproduce your results reported in Table 3 in your arxiv paper. Is it possible to release the pretrained model (SDFNet) and predicted depth and normal used for reproducing Table 3? Thank you very much!

Hi again,

I have some general questions about the repo:

• How many GPU's, GPU memory and RAM do you use to train the network?

  • I get a CUDAout of memory error for 2 GPU's with 12GB each.
  • The network is not so big, so I wonder why this needs so much memory
  • I can only train a model on a single datapoint with single GPU (DataParallel set to device_ids=[0])

• In your paper the NW architecture contains a U-ResNet that predicts depth, normals and silhouette from a RGB image. Where can I find this in your repo? The SDFNet/model.py only contains encoder and 3D decoder.

  • Ok, I think I found this in GenRe-ShapeHD/networks/

• Why do you need 2 different marching cubes algorithms, i.e. libmcubes for occupancy representation and the marching cubes in SDFNet/isosurface?

Best regards, Jan