hzxie / pix2vox Goto Github PK

Dear authors, it would be nice to see how the SOTA model performs on real images. Is it possible to release the model trained on Pix3D?

Hello, Mr. hzxie. Thanks for the shapenet dataset link you provided.In
ShapeNetRendering, each sample contains a rendering_metadata.txt, but after searching online, I only found such a profile (''From the left, they are azimuth, elevation, in-plane rotation, distance, the field of view.''). I am still confused about the 'distance', could you please show me the specific meaning? And if it's convenient, could you please also put on the specific metadata introduction ? I will be grateful for everything you have done.

Hi @hzxie , very cool work! Will the code of Pix2Vox++ be available soon?

您好，您好像只提供了Pix3D和ShapeNet的链接，PASCAL3D的链接好像没有提供，其次是SUN2012 DB的链接是不是也需要修改，所以出现了这个问题？

正在下载数据进行测试.....

If the distance between camera and target is known, is it possible to estimate the size of the 3D reconstruction result obtained by your method? If yes, how to calculate?

Hi, What are the parameters when converting .obj files to .binvox files for Pix3D? I am using the following:
./binvox -d 32 'model.obj'
When I am testing the checkpoint on ShapeNet dataset, I can reproduce the exact results reported in the paper. However, I can not reproduce the results on Pix3D.

1- I was wondering if any other pre-processing (e.g., rotation) should be applied on the binvox files?
2- Should I use the same checkpoint (Pix2Vox-A-ShapeNet.pth) when testing on Pix3D?

As per the reference you mentioned i.e. #11 . If I want the output of all the classes in shapenet dataset then What exactly needs to come in pred_volume?

encoder = Encoder(cfg)
decoder = Decoder(cfg)
refiner = Refiner(cfg)
merger = Merger(cfg)

cfg.CONST.WEIGHTS = '/home/baijinggroup/lss/Pix2Vox/pretrained_weights/Pix2Vox-A-ShapeNet.pth'

checkpoint = torch.load(cfg.CONST.WEIGHTS, map_location=torch.device('cpu'))

fix_checkpoint = {}
fix_checkpoint['encoder_state_dict'] = OrderedDict((k.split('module.')[1:][0], v) for k, v in checkpoint['encoder_state_dict'].items())
fix_checkpoint['decoder_state_dict'] = OrderedDict((k.split('module.')[1:][0], v) for k, v in checkpoint['decoder_state_dict'].items())

epoch_idx = checkpoint['epoch_idx']
encoder.load_state_dict(fix_checkpoint['encoder_state_dict'])
decoder.load_state_dict(fix_checkpoint['decoder_state_dict'])

encoder.eval()
decoder.eval()
refiner.eval()
merger.eval()

img1_path = '/home/baijinggroup/lss/Pix2Vox/datasets/ShapeNetRendering/02691156/1a04e3eab45ca15dd86060f189eb133/rendering/00.png'
#img1_path = '/home/baijinggroup/lss/Pix2Vox/datasets/ShapeNetRendering/04090263/1aa5498ac780331f782611375da5ea9a/rendering/00.png'	
img1_np = np.asarray(Image.open(img1_path))
#img1_np = cv2.imread(img1_path)

sample = np.array([img1_np])

IMG_SIZE = cfg.CONST.IMG_H, cfg.CONST.IMG_W
CROP_SIZE = cfg.CONST.CROP_IMG_H, cfg.CONST.CROP_IMG_W

test_transforms = utils.data_transforms.Compose([
    utils.data_transforms.CenterCrop(IMG_SIZE, CROP_SIZE),
    utils.data_transforms.RandomBackground(cfg.TEST.RANDOM_BG_COLOR_RANGE), 
    utils.data_transforms.Normalize(mean=cfg.DATASET.MEAN, std=cfg.DATASET.STD),
    utils.data_transforms.ToTensor(),
])

rendering_images = test_transforms(rendering_images=sample)
rendering_images = rendering_images.unsqueeze(0)

with torch.no_grad():
    image_features = encoder(rendering_images)
    raw_features, generated_volume = decoder(image_features)

    if cfg.NETWORK.USE_MERGER and epoch_idx >= cfg.TRAIN.EPOCH_START_USE_MERGER:
        generated_volume = merger(raw_features, generated_volume)
    else:
        generated_volume = torch.mean(generated_volume, dim=1)

    if cfg.NETWORK.USE_REFINER and epoch_idx >= cfg.TRAIN.EPOCH_START_USE_REFINER:
        generated_volume = refiner(generated_volume)

generated_volume = generated_volume.squeeze(0)
img_dir = '/home/baijinggroup/lss/Pix2Vox/core/sample_images'
gv = generated_volume.cpu().numpy()
gv_new = np.swapaxes(gv, 2, 1)
rendering_views = utils.binvox_visualization.get_volume_views(gv_new, os.path.join(img_dir), epoch_idx)

This is the input:

This is the model's output:

The result after I added this code np.swapaxes(gv,2,1)：

How to continue training on the weight of the training you provide, that is, how to perform transfer learning?

hello, This is my problem.
At first I thought about the TensorBoard version problem, but the TensorBoard is using 1.8.
can you help me..?

I have done the complete training and also done the testing but the results only show a single class i.e Aeroplane. So how to get other classes results?

If we know the parameters of camera and the distance between camera and target, can we use the voxel coordinates to calculate the size of target?

I wonder how can I visualize the reconstruction results like you do in the original paper.
Examples are shown in the picture.

Can you tell me what voxel visualization tool you are using? Thanks.

Is the Pascal3d dataset to be downloaded as well ,since no link has been provided for downloading it,but a path to it has been set in the code.

Hi great resource, thank you.

Unclear on 4 things though:

1. where is the output being stored, I checked the 'Output' folder but I only see logs there, nothing else!
2. in what format is output being generated? Is it .binvox files?
3. How do we test the model on new/unseen data?
4. Also, I see that ShapeNetRendering dataset consists of multiview images per each object; isn't this code for single-view image to 3D?

错误在于binvox_visualizatrion.py 这个文件。具体错误如下

When I was looking at the source code, I was not sure where the "context-aware fusion module" mentioned in the article was implemented in the code? Is this a few lines of code to achieve this function? Looking forward to your answer, thank you!

voxels_flat = np.transpose(dense_voxel_data, (0, 2, 1)).flatten()
Value Error Axes does not match array

Does anyone know how to save the output as a 3D model?

How to implement Score Normalization of multi-scale context-aware fusion
module in your code? Can you give me a link to implement this function? looking forwarding to your reply.

The random background color range in config.py for training is [[255, 255], [255, 255], [255, 255]]. So the random background color is np.random.randint(255, 256), which is always 255? BTW, which subset of SUN2012 do you use for random BG? Thank you very much.

I am trying to reuse your pre-trained Pix2Vox-A model but the output seems to be of a low quality (close to randomness). Following is the bare minimum of code that I used.

encoder = Encoder(cfg)
decoder = Decoder(cfg)
refiner = Refiner(cfg)
merger = Merger(cfg)

cfg.CONST.WEIGHTS = '/Projects/Pix2Vox/pretrained_weights/Pix2Vox-A-ShapeNet.pth'
checkpoint = torch.load(cfg.CONST.WEIGHTS, map_location=torch.device('cpu'))

fix_checkpoint = {}
fix_checkpoint['encoder_state_dict'] = OrderedDict((k.split('module.')[1:][0], v) for k, v in checkpoint['encoder_state_dict'].items())
fix_checkpoint['decoder_state_dict'] = OrderedDict((k.split('module.')[1:][0], v) for k, v in checkpoint['decoder_state_dict'].items())

epoch_idx = checkpoint['epoch_idx']
encoder.load_state_dict(fix_checkpoint['encoder_state_dict'])
decoder.load_state_dict(fix_checkpoint['decoder_state_dict'])

encoder.eval()
decoder.eval()
refiner.eval()
merger.eval()

img1_path = '/ShapeNetRendering/02691156/1a04e3eab45ca15dd86060f189eb133/rendering/00.png'
img1_np = np.asarray(Image.open(img1_path))

sample = np.array([img1_np])

IMG_SIZE = cfg.CONST.IMG_H, cfg.CONST.IMG_W
CROP_SIZE = cfg.CONST.CROP_IMG_H, cfg.CONST.CROP_IMG_W

test_transforms = utils.data_transforms.Compose([
    utils.data_transforms.CenterCrop(IMG_SIZE, CROP_SIZE),
    utils.data_transforms.RandomBackground(cfg.TEST.RANDOM_BG_COLOR_RANGE),
    utils.data_transforms.Normalize(mean=cfg.DATASET.MEAN, std=cfg.DATASET.STD),
    utils.data_transforms.ToTensor(),
])

rendering_images = test_transforms(rendering_images=sample)
rendering_images = rendering_images.unsqueeze(0)

with torch.no_grad():
    image_features = encoder(rendering_images)
    raw_features, generated_volume = decoder(image_features)

    if cfg.NETWORK.USE_MERGER and epoch_idx >= cfg.TRAIN.EPOCH_START_USE_MERGER:
        generated_volume = merger(raw_features, generated_volume)
    else:
        generated_volume = torch.mean(generated_volume, dim=1)

    if cfg.NETWORK.USE_REFINER and epoch_idx >= cfg.TRAIN.EPOCH_START_USE_REFINER:
        generated_volume = refiner(generated_volume)

For visualization I used the binvox_visualization from utilities:

generated_volume = generated_volume.squeeze(0)

img_dir = '/sample_images'
gv = generated_volume.cpu().numpy()
rendering_views = utils.binvox_visualization.get_volume_views(gv, os.path.join(img_dir),
                                                              epoch_idx)

This is the model's output:

This is the input:

There were few problems with loading the pre-trained weights, as mentioned in other issues, but other than that it seems to work properly except for the quality. I guess I must be missing something?

Hi,

I tried testing the pretrained model on a number of CAD models from the ABC dataset, but the network fails to recover the shapes, even when 5 views are used. I rendered the 2D images using Blender. I wonder if it is the method of rendering that is causing the problem, or the model just cannot generalize to unseen classes. I should also mention that the code worked fine on the customized ShapeNet dataset provided in the repo. Any thoughts? I have attached the model, reconstruction, and the rendered views down below.
Thanks

04.zip

Hello,

Is tensorflow code available for Pix2Vox.
Has anyone tried ONNX to convert it to tensorflow?

Thanks

I have a question about the two fully two layers in this network. These two fully connected layers occupy a lot of parameters. Can you ask, what is the role of these two layers?

Here is the error
...
[WARN] 2020-04-14 14:24:54.128112 Ignore sample 04530566/94a0345790869eace074faa875b76f82 since volume file not exists.
[WARN] 2020-04-14 14:24:54.128126 Ignore sample 04530566/705df680888e17ed8841beb2e0280e42 since volume file not exists.
[WARN] 2020-04-14 14:24:54.128140 Ignore sample 04530566/b4f254a5c49748a2db2b485c26c9f191 since volume file not exists.
[INFO] 2020-04-14 14:24:54.128153 Complete collecting files of the dataset. Total files: 0.
Downloading: "https://download.pytorch.org/models/vgg16_bn-6c64b313.pth" to /root/.cache/torch/checkpoints/vgg16_bn-6c64b313.pth
100% 528M/528M [00:29<00:00, 18.8MB/s]
[INFO] 2020-04-14 14:25:34.705290 Loading weights from ../Pix2Vox-A-ShapeNet.pth ...
[INFO/Process-1] child process calling self.run()
[INFO/Process-1] process shutting down
[INFO/Process-1] process exiting with exitcode 0
/content/Pix2Vox/core/test.py:168: RuntimeWarning: invalid value encountered in double_scalars
mean_iou = np.sum(mean_iou, axis=0) / n_samples
============================ TEST RESULTS ============================
Taxonomy #Sample Baseline t=0.20 t=0.30 t=0.40 t=0.50
Overall Traceback (most recent call last):
File "runner.py", line 93, in
main()
File "runner.py", line 77, in main
test_net(cfg)
File "/content/Pix2Vox/core/test.py", line 192, in test_net
for mi in mean_iou:
TypeError: 'numpy.float64' object is not iterable
[INFO/MainProcess] process shutting down

In the code, the pix3d dataset is regared as '.binvox' format files. But, in fact, there are only '.obj' files and '.mat' files. The Pix3dDataset is wrong.

how to reproduce results. how can we create a demo file to test trained models and visualize results. I tried to reproduce the test results from trained weights but failed.
@hzxie @kondela @aashishbohra10

encoder = Encoder(cfg)
decoder = Decoder(cfg)
refiner = Refiner(cfg)
merger = Merger(cfg)

cfg.CONST.WEIGHTS = pretrained/Pix2Vox-A-ShapeNet.pth
chechpoint = torch.load(cfg.CONST.WEIGHTS, map_location=torch.device('cpu'))

fix_checkpoint = {}
fix_checkpoint['encoder_state_dict'] = orderedDict((k.split('module.')[1:][0], v) for k, v in checkpoint['encoder_state_dict'].items())
fix_checkpoint['decoder_state_dict'] = orderedDict((k.split('module.')[1:][0], v) for k, v in checkpoint['decoder_state_dict'].items())

epoch_idx = checkpoint['epoch_idx']
encoder.load_state_dict(fix_checkpoint['encoder_state_dict'])
decoder.load_state_dict(fix_checkpoint['decoder_state_dict'])

encoder.eval()
decoder.eval()
refiner.eval()
merger.eval()

img1_path = '/datasets/ShapeNetRendering/02691156/1a04e3eab45ca15dd86060f189eb133/rendering/00.png'
img1_np = np.asarray(Image.open(img1_path))

sample = np.array([img1_np])

IMG_SIZE = cfg.CONST.IMG_H, cfg.CONST.IMG_W
CROP_SIZE = cfg.CONST.CROP_IMG_H, cfg.CONST.CROP_IMG_W

test_transforms = utils.data_transforms.Compose([
    utils.data_transforms.CenterCrop(IMG_SIZE, CROP_SIZE),
    utils.data_transforms.RandomBackground(cfg.TEST.RANDOM_BG_COLOR_RANGE), 
    utils.data_transforms.Normalize(mean=cfg.DATASET.MEAN, std=cfg.DATASET.STD),
    utils.data_transforms.ToTensor(),
])

rendering_images = test_transforms(rendering_images=sample)
rendering_images = rendering_images.unsqueeze(0)

with torch.no_grad():
    image_features = encoder(rendering_images)
    raw_features, generated_volum = decoder(image_features)

    if cfg.NETWORK.USE_MERFER and epoch_idx >= cfg.TRAIN.EPOCH_START_USE_REFINER:
        generated_volume = refiner(generated_volume)


generated_volume = generated_volume.squeeze(0)

img_dir= '/output/myresults'
gv = generated_volume.cpu().numpy()
rendering_views = utils.binvox_visualizatino.get_volume_views(gv, os.path.join(img_dir), epoch_idx)

Thank you in advance.

Originally posted by @ahmedshingaly in #28 (comment)

I'm getting this issue with the shapes when i run the command python runner.py --test --weights=Pix2Vox-F-ShapeNet.pth

Hello, do you know how to make shapenet datasets? I want to make a 3D dataset of an object similar to a courier box, but I don't know how to make it.

Thanks for your great work at first~

You calculated IoU of voxels in core/test.py line129-134. I have some questions about that.

Line 133: union = torch.sum(torch.ge(_volume.add(ground_truth_volume), 1)).float()
I think a union includes 3 parts:

1. volume = 1 and gt = 1 => 1+1=2
2. volume = 0 and gt = 1 => 0+1=1
3. volume = 1 and gt = 0 => 1+0=1

But I find that in your code, it only includes 2) and 3). Because you throw away the voxel-pair, whose sum is 2, by torch.ge(add_reslut,1).
Am I wrong somewhere? Looking for reply.

Best regards.

Error in loading state_dict for encoder while trying to load the pretrained weights

In the paper [Pix2Vox++: Multi-scale Context-aware 3D Object Reconstruction from Single and Multiple Images]
it says

We construct a large-scale dataset, named Things3D,
containing 1.68M images of 280K objects collected
from over 39K indoor scenarios.

Is that dataset publicly available? If yes, would you be kind enough and provide a link?

hello, This is my problem.can you help me..?thanks

you can simply use the following code:
pred_volume = ...

with open('/tmp/%s.binvox' % model_id, 'wb') as f:
vox = binvox_rw.Voxels(pred_volume.cpu().numpy(), (32,) * 3, (0,) * 3, 1, 'xyz')
vox.write(f)

hey .. I am not getting where to add this code referred by you in #11?
and also tell what will happen by this code?

Dear author, could you provide us with information about the training that was done to obtain the Pix2Vox-A(457.0 MB) pre-trained model , such as the configuration used, the hardware, etc.

Thanks in advance.

hello~
I'd like to test Pix2Vox++-F and use "git checkout -b Pix2Vox++-F origin/Pix2Vox++-F" in git, but there is something wrong happend: fatal: 'origin/Pix2Vox++-F' is not a commit and a branch 'Pix2Vox++-F' cannot be created from it
I tried "git pull" but it didn't work, could you give me some other advice? thanks.

Dear Haozhe, I also study 3D reconstruction. In my PHD thesis, I did a set of experiments on elliptical orbit reconstruction. I want to test your code on the collected images. But, I spent more than ten days without success. The deadline given by my tutor is approaching. Could I send you a set of images to test for me? Just one reconstruction test, could you help me ? Could you tell me if you agree? You can leave your contact information or contact me. My telephone/wechat number is 18706752562. My email address is [email protected].

Hello, Mr. hzxie.After I modified the file path of the datasets according to README,I started training or testing datasets of ShapeNet.But both reported the same error:"TypeError:not all arguments converted during string formatting".I don't know if it's a path problem or anything else?

[INFO] 2019-11-18 20:14:12.342971 [Epoch 1/250][Batch 478/478] BatchTime = 0.641 (s) DataTime = 0.000 (s) EDLoss = 1.6889 RLoss = 1.5908
[INFO/Process-1] process shutting down
[INFO/Process-2] process shutting down
[INFO/Process-3] process shutting down
[INFO/Process-1] process exiting with exitcode 0
[INFO/Process-2] process exiting with exitcode 0
[INFO/Process-4] process shutting down
[INFO/Process-3] process exiting with exitcode 0
[INFO/Process-4] process exiting with exitcode 0
[INFO] 2019-11-18 20:14:12.717975 Epoch [1/250] EpochTime = 364.328 (s) EDLoss = 2.7254 RLoss = 2.1918
[INFO/Process-5] child process calling self.run()
[INFO/Process-5] process shutting down
[INFO/Process-5] process exiting with exitcode 0
Traceback (most recent call last):
File "C:\Users\A409\AppData\Roaming\Python\Python35\site-packages\PIL\Image.py", line 2645, in fromarray
mode, rawmode = _fromarray_typemap[typekey]
KeyError: ((1, 1, 389), '|u1')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File ".\runner.py", line 87, in
main()
File ".\runner.py", line 68, in main
train_net(cfg)
File "D:\lisaisai\Pix2Vox-master\core\train.py", line 261, in train_net
iou = test_net(cfg, epoch_idx + 1, output_dir, val_data_loader, val_writer, encoder, decoder, refiner, merger)
File "D:\lisaisai\Pix2Vox-master\core\test.py", line 150, in test_net
test_writer.add_image('Test Sample#%02d/Volume Reconstructed' % sample_idx, rendering_views, epoch_idx)
File "C:\Users\A409\AppData\Roaming\Python\Python35\site-packages\tensorboardX\writer.py", line 584, in add_image
image(tag, img_tensor, dataformats=dataformats), global_step, walltime)
File "C:\Users\A409\AppData\Roaming\Python\Python35\site-packages\tensorboardX\summary.py", line 271, in image
image = make_image(tensor, rescale=rescale)
File "C:\Users\A409\AppData\Roaming\Python\Python35\site-packages\tensorboardX\summary.py", line 311, in make_image
image = Image.fromarray(tensor)
File "C:\Users\A409\AppData\Roaming\Python\Python35\site-packages\PIL\Image.py", line 2647, in fromarray
raise TypeError("Cannot handle this data type")
TypeError: Cannot handle this data type
[INFO/MainProcess] process shutting down

Hi, thanks for your wonderful work! But I'm just wondering that why would you use ELU activations in the encoder instead of simply using ReLU? Can you explain that? Thanks a lot!

Mr. hzxie, I'm trying to do something with your Pix2Vox work, but there seems to be some problems caused by the version of pytorch, so could you please show me the version you use?

The attach files show the problem with pytorch 1.0

Greetings,
I faced below error while running runner.py file, any suggestions will be appreciated.

[INFO] 2020-09-07 11:14:58.773195 [Epoch 1/250][Batch 473/478] BatchTime = 0.609 (s) DataTime = 0.000 (s) EDLoss = 1.5209 RLoss = 1.4220
[INFO] 2020-09-07 11:14:59.339636 [Epoch 1/250][Batch 474/478] BatchTime = 0.566 (s) DataTime = 0.001 (s) EDLoss = 1.5411 RLoss = 1.4908
[INFO] 2020-09-07 11:14:59.899355 [Epoch 1/250][Batch 475/478] BatchTime = 0.560 (s) DataTime = 0.000 (s) EDLoss = 1.5617 RLoss = 1.4801
[INFO] 2020-09-07 11:15:03.457158 [Epoch 1/250][Batch 476/478] BatchTime = 3.558 (s) DataTime = 2.985 (s) EDLoss = 1.4925 RLoss = 1.4052
[INFO] 2020-09-07 11:15:04.028943 [Epoch 1/250][Batch 477/478] BatchTime = 0.572 (s) DataTime = 0.000 (s) EDLoss = 1.5314 RLoss = 1.4357
[INFO] 2020-09-07 11:15:04.586913 [Epoch 1/250][Batch 478/478] BatchTime = 0.558 (s) DataTime = 0.001 (s) EDLoss = 1.4898 RLoss = 1.3886
[INFO/Process-2] process shutting down
[INFO/Process-1] process shutting down
[INFO/Process-3] process shutting down
[INFO/Process-2] process exiting with exitcode 0
[INFO/Process-4] process shutting down
[INFO/Process-1] process exiting with exitcode 0
[INFO/Process-3] process exiting with exitcode 0
[INFO/Process-4] process exiting with exitcode 0
[INFO] 2020-09-07 11:15:04.620620 Epoch [1/250] EpochTime = 713.621 (s) EDLoss = 3.1987 RLoss = 2.5285
[INFO/Process-5] child process calling self.run()
[INFO/Process-5] created temp directory /tmp/pymp-mf7icpus
[INFO/Process-5] process shutting down
[INFO/Process-5] process exiting with exitcode 0
Traceback (most recent call last):
File "runner.py", line 93, in
main()
File "runner.py", line 74, in main
train_net(cfg)
File "/home/ahmed/Documents/3DAI/Pix2Vox_hzxie/Pix2Vox/core/train.py", line 264, in train_net
iou = test_net(cfg, epoch_idx + 1, output_dir, val_data_loader, val_writer, encoder, decoder, refiner, merger)
File "/home/ahmed/Documents/3DAI/Pix2Vox_hzxie/Pix2Vox/core/test.py", line 151, in test_net
epoch_idx)
File "/home/ahmed/Documents/3DAI/Pix2Vox_hzxie/Pix2Vox/utils/binvox_visualization.py", line 19, in get_volume_views
ax.set_aspect('equal')
File "/home/ahmed/anaconda3/envs/py37/lib/python3.7/site-packages/mpl_toolkits/mplot3d/axes3d.py", line 324, in set_aspect
"Axes3D currently only supports the aspect argument "
NotImplementedError: Axes3D currently only supports the aspect argument 'auto'. You passed in 'equal'.
[INFO/MainProcess] process shutting down

when debugging it says
The reason for a NotImplementedError being raised is indeed that aspect is not implemented for 3D axes. This was added deliberately in #13474 to tell people about the fact that this feature does not (yet) exist.
is there a workaround?
thank you in advance