jongchyisu / mvcnn_pytorch Goto Github PK

In the command for given dataset num_models is 1000 and num_views is 12.

There is also a line which is
n_models_train = args.num_models*args.num_views

In this settings, n_models_train is 12000, but there are 118116 images in the train set.

I have a dataset in which there are 15 classes (1 instance per class) and there are 720 images per class. In sum, training set contains 10800 images in which 20 different views are represented. In that case, what should be my num_models parameter (I guess num_view parameter should be 20)?

Thanks for sharing the paper. I am very interested in it and doing relative research.

I notice you mentioned knowledge distillation in the paper and it was used to improve the voxel-based 3D CNN and point-based architectures. I think this observation is exciting and I want to try it for my ideas. I have reproduced the classification experiments using your codebase.

But in this repo, the code of knowledge distillation is absent. I understand that many years have passed since you conducted the experiments. Can you try to find this part of code, or provide me some references/github repositories/advice to faciliatate my re-implentations?

It would be appreciated.

Thanks for your great work, I wonder how to directly run stage2 mvcnn without run stage1 first. I would appreciate if you could give some advice.

could you please add the license?! Thank you

Hi,
I have a small remark about the way the python file ImgDataset.py:MultiviewImgDataset(torch.utils.data.Dataset) is constructing the multi_view batch images. In fact, when you use stride = int(12/self.num_views), the trainer has only access to the same poses of an object.
For example, if you use self.num_views = 2, the trainer always get something like :
(car.XXX.001.png and car.XXX.007.png)
or
(cup.XXX.001.png and cup.XXX.007.png)

and this is the same problem for num_views = 3, 4, 6

I think this might be a big issue because there is no randomness in the multi view training stage.
If you validate the system with different poses, how the CNN could get the same performance?

Does anyone have this problem? How do you solve this?

E:\LBKpy\mvcnn_pytorch-master>python train_mvcnn.py -name mvcnn -num_models 1000 -weight_decay 0.001 -num_views 12 -cnn_name vgg11 E:\Anaconda\envs\multiview\lib\site-packages\torchvision\io\image.py:11: UserWarning: Failed to load image Python extension: Could not find module 'E:\Anaconda\envs\multiview\Lib\site-packages\torchvision\image.pyd' (or one of its d ependencies). Try using the full path with constructor syntax. warn(f"Failed to load image Python extension: {e}") WARNING: summary folder already exists!! It will be overwritten!! WARNING: summary folder already exists!! It will be overwritten!! num_train_files: 118116 num_val_files: 29616 Traceback (most recent call last): File "train_mvcnn.py", line 61, in <module> trainer.train(30) File "E:\LBKpy\mvcnn_pytorch-master\tools\Trainer.py", line 50, in train for i, data in enumerate(self.train_loader): File "E:\Anaconda\envs\multiview\lib\site-packages\torch\utils\data\dataloader.py", line 521, in __next__ data = self._next_data() File "E:\Anaconda\envs\multiview\lib\site-packages\torch\utils\data\dataloader.py", line 561, in _next_data data = self._dataset_fetcher.fetch(index) # may raise StopIteration File "E:\Anaconda\envs\multiview\lib\site-packages\torch\utils\data\_utils\fetch.py", line 49, in fetch data = [self.dataset[idx] for idx in possibly_batched_index] File "E:\Anaconda\envs\multiview\lib\site-packages\torch\utils\data\_utils\fetch.py", line 49, in <listcomp> data = [self.dataset[idx] for idx in possibly_batched_index] File "E:\LBKpy\mvcnn_pytorch-master\tools\ImgDataset.py", line 127, in __getitem__ class_id = self.classnames.index(class_name) ValueError: 'modelnet40_images_new_12x' is not in list

when i run the project ,the wrong is 'modelnet40_images_new_12x' is not in list

http://people.cs.umass.edu/~jcsu/papers/shape_recog/render_depth.blend
I cannot download it.

Hello,

I am trying to run the code on a different dataset. I have prepared the dataset and extracted the images correctly. But while training, I get the error

Traceback (most recent call last):
  File "train_mvcnn.py", line 61, in <module>
    trainer.train(10)
  File "/home/bharadwajmanda/mvcnn_pytorch/tools/Trainer.py", line 65, in train
    self.writer.add_scalar('train/train_loss', loss, i_acc+i+1)
  File "/home/bharadwajmanda/anaconda3/envs/mvcnn/lib/python3.6/site-packages/tensorboardX/writer.py", line 341, in add_scalar
    scalar(tag, scalar_value), global_step, walltime)
  File "/home/bharadwajmanda/anaconda3/envs/mvcnn/lib/python3.6/site-packages/tensorboardX/summary.py", line 88, in scalar
    scalar = make_np(scalar)
  File "/home/bharadwajmanda/anaconda3/envs/mvcnn/lib/python3.6/site-packages/tensorboardX/x2num.py", line 27, in make_np
    return check_nan(prepare_pytorch(x))
  File "/home/bharadwajmanda/anaconda3/envs/mvcnn/lib/python3.6/site-packages/tensorboardX/x2num.py", line 40, in prepare_pytorch
    x = x.cpu().numpy()
RuntimeError: CUDA error: device-side assert triggered

Can you please help me with this?

Thanks,
Bharadwaj

Hi, thank you for your code and work.

I compare the dataset you provided and the dataset provided by RotationNet (http://maxwell.cs.umass.edu/mvcnn-data/?C=D;O=A). These two datasets are the same. But RotationNet said this is a subset of ModelNet40. Is it right?

Thank you for your time and attention.

Rainbow

Hello,
I tried to use the render scripts. The shaded works fine for me but the depth isn't, it gives me an image with constant color (see picture attached). I know this was discussed in a previous feed but I don't understand the problem, and I don't see the difference between this script and the shaded script (you mentioned the node editor but I don't know where this is located, I'm a newbie in Blender)
Thanks a lot !

[Blender script for rendering shaded images](http://people.cs.umass.edu/~jcsu/papers/shape_recog/render_shaded_black_bg.blend) and
[Blender script for rendering depth images](http://people.cs.umass.edu/~jcsu/papers/shape_recog/render_depth.blend) point to the same file with different file name in commit ff476b2

What i plan is to train with several views and get class (object type) at test time with 3 views, is it possible?

D:\git_hub\mvcnn_pytorch>python train_mvcnn.py -name mvcnn -num_models 1000 -weight_decay 0.001 -num_views 12 -cnn_name vgg11
WARNING: summary folder already exists!! It will be overwritten!!
WARNING: summary folder already exists!! It will be overwritten!!
Downloading: "https://download.pytorch.org/models/vgg11-bbd30ac9.pth" to C:\Users\Garima/.torch\models\vgg11-bbd30ac9.pth
531456000it [05:07, 1731015.24it/s]
Traceback (most recent call last):
File "train_mvcnn.py", line 54, in
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=64, shuffle=True, num_workers=0)
File "C:\Users\Garima\Anaconda3\envs\fastai-3.7\lib\site-packages\torch\utils\data\dataloader.py", line 802, in init
sampler = RandomSampler(dataset)
File "C:\Users\Garima\Anaconda3\envs\fastai-3.7\lib\site-packages\torch\utils\data\sampler.py", line 64, in init
"value, but got num_samples={}".format(self.num_samples))
ValueError: num_samples should be a positive integeral value, but got num_samples=0

I use the parameters suggested in your paper and for vgg-11, I trained first stage for 30 epoch and second stage for 100 epoch. I got 91.76% per-class accuracy and 94.41% per-instance accuracy. I set learning rate 5e-5 and 1e-5 in first and second stages respectively and weight decay 0.001. Is there something wrong with my setting? Thank u.

I downloaded data from http://supermoe.cs.umass.edu/shape_recog/shaded_images.tar.gz

However, I found that
airplane only has test set.

Is it normal?

When will be the code released for Active Adversarial Domain Adaptation?

There seems solely train script, how can i test?

After successfully training I am struggling on how to test both models.
Do you have the script used for evaluation?

Hello, I would like to know how to concatenate depth image and shaded image? Is it just adding the depth image as alpha channel into the shaded image and processing, or training separately and combining at the view-pooling layer?

When I run the file, there is a cube in the middle of the generated png image, I would appreciate it if you could help, thanks.

Hello! I used the datasets you provided and got the ideal result. But when i try to use the blender script in your project-web to make other experiments（like other view numbers）,the accuracy is much lower. Do you preprocess the data in other method beside the blender script? Thank you!

Traceback (most recent call last):
File "train_mvcnn.py", line 79, in
trainer.train(1)
File "/home/wanghuijuan1/MVCNN/mvcnn_pytorch-master/tools/Trainer.py", line 50, in train
for i, data in enumerate(self.train_loader):
File "/home/wanghuijuan1/anaconda3/envs/torch18/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 517, in next
data = self._next_data()
File "/home/wanghuijuan1/anaconda3/envs/torch18/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 557, in _next_data
data = self._dataset_fetcher.fetch(index) # may raise StopIteration
File "/home/wanghuijuan1/anaconda3/envs/torch18/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/wanghuijuan1/anaconda3/envs/torch18/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/wanghuijuan1/MVCNN/mvcnn_pytorch-master/tools/ImgDataset.py", line 76, in getitem
class_id = self.classnames.index(class_name)
ValueError: 'train' is not in list

Hello! How do you get the VGG-M model pretrained on Imagenet? Thank you very much.

Hello,

I have tried to train the system. While training, I found some error in the ImgDataset.py like below

im = Image.open(self.filepaths[idx]).convert('RGB')

I used cuda 9.2.148 at ubuntu 18.04

Thanks in advance

epoch 1, step 25: train_loss 2.269; train_acc 0.406
epoch 1, step 26: train_loss 2.170; train_acc 0.438
Traceback (most recent call last):
  File "train_mvcnn.py", line 61, in <module>
    trainer.train(30)
  File "/home/stmoon/Test/mvcnn_pytorch/tools/Trainer.py", line 50, in train
    for i, data in enumerate(self.train_loader):
  File "/home/stmoon/Test/mvcnn_pytorch/env/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 615, in __next__
    batch = self.collate_fn([self.dataset[i] for i in indices])
  File "/home/stmoon/Test/mvcnn_pytorch/env/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 615, in <listcomp>
    batch = self.collate_fn([self.dataset[i] for i in indices])
  File "/home/stmoon/Test/mvcnn_pytorch/tools/ImgDataset.py", line 130, in __getitem__
    im = Image.open(self.filepaths[idx]).convert('RGB')
  File "/home/stmoon/Test/mvcnn_pytorch/env/lib/python3.6/site-packages/PIL/Image.py", line 915, in convert
    self.load()
  File "/home/stmoon/Test/mvcnn_pytorch/env/lib/python3.6/site-packages/PIL/ImageFile.py", line 250, in load
    self.load_end()
  File "/home/stmoon/Test/mvcnn_pytorch/env/lib/python3.6/site-packages/PIL/PngImagePlugin.py", line 677, in load_end
    self.png.call(cid, pos, length)
  File "/home/stmoon/Test/mvcnn_pytorch/env/lib/python3.6/site-packages/PIL/PngImagePlugin.py", line 140, in call
    return getattr(self, "chunk_" + cid.decode('ascii'))(pos, length)
  File "/home/stmoon/Test/mvcnn_pytorch/env/lib/python3.6/site-packages/PIL/PngImagePlugin.py", line 356, in chunk_IDAT
    raise EOFError
EOFError

D:\python\anaconda\python.exe D:/_多媒体检索/mvcnn_pytorch-master/train_mvcnn.py
WARNING: summary folder already exists!! It will be overwritten!!
WARNING: summary folder already exists!! It will be overwritten!!
num_train_files: 118116
num_val_files: 29616
Traceback (most recent call last):
File "D:/_多媒体检索/mvcnn_pytorch-master/train_mvcnn.py", line 61, in
trainer.train(30)
File "D:_多媒体检索\mvcnn_pytorch-master\tools\Trainer.py", line 50, in train
for i, data in enumerate(self.train_loader):
File "D:\python\anaconda\lib\site-packages\torch\utils\data\dataloader.py", line 363, in next
data = self._next_data()
File "D:\python\anaconda\lib\site-packages\torch\utils\data\dataloader.py", line 403, in _next_data
data = self._dataset_fetcher.fetch(index) # may raise StopIteration
File "D:\python\anaconda\lib\site-packages\torch\utils\data_utils\fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "D:\python\anaconda\lib\site-packages\torch\utils\data_utils\fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "D:_多媒体检索\mvcnn_pytorch-master\tools\ImgDataset.py", line 127, in getitem
class_id = self.classnames.index(class_name)
ValueError: 'modelnet40_images_new_12x' is not in list

Process finished with exit code 1

Hello ！I have a new problem that the resnet50 architecture get 96.1% ish accuracy twice! I set the parameters as what you mentioned in your paper. I keep all the default settings and only change the num_models to 0 and learning_rate in second stage to 1e-5. Did you get such high results in your runs?

HI Jong-Chyi,
I'm very interested in your work and trying to use your blender script to generate rendered views. But I find that Blender Script for shaded and depth images are exactly the same. Is there something wrong?

And in the Blender Script, it converts the *.off model to *.obj model by using the off2obj, is it an additional python script that used? I tried to find the convert script by google and used airplane_0627.off as an example for rendering, but the result is rather weird, it is

By the way, how to use the Blender Script effectively? I'm really a newbie about blender and I'd appreciate it if you could you show me an example ;).

Thanks very much.

First,thanks for your codde,i have a problem about stage,i don't know the stage1 and stage2's meanings,thank you for your help.

HI Jongchyisu,
I'm trying to use your blender script to generate rendered views. But there is a box in the middle of the image generated by this script .

Sorry I posted such a long code. Can you tell me how to modify the code？
thank you very much

I just used cmd to execute the following code
blender.exe -b -P render.py

render.py
`
import bpy
import os
import math
import numpy as np
import glob

from os import listdir
from os.path import isfile, join

context = bpy.context

models_path = "E:\\data\\ModelNet40"
models = sorted(glob.glob(models_path + "/*/*/*.off"))
nviews = 12

scene = bpy.context.scene

missed_models_list = []

for model_path in models:
    print(model_path)

    command = 'e:/antiprism/off2obj ' + model_path[:-4] + '.off -o ' + model_path[:-4] + '.obj'
    model_path = model_path[:-4] + '.obj'
    os.system(command)

    try:
        bpy.ops.import_scene.obj(filepath=model_path, filter_glob="*.obj")
    except:
        missed_models_list.append(model_path)
        continue

   imported = bpy.context.selected_objects[0]
   bpy.ops.object.origin_set(type='ORIGIN_GEOMETRY')

   maxDimension = 5.0
   scaleFactor = maxDimension / max(imported.dimensions)
   imported.scale = (scaleFactor, scaleFactor, scaleFactor)
   imported.location = (0, 0, 0)

   imported.rotation_mode = 'XYZ'

   views = np.linspace(0, 2 * np.pi, nviews, endpoint=False)
   print(views)

   for i in range(nviews):
       imported.rotation_euler[2] = views[i]
       imported.rotation_euler[0] = np.pi
       filename = model_path.split("/")[-1]
       print(filename)
       bpy.ops.view3d.camera_to_view_selected()
       context.scene.render.filepath = model_path + "_whiteshaded_v" + str(i) + ".png"
       bpy.ops.render.render(write_still=True)

   meshes_to_remove = []
   for ob in bpy.context.selected_objects:
       meshes_to_remove.append(ob.data)
   bpy.ops.object.delete()
   # Remove the meshes from memory too
   for mesh in meshes_to_remove:
       bpy.data.meshes.remove(mesh)

   imported = None
   del imported

`

Does anyone have this problem? How do you solve this?

RuntimeWarning: invalid value encountered in true_divide
  val_mean_class_acc = np.mean((samples_class-wrong_class)/samples_class)
WARNING:root:NaN or Inf found in input tensor.

Hi,

Is there any problem in changing the learning rate and training , changed to 5e-3 from default=5e-5 as for my dataset ,it seems accuracy 1 for sometimes in epoch 1 itself , after changing learning rate , till epoch 2 no such problem. I am having a large training time such as 14 hrs for 1 epoch , cud u pls tell me how to reduce the training time and effectively use ur model for my dataset.

Hello,

I have tried out the code with the provided training and test data "modelnet40_images_new_12x". I have trained my model with the default parameters in the README file:

python train_mvcnn.py -name mvcnn -num_models 1000 -weight_decay 0.001 -num_views 12 -cnn_name vgg11

Like many others I got a 94.2% overall accuracy with the second stage. After that I tried rendering the dataset myself with the provided blender script. As stated in another issues thread I didn't get exactly the same render results as in modelnet40_images_new_12x, because it's not the original blender file. With the rerendered dataset I only archived 91.5% overall accuracy. So I compared the images and noticed that not all classes have the same amount of differences. It seems that the classes are rendered with different camera angles. More precisely it is the rotation around the Y-axis that varies.

From my comparison it seems that e.g. the desks are rendered with ~180.8° and tables are rendered with 181.308°. In fact all objects in alphabetical order from plant to xbox are rendered with 181.308°.
(The angles are >180° because the object and the camera are both upside down in blender coordinate system, in case anyone is wondering.)
I have made several tests and my conclusion is that the network actually considers the camera rotation in some way. Here are two demonstrations for that behavior:

1.) When varying the rotation in the rerendered test data, the orignal model (trained with original modelnet40_images_new_12x data) reacts highly sensitive:

180°	71.2%
180.8°  90.1%
185°    80.1%

When the model is trained with the rerendered dataset (with equal rotation) it's not sensitive at all. The result is always 91.5%.

2.) Desks and tables are very similar objects in the dataset from the start. The model trained with original (modelnet40_images_new_12x) data still gets desks with 96% and tables with 95% right. When using again the original model and changing the angle for the whole test dataset we get:

		desk 	table
180.8°		95.3%	82%
181.308°	40.6%	93%

This further indicates that desk is rendered with 180.8° and table with 181.308° in the original training data and the network expects that in the test data as well.
As expected desk is false classified as table + tv_stand and table is false classified as desk. Similar confusion happens with bookshelf and tv_stand or cup and vase because of the same angles.

Do you have any idea why the camera rotation change happened in the first place? It's weird that the changes happen to be on a per-class basis instead of a per-object basis.

Hi, Dr. Su, sorry to bother you.

I just ran the training process in PyCharm(I use pytorch 1.1 installed by pip19.1, python 3.7, and CUDA 9.0 on Windows) following your instruction in README.md, and I got the CUDA OOM error. I tried to adjust the parameters like batchSize and num_models but it didn't help.

Would you mind telling me how to fix this CUDA OOM issue? What's your GPU capacity used in the exp? Thank you.

Here is the traceback of CUDA OOM error:

  File "train_mvcnn.py", line 66, in <module>
    trainer.train(30)
  File "C:\Work\CompVision\Code\proj\MVCNN\tools\Trainer.py", line 58, in train
    out_data = self.model(in_data)
  File "C:\Users\Karn\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\nn\modules\module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Work\CompVision\Code\proj\MVCNN\models\MVCNN.py", line 65, in forward
    y = self.net_1(x)
  File "C:\Users\Karn\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\nn\modules\module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\Karn\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\nn\modules\container.py", line 92, in forward
    input = module(input)
  File "C:\Users\Karn\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\nn\modules\module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\Karn\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\nn\modules\pooling.py", line 146, in forward
    self.return_indices)
  File "C:\Users\Karn\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\_jit_internal.py", line 133, in fn
    return if_false(*args, **kwargs)
  File "C:\Users\Karn\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\nn\functional.py", line 494, in _max_pool2d
    input, kernel_size, stride, padding, dilation, ceil_mode)
RuntimeError: CUDA out of memory. Tried to allocate 98.00 MiB (GPU 0; 4.00 GiB total capacity; 2.96 GiB already allocated; 68.48 MiB free; 7.42 MiB cached)