weixk2015 / deepsfm Goto Github PK

Hi, Good job and congratulations for the oral in ECCV 2020.
Im new here and had tried to use your scripts for testing the test-datasets. However, I was troubled for some questions and problems. Can you help me solve the followings if you are free? THX.

1. What's the meaning of outputs for PoseNet? In your script pose_test.py, the outputs of PoseNet are then transformed to the value p_matrix. So, what is the meaning of this value? Is this value meaning the variation of relative pose matrix(noise_pose in your codes) ?

2. In your script pose_test.py, what you saved in '%{args}_poses,txt' is the relative pose matrix between target and reference images. Is that right? However，we should read the absolute pose of each image from the last '%{args}_poses,txt'. There seems to be a conflict between the two files.

3. When I tried to use the poses.txt as the initial pose(Not the demon outputs) for the iterations testing, the results are getting worse. I think there may be something wrong with the saved pose file(maybe every iteration, the codes saved wrong pose). Have you ever been in this situation？

I would appreciate it if you could answer these questions! THX

Hi Xingkui,

Thanks for your great work first.

I am curious about the abs_rel and l1_rel used in these SFM papers. If I understand it correctly, l1_rel seems to be just abs_rel (according to DEMON, equation 9). However, regarding abs_rel, I noticed that DeepSFM largely outperforms DPSNet in Sun3D (0.072 vs 0.1470) and RGBD (0.126 vs 0.1470) datasets. In my humble opinion, DPSNet with gt poses should be the upper bound of DeepSFM. I guess DeepSFM's better result is because the iterative refinement, is it right?

Besides, I am trying to report the metrics mentioned below in my work. After getting pose_init from DEMON, my reproduced DeepSFM results are not that good. It would be appreciated if you can provide the results for the datasets in Table 1.

Hi, thank you for the great work!

I was wondering if the MVS training set from the DeMoN dataset is used for training in the paper. I have seen the mvs_train.tgz downloaded by script download_traindata.sh but the preparedata_train.py doesn't process the "mvs_citywall.h5", "mvs_breisach.h5", and "mvs_achteck_turm.h5". Further, in train.txt there are also no scenes from the MVS dataset.

So is the MVS dataset used for training or just for testing the generalization ability?

Hello, Wei. When I run your network (also using your data), I always get a warning "root:NaN or Inf found in input tensor." in a fixed position (in train data 3293), and then the value of loss becomes "nan". Have you ever encountered such a situation? Or do you know what causes this? Some people said that the learning rate may be too high and there may be a gradient explosion. But after I reduce the learning rate (even set it to 0), I will still make mistakes in the same position. The pytorch version I use is 1.7.0 instead of 0.4.0 you use. I wonder if this is the reason?
Looking forward to your reply.

Has this been tested for real time mapping? The paper does not say anything explicitly about this.

Hi Xingkui,

Hope you are doing well. I was wondering how did you plot depth maps for DEMON datasets, as in your paper. Is it

plt.imshow(depth_np, cmap='Greys')

Best,
Jianyuan

Dear Xingkui,
thanks for sharing the code DeepSFM.

Nonetheless, when I try to set up my machine to test it, I cannot install minieigen. I have tried on three different machines, fulfilling the previous requirements with no avail.
I tried using pip, conda to install without success. Would it be possible for you to give me some advise. I am running as you suggest ubuntu 16.04 with all the version requirements you define:

Pytorch 0.4.0
CUDA 9.0
python 3.6.4
scipy
argparse
tensorboardX
progressbar2
path.py
transforms3d
minieigen

Following the error messages, I have installed

libeigen3-dev

but cannot install libboost-python-py36

Would you please help me?

Good afternoon.
I tried to build your repository following the instructions. Having problems with an older version of Python. I used Сonda to solve this problem. But I could not install minieigen using pip install.

in PSNet, if add_geo is open, targets_depth is used to calculate some costs, but how to add geo cost when I don't have any depth info? can I init target depth with zero matrix?

Hi,
Have you published the paper in Chinese?

Thanks very much.

Hi, could you let me know where to download demon_poses.txt and *_demon.npy? I checked https://github.com/lmb-freiburg/demon, but I did not find the way.

Thanks.

Hi, I used your trained model, ground truth pose and depth files to calculate the "abs rel", "abs diff", .... But they are much lower than DPSNet (the last table of DPSNet code, https://github.com/sunghoonim/DPSNet).

Metrics
'abs_rel', 'abs_diff', 'sq_rel', 'rms', 'log_rms', 'a1', 'a2', 'a3'

MVS
DPSNet | 0.0722 0.2095 0.0798 0.4928 0.1527 0.8930 0.9502 0.9760
DeepSFM | 0.0986,0.7810,0.2797,2.1187,0.1470,0.8889,0.9774,0.9916

Sun3D
DPSNet | 0.1470 0.3234 0.1071 0.4269 0.1906 0.7892 0.9317 0.9672
DeepSFM | 0.0723,0.9464,0.2570,1.4316,0.1049,0.9598,0.9887,0.9942

RGBD
DPSNet | 0.1538 0.5235 0.2149 0.7226 0.2263 0.7842 0.8959 0.9402
DeepSFM | 0.1353,3.8738,2.2566,18.1695,0.2085,0.8909,0.9722,0.9836

Scenes
DPSNet | 0.0558 0.2430 0.1435 0.7136 0.1396 0.9502 0.9726 0.9804
DeepSFM | 0.0668,1.0298,0.2532,1.9122,0.1057,0.9621,0.9906,0.9961

I used ground truth poses.txt and *.npy depth files, which should give a better performance.
But from the above comparison, I found in the first 5 metrics (lower better), at most cases, DeepSFM is worse than DPS, especially DeepSFM on RGBD.
This seems a bit wired.
Is there any issues about this problem or could you give me some tips as I may miss something?

Hello.!

I'm wondering why are the relative poses in pose_train.py and pose_sequence_folder.py and pose_test.py are inconsistent.

For example, exp_pose ,pose_rel , and pose_rel _pd.

1. exp_pose = torch.matmul(inv(pose[:, j]), noise_pose[:, j]) in pose_train.py

2. pose_rel = pose_src @ np.linalg.inv(pose_tgt) in pose_sequence_folder.py

3. pose_rel_pd = pose_src_pd @ np.linalg.inv(pose_tgt_pd) in pose_sequence_folder.py

4. p_rel_pose[:, j] = torch.matmul(noise_pose[:, j], inv(p_matrix)) in pose_test.py

I think the matrix multiplication to find out relative poses are should be inv(camera_1_pose) X (camera_2_pose).

But Why the multiplication's order is not consistent?

Please answer it. Thanks. :)

Hi, thank you for releasing the source code for such great work. I am new in this field and please accept my apologies for asking very basic questions.

You told in issue #6, to run DeMoN to get poses_demon.txt and respective depth maps. However, it seems that the DeMoN evaluation (for image-1 as reference and image-2 as source images ) only outputs the depth map for the first image and the relative pose for the second image. Can you please elaborate on how to generate the depth map and pose for the second image? e.g. by simply swapping the input image pair?

Second, in the provided examples (poses.txt), there are two poses there, can you please elaborate which pose corresponds to which image, i.e., line-1 in poses.txt correspond to the relative pose of image-1 with reference to image-2?

Third, the translation values in the poses.txt seem to be normalized between (0,1), which in the ground truth of MVS dataset are in higher values. Can you please elaborate on how they are normalized? or what pre-processing is used?

Thank you.

Hi, recently I study the code of DPSNet and yours DeepSFM, and I found when running the training code, it will cost the GPU memory too much. For example, it will cost 20+G on V100 with batch size= 1 with only the PSNet , which is so weird. I think this maybe concerned with the version of pytorch, because DPSNet and DeepSFM all required pytorch 0.4.0 but I use the pytorch 1.6.0, So I want to confirm that the exactly version of pytorch is important to reporduce the expriment.