Benjamin Graham, David Novotny
3DV 2020

This is the official implementation of RidgeSfM: Structure from Motion via Robust Pairwise Matching Under Depth Uncertainty in PyTorch.

Link to paper | Poster

Below we illustrate the depth uncertainty factors of variation for a frame from scene 0708.

Top left: an input image. Bottom left: the predicted depth. Middle and right: We use SVD to reduce the 32 FoV planes down to 12 planes, and display them as 4 RGB images; each of the 4x3 color planes represents one factor of variation.

We applied RidgeSfM to a short video taken using a mobile phone camera. There is no ground truth pose, so the bottom right hand corner of the video is blank.

We trained a depth prediction network on the KITTI depth prediction training set. We then processed videos from the KITTI Visual Odometry dataset. We used the 'camera 2' image sequences, cropping the input to RGB images of size 1216x320. We used R2D2 as the keypoint detector. We used a frame skip rate of k=3. The scenes are larger spatially, so for visualization we increased the number of K-Means centroids to one million.

• Download SuperPoint to ridgesfm/

wget https://github.com/magicleap/SuperGluePretrainedNetwork/blob/master/models/weights/superpoint_v1.pth?raw=true -O ridgesfm/weights/superpoint_v1.pth
wget https://raw.githubusercontent.com/magicleap/SuperGluePretrainedNetwork/master/models/superpoint.py -O ridgesfm/superpoint.py

• To download one test scene:
  • Download scannet.zip to RidgeSfM/ridgesfm/data and check that RidgeSfM/ridgesfm/data/scannet/test_scenes/0707_00/seq=0707_00-maxframes=300-frameskip=10-start=1.pth exists.
  • Run python ridgesfm.py scene.n=0 scene.frameskip=10
• To prepare the full ScanNet dataset:
  • Download the ScanNet dataset to ridgesfm/data/scannet_sens/[train|test]/
  • Download SensReader to ridgesfm/data
  • Run bash prepare_scannet.sh in ridgesfm/data/
  • Run python ridgesfm.py scene.n=x scene.frameskip=y with x = 0,1,...,98 or 99 and y = 1,3,10 or 30

• calibrate your camera using calibrate/calibrate.ipynb
• then run python ridgesfm.py scenes=calibrate/ scene.n=0 scene.frameskip=10

• Python 3.7+
• PyTorch and TorchVision
• Faiss
• PyKeOps
• OpenCV
• imageio and imageio-ffmpeg
• PyTorch3D

RidgeSfM is CC-BY-NC licensed, as found in the LICENSE file. Terms of use. Privacy

If you find this code useful in your research then please cite:

@InProceedings{ridgesfm2020,
    author       = "Benjamin Graham and David Novotny",
    title        = "Ridge{S}f{M}: Structure from Motion via Robust Pairwise Matching Under Depth Uncertainty",
    booktitle    = "International Conference on 3D Vision (3DV)",
    year         = "2020",
}