Segmentation-Guided Self-Supervised Monocular Depth Estimation

In this project, semantic segmentation is used as a booster to the depth estimation task. Segmentation masks that separate foreground objects from background are created from full scale disparity maps through a U-net architecture. In addition to minimum reprojection loss and edge-aware smoothness loss, pixel-wise weighted cross entropy loss is used as an additional supervision signal.

You can find the dataset (KITTI raw) from this link

You can find the pretrained baseline and SegDepth network on KITTI raw subset dataset from this link

I performed all experiments on NVIDIA Geforce 1050-TI Training baseline model on eigen_zhou_subset took 2.5 hours approximately. Training improved model on eigen_zhou_subset took 7 hours approximately.

Splits

I used subset of eigen_zhou_subset split. You can find it under splits/ folder.

In this section, necessary instructions to run scripts are described.

Warning

Before downloading the dataset (step 1), you will get an error if you try to run a script below. Since downloading all the dataset is time consuming, you can skip to Toy Examples section to run with a very few samples that are already in kitti_data folder without any error.

You can download training subset data by typing the commands below:

wget -i splits/kitti_archives_to_download_subset.txt -P kitti_data/
cd kitti_data
unzip "*.zip"
cd ..

For creating segmentation masks, create and use the conda environment specified below:

conda create -n csm python=3.6.6
conda activate csm
pip install torch torchvision
pip install opencv-python
pip install numpy
python create_segmentation_masks.py

Follow the steps below:

conda create -n segdepth python=3.6.6
conda activate segdepth
conda install pytorch=0.4.1 torchvision=0.2.1 -c pytorch
conda install scikit-image
conda install opencv=3.3.1

By default models are saved to tmp/<model_name>. This can be changed with the --log_dir flag.

python train.py --model_name mono_model

You can predict depth for a single image with (change model path according to your case):

python test_simple.py --image_path assets/test_image.jpg --model_name mono_model --model_path tmp/mono_model/models/weights_19/

To prepare the ground truth depth maps run:

python export_gt_depth.py --data_path kitti_data --split eigen

...assuming that you have placed the KITTI dataset in the default location of ./kitti_data/.

The following example command evaluates the epoch 19 weights of a model named mono_model:

python evaluate_depth.py --load_weights_folder tmp/mono_model/models/weights_19/ --eval_mono

In this section, you can run the scripts above with a very few data without any downloading process.

Follow the steps below:

conda create -n segdepth python=3.6.6
conda activate segdepth
conda install pytorch=0.4.1 torchvision=0.2.1 -c pytorch
conda install scikit-image
conda install opencv=3.3.1
python train.py --model_name mono_model --example_run --batch_size 1

To prepare the ground truth depth maps run:

python export_gt_depth.py --data_path kitti_data --split eigen --test_file test_files_example.txt

assuming that you have placed the KITTI dataset in the default location of ./kitti_data/.

The following example command evaluates the epoch 19 weights of a model named mono_model:

python evaluate_depth.py --load_weights_folder tmp/mono_model/models/weights_19/ --eval_mono --test_file test_files_example.txt