I have trained VPGNet, but i don't know how to validate the codel by visualizing outputs. Do you smaple code to work on each image and visualize lane detection output? Kindly shared the code if you have.

Issue summary

Make runtest failed on Ubuntu 16.04 and CUDA 10.0

*** Aborted at 1573612683 (unix time) try "date -d @1573612683" if you are using GNU date ***
PC: @ 0x7fad4225218e caffe::CuDNNConvolutionLayer<>::LayerSetUp()
*** SIGFPE (@0x7fad4225218e) received by PID 27175 (TID 0x7fad4d5b9740) from PID 1109729678; stack trace: ***
@ 0x7fad41776390 (unknown)
@ 0x7fad4225218e caffe::CuDNNConvolutionLayer<>::LayerSetUp()
@ 0x4a10b4 caffe::Layer<>::SetUp()
@ 0x7b36d8 caffe::CuDNNConvolutionLayerTest_TestSimpleConvolutionCuDNN_Test<>::TestBody()
@ 0x88cc8c testing::internal::HandleSehExceptionsInMethodIfSupported<>()
@ 0x887d99 testing::internal::HandleExceptionsInMethodIfSupported<>()
@ 0x8736d4 testing::Test::Run()
@ 0x873ec6 testing::TestInfo::Run()
@ 0x874511 testing::TestCase::Run()
@ 0x8799bf testing::internal::UnitTestImpl::RunAllTests()
@ 0x88de69 testing::internal::HandleSehExceptionsInMethodIfSupported<>()
@ 0x888a4a testing::internal::HandleExceptionsInMethodIfSupported<>()
@ 0x8785ce testing::UnitTest::Run()
@ 0x496d9b main
@ 0x7fad413bb830 __libc_start_main
@ 0x496429 _start
@ 0x0 (unknown)
make: *** [Makefile:478: runtest] Floating point exception (core dumped)

Steps to reproduce

make clean
make all -j16
make test -j16
make runtest

Tried solutions

modify the makefile.config

## Refer to http://caffe.berkeleyvision.org/installation.html
# Contributions simplifying and improving our build system are welcome!

# cuDNN acceleration switch (uncomment to build with cuDNN).
USE_CUDNN := 1

# CPU-only switch (uncomment to build without GPU support).
# CPU_ONLY := 1

# To customize your choice of compiler, uncomment and set the following.
# N.B. the default for Linux is g++ and the default for OSX is clang++
# CUSTOM_CXX := g++

# CUDA directory contains bin/ and lib/ directories that we need.
CUDA_DIR := /usr/local/cuda-10.0
# On Ubuntu 14.04, if cuda tools are installed via
# "sudo apt-get install nvidia-cuda-toolkit" then use this instead:
# CUDA_DIR := /usr

# CUDA architecture setting: going with all of them.
# For CUDA < 6.0, comment the *_50 lines for compatibility.
CUDA_ARCH := -gencode arch=compute_30,code=sm_30 \
		-gencode arch=compute_35,code=sm_35 \
		-gencode arch=compute_50,code=sm_50 \
		-gencode arch=compute_52,code=sm_52 \
        	-gencode arch=compute_60,code=sm_60 \
        	-gencode arch=compute_61,code=sm_61 \
        	-gencode arch=compute_61,code=compute_61 

# BLAS choice:
# atlas for ATLAS (default)
# mkl for MKL
# open for OpenBlas
BLAS := atlas
# Custom (MKL/ATLAS/OpenBLAS) include and lib directories.
# Leave commented to accept the defaults for your choice of BLAS
# (which should work)!
# BLAS_INCLUDE := /path/to/your/blas
# BLAS_LIB := /path/to/your/blas

# Homebrew puts openblas in a directory that is not on the standard search path
# BLAS_INCLUDE := $(shell brew --prefix openblas)/include
# BLAS_LIB := $(shell brew --prefix openblas)/lib

# This is required only if you will compile the matlab interface.
# MATLAB directory should contain the mex binary in /bin.
MATLAB_DIR := /usr/local/MATLAB/R2018b
# MATLAB_DIR := /Applications/MATLAB_R2012b.app

# NOTE: this is required only if you will compile the python interface.
# We need to be able to find Python.h and numpy/arrayobject.h.
PYTHON_INCLUDE := /usr/include/python3.5 \
		/home/liying/.local/lib/python3.5/site-packages/numpy/core/include/numpy 	
# Anaconda Python distribution is quite popular. Include path:
# Verify anaconda location, sometimes it's in root.
# ANACONDA_HOME := $(HOME)/anaconda
# PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
		# $(ANACONDA_HOME)/include/python2.7 \
		# $(ANACONDA_HOME)/lib/python2.7/site-packages/numpy/core/include \

# We need to be able to find libpythonX.X.so or .dylib.
PYTHON_LIB := /usr/lib
# PYTHON_LIB := $(ANACONDA_HOME)/lib

# Homebrew installs numpy in a non standard path (keg only)
# PYTHON_INCLUDE += $(dir $(shell python -c 'import numpy.core; print(numpy.core.__file__)'))/include
# PYTHON_LIB += $(shell brew --prefix numpy)/lib

# Uncomment to support layers written in Python (will link against Python libs)
# WITH_PYTHON_LAYER := 1

# Whatever else you find you need goes here.
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial /usr/local/include/opencv /usr/local/include/opencv2
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu/hdf5/serial

# If Homebrew is installed at a non standard location (for example your home directory) and you use it for general dependencies
# INCLUDE_DIRS += $(shell brew --prefix)/include
# LIBRARY_DIRS += $(shell brew --prefix)/lib

# Uncomment to use `pkg-config` to specify OpenCV library paths.
# (Usually not necessary -- OpenCV libraries are normally installed in one of the above $LIBRARY_DIRS.)
USE_PKG_CONFIG := 1

BUILD_DIR := build
DISTRIBUTE_DIR := distribute

# Uncomment for debugging. Does not work on OSX due to https://github.com/BVLC/caffe/issues/171
# DEBUG := 1

# The ID of the GPU that 'make runtest' will use to run unit tests.
TEST_GPUID := 0

# enable pretty build (comment to see full commands)
Q ?= @

System configuration

• Operating system: Ubuntu 16.04
• Compiler: GNU make 4.2
• CUDA version (if applicable): CUDA 10.0
• CUDNN version (if applicable): CUDNN 7.6.2
• BLAS: ATLAS
• Python version (if using pycaffe): python 3.5
• MATLAB version (if using matcaffe): R2018b

Our ENet-Label-Torch has been released. More details can be found in my repo.

Key features:

(1) ENet-label is a light-weight lane detection model based on ENet and adopts self attention distillation (more details can be found in our paper which will be published soon).

(2) It has 20 × fewer parameters and runs 10 × faster compared to the state-of-the-art SCNN, and achieves 72.0 (F1-measure) on CULane testing set (better than SCNN which achieves 71.6).

(Do not hesitate to try our model!!!)

Performance on CULane testing set (F1-measure):

I have implemented SCNN using Tensorflow and put the full codes here. You can test the code in popular lane detection benchmarks like TuSimple, CULane and BDD100K or your custom dataset with minor modification. Welcome to raising issues if you have problems in reproducing the results. My code is based on LaneNet and SCNN-Torch.

I have a research on the lane detection , and do some work about it , could you open your code, that i can do some test on it , thank you !

I am now doing the evaluation of the results ,in your paper you said: we compute the minimum distance from the center of each cell to the sampled lane points for every cell. If the minimum distance is within the boundary R, we mark these sampled points as true positive and the corresponding grid cell as detected.
how about the R and the class FP?

Hi,

Can you share a pretrained model, please?

Hi @SeokjuLee ,
Thanks you for your amazing works.

Would you mind releasing the model for caltech-lane dataset or your dataset ?
I would like to use it as pre-trained model in my personal side-project. Please~ :)

How can I find each label name of 64 multi-labels?

@SeokjuLee

How can I use my own dataset ? My dataset is some pictures with polygon labels (pixel-level , store in .json with the polygon's coordinates value ). I tried to figure out how you deal with the original pictures with labels, but I ... emmmm, I'm still confused with your implement of DriveData ...

So can I use my own data in a simple way ?

Thx a lot ~

Hi,

I'm a little confused since most of the files are solely about caffe. Is there a way to test the vpgnet on an image or video from other data sets (kitti, culane, etc)? If so, which script would I run, with the image/video as a file input?

Hi, SeokjuLee, Recently, I am also doing some job about lane and road mask detection and reconnition. I am so amazing about your job about it. Now I have a question to ask you for some help. How do you annotate you trainning data set ? what you steps are?

Hi, I found in your paper you declared your algorithm speed could achieve to 20Fps, so I am confused whether the speed you declared is just the speed of the network inference time or with the post process included clustering and curve fitting? If it is with the post-process, could you share your post process code? Thanks for your good job!

@SeokjuLee
My equipment configuration does not allow me to train a model and I'm very new to computer vision and deep-learning things. I will be very appreciate if there's a well-trained model available. Not mean to be lazy ^-^

@SeokjuLee
VPGNet/caffe/tools/convert_driving_data.cpp
[line 115]
int resize_height = std::max(0, FLAGS_resize_height); ///480+32=512
int resize_width = std::max(0, FLAGS_resize_width); ///640+32=672

original images are resized to 672*512, should coordinates of lines be reseized also??

[line 96]
lines.push_back(std::make_pair(filename, boxes));

I was able to run make_lmdb.sh with cuda8, however, this shell gives me this error when running with cuda 9:

E0731 17:34:52.597481 20328 common.cpp:103] Cannot create Cublas handle. Cublas won't be available.
E0731 17:34:52.653252 20328 common.cpp:110] Cannot create Curand generator. Curand won't be available.

I'm not sure if this is the problem of cuda version, or something else.
This error alone does not prevent the shell from running, but it makes train.sh aborted:

./train.sh: line 1: 20354 Aborted                 (core dumped) ../../build/tools/caffe train --solver=./solver.prototxt >> ./output/output.log 2>&1

Any thoughts?
Thank you!

In the data layer, layer output is list as below:
data -> data 32 3 480 640 (29491200)
data -> label 32 8 120 160 (4915200)
data -> type 32 1 60 80 (153600)

I know the shape of "type" is depended on "catalog_resolution"
But I can't find out where shape of "label" is depended.
If I want to change the output shape of binary-mask, I should change shape of "label" firstly, how can I change "data -> label 32 8 120 160 (4915200)" to "data -> label 32 8 240 320 (4915200)"?

VPGNet dataset is available.
Please fill out a form for the download link, and check the README.
Thanks.

which program can be use to test picture

I want to know how to test it? I just finish the train.Thanks.

@SeokjuLee hi，In section 4.4 of this paper，how to understand the process of point sampling：“First, we subsample local peaks from the region where the probability of lane channels from the multi-label task is high.”，such as：
1、how to find "the region"? in one of the categories's feature map (60×80) or in original image (640×480)？
2、how to understand the peaks？what does its horizontal and vertical coordinates stand for？

thank you very much.

Several days ago, I reproduced the result of VPGNet and I put some frequently met installation problems here to facilitate other researchers:

1. Compile caffe:
   If you use anaconda python, please uncomment the corresponding lines in Makefile.config (Line 55 ~ 58) and DO NOT uncomment Line 5 (# USE_CUDNN := 1) since current version of VPGNet does not use cudnn.

2. Run make_lmdb.sh:
   If you meet problems about cudnn, please recompile caffe without cudnn. Besides, in my case, I need to write the absolute path of LMDB_train and LMDB_test in make_lmdb.sh.

Regards,
Cardwing

The output size of multi label is 60 * 80, but the output size of grid box and object mask is 120 * 180. Can you explain the reason for this setting and how to generate grid box and object box through 8 * 8 grid? Thank you again for your work!

Hello everyone.

I would like to know how I can visualize the output of the multi-label with an example image like it is done here:

#8 (comment)

Please ask installation, training and test issues in this panel.

Hi everyone
I want to try to redo SeokjuLee's VPGNet Project. I try to compile the caffe and i failed. But when i try to compile caffe from https://github.com/BVLC/caffe.git i found no problem and success. I figure the problem is because some modification in SeokjuLee's Caffe. there are some variables that are not declared so the compiling is failed. the content of some file also different from BVLC's Caffe
I'm using CUDA toolkit 10, Opencv 4, and Python 3.8
For anyone who have tried to redo this project, could you send me your caffe file? or is it because different version of my Opencv?

Dear authors,
#1 Can you show me how to perform training on your VPGNet dataset?
I don't know how to generate the dataset_list file of your VPGNet dataset in order to run the make_lmbd.sh

#2 The Caltech Lanes dataset seems to be removed from the official website. Can you share a snapshot of the dataset.

Thank you very much

I am implementing VPGNET in keras. Task 2,3,4 are classification tasks, so they are learning and inferring okay. But grid box regression task isn't learning. It has a constant loss. I am using linear activation function on the last layer and using Mean Absolute error L1 loss function for bounding box regression task as per paper. I am not normalising any bounding box coordinates, so their range is 0-640 (width) and 0-480 height. The training MAE loss starts off really high, 17.5 to be exact and then stays there.

I tried to edit the make_lmdb.sh file as following:
../../build/tools/convert_driving_data /media/aquarius/Backup/VPGNet/caltech-lanes-dataset/cordova1/ /media/aquarius/Backup/VPGNet/caltech-lanes-dataset/cordova1/list.txt LMDB_train
../../build/tools/compute_driving_mean LMDB_train ./driving_mean_train.binaryproto lmd
../../build/tools/convert_driving_data /media/aquarius/Backup/VPGNet/caltech-lanes-dataset/cordova2/ /media/aquarius/Backup/VPGNet/caltech-lanes-dataset/cordova2/list.txt LMDB_test
but it seem doesn't work.
Here is my output:
./make_lmdb.sh
E1227 16:49:05.001368 4610 convert_driving_data.cpp:73] argv[1]: /media/aquarius/Backup/VPGNet/caltech-lanes-dataset/cordova1/
E1227 16:49:05.001581 4610 convert_driving_data.cpp:74] argv[2]: /media/aquarius/Backup/VPGNet/caltech-lanes-dataset/cordova1/list.txt
E1227 16:49:05.264470 4610 convert_driving_data.cpp:145] Total to be processed: 0.
F1227 16:49:05.354425 4617 compute_driving_mean.cpp:68] Unknown db backend lmd
*** Check failure stack trace: ***
@ 0x7f38d4fba5cd google::LogMessage::Fail()
@ 0x7f38d4fbc433 google::LogMessage::SendToLog()
@ 0x7f38d4fba15b google::LogMessage::Flush()
@ 0x7f38d4fbce1e google::LogMessageFatal::~LogMessageFatal()
@ 0x402c73 main
@ 0x7f38d3ed9830 __libc_start_main
@ 0x402f39 _start
@ (nil) (unknown)
./make_lmdb.sh: line 7: 4617 Aborted (core dumped) ../../build/tools/compute_driving_mean LMDB_train ./driving_mean_train.binaryproto lmd
E1227 16:49:05.457511 4619 convert_driving_data.cpp:73] argv[1]: /media/aquarius/Backup/VPGNet/caltech-lanes-dataset/cordova2/
E1227 16:49:05.457718 4619 convert_driving_data.cpp:74] argv[2]: /media/aquarius/Backup/VPGNet/caltech-lanes-dataset/cordova2/list.txt
E1227 16:49:05.694105 4619 convert_driving_data.cpp:145] Total to be processed: 0.

Could you explain to me more details about this process? I am a newbie in caffe so it took me more than 3 days to fix this and nothing happened.
Thank you in advance.

hello, @SeokjuLee ,when trying to bash make_lmdb.sh,it always reminds me of such mistakes. What should I do to solve them?
compute_driving_mean.cpp:58] Check failed: mdb_env_open(mdb_env, argv[1], 0x20000, 0664) == 0 (2 vs. 0) mdb_env_open failed

Hi @SeokjuLee,

I have been using VPGNet to do some lane detection on our own dataset. For the test loss, we are getting around 3.5. I'm just curious what would be a good number for the test loss?

Thanks!

I am looking forward to meet the dataset. But I don't understand the grid-level annotations, what means?

@SeokjuLee Thanks for sharing this awesome work.
I have run the caffe version on a Tesla M40 GPU，CUDA version 8.0，cuDNN version v5. when compile it，there is an error：
./include/caffe/util/cudnn.hpp:124:41: error: too few arguments to function ‘cudnnStatus_t cudnnSetPooling2dDescriptor(cudnnPoolingDescriptor_t, cudnnPoolingMode_t, cudnnNanPropagation_t, int, int, int, int, int, int)’pad_h, pad_w, stride_h, stride_w));
solution：
1、Because the cuDNN head file is too old，so replace the file ./include/caffe/util/cudnn.hpp with /usr/local/cuda/include/cudnn.h
2、 editing "caffe/cmake/Dependencies.cmake", at line 29
replace list(APPEND Caffe_LINKER_LIBS ${HDF5_LIBRARIES}) with list(APPEND Caffe_LINKER_LIBS ${HDF5_LIBRARIES} ${HDF5_HL_LIBRARIES})

I have trained a caffemodel from your tutorial, but the test result is bad. I think it is because of little dataset. Could you please tell me where I can download the lane and road marking benchmark which consists of about 20,000 images or give me the caffemodel described in your paper? Thank you very much!

Is the caffe used in this project different from the standard caffe?(do your add some special layers?)
can i replace the caffe in this project with the newest caffe?

Hello,I have finished train and test. Here is a output.log of test.How can I visualize test results without post-process.

Hi, SeokjuLee
Thanks for your remarkable work.
I download your code from here and implemented training and deploying according to your Baseline Usage step by step completely.
I Compiled your caffe successfully on my ubuntu16.04.
I used all of your Caltech Lanes Dataset for training.
And of course no any change of your training and deploying prototxt and any other files in VPGnet.

But the 64X60X80 output of my 'multi-label' was not able to detected anything.
The first 60X80 channel of 'multi-label' is the background class. Each elements in this channel is above 0.999 at least (training for 3000 steps). And I know it's meaning that nothing to be detected by VPGnet.
Nothing to be detected, but by the training log, the pixel-acc and type-acc are all above 0.98. Does it meaning that Label data is nothing? But the training data preparation was according to your Baseline Usage step by step completely.

I have confused by this problems for several days, is there anything I missed?
I tried training and deploying according to your Baseline Usage step by step completely and didn't changed any files in VPGnet.

Did you meet this kind of problem? Or is there something missed in my implementation?

does somebody have get the dataset and pretrained model ?

I have trained the net two times with similar data which has two different type labels .One of the two type is Lane,conclude broken yellow, broken white and so on.Another type is about roadmarking,conclude straight arrow,right arrow and so on.But the test results were so strange.The lane detection result was good,such as pic 1.But the detection of roadmarking was beyond my expectation，ａlmost nothing cｏｕｌｄ be detected，such as pic 2. I want to know why are there such a big difference. Can you explain the reason for me？ Thanks a lot @SeokjuLee

ｐｉｃ　１

ｐｉｃ　２

Hi,
I'm getting the below error while I try to train the model. I checked in internet, the sources say that it can happen because of multiple cuda versions. I delete other cudas and tried, the issue still persists. Can you please help me get through this?

F0402 17:15:39.522164 20718 math_functions.cu:422] Check failed: status == CURAND_STATUS_SUCCESS (201 vs. 0) CURAND_STATUS_LAUNCH_FAILURE
*** Check failure stack trace: ***
@ 0x7f4e3e0b35cd google::LogMessage::Fail()
@ 0x7f4e3e0b5433 google::LogMessage::SendToLog()
@ 0x7f4e3e0b315b google::LogMessage::Flush()
@ 0x7f4e3e0b5e1e google::LogMessageFatal::~LogMessageFatal()
@ 0x7f4e3e80703a caffe::caffe_gpu_rng_uniform()
@ 0x7f4e3e80b2d2 caffe::DropoutLayer<>::Forward_gpu()
@ 0x7f4e3e7cfea2 caffe::Net<>::ForwardFromTo()
@ 0x7f4e3e7cffc7 caffe::Net<>::ForwardPrefilled()
@ 0x7f4e3e7f498d caffe::Solver<>::Step()
@ 0x7f4e3e7f551a caffe::Solver<>::Solve()
@ 0x40bbfb train()
@ 0x408408 main
@ 0x7f4e3d53e830 __libc_start_main
@ 0x408ab9 _start
@ (nil) (unknown)
Aborted (core dumped)

Here is an additional explanation about the lane post-processing. There are four steps basically. First, sampling seed points with the lane heat map from the multi-label task. Then, cluster the seeds in the IPM-ed coordinate with our clustering method (“We sequentially decide the cluster by the pixel distance. After sorting the points by the vertical index, we stack the point in a bin if there is a close point among the top of the existing bins. Otherwise, we create a new bin for a new cluster. By doing this, we can reduce the time complexity of the clustering.”). Lastly, we did polynomial line fitting for each cluster. If the cluster is near the VP, we include that VP while clustering for the stability. Related contents are described in our paper, Section 4.4.

Hi seokju,

I have similar problem like the picture below

Even though I tried SCNN but I believe the postprocessing step is similar.
below is my result

the model confuse itself to see points from different lanes. Thus I want to cluster all points again using IPM to seperate the points near VP.

May i know which 4 src points did you use in cv2.getperspective() function. Since the prediction of mine has a large variety, I have some trouble finding these 4 points which encloses all points found.
Could you share some insight how you find these 4 points ?

Best regards,

ZX

as the paper said. can the first task VPP be provide?

I would like to know which version of cudnn you are using???

CXX/LD -o .build_release/tools/compute_image_mean.bin
.build_release/lib/libcaffe.so: undefined reference to void caffe::hdf5_load_nd_dataset<double>(int, char const*, int, int, caffe::Blob<double>*)' .build_release/lib/libcaffe.so: undefined reference to void caffe::hdf5_load_nd_dataset(int, char const*, int, int, caffe::Blob*)'
.build_release/lib/libcaffe.so: undefined reference to caffe::hdf5_get_name_by_idx[abi:cxx11](int, int)' .build_release/lib/libcaffe.so: undefined reference to void caffe::hdf5_save_nd_dataset(int, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, caffe::Blob const&, bool)'
.build_release/lib/libcaffe.so: undefined reference to caffe::hdf5_get_num_links(int)' .build_release/lib/libcaffe.so: undefined reference to void caffe::hdf5_save_nd_dataset(int, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, caffe::Blob const&, bool)'
collect2: error: ld returned 1 exit status
Makefile:568: recipe for target '.build_release/tools/compute_image_mean.bin' failed
make: *** [.build_release/tools/compute_image_mean.bin] Error 1

Hi, I want to run the make_lmdb.sh file, but I got the following errors:
make_lmdb.sh: line 3: ../../build/tools/convert_driving_data: No such file or directory
make_lmdb.sh: line 4: ../../build/tools/compute_driving_mean: No such file or directory
make_lmdb.sh: line 6: ../../build/tools/convert_driving_data: No such file or directory

I checked the directories and find out that there is no folder named build, and the tools folder is under caffe. So I want to ask what does ../../build mean?

BTW, I use windows so I run this file on Git.

src/caffe/layers/contrastive_loss_layer.cpp: In instantiation of ‘void caffe::ContrastiveLossLayer::Forward_cpu(const std::vector<caffe::Blob>&, const std::vector<caffe::Blob>&) [with Dtype = float]’:
src/caffe/layers/contrastive_loss_layer.cpp:118:1: required from here
src/caffe/layers/contrastive_loss_layer.cpp:56:30: error: no matching function for call to ‘max(float, double)’

Hi @SeokjuLee

Thanks you for your wonderful works.

Would you mind posting a license? (such as apache-license-2.0) I would like to use your code in my work.

Thank you very much.