libfuzzer_example_test fails on bionic about libprotobuf-mutator HOT 7 OPEN

I had the same problem with failing tests (due to zero exit code instead of 77). It's because the example bugs are not triggered. I was using clang 8.0.0 so had no issues with the len_control flag.

I found that the tests only pass if I build with -DLIB_PROTO_MUTATOR_WITH_ASAN=ON as per the travis CI builds: https://travis-ci.org/google/libprotobuf-mutator/jobs/533188543/config

I'm not sure why this is the case.

from libprotobuf-mutator.

For me those tests fail on ubuntu 18.04 with clang 8.0.0 installed. The len_control flag is available there. As I understand, -DLIB_PROTO_MUTATOR_WITH_ASAN=ON instruments the protobuf mutator itself, so that the fuzzer picks up coverage information there too. While I do not think this is intended behaviour for regular fuzzing jobs, this also lets the tests pass for me.

Still I think this Issue is a major bug, since somehow the coverage guidance in these tests does not seem to work.

from libprotobuf-mutator.

So I did some further research on the issue and in order for the test to succeed (and therefore fuzzing to work as intended) the -fsanitize=address flag has to be added on the libfuzzer executables at link time, see:
master...dende:master

I still do not think this is intended behaviour and I am not sure whether I had these issues when I last checked out the project around November 2018

Edit: Well actually the tests succeed with either of -fsanitize=address,thread,memory.

While with address and thread the example bug gets triggered, the memory sanitizer complains about something else:

Uninitialized bytes in __interceptor_memcmp at offset 15 inside [0x7030000016b0, 21)
==12166==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x7fbd5be3abc7 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::compare(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0x126bc7)
    #1 0x5cc7ac in bool std::operator<<char, std::char_traits<char>, std::allocator<char> >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (/home/c/git/libprotobuf-mutator/build/examples/libfuzzer/libfuzzer_bin_example+0x5cc7ac)
    #2 0x5cc600 in std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const (/home/c/git/libprotobuf-mutator/build/examples/libfuzzer/libfuzzer_bin_example+0x5cc600)
    #3 0x6417f3 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::pair<void const*, int> >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::pair<void const*, int> > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::pair<void const*, int> > > >::_M_upper_bound(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::pair<void const*, int> > >*, std::_Rb_tree_node_base*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (/home/c/git/libprotobuf-mutator/build/examples/libfuzzer/libfuzzer_bin_example+0x6417f3)
    #4 0x641790 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::pair<void const*, int> >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::pair<void const*, int> > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::pair<void const*, int> > > >::upper_bound(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (/home/c/git/libprotobuf-mutator/build/examples/libfuzzer/libfuzzer_bin_example+0x641790)
    #5 0x64170c in std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<void const*, int>, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::pair<void const*, int> > > >::upper_bound(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (/home/c/git/libprotobuf-mutator/build/examples/libfuzzer/libfuzzer_bin_example+0x64170c)
    #6 0x64144a in google::protobuf::SimpleDescriptorDatabase::DescriptorIndex<std::pair<void const*, int> >::FindLastLessOrEqual(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (/home/c/git/libprotobuf-mutator/build/examples/libfuzzer/libfuzzer_bin_example+0x64144a)
    #7 0x6402ac in google::protobuf::SimpleDescriptorDatabase::DescriptorIndex<std::pair<void const*, int> >::AddSymbol(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::pair<void const*, int>) (/home/c/git/libprotobuf-mutator/build/examples/libfuzzer/libfuzzer_bin_example+0x6402ac)
    #8 0x63868a in google::protobuf::SimpleDescriptorDatabase::DescriptorIndex<std::pair<void const*, int> >::AddFile(google::protobuf::FileDescriptorProto const&, std::pair<void const*, int>) (/home/c/git/libprotobuf-mutator/build/examples/libfuzzer/libfuzzer_bin_example+0x63868a)
    #9 0x636749 in google::protobuf::EncodedDescriptorDatabase::Add(void const*, int) (/home/c/git/libprotobuf-mutator/build/examples/libfuzzer/libfuzzer_bin_example+0x636749)
    #10 0x5937d2 in google::protobuf::DescriptorPool::InternalAddGeneratedFile(void const*, int) (/home/c/git/libprotobuf-mutator/build/examples/libfuzzer/libfuzzer_bin_example+0x5937d2)
    #11 0x65898f in google::protobuf::(anonymous namespace)::AddDescriptorsImpl(google::protobuf::internal::DescriptorTable const*) (/home/c/git/libprotobuf-mutator/build/examples/libfuzzer/libfuzzer_bin_example+0x65898f)
    #12 0x6588d3 in google::protobuf::internal::AddDescriptors(google::protobuf::internal::DescriptorTable const*) (/home/c/git/libprotobuf-mutator/build/examples/libfuzzer/libfuzzer_bin_example+0x6588d3)
    #13 0x421b5f in __cxx_global_var_init.23 (/home/c/git/libprotobuf-mutator/build/examples/libfuzzer/libfuzzer_bin_example+0x421b5f)
    #14 0x421b7d in _GLOBAL__sub_I_descriptor.pb.cc (/home/c/git/libprotobuf-mutator/build/examples/libfuzzer/libfuzzer_bin_example+0x421b7d)
    #15 0x6ebc9c in __libc_csu_init (/home/c/git/libprotobuf-mutator/build/examples/libfuzzer/libfuzzer_bin_example+0x6ebc9c)
    #16 0x7fbd5ad63b27 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:266
    #17 0x422349 in _start (/home/c/git/libprotobuf-mutator/build/examples/libfuzzer/libfuzzer_bin_example+0x422349)

SUMMARY: MemorySanitizer: use-of-uninitialized-value (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0x126bc7) in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::compare(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const
Exiting

from libprotobuf-mutator.

I wonder if this test can be safely ignored or there might be something wrong if I ignore this.

from libprotobuf-mutator.

Tests also failing on ArchLinux with clang 9.0.0 installed.
I also think this Issue is a major bug, as dende mentioned.

I also run the test on clang 10.0.0. and it also failed.

`Running tests...
Test project /home/sirko/Projects/libprotobuf-mutator/build
Start 1: test.protobuf_mutator_test_0
1/20 Test #1: test.protobuf_mutator_test_0 ..... Passed 4.38 sec
Start 2: test.protobuf_mutator_test_1
2/20 Test #2: test.protobuf_mutator_test_1 ..... Passed 2.36 sec
Start 3: test.protobuf_mutator_test_2
3/20 Test #3: test.protobuf_mutator_test_2 ..... Passed 3.16 sec
Start 4: test.protobuf_mutator_test_3
4/20 Test #4: test.protobuf_mutator_test_3 ..... Passed 9.45 sec
Start 5: test.protobuf_mutator_test_4
5/20 Test #5: test.protobuf_mutator_test_4 ..... Passed 9.24 sec
Start 6: test.protobuf_mutator_test_5
6/20 Test #6: test.protobuf_mutator_test_5 ..... Passed 6.10 sec
Start 7: test.protobuf_mutator_test_6
7/20 Test #7: test.protobuf_mutator_test_6 ..... Passed 4.09 sec
Start 8: test.protobuf_mutator_test_7
8/20 Test #8: test.protobuf_mutator_test_7 ..... Passed 6.83 sec
Start 9: test.protobuf_mutator_test_8
9/20 Test #9: test.protobuf_mutator_test_8 ..... Passed 3.79 sec
Start 10: test.protobuf_mutator_test_9
10/20 Test #10: test.protobuf_mutator_test_9 ..... Passed 23.76 sec
Start 11: test.protobuf_mutator_test_10
11/20 Test #11: test.protobuf_mutator_test_10 .... Passed 5.03 sec
Start 12: test.protobuf_mutator_test_11
12/20 Test #12: test.protobuf_mutator_test_11 .... Passed 4.92 sec
Start 13: test.protobuf_mutator_test_12
13/20 Test #13: test.protobuf_mutator_test_12 .... Passed 9.52 sec
Start 14: test.protobuf_mutator_test_13
14/20 Test #14: test.protobuf_mutator_test_13 .... Passed 5.14 sec
Start 15: test.protobuf_mutator_test_14
15/20 Test #15: test.protobuf_mutator_test_14 .... Passed 8.18 sec
Start 16: test.protobuf_mutator_test_15
16/20 Test #16: test.protobuf_mutator_test_15 .... Passed 15.24 sec
Start 17: test.protobuf_libfuzzer_test
17/20 Test #17: test.protobuf_libfuzzer_test ..... Passed 0.04 sec
Start 18: test.expat_example_test
Could not find executable expat_example_test
Looked in the following places:
expat_example_test
expat_example_test
Release/expat_example_test
Release/expat_example_test
Debug/expat_example_test
Debug/expat_example_test
MinSizeRel/expat_example_test
MinSizeRel/expat_example_test
RelWithDebInfo/expat_example_test
RelWithDebInfo/expat_example_test
Deployment/expat_example_test
Deployment/expat_example_test
Development/expat_example_test
Development/expat_example_test
Unable to find executable: expat_example_test
18/20 Test #18: test.expat_example_test ..........***Not Run 0.00 sec
Start 19: test.libfuzzer_example_test
Could not find executable libfuzzer_example_test
Looked in the following places:
libfuzzer_example_test
libfuzzer_example_test
Release/libfuzzer_example_test
Release/libfuzzer_example_test
Debug/libfuzzer_example_test
Debug/libfuzzer_example_test
MinSizeRel/libfuzzer_example_test
MinSizeRel/libfuzzer_example_test
RelWithDebInfo/libfuzzer_example_test
RelWithDebInfo/libfuzzer_example_test
Deployment/libfuzzer_example_test
Deployment/libfuzzer_example_test
Development/libfuzzer_example_test
Development/libfuzzer_example_test
Unable to find executable: libfuzzer_example_test
19/20 Test #19: test.libfuzzer_example_test ......***Not Run 0.00 sec
Start 20: test.libxml2_example_test
Could not find executable libxml2_example_test
Looked in the following places:
libxml2_example_test
libxml2_example_test
Release/libxml2_example_test
Release/libxml2_example_test
Debug/libxml2_example_test
Debug/libxml2_example_test
MinSizeRel/libxml2_example_test
MinSizeRel/libxml2_example_test
RelWithDebInfo/libxml2_example_test
RelWithDebInfo/libxml2_example_test
Deployment/libxml2_example_test
Deployment/libxml2_example_test
Development/libxml2_example_test
Development/libxml2_example_test
Unable to find executable: libxml2_example_test
20/20 Test #20: test.libxml2_example_test ........***Not Run 0.00 sec

85% tests passed, 3 tests failed out of 20

Total Test time (real) = 121.25 sec

The following tests FAILED:
18 - test.expat_example_test (Not Run)
19 - test.libfuzzer_example_test (Not Run)
20 - test.libxml2_example_test (Not Run)
Errors while running CTest
make: *** [Makefile:108: test] Fehler 8
make test 107,43s user 0,17s system 88% cpu 2:01,38 total
`

from libprotobuf-mutator.

So I did some further research on the issue and in order for the test to succeed (and therefore fuzzing to work as intended) the -fsanitize=address flag has to be added on the libfuzzer executables at link time, see:
master...dende:master

I still do not think this is intended behaviour and I am not sure whether I had these issues when I last checked out the project around November 2018

Edit: Well actually the tests succeed with either of -fsanitize=address,thread,memory.

While with address and thread the example bug gets triggered, the memory sanitizer complains about something else:

Uninitialized bytes in __interceptor_memcmp at offset 15 inside [0x7030000016b0, 21)
==12166==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x7fbd5be3abc7 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::compare(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0x126bc7)
    #1 0x5cc7ac in bool std::operator<<char, std::char_traits<char>, std::allocator<char> >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (/home/c/git/libprotobuf-mutator/build/examples/libfuzzer/libfuzzer_bin_example+0x5cc7ac)
    #2 0x5cc600 in std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >::operator()(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const (/home/c/git/libprotobuf-mutator/build/examples/libfuzzer/libfuzzer_bin_example+0x5cc600)
    #3 0x6417f3 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::pair<void const*, int> >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::pair<void const*, int> > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::pair<void const*, int> > > >::_M_upper_bound(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::pair<void const*, int> > >*, std::_Rb_tree_node_base*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (/home/c/git/libprotobuf-mutator/build/examples/libfuzzer/libfuzzer_bin_example+0x6417f3)
    #4 0x641790 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::pair<void const*, int> >, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::pair<void const*, int> > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::pair<void const*, int> > > >::upper_bound(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (/home/c/git/libprotobuf-mutator/build/examples/libfuzzer/libfuzzer_bin_example+0x641790)
    #5 0x64170c in std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<void const*, int>, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::pair<void const*, int> > > >::upper_bound(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (/home/c/git/libprotobuf-mutator/build/examples/libfuzzer/libfuzzer_bin_example+0x64170c)
    #6 0x64144a in google::protobuf::SimpleDescriptorDatabase::DescriptorIndex<std::pair<void const*, int> >::FindLastLessOrEqual(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (/home/c/git/libprotobuf-mutator/build/examples/libfuzzer/libfuzzer_bin_example+0x64144a)
    #7 0x6402ac in google::protobuf::SimpleDescriptorDatabase::DescriptorIndex<std::pair<void const*, int> >::AddSymbol(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::pair<void const*, int>) (/home/c/git/libprotobuf-mutator/build/examples/libfuzzer/libfuzzer_bin_example+0x6402ac)
    #8 0x63868a in google::protobuf::SimpleDescriptorDatabase::DescriptorIndex<std::pair<void const*, int> >::AddFile(google::protobuf::FileDescriptorProto const&, std::pair<void const*, int>) (/home/c/git/libprotobuf-mutator/build/examples/libfuzzer/libfuzzer_bin_example+0x63868a)
    #9 0x636749 in google::protobuf::EncodedDescriptorDatabase::Add(void const*, int) (/home/c/git/libprotobuf-mutator/build/examples/libfuzzer/libfuzzer_bin_example+0x636749)
    #10 0x5937d2 in google::protobuf::DescriptorPool::InternalAddGeneratedFile(void const*, int) (/home/c/git/libprotobuf-mutator/build/examples/libfuzzer/libfuzzer_bin_example+0x5937d2)
    #11 0x65898f in google::protobuf::(anonymous namespace)::AddDescriptorsImpl(google::protobuf::internal::DescriptorTable const*) (/home/c/git/libprotobuf-mutator/build/examples/libfuzzer/libfuzzer_bin_example+0x65898f)
    #12 0x6588d3 in google::protobuf::internal::AddDescriptors(google::protobuf::internal::DescriptorTable const*) (/home/c/git/libprotobuf-mutator/build/examples/libfuzzer/libfuzzer_bin_example+0x6588d3)
    #13 0x421b5f in __cxx_global_var_init.23 (/home/c/git/libprotobuf-mutator/build/examples/libfuzzer/libfuzzer_bin_example+0x421b5f)
    #14 0x421b7d in _GLOBAL__sub_I_descriptor.pb.cc (/home/c/git/libprotobuf-mutator/build/examples/libfuzzer/libfuzzer_bin_example+0x421b7d)
    #15 0x6ebc9c in __libc_csu_init (/home/c/git/libprotobuf-mutator/build/examples/libfuzzer/libfuzzer_bin_example+0x6ebc9c)
    #16 0x7fbd5ad63b27 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:266
    #17 0x422349 in _start (/home/c/git/libprotobuf-mutator/build/examples/libfuzzer/libfuzzer_bin_example+0x422349)

SUMMARY: MemorySanitizer: use-of-uninitialized-value (/usr/lib/x86_64-linux-gnu/libstdc++.so.6+0x126bc7) in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::compare(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) const
Exiting

For anyone stumbling upon this issue, I can confirm that the steps in master...dende:master solved the issue for me on Ubuntu 18.04.3 with the default clang installation (version 6.0.0). Thanks dende!

from libprotobuf-mutator.

I still get the errors in my libfuzzer_example_test after applying @dende's fix.

[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from LibFuzzerExampleTest
[ RUN ] LibFuzzerExampleTest.Text
ASAN_OPTIONS=detect_leaks=0 ./libfuzzer_example -detect_leaks=0 -len_control=0 -max_len=1000 -runs=10000000 -artifact_prefix=/tmp/libxml2_example_test_am3CxW/ /tmp/libxml2_example_test_am3CxW/
sh: 1: ./libfuzzer_example: not found
../examples/libfuzzer/libfuzzer_example_test.cc:28: Failure
Expected equality of these values:
kDefaultLibFuzzerError
Which is: 77
GetError(RunFuzzer("libfuzzer_example", 1000, 10000000))
Which is: 127
[ FAILED ] LibFuzzerExampleTest.Text (3 ms)
[ RUN ] LibFuzzerExampleTest.Binary
ASAN_OPTIONS=detect_leaks=0 ./libfuzzer_bin_example -detect_leaks=0 -len_control=0 -max_len=1000 -runs=10000000 -artifact_prefix=/tmp/libxml2_example_test_x846tA/ /tmp/libxml2_example_test_x846tA/
sh: 1: ./libfuzzer_bin_example: not found
../examples/libfuzzer/libfuzzer_example_test.cc:33: Failure
Expected equality of these values:
kDefaultLibFuzzerError
Which is: 77
GetError(RunFuzzer("libfuzzer_bin_example", 1000, 10000000))
Which is: 127
[ FAILED ] LibFuzzerExampleTest.Binary (3 ms)
[----------] 2 tests from LibFuzzerExampleTest (6 ms total)

[----------] Global test environment tear-down
[==========] 2 tests from 1 test suite ran. (6 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] LibFuzzerExampleTest.Text
[ FAILED ] LibFuzzerExampleTest.Binary

from libprotobuf-mutator.