snaipe / criterion Goto Github PK

When I do noting in the unit test,it seems to have memory leak problem? For example:

#include <criterion/criterion.h>
Test(Hello,Hello)
{

}

Please confirm the problem!!!

Valgrind reports g_worker_pipe memory leak never being freed. Should it be freed in cleanup?

https://github.com/Snaipe/Criterion/blob/bleeding/src/core/runner.c#L412

This results in a segfault when too many assertions were made during the lifetime of the program.

The current sample cannot be used in the regression tests as its output rely on undefined behaviour (since we're illustrating what happens when computing INT_MIN / -1); also, one of the parameters happens to be a pointer to some data, which with ASRL does not have the same value across executions.

Everyone loves colored output. ~

This would allow a test to print a message through criterion's logger.

Also physically group the tests by suite (and not just sort them) for more flexibility

• print test tree
• run a subset of the tests
• exit after the first failure

It seems that gcov generates no coverage report for forked children. A bit of googling on the matter yielded no results -- maybe someone knows how to solve this ?

Relevant stackoverflow question.

This is just a reminder to eliminate all leaks before hitting the new release (parallelisation might have spawned a few)

• assertEquals
• assertArrayEquals
• assertStringEquals
• assertNot
• assertNotEquals
• assertArrayNotEquals
• assertStringNotEquals

This will enable more compatibility with other testing tools and should not be too difficult to implement.

This might be tricky since libintl is not available by default on windows/osx/freebsd. Ideally you should be able to activate internationalization on the target binary, not the library itself.

Currently, the extended globbing is transformed by the algorithm in extmatch.c to a perl regexp, which is then passed to PCRE.

It would probably be better to cut the middle ground and do the parsing ourselves -- this would remove PCRE as a dependency and provide --pattern on all platforms.

The following aborts needs to output an error message: (as of f1dfff5)

• src/compat/alloc.c:55
• src/compat/process.c:137
• src/compat/pipe.h:63
• src/core/runner.c:393
• src/core/worker.c:72
• src/core/worker.c:105
• src/io/event.c:56

This would allow C++ users to use STL containers with cr_malloc/cr_free:

std::vector<int, criterion::allocator<int>> vec;
vec.push_back(42); // invokes cr_malloc(sizeof (int))

Some people want a full report, others want only failures. Either way, we need verbosity levels.

If possible, could Criterion print the name of the test on crash during start up or teardown. When debugging regressions, it can become painful to figure out which test actually crashed when it is a small subset of the tests that have regressed.

At the moment, the library only compiles (and runs) with cygwin.
To port the library, these functions need to be implemented:

• fork
• pipe

fork() is tricky, since there is no direct equivalent in WinAPI. The steps needed to achieve the same functionality (for an innefficient full-copy fork) are as follow:

• Start the process stopped
• Copy over the heap, stack, data sections, and setjmp() data
• longjmp() to the address where the forking process happens

pipe() will need some refactoring to bind file handles instead of file descriptors, and be adapted to use an anonymous pipe with winapi.

While using Criterion something that we find would be useful is the option for a negation pattern. The option to selectively run certain tests exists, but of course doesn't support full regular expression syntax. Rather than needing to specifically disable in the source it'd be cool if we could do the following:

./test --exclude-pattern="suite_a/*"

To run all tests from suites other than the suite_a suite.

To my knowledge this isn't possible with the current --pattern option, but I could be mistaken.

Hello,

can you add binaries to the new release 1.2.1 (at least linux)? Without it's difficult to use within CI.

Thanks.

Signals can be tested, exit statuses should be, too.

While attempting the libsass test progress cross platform, I found that libintl.h cannot be found by CMake on FreeBSD, although gettext is installed. Following are the steps to repro:

• Installed FreeBSD 10.1, followed by portsnap fetch && portsnap extract.
• mkdir ~/projects && cd ~/projects
• git clone https://github.com/Snaipe/Criterion && cd Criterion
• cmake . && cmake --build .

Log reads:

[...]
-- Configuring done
-- Generating done
-- Build files have been written to: /root/test/Criterion
[ 21%] Built target dyncall_s
[ 25%] Built target csptr
[ 28%] Building C object CMakeFiles/criterion.dir/src/core/runner.c.o
In file included from /root/projects/Criterion/src/core/runner.c:33:
/root/projects/Criterion/src/string/i18n.h:10:12: fatal error: 'libintl.h' file not found
#  include <libintl.h>
           ^
1 error generated.
CMakeFiles/criterion.dir/build.make:100: recipe for target 'CMakeFiles/criterion.dir/src/core/runner.c.o' failed
gmake[2]: *** [CMakeFiles/criterion.dir/src/core/runner.c.o] Error 1
CMakeFiles/Makefile2:61: recipe for target 'CMakeFiles/criterion.dir/all' failed
gmake[1]: *** [CMakeFiles/criterion.dir/all] Error 2
Makefile:146: recipe for target 'all' failed
gmake: *** [all] Error 2

But I can find it in csh:

root@fbsd-pc:~/projects/Criterion # find / -name libintl.h
/usr/local/include/libintl.h

When compiling test cases with anything other than -O0, a failed assert causes a program crash, instead of just a failure. Simple example:

#include <criterion/criterion.h>

Test(cat, meow)
{
    cr_assert_eq(strlen("meow"), 5);
}

gcc -O0 test.c -o test -lcriterion
# OK
./test
gcc -O1 test.c -o test -lcriterion
# Segfault
./test

I want Criterion to show some information about tests in suites? such as test case names!

A C++11-compatible interface should be doable with a bit of macro tricks.

This is not a critical feature, and can be dropped if the effort is not worth the results.

A paper on the basics of theories for TDD can be found here

On Windows x64 with VS2015, current bleeding branch (ref. 4cc826e) chokes on running theories test with following commands sequence:

:: in cmd
c: && mkdir \temp && cd \temp
git clone https://github.com/Snaipe/Criterion --recursive
cd Criterion
cmake . -DCTESTS=ON
cmake --build .
cmake --build . --target criterion_tests
ctest

Following dialog is shown, when building with Debug configuration and debugging with IDE suggests that it is aborting from src/core/worker.c#L72 (in release, it shows a different more cryptic message):

Here is the call stack:

    ucrtbased.dll!issue_debug_notification(const wchar_t * const message) Line 125  C++
    ucrtbased.dll!__acrt_report_runtime_error(const wchar_t * message) Line 142 C++
    ucrtbased.dll!abort() Line 51   C++
>   criterion.dll!worker_read_event(worker_set * workers, _iobuf * pipe) Line 72    C
    criterion.dll!run_tests_async(criterion_test_set * set, criterion_global_stats * stats) Line 357    C
    criterion.dll!criterion_run_all_tests_impl(criterion_test_set * set) Line 396   C
    criterion.dll!criterion_run_all_tests(criterion_test_set * set) Line 419    C
    criterion.dll!main(int argc, char * * argv) Line 9  C
    theories.c.bin.exe!invoke_main() Line 74    C++
    theories.c.bin.exe!__scrt_common_main_seh() Line 264    C++
    theories.c.bin.exe!__scrt_common_main() Line 309    C++
    theories.c.bin.exe!mainCRTStartup() Line 17 C++
    kernel32.dll!@BaseThreadInitThunk@12�() Unknown
    ntdll.dll!__RtlUserThreadStart()    Unknown
    ntdll.dll!__RtlUserThreadStart@8�() Unknown

while still at breakpoints, here is how the locals look like:

+       workers 0x0055f8cc {workers=0x0074a3f0 {0x00000000 <NULL>} max_workers=8 }  worker_set *
+       pipe    0x00754e50 {_Placeholder=0x00754e65 }   _iobuf *
+       ev  0x00756100 {pid=144115188075855918 kind=-1006632960 data=0x00000000 ...}    event *

• Linux
  • Compile library
  • Compile tests
• FreeBSD
  • Compile library
  • Compile tests
• OS X
  • Compile library
  • Compile tests
• Windows
  • MinGW
    • Compile library
    • Compile tests
  • Cygwin
    • Compile library
    • Compile tests
  • MSVC (compile tests only)

This would mean copying the section holding the tests metadata to a separate table and expose it to the report hook.

Some functions and most of the macros in criterion/common.h are left unprefixed and are candidates for identifier clashes.

Almost all of these are part of criterion's internals, so it shouldn't be really impacting to chance.

ABI will most probably break, but we're a test framework, and compiled tests are hardly part of a release.

I was trying to see if my own code was freeing properly so naturally I tried out Valgrind. Is this normal, or unexpected output?

==36733== 40 bytes in 1 blocks are possibly lost in loss record 8 of 18
==36733==    at 0x4C28F09: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==36733==    by 0x408E1F: smalloc_impl (mman.c:89)
==36733==    by 0x40AD8F: get_current_process (posix-compat.c:198)
==36733==    by 0x4097DA: set_runner_process (process.c:42)
==36733==    by 0x409652: criterion_run_all_tests (runner.c:248)
==36733==    by 0x4029CB: main (main.c:186)
==36733==
==36733== 40 bytes in 1 blocks are possibly lost in loss record 9 of 18
==36733==    at 0x4C28F09: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==36733==    by 0x408E1F: smalloc_impl (mman.c:89)
==36733==    by 0x40AD0F: stdpipe (posix-compat.c:190)
==36733==    by 0x409889: spawn_test_worker (process.c:79)
==36733==    by 0x40917D: run_test (runner.c:183)
==36733==    by 0x4096F1: criterion_run_all_tests (runner.c:115)
==36733==    by 0x4029CB: main (main.c:186)
==36733==
==36733== 48 bytes in 1 blocks are possibly lost in loss record 10 of 18
==36733==    at 0x4C28F09: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==36733==    by 0x408E1F: smalloc_impl (mman.c:89)
==36733==    by 0x4095FF: criterion_init (runner.c:88)
==36733==    by 0x409664: criterion_run_all_tests (runner.c:233)
==36733==    by 0x4029CB: main (main.c:186)
==36733==
==36733== 64 bytes in 1 blocks are possibly lost in loss record 11 of 18
==36733==    at 0x4C28F09: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==36733==    by 0x408E1F: smalloc_impl (mman.c:89)
==36733==    by 0x40AA0F: new_ordered_set (ordered-set.c:45)
==36733==    by 0x4094FB: criterion_init (runner.c:65)
==36733==    by 0x409664: criterion_run_all_tests (runner.c:233)
==36733==    by 0x4029CB: main (main.c:186)
==36733==
==36733== 96 bytes in 1 blocks are possibly lost in loss record 12 of 18
==36733==    at 0x4C28F09: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==36733==    by 0x408E1F: smalloc_impl (mman.c:89)
==36733==    by 0x409DEA: test_stats_init (stats.c:70)
==36733==    by 0x409151: run_test (runner.c:173)
==36733==    by 0x4096F1: criterion_run_all_tests (runner.c:115)
==36733==    by 0x4029CB: main (main.c:186)
==36733==
==36733== 112 bytes in 1 blocks are possibly lost in loss record 13 of 18
==36733==    at 0x4C28F09: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==36733==    by 0x408E1F: smalloc_impl (mman.c:89)
==36733==    by 0x409C77: stats_init (stats.c:48)
==36733==    by 0x409674: criterion_run_all_tests (runner.c:237)
==36733==    by 0x4029CB: main (main.c:186)
==36733==
==36733== 432 bytes in 6 blocks are possibly lost in loss record 14 of 18
==36733==    at 0x4C28F09: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==36733==    by 0x408E1F: smalloc_impl (mman.c:89)
==36733==    by 0x40AB00: insert_ordered_set (ordered-set.c:61)
==36733==    by 0x409581: criterion_init (runner.c:78)
==36733==    by 0x409664: criterion_run_all_tests (runner.c:233)
==36733==    by 0x4029CB: main (main.c:186)
==36733==
==36733== 648 bytes in 9 blocks are possibly lost in loss record 15 of 18
==36733==    at 0x4C28F09: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==36733==    by 0x408E1F: smalloc_impl (mman.c:89)
==36733==    by 0x40AB00: insert_ordered_set (ordered-set.c:61)
==36733==    by 0x40953C: criterion_init (runner.c:71)
==36733==    by 0x409664: criterion_run_all_tests (runner.c:233)
==36733==    by 0x4029CB: main (main.c:186)
==36733==
==36733== 768 bytes in 6 blocks are possibly lost in loss record 16 of 18
==36733==    at 0x4C28F09: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==36733==    by 0x408E1F: smalloc_impl (mman.c:89)
==36733==    by 0x409D2B: suite_stats_init (stats.c:58)
==36733==    by 0x4096A7: criterion_run_all_tests (runner.c:109)
==36733==    by 0x4029CB: main (main.c:186)
==36733==
==36733== 896 bytes in 14 blocks are possibly lost in loss record 17 of 18
==36733==    at 0x4C28F09: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==36733==    by 0x408E1F: smalloc_impl (mman.c:89)
==36733==    by 0x40AA0F: new_ordered_set (ordered-set.c:45)
==36733==    by 0x409623: criterion_init (runner.c:80)
==36733==    by 0x409664: criterion_run_all_tests (runner.c:233)
==36733==    by 0x4029CB: main (main.c:186)
==36733==
==36733== 3,040 bytes in 38 blocks are possibly lost in loss record 18 of 18
==36733==    at 0x4C28F09: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==36733==    by 0x408E1F: smalloc_impl (mman.c:89)
==36733==    by 0x40AB00: insert_ordered_set (ordered-set.c:61)
==36733==    by 0x4095A2: criterion_init (runner.c:82)
==36733==    by 0x409664: criterion_run_all_tests (runner.c:233)
==36733==    by 0x4029CB: main (main.c:186)

That's for a single test. (Using --pattern to pick out only one test.)

Edited to contain the full output.

It seems that msdn lied and that positional parameters are in fact not supported (or at least not working for both vsprintf and _vsprintf_p).

Internationalization on windows will have to go down the drain at the moment.

This is because any compilation will fail without at least one test.

To reproduce:

Test(sample, simple) {
    auto& f_cin = criterion::get_redirected_cin();
    f_cin << "Foo";
}

This snippet will not feed the "Foo" string to f_cin, but its pointer.
This is horribly counter intuitive and violates the principle of least surprise.

A simple workaround to this would be to simply replace "Foo" by std::string("Foo").

AFAIK, the issue only concerns Windows.

This would assert that all regression tests also pass on optimized targets and prevent things like #46 to happen.

• Appveyor
• Travis CI

the assert macro conflicts with the one provided by the standard assert.h. This can be a problem when both are included in any order.

• Should I provide an alias for assert ? (maybe criterion_assert)
• Should I undefine the assert from assert.h and replace it with mine ?

also, in the event that criterion.h is included before assert.h, it should define _ASSERT_H to let the posix header replace assert itself (The MSVC header seem to replace it regardless)

The test statistics lack a proper time measurement & timeout management systems.

• time measurement should probably be done using clock(3), since it is portable with OS X. the result of gettimeofday() can change if the system wall-clock changed, and clock_gettime forces the user to compile with -lrc, and is linux-only for the moment. Edit: clock(3) may be portable, but provide wall-clock time and precision is in the order of seconds. Functions to query the system's monotonic clock shall be used.
• timeouts may be done with SIGALRM, but there might be more portable alternatives.
  Edit: SIGALRM might be used to implement sleep(3), a signal in the range of SIGRTMIN - SIGRTMAX should be used (see timer_create(2) on linux)

Hello,

In our current effort of switching from Ruby-based test runner to Criterion, I have encountered a scenario where we are alerting the observer about a TODO test, which unexpectedly passes:
https://github.com/sass/sass-spec/blob/6d3835/lib/sass_spec/test.rb#L26 &
https://github.com/sass/sass-spec/blob/6d3835/lib/sass_spec/test.rb#L37

In order to replicate that effect with Criterion, we have experimented with ASSERT report hook which although emits the status of assertion test->passed, but does not provide context for Theory's arguments or expected/actual result. Analogous to struct criterion_theory_stats for theory, it would be of great help if the existing struct criterion_assert_stats or a new struct criterion_theory_assert_stats let us exercise:

// Supposedly, it requires a separate `THEORY_ASSERT`
ReportHook(THEORY_ASSERT)(struct criterion_theory_assert_stats *test) {
  if(test->passed && should_test_todo && strstr(test->formatted_args, "-todo-")) {
    // alert unexpectedness 
  }
}

Nitpick:
On a related note, it seems like test->message always contains the error string, even when test->passed is true.

MSVC 12.0 supports c99 designated initializers, so it should not be too hard to make the headers compatible.
Note: check for _MSC_VER >= 1800.

Section attributes can be replaced with the following:

__pragma(data_seg(push))
__pragma(section("section_name", read))
__declspec(allocate("section_name"))
int custom_section_var;

__pragma(data_seg(pop))

Not sure if this is expected behavior, but upon failures I'll get output of this sort:

[====] Synthesis: Tested: 12 | Passing: 0 | Failing: 13 | Crashing: 5 

The failing number is larger than the number of tests listed as "Tested". (I can confirm there are only 12)

This also seems to comes up when using the .disabled option. The "Tested" stat will be the full number, but the sum of passing and failing will approximate the correct number of tests actually run (non-disabled).

Currently, this prevents users from compiling their tests with clang++.

stdio_sync_filebuf is just a wrapper over a FILE* that implements the interface functions of a std::filebuf.

Assertion macro in questions is referred to here:
http://criterion.readthedocs.org/en/latest/starter.html#asserting-things

But upon building:

warning: implicit declaration of function ‘expect_null’

Recent testing on OS X yosemite shows that criterion does not seem to correctly iterate the tests.
More testing needs to be done; having travis OS X build will certainly help a lot.

Ever since the shift to a shared library build, windows builds are broken because (it seems) of autotools on cygwin. If I don't come with a proper solution, I will probably need to switch to CMake.

Not only is assert deprecated, it has been removed by 461c56e.

The screencast should be updated to use cr_assert instead of assert.

Also, the output format has changed since then.