snaipe / criterion Goto Github PK
View Code? Open in Web Editor NEWA cross-platform C and C++ unit testing framework for the 21st century
License: MIT License
A cross-platform C and C++ unit testing framework for the 21st century
License: MIT License
Assertion macro in questions is referred to here:
http://criterion.readthedocs.org/en/latest/starter.html#asserting-things
But upon building:
warning: implicit declaration of function ‘expect_null’
This would allow a test to print a message through criterion's logger.
On Windows x64 with VS2015, current bleeding branch (ref. 4cc826e) chokes on running theories test with following commands sequence:
:: in cmd
c: && mkdir \temp && cd \temp
git clone https://github.com/Snaipe/Criterion --recursive
cd Criterion
cmake . -DCTESTS=ON
cmake --build .
cmake --build . --target criterion_tests
ctest
Following dialog is shown, when building with Debug configuration and debugging with IDE suggests that it is aborting from src/core/worker.c#L72 (in release, it shows a different more cryptic message):
Here is the call stack:
ucrtbased.dll!issue_debug_notification(const wchar_t * const message) Line 125 C++
ucrtbased.dll!__acrt_report_runtime_error(const wchar_t * message) Line 142 C++
ucrtbased.dll!abort() Line 51 C++
> criterion.dll!worker_read_event(worker_set * workers, _iobuf * pipe) Line 72 C
criterion.dll!run_tests_async(criterion_test_set * set, criterion_global_stats * stats) Line 357 C
criterion.dll!criterion_run_all_tests_impl(criterion_test_set * set) Line 396 C
criterion.dll!criterion_run_all_tests(criterion_test_set * set) Line 419 C
criterion.dll!main(int argc, char * * argv) Line 9 C
theories.c.bin.exe!invoke_main() Line 74 C++
theories.c.bin.exe!__scrt_common_main_seh() Line 264 C++
theories.c.bin.exe!__scrt_common_main() Line 309 C++
theories.c.bin.exe!mainCRTStartup() Line 17 C++
kernel32.dll!@BaseThreadInitThunk@12�() Unknown
ntdll.dll!__RtlUserThreadStart() Unknown
ntdll.dll!__RtlUserThreadStart@8�() Unknown
while still at breakpoints, here is how the locals look like:
+ workers 0x0055f8cc {workers=0x0074a3f0 {0x00000000 <NULL>} max_workers=8 } worker_set *
+ pipe 0x00754e50 {_Placeholder=0x00754e65 } _iobuf *
+ ev 0x00756100 {pid=144115188075855918 kind=-1006632960 data=0x00000000 ...} event *
Signals can be tested, exit statuses should be, too.
This would assert that all regression tests also pass on optimized targets and prevent things like #46 to happen.
While attempting the libsass test progress cross platform, I found that libintl.h cannot be found by CMake on FreeBSD, although gettext is installed. Following are the steps to repro:
portsnap fetch && portsnap extract
.mkdir ~/projects && cd ~/projects
git clone https://github.com/Snaipe/Criterion && cd Criterion
cmake . && cmake --build .
Log reads:
[...]
-- Configuring done
-- Generating done
-- Build files have been written to: /root/test/Criterion
[ 21%] Built target dyncall_s
[ 25%] Built target csptr
[ 28%] Building C object CMakeFiles/criterion.dir/src/core/runner.c.o
In file included from /root/projects/Criterion/src/core/runner.c:33:
/root/projects/Criterion/src/string/i18n.h:10:12: fatal error: 'libintl.h' file not found
# include <libintl.h>
^
1 error generated.
CMakeFiles/criterion.dir/build.make:100: recipe for target 'CMakeFiles/criterion.dir/src/core/runner.c.o' failed
gmake[2]: *** [CMakeFiles/criterion.dir/src/core/runner.c.o] Error 1
CMakeFiles/Makefile2:61: recipe for target 'CMakeFiles/criterion.dir/all' failed
gmake[1]: *** [CMakeFiles/criterion.dir/all] Error 2
Makefile:146: recipe for target 'all' failed
gmake: *** [all] Error 2
But I can find it in csh:
root@fbsd-pc:~/projects/Criterion # find / -name libintl.h
/usr/local/include/libintl.h
I want Criterion to show some information about tests in suites? such as test case names!
The following abort
s needs to output an error message: (as of f1dfff5)
When compiling test cases with anything other than -O0, a failed assert causes a program crash, instead of just a failure. Simple example:
#include <criterion/criterion.h>
Test(cat, meow)
{
cr_assert_eq(strlen("meow"), 5);
}
gcc -O0 test.c -o test -lcriterion
# OK
./test
gcc -O1 test.c -o test -lcriterion
# Segfault
./test
It seems that gcov generates no coverage report for forked children. A bit of googling on the matter yielded no results -- maybe someone knows how to solve this ?
Relevant stackoverflow question.
Currently, the extended globbing is transformed by the algorithm in extmatch.c
to a perl regexp, which is then passed to PCRE.
It would probably be better to cut the middle ground and do the parsing ourselves -- this would remove PCRE as a dependency and provide --pattern
on all platforms.
Not sure if this is expected behavior, but upon failures I'll get output of this sort:
[====] Synthesis: Tested: 12 | Passing: 0 | Failing: 13 | Crashing: 5
The failing number is larger than the number of tests listed as "Tested". (I can confirm there are only 12)
This also seems to comes up when using the .disabled option. The "Tested" stat will be the full number, but the sum of passing and failing will approximate the correct number of tests actually run (non-disabled).
A paper on the basics of theories for TDD can be found here
Hello,
can you add binaries to the new release 1.2.1 (at least linux)? Without it's difficult to use within CI.
Thanks.
Some functions and most of the macros in criterion/common.h
are left unprefixed and are candidates for identifier clashes.
Almost all of these are part of criterion's internals, so it shouldn't be really impacting to chance.
ABI will most probably break, but we're a test framework, and compiled tests are hardly part of a release.
Some people want a full report, others want only failures. Either way, we need verbosity levels.
This is because any compilation will fail without at least one test.
To reproduce:
Test(sample, simple) {
auto& f_cin = criterion::get_redirected_cin();
f_cin << "Foo";
}
This snippet will not feed the "Foo"
string to f_cin
, but its pointer.
This is horribly counter intuitive and violates the principle of least surprise.
A simple workaround to this would be to simply replace "Foo"
by std::string("Foo")
.
AFAIK, the issue only concerns Windows.
Also physically group the tests by suite (and not just sort them) for more flexibility
Currently, this prevents users from compiling their tests with clang++
.
stdio_sync_filebuf is just a wrapper over a FILE* that implements the interface functions of a std::filebuf.
This would allow C++ users to use STL containers with cr_malloc/cr_free:
std::vector<int, criterion::allocator<int>> vec;
vec.push_back(42); // invokes cr_malloc(sizeof (int))
This results in a segfault when too many assertions were made during the lifetime of the program.
I was trying to see if my own code was freeing properly so naturally I tried out Valgrind. Is this normal, or unexpected output?
==36733== 40 bytes in 1 blocks are possibly lost in loss record 8 of 18
==36733== at 0x4C28F09: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==36733== by 0x408E1F: smalloc_impl (mman.c:89)
==36733== by 0x40AD8F: get_current_process (posix-compat.c:198)
==36733== by 0x4097DA: set_runner_process (process.c:42)
==36733== by 0x409652: criterion_run_all_tests (runner.c:248)
==36733== by 0x4029CB: main (main.c:186)
==36733==
==36733== 40 bytes in 1 blocks are possibly lost in loss record 9 of 18
==36733== at 0x4C28F09: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==36733== by 0x408E1F: smalloc_impl (mman.c:89)
==36733== by 0x40AD0F: stdpipe (posix-compat.c:190)
==36733== by 0x409889: spawn_test_worker (process.c:79)
==36733== by 0x40917D: run_test (runner.c:183)
==36733== by 0x4096F1: criterion_run_all_tests (runner.c:115)
==36733== by 0x4029CB: main (main.c:186)
==36733==
==36733== 48 bytes in 1 blocks are possibly lost in loss record 10 of 18
==36733== at 0x4C28F09: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==36733== by 0x408E1F: smalloc_impl (mman.c:89)
==36733== by 0x4095FF: criterion_init (runner.c:88)
==36733== by 0x409664: criterion_run_all_tests (runner.c:233)
==36733== by 0x4029CB: main (main.c:186)
==36733==
==36733== 64 bytes in 1 blocks are possibly lost in loss record 11 of 18
==36733== at 0x4C28F09: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==36733== by 0x408E1F: smalloc_impl (mman.c:89)
==36733== by 0x40AA0F: new_ordered_set (ordered-set.c:45)
==36733== by 0x4094FB: criterion_init (runner.c:65)
==36733== by 0x409664: criterion_run_all_tests (runner.c:233)
==36733== by 0x4029CB: main (main.c:186)
==36733==
==36733== 96 bytes in 1 blocks are possibly lost in loss record 12 of 18
==36733== at 0x4C28F09: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==36733== by 0x408E1F: smalloc_impl (mman.c:89)
==36733== by 0x409DEA: test_stats_init (stats.c:70)
==36733== by 0x409151: run_test (runner.c:173)
==36733== by 0x4096F1: criterion_run_all_tests (runner.c:115)
==36733== by 0x4029CB: main (main.c:186)
==36733==
==36733== 112 bytes in 1 blocks are possibly lost in loss record 13 of 18
==36733== at 0x4C28F09: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==36733== by 0x408E1F: smalloc_impl (mman.c:89)
==36733== by 0x409C77: stats_init (stats.c:48)
==36733== by 0x409674: criterion_run_all_tests (runner.c:237)
==36733== by 0x4029CB: main (main.c:186)
==36733==
==36733== 432 bytes in 6 blocks are possibly lost in loss record 14 of 18
==36733== at 0x4C28F09: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==36733== by 0x408E1F: smalloc_impl (mman.c:89)
==36733== by 0x40AB00: insert_ordered_set (ordered-set.c:61)
==36733== by 0x409581: criterion_init (runner.c:78)
==36733== by 0x409664: criterion_run_all_tests (runner.c:233)
==36733== by 0x4029CB: main (main.c:186)
==36733==
==36733== 648 bytes in 9 blocks are possibly lost in loss record 15 of 18
==36733== at 0x4C28F09: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==36733== by 0x408E1F: smalloc_impl (mman.c:89)
==36733== by 0x40AB00: insert_ordered_set (ordered-set.c:61)
==36733== by 0x40953C: criterion_init (runner.c:71)
==36733== by 0x409664: criterion_run_all_tests (runner.c:233)
==36733== by 0x4029CB: main (main.c:186)
==36733==
==36733== 768 bytes in 6 blocks are possibly lost in loss record 16 of 18
==36733== at 0x4C28F09: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==36733== by 0x408E1F: smalloc_impl (mman.c:89)
==36733== by 0x409D2B: suite_stats_init (stats.c:58)
==36733== by 0x4096A7: criterion_run_all_tests (runner.c:109)
==36733== by 0x4029CB: main (main.c:186)
==36733==
==36733== 896 bytes in 14 blocks are possibly lost in loss record 17 of 18
==36733== at 0x4C28F09: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==36733== by 0x408E1F: smalloc_impl (mman.c:89)
==36733== by 0x40AA0F: new_ordered_set (ordered-set.c:45)
==36733== by 0x409623: criterion_init (runner.c:80)
==36733== by 0x409664: criterion_run_all_tests (runner.c:233)
==36733== by 0x4029CB: main (main.c:186)
==36733==
==36733== 3,040 bytes in 38 blocks are possibly lost in loss record 18 of 18
==36733== at 0x4C28F09: malloc (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)
==36733== by 0x408E1F: smalloc_impl (mman.c:89)
==36733== by 0x40AB00: insert_ordered_set (ordered-set.c:61)
==36733== by 0x4095A2: criterion_init (runner.c:82)
==36733== by 0x409664: criterion_run_all_tests (runner.c:233)
==36733== by 0x4029CB: main (main.c:186)
That's for a single test. (Using --pattern to pick out only one test.)
Edited to contain the full output.
The test statistics lack a proper time measurement & timeout management systems.
MSVC 12.0 supports c99 designated initializers, so it should not be too hard to make the headers compatible.
Note: check for _MSC_VER >= 1800
.
Section attributes can be replaced with the following:
__pragma(data_seg(push))
__pragma(section("section_name", read))
__declspec(allocate("section_name"))
int custom_section_var;
__pragma(data_seg(pop))
Ever since the shift to a shared library build, windows builds are broken because (it seems) of autotools on cygwin. If I don't come with a proper solution, I will probably need to switch to CMake.
A C++11-compatible interface should be doable with a bit of macro tricks.
This is not a critical feature, and can be dropped if the effort is not worth the results.
This might be tricky since libintl is not available by default on windows/osx/freebsd. Ideally you should be able to activate internationalization on the target binary, not the library itself.
This will enable more compatibility with other testing tools and should not be too difficult to implement.
Everyone loves colored output. ~
If possible, could Criterion print the name of the test on crash during start up or teardown. When debugging regressions, it can become painful to figure out which test actually crashed when it is a small subset of the tests that have regressed.
Valgrind reports g_worker_pipe memory leak never being freed. Should it be freed in cleanup?
https://github.com/Snaipe/Criterion/blob/bleeding/src/core/runner.c#L412
Hello,
In our current effort of switching from Ruby-based test runner to Criterion, I have encountered a scenario where we are alerting the observer about a TODO test, which unexpectedly passes:
https://github.com/sass/sass-spec/blob/6d3835/lib/sass_spec/test.rb#L26 &
https://github.com/sass/sass-spec/blob/6d3835/lib/sass_spec/test.rb#L37
In order to replicate that effect with Criterion, we have experimented with ASSERT report hook which although emits the status of assertion test->passed
, but does not provide context for Theory's arguments or expected/actual result. Analogous to struct criterion_theory_stats
for theory, it would be of great help if the existing struct criterion_assert_stats
or a new struct criterion_theory_assert_stats
let us exercise:
// Supposedly, it requires a separate `THEORY_ASSERT`
ReportHook(THEORY_ASSERT)(struct criterion_theory_assert_stats *test) {
if(test->passed && should_test_todo && strstr(test->formatted_args, "-todo-")) {
// alert unexpectedness
}
}
Nitpick:
On a related note, it seems like test->message
always contains the error string, even when test->passed
is true
.
This would mean copying the section holding the tests metadata to a separate table and expose it to the report hook.
Not only is assert
deprecated, it has been removed by 461c56e.
The screencast should be updated to use cr_assert
instead of assert
.
Also, the output format has changed since then.
The current sample cannot be used in the regression tests as its output rely on undefined behaviour (since we're illustrating what happens when computing INT_MIN / -1
); also, one of the parameters happens to be a pointer to some data, which with ASRL does not have the same value across executions.
This is just a reminder to eliminate all leaks before hitting the new release (parallelisation might have spawned a few)
the assert
macro conflicts with the one provided by the standard assert.h
. This can be a problem when both are included in any order.
assert
? (maybe criterion_assert
)assert.h
and replace it with mine ?also, in the event that criterion.h is included before assert.h, it should define _ASSERT_H
to let the posix header replace assert itself (The MSVC header seem to replace it regardless)
While using Criterion something that we find would be useful is the option for a negation pattern. The option to selectively run certain tests exists, but of course doesn't support full regular expression syntax. Rather than needing to specifically disable in the source it'd be cool if we could do the following:
./test --exclude-pattern="suite_a/*"
To run all tests from suites other than the suite_a
suite.
To my knowledge this isn't possible with the current --pattern
option, but I could be mistaken.
It seems that msdn lied and that positional parameters are in fact not supported (or at least not working for both vsprintf
and _vsprintf_p
).
Internationalization on windows will have to go down the drain at the moment.
Recent testing on OS X yosemite shows that criterion does not seem to correctly iterate the tests.
More testing needs to be done; having travis OS X build will certainly help a lot.
At the moment, the library only compiles (and runs) with cygwin.
To port the library, these functions need to be implemented:
fork() is tricky, since there is no direct equivalent in WinAPI. The steps needed to achieve the same functionality (for an innefficient full-copy fork) are as follow:
pipe() will need some refactoring to bind file handles instead of file descriptors, and be adapted to use an anonymous pipe with winapi.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.