libnonius / nonius Goto Github PK
View Code? Open in Web Editor NEWA C++ micro-benchmarking framework
Home Page: https://nonius.io
License: Creative Commons Zero v1.0 Universal
A C++ micro-benchmarking framework
Home Page: https://nonius.io
License: Creative Commons Zero v1.0 Universal
I evaluated several benchmark libraries for C++.
Right at the moment I could not find one benchmark library which is really header only.
For the next release, I wish you could get rid of the boost dependency and make the library a really header only library.
The test framework Catch is also header only, but does not contain a benchmark functionality.
Your library can fill this part. I also think a lot of users would like to simply add the header and start benching.
PS: what is the reason for the pthread linking on linux? Maybe you can also fix this (with std::thread?)
PPS: I also does not see an explanation for linking against boost::chrono on windows, why is this the case?
When I go to https://nonius.io in safari on OSX 10.12 beta 3 the website fails to load with the error shown below:
In the benchmarks below, to_string(4.2) with sleep
always runs with 1 iteration, while to_string(4.2) with no sleep
runs with around 100 iterations. This is because nonius::benchmark::prepare() computes the number of iterations based on the duration of the entire benchmark function, not the duration of the measure() function. This is a problem if the code prior to the measure() is expensive. In this example, the reported mean of the sleep version is 5.5 _micro_seconds, while the mean of the no-sleep version is 196 _nano_seconds!
#define NONIUS_RUNNER
#include <nonius.h++>
#include <thread>
#include <chrono>
NONIUS_BENCHMARK("to_string(4.2) with sleep", [] (nonius::chronometer chronometer) {
std::this_thread::sleep_for(std::chrono::milliseconds(1));
return chronometer.measure([] {
return std::to_string(4.2);
});
})
NONIUS_BENCHMARK("to_string(4.2) with no sleep", [] (nonius::chronometer chronometer) {
return chronometer.measure([] {
return std::to_string(4.2);
});
})
The VC++ build is really broken at the moment.
Given #36, it is only natural to automate verification of fitness to different models, in order to estimate the likelihood of some code belonging to a particular complexity class.
Clang 3.7 is out. The build system should automatically test it.
Support KDE outputs.
Now that #54 is merged, the documentation should reflect that highcharts is no longer used.
The websites http://nonius.io/ and http://flamingdangerzone.com/nonius are currently not loading for me. Everything else works fine.
If benchmarks can be parameterized, once can run the same code over inputs of different sizes, for example, and check how some algorithm scales with input.
It looks like that in 3rd sample and up code:
#define NONIUS_RUNNER
#include <nonius.h++>
must be replaced with:
#define NONIUS_RUNNER
#include <nonius/nonius_single.h++>
The current release (1.1.0) will bring up again compiler warnings for VSx64.
// line 1541, the 'n' variable is std::size_t
double outlier_variance = detail::outlier_variance(mean_estimate, stddev_estimate, n);
To avoid these kind of problems in the feature, please add support for CI for nonius:
See following template as starting point: https://github.com/headmyshoulder/cit
For Visual Studio use: http://www.appveyor.com/
test/faster_than_clock.c++:22: FAILED:
due to unexpected exception with message:
could not measure benchmark, maybe it was optimized away
This test is failing for me when compiled with msvc2015 and mingw64 g++. I've tried it on multiple machines. It seems to be OS specific rather than compiler or hardware.
Some instructions about using the build scripts for people wanting to tweak or contribute to nonius are needed.
I want to ignore in the measurement the destruction of an object. I know the special objects: nonius::storage_for
& nonius::destructable_object
But I think it will not help me, I have following case:
MyObject obj = method_generate_myobject()
The method method_generate_myobject()
will generate an instance of MyObject
with automatica storage. The method has access to a private ctor which I cannot invoke directly.
How can I ignore with nonius the destruction of MyObject
during the benchmark run?
Make it available for VS using native Nuget package
The benchmark authoring guide needs to explain that the user is still responsible for certain characteristics of the environment that nonius cannot control nor account for, and also explain the extent of nonius' interaction with those.
The example1.c++
and example2.c++
compile fine, but example3.c++
, example4.c++
and example5.c++
fail:
certik@redhawk:~/repos/nonius/examples(stable)$ g++ -g -std=c++11 -I../include -I$HASHSTACK/include example1.c++ -pthread
certik@redhawk:~/repos/nonius/examples(stable)$ g++ -g -std=c++11 -I../include -I$HASHSTACK/include example2.c++ -pthread
certik@redhawk:~/repos/nonius/examples(stable)$ g++ -g -std=c++11 -I../include -I$HASHSTACK/include example3.c++ -pthread
/usr/lib/../lib64/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
collect2: error: ld returned 1 exit status
certik@redhawk:~/repos/nonius/examples(stable)$ g++ -g -std=c++11 -I../include -I$HASHSTACK/include example4.c++ -pthread
/usr/lib/../lib64/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
collect2: error: ld returned 1 exit status
certik@redhawk:~/repos/nonius/examples(stable)$ g++ -g -std=c++11 -I../include -I$HASHSTACK/include example5.c++ -pthread
/usr/lib/../lib64/crt1.o: In function `_start':
(.text+0x20): undefined reference to `main'
collect2: error: ld returned 1 exit status
Also I had to apply the following patch to make them compile:
diff --git a/examples/example1.c++ b/examples/example1.c++
index 01e66bc..0dda421 100644
--- a/examples/example1.c++
+++ b/examples/example1.c++
@@ -1,4 +1,4 @@
-#include <nonius.h++>
+#include <nonius/nonius.h++>
#include <iterator>
#include <string>
diff --git a/examples/example2.c++ b/examples/example2.c++
index 2ad5a0c..bcca1f6 100644
--- a/examples/example2.c++
+++ b/examples/example2.c++
@@ -1,4 +1,4 @@
-#include <nonius.h++>
+#include <nonius/nonius.h++>
#include <iterator>
#include <string>
diff --git a/examples/example3.c++ b/examples/example3.c++
index f41a51f..24b7b7a 100644
--- a/examples/example3.c++
+++ b/examples/example3.c++
@@ -1,5 +1,5 @@
#define NONIUS_RUNNER
-#include <nonius.h++>
+#include <nonius/nonius.h++>
NONIUS_BENCHMARK("to_string(42)", []{
return std::to_string(42);
diff --git a/examples/example4.c++ b/examples/example4.c++
index 727cbaa..00c51c5 100644
--- a/examples/example4.c++
+++ b/examples/example4.c++
@@ -1,5 +1,5 @@
#define NONIUS_RUNNER
-#include <nonius.h++>
+#include <nonius/nonius.h++>
#include <list>
#include <forward_list>
diff --git a/examples/example5.c++ b/examples/example5.c++
index 05a5482..0ac1062 100644
--- a/examples/example5.c++
+++ b/examples/example5.c++
@@ -1,5 +1,5 @@
#define NONIUS_RUNNER
-#include <nonius.h++>
+#include <nonius/nonius.h++>
#include <string>
#include <vector>
Moving from rev 74b2de7 to ce74266
breaks linking my nonius test program with link errors:
/tmp/direct-simple-8546e9.o: In function `nonius::timing<boost::chrono::steady_clock::duration, nonius::detail::complete_type<std::result_of<nonius::detail::repeater<nonius::now<boost::chrono::steady_clock> > (int)>::type>::type> nonius::detail::run_for_at_least<boost::chrono::steady_clock, nonius::detail::repeater<nonius::now<boost::chrono::steady_clock> > >(boost::chrono::steady_clock::duration, int, nonius::detail::repeater<nonius::now<boost::chrono::steady_clock> >&&)':
direct-simple.cpp:(.text._ZN6nonius6detail16run_for_at_leastIN5boost6chrono12steady_clockENS0_8repeaterINS_3nowIS4_EEEEEENS_6timingINT_8durationENS0_13complete_typeINSt9result_ofIFT0_iEE4typeEE4typeEEESB_iOSE_[_ZN6nonius6detail16run_for_at_leastIN5boost6chrono12steady_clockENS0_8repeaterINS_3nowIS4_EEEEEENS_6timingINT_8durationENS0_13complete_typeINSt9result_ofIFT0_iEE4typeEE4typeEEESB_iOSE_]+0x15): undefined reference to `boost::chrono::steady_clock::now()'
direct-simple.cpp:(.text._ZN6nonius6detail16run_for_at_leastIN5boost6chrono12steady_clockENS0_8repeaterINS_3nowIS4_EEEEEENS_6timingINT_8durationENS0_13complete_typeINSt9result_ofIFT0_iEE4typeEE4typeEEESB_iOSE_[_ZN6nonius6detail16run_for_at_leastIN5boost6chrono12steady_clockENS0_8repeaterINS_3nowIS4_EEEEEENS_6timingINT_8durationENS0_13complete_typeINSt9result_ofIFT0_iEE4typeEE4typeEEESB_iOSE_]+0x1d): undefined reference to `boost::chrono::steady_clock::now()'
direct-simple.cpp:(.text._ZN6nonius6detail16run_for_at_leastIN5boost6chrono12steady_clockENS0_8repeaterINS_3nowIS4_EEEEEENS_6timingINT_8durationENS0_13complete_typeINSt9result_ofIFT0_iEE4typeEE4typeEEESB_iOSE_[_ZN6nonius6detail16run_for_at_leastIN5boost6chrono12steady_clockENS0_8repeaterINS_3nowIS4_EEEEEENS_6timingINT_8durationENS0_13complete_typeINSt9result_ofIFT0_iEE4typeEE4typeEEESB_iOSE_]+0x41): undefined reference to `boost::chrono::steady_clock::now()'
direct-simple.cpp:(.text._ZN6nonius6detail16run_for_at_leastIN5boost6chrono12steady_clockENS0_8repeaterINS_3nowIS4_EEEEEENS_6timingINT_8durationENS0_13complete_typeINSt9result_ofIFT0_iEE4typeEE4typeEEESB_iOSE_[_ZN6nonius6detail16run_for_at_leastIN5boost6chrono12steady_clockENS0_8repeaterINS_3nowIS4_EEEEEENS_6timingINT_8durationENS0_13complete_typeINSt9result_ofIFT0_iEE4typeEE4typeEEESB_iOSE_]+0x51): undefined reference to `boost::chrono::steady_clock::now()'
direct-simple.cpp:(.text._ZN6nonius6detail16run_for_at_leastIN5boost6chrono12steady_clockENS0_8repeaterINS_3nowIS4_EEEEEENS_6timingINT_8durationENS0_13complete_typeINSt9result_ofIFT0_iEE4typeEE4typeEEESB_iOSE_[_ZN6nonius6detail16run_for_at_leastIN5boost6chrono12steady_clockENS0_8repeaterINS_3nowIS4_EEEEEENS_6timingINT_8durationENS0_13complete_typeINSt9result_ofIFT0_iEE4typeEE4typeEEESB_iOSE_]+0x5b): undefined reference to `boost::chrono::steady_clock::now()'
/tmp/direct-simple-8546e9.o:direct-simple.cpp:(.text._ZN6nonius6detail16run_for_at_leastIN5boost6chrono12steady_clockENS0_8repeaterINS_3nowIS4_EEEEEENS_6timingINT_8durationENS0_13complete_typeINSt9result_ofIFT0_iEE4typeEE4typeEEESB_iOSE_[_ZN6nonius6detail16run_for_at_leastIN5boost6chrono12steady_clockENS0_8repeaterINS_3nowIS4_EEEEEENS_6timingINT_8durationENS0_13complete_typeINSt9result_ofIFT0_iEE4typeEE4typeEEESB_iOSE_]+0x6a): more undefined references to `boost::chrono::steady_clock::now()' follow
Is this intentional? I hoped it could stay fully header-only.
Clang 3.5
Boost 1_57
Grouping benchmarks into suites enables certain functionality like having different sets of parameters for different benchmarks.
This would help other projects grab this tool easily into their testing projects as a dependency.
This is a typical benchmark code I use with nonius:
nonius::benchmark my_benchmark()
{
return nonius::benchmark("title", [](nonius::chronometer meter)
{
// here I do some heavy setup; before actually running the benchmark
std::vector<int> foo(10);
meter.measure([&]()
{
my_method()
});
});
}
I've increased the size of the std::vector
to 1000
, and this has lead to higher sample values, although the object is not used at all. For my point of view I always thought nonius will measure only the time of the function call meter.measure([&]()
. But this seems to be not the case.
Is this a bug or a feature? Or am I totally wrong?
https://github.com/rmartinho/nonius/blob/devel/include/nonius/detail/stats.h++ uses boost::math::quantile
, boost::math::normal
, and boost::math::cdf
.
These need to be replaced with non-boost equivalents to remove the boost dependency.
https://github.com/rmartinho/nonius/blob/devel/include/nonius/detail/cpptempl.h uses boost::lexical_cast
, boost::trim_copy_if
, boost::trim_copy
, boost::is_space
, boost::split
, and boost::starts_with
. If the boost dependencies are to go away, these need to be replaced with non-boost equivalents.
I get a lot of compiler warning while compiling the 64 bit version of nonius.
Version: 1.0.0
e.g. something like this:
int quotes = std::count(first, last, '"');
Full MSVC 2013 64 bit output:
.../nonius/reporters/csv_reporter.h++(102): warning C4244: 'initializing' : conversion from '__int64' to 'int', possible loss of data
1>.../nonius/detail/escape.h++(32): warning C4244: 'initializing' : conversion from '__int64' to 'int', possible loss of data
1>.../nonius/reporters/junit_reporter.h++(94): warning C4244: 'initializing' : conversion from '__int64' to 'int', possible loss of data
1>.../nonius/detail/argparse.h++(73): warning C4267: 'return' : conversion from 'size_t' to 'int', possible loss of data
1> .../nonius/detail/argparse.h++(79) : see reference to function template instantiation 'int nonius::detail::get_max_width<std::_Vector_const_iterator<std::_Vector_val<std::_Simple_types<nonius::detail::option>>>,nonius::detail::<<::<lambda_ebcb31a12dfc7293f4c5c592439004ad>>(Iterator,Iterator,Projection)' being compiled
1> with
1> [
1> Iterator=std::_Vector_const_iterator<std::_Vector_val<std::_Simple_types<nonius::detail::option>>>
1> , Projection=nonius::detail::<<::<lambda_ebcb31a12dfc7293f4c5c592439004ad>
1> ]
1>.../nonius/detail/stats.h++(199): warning C4244: 'initializing' : conversion from '__int64' to 'int', possible loss of data
1> .../nonius/detail/analyse.h++(33) : see reference to function template instantiation 'nonius::detail::bootstrap_analysis nonius::detail::analyse_samples<std::_Vector_iterator<std::_Vector_val<std::_Simple_types<double>>>>(double,int,Iterator,Iterator)' being compiled
1> with
1> [
1> Iterator=std::_Vector_iterator<std::_Vector_val<std::_Simple_types<double>>>
1> ]
1> .../nonius/go.h++(88) : see reference to function template instantiation 'nonius::sample_analysis<boost::chrono::duration<double,boost::ratio<0x01,0x03b9aca00>>> nonius::detail::analyse<boost::chrono::duration<double,boost::ratio<0x01,0x03b9aca00>>,std::_Vector_iterator<std::_Vector_val<std::_Simple_types<boost::chrono::duration<double,boost::ratio<0x01,0x03b9aca00>>>>>>(nonius::configuration,nonius::environment<boost::chrono::duration<double,boost::ratio<0x01,0x03b9aca00>>>,Iterator,Iterator)' being compiled
1> with
1> [
1> Iterator=std::_Vector_iterator<std::_Vector_val<std::_Simple_types<boost::chrono::duration<double,boost::ratio<0x01,0x03b9aca00>>>>>
1> ]
1> .../nonius/go.h++(130) : see reference to function template instantiation 'void nonius::go<nonius::default_clock,std::_Vector_iterator<std::_Vector_val<std::_Simple_types<nonius::benchmark>>>>(nonius::configuration,Iterator,Iterator,nonius::reporter &)' being compiled
1> with
1> [
1> Iterator=std::_Vector_iterator<std::_Vector_val<std::_Simple_types<nonius::benchmark>>>
1> ]
1> .../nonius/main.h++(145) : see reference to function template instantiation 'void nonius::go<nonius::default_clock>(nonius::configuration,nonius::benchmark_registry &,nonius::reporter_registry &)' being compiled
1>.../nonius/detail/stats.h++(80): warning C4244: 'initializing' : conversion from '__int64' to 'int', possible loss of data
1> .../nonius/detail/estimate_clock.h++(64) : see reference to function template instantiation 'double nonius::detail::mean<std::_Vector_iterator<std::_Vector_val<std::_Simple_types<double>>>>(Iterator,Iterator)' being compiled
1> with
1> [
1> Iterator=std::_Vector_iterator<std::_Vector_val<std::_Simple_types<double>>>
1> ]
1> .../nonius/go.h++(42) : see reference to function template instantiation 'nonius::environment_estimate<boost::chrono::duration<double,boost::ratio<0x01,0x03b9aca00>>> nonius::detail::estimate_clock_resolution<Clock>(int)' being compiled
1> with
1> [
1> Clock=nonius::default_clock
1> ]
1> .../nonius/go.h++(73) : see reference to function template instantiation 'nonius::environment<boost::chrono::duration<double,boost::ratio<0x01,0x03b9aca00>>> nonius::detail::measure_environment<Clock>(nonius::reporter &)' being compiled
1> with
1> [
1> Clock=nonius::default_clock
1> ]
1>.../nonius/detail/stats.h++(42): warning C4244: 'initializing' : conversion from '__int64' to 'int', possible loss of data
1> .../nonius/detail/stats.h++(58) : see reference to function template instantiation 'double nonius::detail::weighted_average_quantile<std::_Vector_iterator<std::_Vector_val<std::_Simple_types<double>>>>(int,int,Iterator,Iterator)' being compiled
1> with
1> [
1> Iterator=std::_Vector_iterator<std::_Vector_val<std::_Simple_types<double>>>
1> ]
1> .../nonius/detail/analyse.h++(34) : see reference to function template instantiation 'nonius::outlier_classification nonius::detail::classify_outliers<std::_Vector_iterator<std::_Vector_val<std::_Simple_types<double>>>>(Iterator,Iterator)' being compiled
1> with
1> [
1> Iterator=std::_Vector_iterator<std::_Vector_val<std::_Simple_types<double>>>
1> ]
1>.../nonius/detail/stats.h++(97): warning C4244: 'initializing' : conversion from '__int64' to 'int', possible loss of data
1> .../nonius/detail/stats.h++(208) : see reference to function template instantiation 'nonius::detail::sample nonius::detail::resample<std::mt19937,Iterator,double(__cdecl *const )(Iterator,Iterator)>(URng &,int,Iterator,Iterator,Estimator &)' being compiled
1> with
1> [
1> Iterator=std::_Vector_iterator<std::_Vector_val<std::_Simple_types<double>>>
1> , URng=std::mt19937
1> , Estimator=double (__cdecl *const )(std::_Vector_iterator<std::_Vector_val<std::_Simple_types<double>>>,std::_Vector_iterator<std::_Vector_val<std::_Simple_types<double>>>)
1> ]
1> D:\Programmierung\IDE\Microsoft Visual Studio 12.0\VC\include\xmemory0(577) : while compiling class template member function 'std::_List_node<std::pair<const _Kty,_Ty>,void *> *std::allocator<_Other>::allocate(std::allocator<_Other>::size_type)'
1> with
1> [
1> _Kty=std::string
1> , _Ty=std::string
1> , _Other=std::_List_node<std::pair<const std::string,std::string>,void *>
1> ]
1> D:\Programmierung\IDE\Microsoft Visual Studio 12.0\VC\include\xmemory0(847) : see reference to function template instantiation 'std::_List_node<std::pair<const _Kty,_Ty>,void *> *std::allocator<_Other>::allocate(std::allocator<_Other>::size_type)' being compiled
1> with
1> [
1> _Kty=std::string
1> , _Ty=std::string
1> , _Other=std::_List_node<std::pair<const std::string,std::string>,void *>
1> ]
1> D:\Programmierung\IDE\Microsoft Visual Studio 12.0\VC\include\xmemory0(749) : see reference to class template instantiation 'std::allocator<_Other>' being compiled
1> with
1> [
1> _Other=std::_List_node<std::pair<const std::string,std::string>,void *>
1> ]
1> D:\Programmierung\IDE\Microsoft Visual Studio 12.0\VC\include\list(484) : see reference to class template instantiation 'std::_Wrap_alloc<std::allocator<_Other>>' being compiled
1> with
1> [
1> _Other=std::_List_node<std::pair<const std::string,std::string>,void *>
1> ]
1> D:\Programmierung\IDE\Microsoft Visual Studio 12.0\VC\include\list(690) : see reference to class template instantiation 'std::_List_base_types<_Ty,_Alloc>' being compiled
1> with
1> [
1> _Ty=std::pair<const std::string,std::string>
1> , _Alloc=std::allocator<std::pair<const std::string,std::string>>
1> ]
1> D:\Programmierung\IDE\Microsoft Visual Studio 12.0\VC\include\list(814) : see reference to class template instantiation 'std::_List_alloc<false,std::_List_base_types<_Ty,_Alloc>>' being compiled
1> with
1> [
1> _Ty=std::pair<const std::string,std::string>
1> , _Alloc=std::allocator<std::pair<const std::string,std::string>>
1> ]
1> D:\Programmierung\IDE\Microsoft Visual Studio 12.0\VC\include\list(862) : see reference to class template instantiation 'std::_List_buy<_Ty,_Alloc>' being compiled
1> with
1> [
1> _Ty=std::pair<const std::string,std::string>
1> , _Alloc=std::allocator<std::pair<const std::string,std::string>>
1> ]
1> D:\Programmierung\IDE\Microsoft Visual Studio 12.0\VC\include\xhash(269) : see reference to class template instantiation 'std::list<std::pair<const _Kty,_Ty>,std::allocator<std::pair<const _Kty,_Ty>>>' being compiled
1> with
1> [
1> _Kty=std::string
1> , _Ty=std::string
1> ]
1> D:\Programmierung\IDE\Microsoft Visual Studio 12.0\VC\include\unordered_map(81) : see reference to class template instantiation 'std::_Hash<std::_Umap_traits<_Kty,_Ty,std::_Uhash_compare<_Kty,_Hasher,_Keyeq>,_Alloc,false>>' being compiled
1> with
1> [
1> _Kty=std::string
1> , _Ty=std::string
1> , _Hasher=std::hash<std::string>
1> , _Keyeq=std::equal_to<std::string>
1> , _Alloc=std::allocator<std::pair<const std::string,std::string>>
1> ]
1> .../nonius/detail/argparse.h++(117) : see reference to class template instantiation 'std::unordered_map<std::string,std::string,std::hash<_Kty>,std::equal_to<_Kty>,std::allocator<std::pair<const _Kty,_Ty>>>' being compiled
1> with
1> [
1> _Kty=std::string
1> , _Ty=std::string
1> ]
1>.../nonius/detail/stats.h++(131): warning C4244: 'initializing' : conversion from '__int64' to 'int', possible loss of data
1> .../nonius/detail/stats.h++(209) : see reference to function template instantiation 'nonius::estimate<double> nonius::detail::bootstrap<Iterator,double(__cdecl *const &)(Iterator,Iterator)>(double,Iterator,Iterator,const nonius::detail::sample &,Estimator)' being compiled
1> with
1> [
1> Iterator=std::_Vector_iterator<std::_Vector_val<std::_Simple_types<double>>>
1> , Estimator=double (__cdecl *const &)(std::_Vector_iterator<std::_Vector_val<std::_Simple_types<double>>>,std::_Vector_iterator<std::_Vector_val<std::_Simple_types<double>>>)
1> ]
1>.../nonius/detail/stats.h++(145): warning C4267: 'initializing' : conversion from 'size_t' to 'int', possible loss of data
1>.../nonius/detail/stats.h++(114): warning C4244: 'initializing' : conversion from '__int64' to 'int', possible loss of data
1> .../nonius/detail/stats.h++(134) : see reference to function template instantiation 'nonius::detail::sample nonius::detail::jackknife<double(__cdecl *const &)(Iterator,Iterator),Iterator>(Estimator,Iterator,Iterator)' being compiled
1> with
1> [
1> Iterator=std::_Vector_iterator<std::_Vector_val<std::_Simple_types<double>>>
1> , Estimator=double (__cdecl *const &)(std::_Vector_iterator<std::_Vector_val<std::_Simple_types<double>>>,std::_Vector_iterator<std::_Vector_val<std::_Simple_types<double>>>)
1> ]
1> .../nonius/detail/stats.h++(209) : see reference to function template instantiation 'nonius::estimate<double> nonius::detail::bootstrap<Iterator,double(__cdecl *const &)(Iterator,Iterator)>(double,Iterator,Iterator,const nonius::detail::sample &,Estimator)' being compiled
1> with
1> [
1> Iterator=std::_Vector_iterator<std::_Vector_val<std::_Simple_types<double>>>
1> , Estimator=double (__cdecl *const &)(std::_Vector_iterator<std::_Vector_val<std::_Simple_types<double>>>,std::_Vector_iterator<std::_Vector_val<std::_Simple_types<double>>>)
1> ]
Additional:
Please get also rid of the C-Style casts: e.g. (int)(variable)
use instead static_cast(), That you can easily search for those casts and its also valid C++.
I've created a Conan-package for Nonius here. It works for me on Mac OS X, but I have no way of testing it on Windows.
Also, in order to configure the Boost dependency better, it would be useful to know for which versions of Visual Studio specifically it is required.
I'm happy for any feedback. If you have no objections, once it's confirmed working on Windows, I'd publish it in the Conan central repository.
Sometimes things other than time need to be measured. Nonius should support collecting additional metrics like, e.g., cache misses and branch mispredictions. Ideally this would be a generically extensible feature that users can use to collect any custom metrics they desire.
It will be nice to have output generator for markdown tables. (esp with sorting options).
Chandler Carruth introduced two interesting functions to control compiler optimization in fine-grained and unintrusive manner in his talk at CppCon2015. Nonius should provide these out of the box.
I wanted to compare to ways of checking whether a boost variant is empty
However, the which()==0
variant consistently eludes the benchmark algorithm by taking too much time to result in useful measurements. The name of testcase is which
and the benchmark results in the following messages:
benchmarking which
which failed to run successfully
benchmark aborted
As you suggested in the lounge:
(And I like that nonius has trouble benchmarking such code because it tells you that even if you coax some results out of it, they might be meaningless)
it might indicate code that's hard to benchmark.
Could you recommend a way to get useful comparison here? Perhaps a rearrangement or a tweak (warmup_time e.g.).
I'd actually be happy if I could clearly see that the test fails to "converge" because it always takes negligible time - because then we have a clear winner if the other variant runs in non-negligible time. I guess in this respect this issue is close to Issue #14
Here's my test program
#include <boost/variant.hpp>
#include <nonius/benchmark.h++>
#include <nonius/main.h++>
namespace detail {
struct is_blank : boost::static_visitor<bool> {
constexpr is_blank() = default;
constexpr bool operator()(boost::blank const&) const { return true; }
template<typename T>
constexpr bool operator()(T const&) const { return false; }
};
}
template <typename... Ts>
constexpr bool is_blank(boost::variant<Ts...> const& v) {
return boost::apply_visitor(detail::is_blank(), v);
}
#include <algorithm>
namespace /*statics*/ {
// initialization phase not part of benchmarks
using V = boost::variant<boost::blank, bool, int>;
static auto const test_data = [] {
std::vector<V> many;
std::generate_n(back_inserter(many), 1ul<<12, []()-> V {
V r;
switch (rand()%3) {
case 1: r = 42; break;
case 2: r = false; break;
}
return r; // NRVO (not critical)
});
return many;
}();
}
NONIUS_BENCHMARK("apply_visitor", [](nonius::chronometer c) {
c.measure([&](int i) {
return is_blank(test_data[ i % test_data.size() ]);
});
})
NONIUS_BENCHMARK("which", [](nonius::chronometer c) {
//for (auto & el : test_data) std::cout << (0 == el.which()) << " ";
//std::cout << '\n';
c.measure([&](int i) {
return (test_data[ i % test_data.size() ].which() == 0);
});
})
Typical output:
$ ./test
clock resolution: mean is 17.7222 ns (81920002 iterations)
benchmarking apply_visitor
collecting 100 samples, 1148 iterations each, in estimated 1722 μs
mean: 13.6744 ns, lb 13.541 ns, ub 14.0019 ns, ci 0.95
std dev: 1.05495 ns, lb 0.508851 ns, ub 1.72528 ns, ci 0.95
found 9 outliers among 100 samples (9%)
variance is severely inflated by outliers
benchmarking which
which failed to run successfully
benchmark aborted
Running benchmarks on my machine usually† ends up in an internal error:
benchmarking to_string(42)
collecting 100 samples, 3 iterations each, in estimated 1173 μs
PANIC: clock is on fire
Error in function boost::math::erfc_inv<double>(double, double): Overflow Error
Upon looking it was found out that the issue had to do with prob_n
being zero which led to an overflow error.
The reason for prob_n
being zero is because resample
was an array full of negative values (-10
) and none were different.
Upon further inspection, it turned out that the samples
vector was also full of negative values for some reason. Which isn't supposed to happen. Looking into what made the samples
negative led to this code:
auto elapsed = model.finished - model.started;
return ((elapsed - env.clock_cost.mean) / plan.iterations_per_sample);
It just so happens that elapsed
ends up being 0 too or somehow less than env.clock_cost.mean
which makes the value negative setting the whole explosion above to happen.
Using Boost.Chrono instead of std::chrono
seems to fix the issue. The true resolutions for each:
std::chrono
: clock resolution: mean is 11.6849 ns (40960002 iterations)However I don't really think std::chrono
is the issue since it's able to have nanosecond resolution required.
N.B.: You've given yourself a note:
I do need to guard against that, though. Since an empty run (i.e., just the cost of the clock) can possibly run faster than the mean clock cost, I need to clamp to 0.
Clamp to zero, and guard against uniform samples.
†: Sometimes it doesn't give this error but the statistics seems wrong. Using --no-analysis seems to work without triggering the error.
Now that #26 was merged, nonius::main
should be part of the documented interface.
The changes in #65 need to be documented.
Hey,
I've tried to compile the examples but I'm getting undefined reference to main
git log shows : 78df23a
g++ -o example2 example2.c++ --std=c++14 -I/home/rsoeldner/custom/nonius/include -I/home/rsoeldner/custom/nonius/include/nonius -lpthread
/usr/lib/../lib64/crt1.o: In function `_start':
/home/abuild/rpmbuild/BUILD/glibc-2.19/csu/../sysdeps/x86_64/start.S:118: undefined reference to `main'
collect2: error: ld returned 1 exit status
The interface for automatically register benchmarks is the following:
NONIUS_BENCHMARK("My Benchmark", [](nonius::chronometer meter)
{
//...
})
Because of that it is not possible to use #ifdef inside the benchmark itself.
it looks also not nice, see the end of the braces? })
This makes it harder for the client writing benchmarks.
I came up with following idea:
NONIUS_BENCHMARK("Group Sorting", "Sort 1")
{
meter.measure([&]()
{
// benchmark code
}
}
You can look in the testing framework catch, which you already use, how it is implemented there, but it's pretty easy. Basically the macro has to expand to a temporary function which gets as argument this nonius::chronometer meter
Additionally:
Like you can see, I would like to group benchmarks together, but this is another big request, but you should keep this in mind when redesigning the interface.
#65 is a breaking change, so it needs a migration strategy.
Enable the default runner to use more than one reporter at once. This allows, e.g., getting the "standard" output while collecting all samples in CSV, and/or while also generating charts.
Nonius should make a best effort attempt to collect useful available system information.
Compiling a tiny test project consisting only of example1.c++, I get the following error:
In file included from /home/claravu/nonius/include/nonius/configuration.h++:17:0,
from /home/claravu/nonius/include/nonius/benchmark.h++:18,
from /home/claravu/nonius/include/nonius/nonius.h++:18,
from /home/claravu/nonius/include/nonius/nonius_single.h++:14,
from /home/claravu/test_nonius/example1.c++:36:
/home/claravu/nonius/include/nonius/param.h++: In member function ‘nonius::param nonius::param::parse(const string&) const’:
/home/claravu/nonius/include/nonius/param.h++:103:38: error: use of deleted function ‘std::basic_stringstream::basic_stringstream(const std::basic_stringstream&)’
auto ss = std::stringstream{s};
Changing the offending line to:
std::stringstream ss;
ss.str(s);
Made the error go away, but I'm not 100% sure this it true to the intent. The error does not occur in v1.1.2.
I'm trying out nonius for benchmarking on a new project, looks pretty good so far!
Per this
MSVC should be added to the build system for automatic testing.
Autocorrelation analysis could help identify problems with benchmarks, or simply interesting behaviour in some code.
I have encountered a problem after updating nonious to the latest version of branch devel. Benchmarks that supposed to run very fast cause a freezing. For example
#include <memory>
#include <nonius/nonius.h++>
#include <nonius/main.h++>
NONIUS_BENCHMARK("shared_ptr<int> default constructor", [](nonius::chronometer meter)
{
nonius::storage_for<std::shared_ptr<int>> storage;
meter.measure([&storage] {
storage.construct();
});
});
produce the following output and then freezes:
clock resolution: mean is 16.1415 ns (40960002 iterations)
benchmarking shared_ptr default constructor
I guess that the bug appeared after latest commits in may/june because same benchmark had been running successfully before.
I compile it with gcc 4.9.3 on ubuntu 14.04. My machine is x64 Intel® Core™ i7-4790 CPU @ 3.60GHz × 8 with 16 Gb RAM.
I have a problem on a PC, which is quite fast. A benchmark failed, because it runs too fast.
The benchmark basically measure the time of a simple function call (a void function which does nothing)
obj.method_call();
On another machine I can run the benchmark (mean: 0.3ns).
How should I handle this kind of problem in general?
Another problem which I've seen is: For certain benchmarks I see sometimes sample ranges alternating between 0.0 ns and 300 ns. Is this a known problem? What can I do about it?
To get a correct result, I have to run the benchmark twice.
Hot code performs differently from cold code. Caches and stuff.
Currently execution order goes like this: benchmark0/sample0, benchmark0/sample1, ..., benchmark0/sampleN, benchmark1/sample0, benchmark1/sample1, ..., benchmark1/sampleN, ..., benchmarkM/sample0, benchmarkM/sample1, ..., benchmarkM/sampleN. This means that all executions of a single benchmark are sequenced together, which can lead to differences from caching effects.
It should be possible to execute different orders, like: benchmark0/sample0, benchmark1/sample0, ..., benchmarkM/sample0, benchmark0/sample1, benchmark1/sample1, ..., benchmarkN/sample1, ..., benchmark0/sampleN, benchmark1/sampleN, ..., benchmarkM/sampleN. This would alternate sampling between different benchmarks and reduce cache effects of locality.
Random ordering is also a possibility.
Hi!
Sorry for being the legalist party pooper...
As part of my work for #36 I was looking at how the Highcharts to see how it works... When reading the code in the template distributed in Nonius, I does not say what license Highcharts has, it just claims that the license is at www.highcharts.com/license. When one visits that page, one is redirected to a shop to buy a proprietary license.
Somewhere hidden there is an option to buy get a non-commercial license, based on Creative Commons Non Commercial. By reading their FAQ: https://shop.highsoft.com/faq#Non-Commercial-0 it seems to me that the way Nonius uses it is not following their terms, or at least the way they interpret them. E.g.
Non-Commercial Redistribution
You are allowed to distribute a Highsoft software product with non-commercial packages given that you fulfill two conditions:
- Emphasize to your users that Highsoft software products are not free for commercial and Governmental use. You can do this on your download page or when your users activate a Highsoft Software Product in your application.
- Provide a link back to highcharts.com in the same location.
May I use your Software under the Non-Commercial License for Open Source Projects?
Although Highsoft´s Software have open source codes, our software is not licensed as an open source software and are unfortunately not compatible with any open source software license like Apache 2 or any GPL. See also Non-Commercial Redistribution
I think that the whole licensing of that library is a mess. It is impossible to know under what license was the library distributed when it was integrated in Nonius, and if those terms allowed rewording. Was this Non-Commercial clause added after-the-fact?
Since their licensing is messy, seems hostile to open-source and even unclear for commercial purposes (they make interpretations in the FAQ that are not sure are correct from understanding CC-NC), I suggest replacing this library altogether. I don't have alternative suggestions for similar libraries, but I am going to look for them now and try to make a recommendation, and maybe implement it myself.
Cheers!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.