Hi,

I'm working on the ApprovalTests.cpp integration and I've ran into a couple issues:

• I need the filename from where the test is implemented (caller function). assertion_pass and assertion_fail have that information but not test_begin. As this information needs to be set before the test is actually run, test_begin needs to have that information. I've been looking at the UT code, but haven't quite figure out the best way to do it.

• In ApprovalTests.cpp, when a test fails it throws an exception which import information inside the exception (e.what()). So when a test throws an exception I would like to check it's contents (and possibility it's type), not simply see if the test threw an exception or not, as it's implemented now. What do you think it would be the best approach here. Simply to add a std::string to the exception struct and store the exception data there? Again not sure the best approach to change the code here.

  Any help would be appreciated. If you want to take a look at the integration between ApprovalTests.cpp and UT take a look at this pull request at the UTApprovals.h file.

  Thanks!

Is it possible to have the predicate printed for constant expressions when there is a failure?

Take the following code as an example, which is taken from the tutorial but evaluated at compile-time.

int main() {
  using namespace boost::ut;
  expect(constant<1_i == 2>);
}

Is it possible to get output similar to the following at run-time where the predicate is shown on failure?

main.cpp:4:FAILED [1 == 2]
===============================================================================
tests:   0 | 0 failed
asserts: 1 | 0 passed | 1 failed

Expected Behavior

Did not find implementation for pattern "Given I have a victor"
Did not find implementation for pattern "When I resize begger"
Did not find implementation for pattern "Then The size suould increase"
All tests passed (3 fatal in 1 tests)

Actual Behavior

All tests passed (0 asserts in 1 tests)

Steps to Reproduce the Problem

https://godbolt.org/z/xxEEGo

int main() {
using namespace boost::ut;

bdd::gherkin::steps steps = [](auto& steps) {
steps.feature("Vector") = [&] {
steps.scenario("*") = [&] {
steps.given("I have a vector") = [&] {
std::vector v(5);
expect((5_ul == std::size(v)) >> fatal);
steps.when("I resize bigger") = [&] { v.resize(10); };
steps.then("The size should increase") = [&] { expect(10_ul == std::size(v)); };
};
};
};
};

"Vector"_test = steps |
R"(
Feature: Vector
Scenario: Resize
Given I have a victor
When I resize begger
Then The size suould increase
)";
}

Specifications

• Version: master
• Platform:
• Subsystem:

In the following example:

"args"_test =
   [](const auto& arg) {
      expect(arg >= 1_i);
    }
  | std::vector{1, 2, 3};

You use a custom integer literal. Why? I noticed there is afair amount of code in the library to support this so I was just curious what its purpose is as it works fine without the literals.

Expected Behavior

That this code compiles:

        catch (const std::exception& e)
        {
            expect(approvalMissingExceptionReceived == true_b)
                << "Unexpected exception received: " << e.what();
        }

Actual Behavior

Compiler error:

.../ApprovalTests.cpp/tests/UT_Tests/UTApprovalTestTests.cpp:34:56: error: use of undeclared identifier 'true_b'
            expect(approvalMissingExceptionReceived == true_b)

Steps to Reproduce the Problem

With v 1.1.7:

• write a test using the following:

expect(false_b == true_b);

With v1.1.8:

• Compile that test, and observe it fail to compile

Specifications

• Version: 1.1.7
• Platform: macOS 10.15.7
• Subsystem:

/opt/local/bin/clang-mp-9.0 --version
clang version 9.0.1 
Target: x86_64-apple-darwin19.6.0
Thread model: posix
InstalledDir: /opt/local/libexec/llvm-9.0/bin

Would it be possible to disable tests running as part of the build? It makes it difficult to incorporate this project as sub-build when failing tests block building the rest of the project. It would be nice to be able to then run all of the tests with a CTest invocation after the build step.

When configuring the [Boost].µT project with CMake, I receive the following warning:

CMake Warning (dev) at cmake-build-debug/_deps/ut-src/CMakeLists.txt:41 (elseif):
  Policy CMP0054 is not set: Only interpret if() arguments as variables or
  keywords when unquoted.  Run "cmake --help-policy CMP0054" for policy
  details.  Use the cmake_policy command to set the policy and suppress this
  warning.

  Quoted variables like "MSVC" will no longer be dereferenced when the policy
  is set to NEW.  Since the policy is not set the OLD behavior will be used.

It looks like setting the minimum CMake version to 3.1 would silence this, according to the documentation of CMP0054.

Expected Behavior

If C++17* support (*with limitations) is advertised, things should mostly work when in C++17 mode.

Actual Behavior

Even

int main() {
  boost::ut::expect(true);
}

fails to compile with MSVC saying:

[build] C:\Users\mnagy\Source\Repos\ut\.vscode\install\include\boost/ut.hpp(1402): error C7555: use of designated initializers requires at least '/std:c++latest'
[build] C:\Users\mnagy\Source\Repos\ut\.vscode\install\include\boost/ut.hpp(1402): note: This diagnostic occurred in the compiler generated function 'auto boost::ext::ut::v1_1_8::runner<TReporter,MaxPathSize>::on(boost::ext::ut::v1_1_8::events::test<Ts...>)'

This issue is not unique to MSVC, even Appveyor build with clang-cl have output such as:

[57/112] Building CXX object example\CMakeFiles\boost_ut_parameterized.dir\parameterized.cpp.obj
..\example\parameterized.cpp(30,20): warning: explicit template parameter list for lambdas is a C++20 extension [-Wc++20-extensions]
  "types"_test = []<class T>() {
                   ^

When the entry points to testing use C++20 features, I naturally don't expect Boost UT to work in C++17 mode, but most features should have C++17 workarounds (hence the asterisk "with limitations").

Either decide whether the library supports C++17 or not. If yes, actually specify/mark which features are known to not work. Run strict C++17 builds in CI (-Werror) and disable such tests on purpose if there are no workarounds implemented.

BTW, this regression was introduced in v1.1.5, as installing tagged version of v1.1.4 compiles the simplest example as intended.

Steps to Reproduce the Problem

cmake_minimum_required(VERSION 3.0)

project(Test-Boost.UT
  LANGUAGES CXX
)

if (MSVC)
  string(REGEX REPLACE "/W[0-9]" "" CMAKE_CXX_FLAGS ${CMAKE_CXX_FLAGS})
endif (MSVC)

find_package(ut REQUIRED)

add_executable(Test1 Test1.cpp)

target_link_libraries(Test1
  PRIVATE
    boost::ut
)

target_compile_options(Test1
  PRIVATE
    $<$<CXX_COMPILER_ID:MSVC>:
      /W4          # Turn on all (sensible) warnings
      /permissive- # Turn on strict language conformance
      /EHsc        # Specify exception handling model
    >
)

2. C:\Kellekek\Microsoft\VisualStudio\2019\BuildTools\VC\Tools\MSVC\14.26.28801\bin\Hostx64\x64\cl.exe /nologo /TP -IC:\Users\mnagy\Source\Repos\ut\.vscode\install\include /DWIN32 /D_WINDOWS /GR /EHsc /MDd /Zi /Ob0 /Od /RTC1 /W4 /permissive- /EHsc /std:c++17 /showIncludes /FoCMakeFiles\Test1.dir\Test1.cpp.obj /FdCMakeFiles\Test1.dir\ /FS -c ..\..\Test1.cpp

Specifications

• Version: 1.1.5+
• Platform: Windows
• Subsystem: ?

This is more of a question than a bug report, as I haven't managed to narrow down my configuration enough to give you a minimal repro.

The question is: is the comma operator in this code intentional, and if so, could a static cast be added to silence the warning?

In some of my builds, where I treat warnings as errors, I am getting this:

/Users/clare/Documents/develop/ApprovalTests/ApprovalTests.cpp.CMakeSamples/ut/include/boost/ut.hpp:210:9: error: possible misuse of comma operator here [-Werror,-Wcomma]
    ++pi, ++si;
        ^
/Users/clare/Documents/develop/ApprovalTests/ApprovalTests.cpp.CMakeSamples/ut/include/boost/ut.hpp:210:5: note: cast expression to void to silence warning
    ++pi, ++si;
    ^~~~
    static_cast<void>( )
1 error generated.

Specifications

• Version: bba087c
• Platform: macOS
• Subsystem:

/opt/local/bin/clang-mp-9.0 --version
clang version 9.0.1 
Target: x86_64-apple-darwin19.6.0
Thread model: posix
InstalledDir: /opt/local/libexec/llvm-9.0/bin

Expected Behavior

A function with a throws / nothrow assertion should compile with the Visual C++ Compiler when given a lambda which uses a constexpr std::array variable... right? This is a very specific issue 😩

#include <array>
#include <boost/ut.hpp>

int main() {
  using namespace boost::ut;
  "example_test"_test = [] {
    constexpr std::array<int, 1> i{{10}};
    expect(nothrow([] { return i.front(); }));
  };
}

It is also interesting to note this only occurs in a test case and not when used directly from main, hence the following compiles:

#include <array>
#include <boost/ut.hpp>

int main() {
  using namespace boost::ut;
  expect(nothrow([] {
    constexpr std::array<int, 1> i{{10}};
    return i.front();
  }));
}

Using a constexpr int instead of a std::array also compiles:

#include <boost/ut.hpp>

int main() {
  using namespace boost::ut;
  "example_test"_test = [] {
    constexpr int i{10};
    expect(nothrow([] { return i; }));
  };
}

Perhaps this is a bug in the MSVC compiler?

Actual Behavior

Error C2326 is given stating that the lambda function cannot access the constexpr variable.

I can workaround this issue by putting the constexpr variable within the lambda itself, as follows:

#include <array>
#include <boost/ut.hpp>

int main() {
  using namespace boost::ut;
  "example_test"_test = [] {
    expect(nothrow([] {
      constexpr std::array<int, 1> i{{10}};
      return i.front();
    }));
  };
}

Steps to Reproduce the Problem

Compiling the preceding snippets above using C++20 / MSVC 16.4.4 should showcase the error.
By the way, Clang compiles the problematic code just fine.

Specifications

• Version: 1.1.6
• Platform: Windows
• Subsystem: MSVC 16.4.4

Expected Behavior

Boost.ut builds and tests pass with VS2019 MSVC with clang-cl compiler

I know this, as we have been building that with the Approval Tests integration, which works file with this compiler.

Actual Behavior

Builds fail, with the error:

  Building Custom Rule D:/a/ApprovalTests.cpp.Builds/ApprovalTests.cpp.Builds/_deps/ut-src/example/CMakeLists.txt
clang-cl : error : unknown argument ignored in clang-cl: '-pedantic' [-Werror,-Wunknown-argument] [D:\a\ApprovalTests.cpp.Builds\ApprovalTests.cpp.Builds\_deps\ut-build\example\BDD.vcxproj]
clang-cl : error : unknown argument ignored in clang-cl: '-pedantic-errors' [-Werror,-Wunknown-argument] [D:\a\ApprovalTests.cpp.Builds\ApprovalTests.cpp.Builds\_deps\ut-build\example\BDD.vcxproj]

For an example failure, see https://github.com/claremacrae/ApprovalTests.cpp.Builds/runs/370748554

Steps to Reproduce the Problem

1. cmake -G "Visual Studio 16 2019" -T "clangcl"
2. build all of Boost.ut, including examples

Specifications

• Version: 8004290
• Platform: 'The CXX compiler identification is Clang 9.0.0 with MSVC-like command-line'
• Subsystem: ?

Nature of fix

See https://github.com/approvals/ApprovalTests.cpp/blob/v.7.0.0/tests/UT_Tests/CMakeLists.txt#L17 for the type of check to use...

Someone previously reported this problem with ApprovalTests.cpp on this environment, and provided a fix which used windows-style compiler warnings if using clang-cl...

Expected Behavior

find_package(ut) # works

Actual Behavior

cmake -S test -B build/test -G Ninja -DCMAKE_PREFIX_PATH=/Users/clausklein/Workspace/cpp/ut/root -DTEST_INSTALLED_VERSION=1
-- The CXX compiler identification is GNU 10.2.0
-- Checking whether CXX compiler has -isysroot
-- Checking whether CXX compiler has -isysroot - yes
-- Checking whether CXX compiler supports OSX deployment target flag
-- Checking whether CXX compiler supports OSX deployment target flag - yes
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/local/bin/g++-10 - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Error at CMakeLists.txt:11 (find_package):
  Could not find a configuration file for package "ut" that is compatible
  with requested version "1.1".

  The following configuration files were considered but not accepted:

    /Users/clausklein/Workspace/cpp/ut/root/lib/cmake/ut/ut-config.cmake, version: unknown

-- Configuring incomplete, errors occurred!

Steps to Reproduce the Problem

diff --git a/test/CMakeLists.txt b/test/CMakeLists.txt
index b00892a..00cb6fb 100644
--- a/test/CMakeLists.txt
+++ b/test/CMakeLists.txt
@@ -4,5 +4,12 @@
# Distributed under the Boost Software License, Version 1.0.
# (See accompanying file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
#
+cmake_minimum_required(VERSION 3.14...3.20)
+project(boost.ut LANGUAGES CXX)
+
+if(TEST_INSTALLED_VERSION)
+    find_package(ut 1.1 REQUIRED)
+endif()
+
add_subdirectory(ut)
add_subdirectory(ft)

Specifications

• Version: 1.1.8
• Platform: Apple OSX
• Subsystem: Clang

I am very interested in using your library. I would like to use FetchContent and have access to your library through add_subdirectory. However, your library does not create a convenience target to use with target_link_libraries. While it is little effort for me to add the necessary include directory myself, I would much rather just "link" against your header only library like so:

target_link_libraries(MyTarget PRIVATE Boost::ut)

I have included an import prefix to mirror Boost's component linking behavior.
Defining an interface library through CMake with the appropriate include directories will make this possible.
The changes might look like the following.

add_library(ut INTERFACE)
add_library(Boost::ut ALIAS ut)
target_include_directories(ut INTERFACE $<BUILD_INTERFACE:${PROJECT_SOURCE_DIR}/include>)

This code also paves the way for the interface target to be exported and included with find_package.

Hi,

I really like the [Boost].UT framework. It's clean and easy to use and does away with macros. Thanks for developing it!

Anyways, I want to use approval tests (https://github.com/approvals/ApprovalTests.cpp) in a project I'm working on, and I'm implementing the integration between UT <> ApprovalTests.cpp. I followed the guidelines (https://github.com/approvals/ApprovalTests.cpp/blob/master/doc/SupportingNewTestFramework.md#top) and the UT reporter example (https://godbolt.org/z/gsAPKg). I made this simple example:

#include <string>

#include "ut.hpp"
#include "ApprovalTests.hpp"

namespace ut = boost::ut;

namespace ApprovalTests
{
    namespace cfg {
        class reporter : public ut::reporter<ut::printer> {
            private:
                TestName currentTest;

            public:
                auto on(ut::events::test_begin test_begin) -> void {
                    std::string name = test_begin.name;
                    currentTest.sections.emplace_back(name);
                    ut::reporter<ut::printer>::on(test_begin);
                }
                auto on(ut::events::test_run test_run) -> void { 
                    ut::reporter<ut::printer>::on(test_run);
                }
                auto on(ut::events::test_skip test_skip) -> void { ut::reporter<ut::printer>::on(test_skip); }
                auto on(ut::events::test_end test_end) -> void {
                    while (!currentTest.sections.empty()) {
                        currentTest.sections.pop_back();
                    }
                    ut::reporter<ut::printer>::on(test_end);
                }
                template <class TMsg>
                auto on(ut::events::log<TMsg>) -> void {}
                template <class TLocation, class TExpr>
                auto on(ut::events::assertion_pass<TLocation, TExpr> location) -> void {
                    currentTest.setFileName(location.location.file_name());
                    ApprovalTestNamer::currentTest(&currentTest);
                    ut::reporter<ut::printer>::on(location);
                }
                template <class TLocation, class TExpr>
                auto on(ut::events::assertion_fail<TLocation, TExpr> fail) -> void { ut::reporter<ut::printer>::on(fail); }
                auto on(ut::events::fatal_assertion fatal) -> void { ut::reporter<ut::printer>::on(fatal); }
                auto on(ut::events::exception exception) -> void { ut::reporter<ut::printer>::on(exception); }
                auto on(ut::events::summary summary) -> void { ut::reporter<ut::printer>::on(summary); }
        };
    }  // namespace cfg
}

template <>
auto ut::cfg<ut::override> = ut::runner<ApprovalTests::cfg::reporter>{};

int main()
{
    using namespace ut;
    using namespace ApprovalTests;

	"Approval"_test = []() {
        //expect(true);
        expect(nothrow([] { Approvals::verify("Approval Tests can verify text via the golder master method"); }));
	};

}

The issue is unless the line containing expect(true); is uncommented it doesn't work. It complains that currentTest was not configured. This seems more like a UT issue than an ApprovalTests issue.

Can anyone help? I'm not familiar enough with the UT code logic to track this issue. Or am I wrong and this is an ApprovalTests.cpp issue?

I appreciate any help.

This is a report about some issues that occurred as I worked through the Tutorial.

Environment

OS: bento/ubuntu-20.04

Compiler: gcc version 9.3.0 (Ubuntu 9.3.0-10ubuntu2)
          c++ -> x86_64-linux-gnu-g++-9

Project Directory Structure

./ut-tutorial/
  |
  ├── ut.hpp
  |
  └── main.cpp

Tutorial Executions: Step 0 "Get it..."

import boost.ut;

int main() { }

$ c++ -std=c++2a main.cpp && ./a.out
main.cpp:1:1: error: ‘import’ does not name a type
    1 | import boost.ut;
      | ^~~~~~

Compilation with flag -std=c++2a  and statement import boost.ut; failed.

#include "ut.hpp"

int main() { }

$  c++ -std=c++17 main.cpp
In file included from main.cpp:1:
ut.hpp: In member function ‘void boost::ut::v1_1_7::bdd::gherkin::steps::next(const TPattern&)’:
ut.hpp:2457:22: warning: range-based ‘for’ loops with initializer only available with ‘-std=c++2a’ or ‘-std=gnu++2a’
 2457 |     for (auto i = 0; const auto& step : gherkin_) {
      |                      ^~~~~

Compilation with flag "-std=c++17" results in a warning flag.

$ c++ -std=c++2a main.cpp && ./a.out
-bash: ./a.out: Permission denied
$ ./a.out
$

The tutorial statement $CXX main.cpp && ./a.out assumes something about permissions I needed to separate the commands. I found nothing on Google on how to address this. How did the combined command work for you?

⋮

Tutorial Executions: Step 2 "Group it..."

⋮

#include "ut.hpp"
#include <vector>

int main() {
  using namespace boost::ut;
  "[vector]"_test = [] {
    std::vector<int> v(5);

    !expect(5_ul == size(v));

    should("resize bigger") = [v] {
      mut(v).resize(10);
      expect(10_ul == size(v));
    };

    !expect(5_ul == size(v));

    should("resize smaller") = [=]() mutable {
      v.resize(0);
      expect(0_ul == size(v));
    };
  };
}

$  c++ -std=c++2a main.cpp
$ ./a.out
All tests passed (4 asserts in 1 tests)
$

$ c++ -std=c++2a -Wall main.cpp
In file included from main.cpp:1:
ut.hpp: In instantiation of ‘void boost::ut::v1_1_7::bdd::gherkin::steps::next(const TPattern&) [with TPattern = std::__cxx11::basic_string<char>]’:
ut.hpp:2374:35:   required from here
ut.hpp:2458:15: warning: comparison of integer expressions of different signedness: ‘int’ and ‘long unsigned int’ [-Wsign-compare]
 2458 |       if (i++ == step_) {
      |           ~~~~^~~~~~~~

$ ./a.out
All tests passed (4 asserts in 1 tests)
$

Compilation with warning flag "-Wall" resulted in a warning about comparing signed and unsigned integers.

Expected Behavior

Expected example/reporters to be built and run without a problem

Actual Behavior

It segfaults with the following backtrace

#112224 0x0000000000407b2d in boost::ut::v1_1_7::detail::neq_<std::basic_string_view<char, std::char_traits >, std::basic_string_view<char, std::char_traits > >::neq_ (this=0x7fffffffccb0, lhs=..., rhs=...) at ../include/boost/ut.hpp:766
#112225 0x0000000000406fa8 in boost::ut::v1_1_7::operators::operator!= (lhs=..., rhs=...) at ../include/boost/ut.hpp:1860
#112226 0x0000000000407abe in ZZN5boost2ut6v1_1_76detail4neq_ISt17basic_string_viewIcSt11char_traitsIcEES7_EC4ERKS7_SA_ENKUlvE_clEv (this=0x7fffffffcd90) at ../include/boost/ut.hpp:764
#112227 0x0000000000407b2d in boost::ut::v1_1_7::detail::neq<std::basic_string_view<char, std::char_traits >, std::basic_string_view<char, std::char_traits > >::neq_ (this=0x7fffffffcd90, lhs=..., rhs=...) at ../include/boost/ut.hpp:766
#112228 0x0000000000406fa8 in boost::ut::v1_1_7::operators::operator!= (lhs=..., rhs=...) at ../include/boost/ut.hpp:1860
#112229 0x00000000004066dd in operator() (__closure=0x0) at ../example/cfg/reporter.cpp:41
#112230 0x000000000040671d in _FUN () at ../example/cfg/reporter.cpp:42
#112231 0x00000000004072d7 in boost::ut::v1_1_7::events::test<void ()(), boost::ut::v1_1_7::none>::run_impl (test=0x40670f <_FUN()>) at ../include/boost/ut.hpp:504
#112232 0x0000000000407301 in boost::ut::v1_1_7::events::test<void ()(), boost::ut::v1_1_7::none>::operator() (this=0x7fffffffcf10) at ../include/boost/ut.hpp:500
#112233 0x0000000000407504 in boost::ut::v1_1_7::runner<cfg::reporter, 16>::on<void ()(), boost::ut::v1_1_7::none> (this=0x60c280 <boost::ut::v1_1_7::cfgboost::ut::v1_1_7::override>, test=...) at ../include/boost/ut.hpp:1364
#112234 0x0000000000407678 in boost::ut::v1_1_7::detail::on<, boost::ut::v1_1_7::events::test<void ()(), boost::ut::v1_1_7::none> > (event=<unknown type in /home/furkan/.local/cpp/ut/build/example/reporter, CU 0x0, DIE 0xaa46>) at ../include/boost/ut.hpp:1563
#112235 0x00000000004076d7 in boost::ut::v1_1_7::detail::test::operator=<>(boost::ut::v1_1_7::detail::test_location<void (*)()>) (this=0x7fffffffd030, test=...) at ../include/boost/ut.hpp:1586
#112236 0x0000000000406794 in main () at ../example/cfg/reporter.cpp:39

Specifications

• Version: g++ 10.0.1 (20200327)
• Platform: Linux
• Subsystem: Centos 7

Edit: When I comment those operators, this time I get the same error in example/expect but this time string instead of string_view

Could you add an option to disable the compilation of the examples/tests so that a make install just installs the header into a desired prefix with no compilation steps?

Expected Behavior

The following should compile and work:

    "test bool"_test = [] {
        std::unique_ptr<int> ptr;
        expect(ptr);
    };

Actual Behavior

This does not compile:

/home/user/working/bsl/tests/ifarray.cpp: In lambda function:
/home/user/working/bsl/tests/ifarray.cpp:66:19: error: no matching function for call to ‘expect(std::unique_ptr<int>&)’
   66 |         expect(ptr);
      |                   ^
In file included from /home/user/working/bsl/tests/ifarray.cpp:26:
/home/user/working/bsl/tests/ut.h:1606:9: note: candidate: ‘template<class TExpr, typename std::enable_if<(is_op_v<TExpr> || is_same_v<bool, TExpr>), int>::type <anonymous> > constexpr auto boost::ut::v1_0_0::expect(const TExpr&, const std::experimental::fundamentals_v2::source_location&)’
 1606 |         expect(
      |         ^~~~~~
/home/user/working/bsl/tests/ut.h:1606:9: note:   template argument deduction/substitution failed:
/home/user/working/bsl/tests/ut.h:1604:79: error: no type named ‘type’ in ‘struct std::enable_if<false, int>’
 1604 |                 type_traits::is_op_v<TExpr> or std::is_same_v<bool, TExpr>> = 0>
      |                                                                               ^
make[2]: *** [tests/CMakeFiles/test_ifarray.dir/build.make:63: tests/CMakeFiles/test_ifarray.dir/ifarray.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:246: tests/CMakeFiles/test_ifarray.dir/all] Error 2
make: *** [Makefile:130: all] Error 2

Steps to Reproduce the Problem

1. Just compile the above test

Specifications

• Version: master
• Platform: Fedora 31
• Subsystem: ???

Printing values when a test fails doesn't seem to work with BOOST_UT_FORWARD.
Given the two files:
a.cpp:

#define BOOST_UT_IMPLEMENTATION
#include <boost/ut.hpp>
int main () {}

and b.cpp

#define BOOST_UT_FORWARD
#include <boost/ut.hpp>
using namespace boost::ut;
suite b = []
{
    expect (that % 1 == 2);
};

and the command line

g++ -std=c++2a  -o a a.cpp b.cpp && ./a

I get

  b.cpp:8:FAILED [false]
===============================================================================
tests:   0 | 0 failed
asserts: 1 | 0 passed | 1 failed

The values of the test are not printed. Move the test into a.cpp (or put some tests into this file) and the values are printed. E.g. replace a.cpp above with

#define BOOST_UT_IMPLEMENTATION
#include <boost/ut.hpp>
    using namespace boost::ut;
suite a = []
{
    expect (that % 1 == 2);
};
int main () {}

and the result is

  a.cpp:6:FAILED [1 == 2]
  b.cpp:6:FAILED [1 == 2]
===============================================================================
tests:   0 | 0 failed
asserts: 2 | 0 passed | 2 failed

i.e. both tests print their values.

Example

A fatal assertion with logged output for failure.

expect((true == false) >> fatal) << "Incorrect!";

Expected Behavior

When the assertion fails, "Incorrect!" should be printed.

FAILED [false] Incorrect!

Actual Behavior

When the assertion fails, "Incorrect!" is not printed.

FAILED [false]

Steps to Reproduce the Problem

1. A failing fatal assertion with streamed output will fail to print the output.

Specifications

• Version: v1.1.8
• Platform: macOS 10.15 Catalina
• Subsystem: MacPorts, LLVM Clang 10.0.1

https://github.com/boost-experimental/ut/blob/dde2cba5123444faa82e5c17fcd99f68fe991d89/include/boost/ut.hpp#L181

On https://github.com/boost-experimental/ut/blob/dde2cba5123444faa82e5c17fcd99f68fe991d89/include/boost/ut.hpp#L176 is was checked for emptiness. So, it is redundant on line 181

Expected Behavior

All tests passed (1 asserts in 1 tests)

Actual Behavior

[32mAll tests passed[0m (1 asserts in 1 tests)

Steps to Reproduce the Problem

1.Download a modern Mingw (GCC-10/Clang-11) compiler from http://winlibs.com/
1.No installation required
1.Run test: "run"_test = [] {expect(42_i==42);};

Specifications

• Version: 1.1.8
• Platform: OS Windows
• Subsystem:

Thanks!

Expected Behavior

The tests should allow the extraction of coverage data.

Actual Behavior

That's not the case.
Coverage data is extracted from the rest of the program (so it's not a problem of options) but not from anything that runs within the tests. It worked just fine with doctest too but nothing with UT.
The gcno / gcda files are found just fine by lcov, it captures everything that ran outside of UT, but nothing from inside (either from a suite or not).
Maybe an option I missed somewhere?

Steps to Reproduce the Problem

Not sure, it fails every time on my system and I haven't done anything particularly strange.
I'm using exclusively the BDD but tests made outside of BDD don't show up either.

Specifications

• Version: 2020/12/11 (commit 572f180)
• Platform: Linux
• Subsystem: Arch
  With gcc g++ (GCC) 11.0.0 20201016 (experimental) [devel/c++-modules ea1205e3bbe??M:20201020-1848], c++20 activated.

Expected Behavior

ut::log supporting at least std::string.

Actual Behavior

ut::log supports only compile-time known char[] string literals.
utility::string_view duplicates some std::string_view functionality, but is incomplete and not convertible.

Specifications

• Version: latest master (b937560)
• Platform: Windows Clang/LLVM 10 HEAD & MSVC 19.24.28307.1

I am not a language lawyer, but I think it would be nice if mut was documented in a sense that if it is technically UB(as used in examples in tutorial), but it works in practice then that is is mentioned (in docs or code comments).

cppref says this:

const_cast makes it possible to form a reference or pointer to non-const type that is actually referring to a const object or a reference or pointer to non-volatile type that is actually referring to a volatile object. Modifying a const object through a non-const access path and referring to a volatile object through a non-volatile glvalue results in undefined behavior.

Apologies in advance if I misunderstood the cppref quote.

Expected Behavior

The runner example should build and run successfully.

Actual Behavior

The runner example program segfaults.

Steps to Reproduce the Problem

1. mkdir build
2. cmake ..
3. cmake --build .

Specifications

• Version: Current master (commit hash 35ac252)
• Platform: macOS 10.15 Catalina
• Toolchain: CMake 3.16.2 / Ninja / XCode / MacPorts LLVM Clang 9.0.0

In case you're interested in upcoming versions of GCC, ut fails to compile with g++ (GCC) 10.0.0 20200108 (experimental)

$ g++ -std=c++2a -Wall -Wextra -I../../external/boost_experimental/ut/include   -c -o tests.o tests.cpp
In file included from tests.cpp:5:
../../external/boost_experimental/ut/include/boost/ut.hpp: In instantiation of ‘constexpr boost::ut::v1_1_6::detail::neq_<TLhs, TRhs>::neq_(const TLhs&, const TRhs&) [with TLhs = std::basic_string_view<char>; TRhs = std::basic_string_view<char>]’:
../../external/boost_experimental/ut/include/boost/ut.hpp:1747:31:   required from here
../../external/boost_experimental/ut/include/boost/ut.hpp:614:30: error: return type of ‘constexpr auto boost::ut::v1_1_6::operators::operator==(std::string_view, std::string_view)’ is not ‘bool’
  614 |             return get(lhs_) != get(rhs_);
      |                    ~~~~~~~~~~^~~~~~~~~~~~
../../external/boost_experimental/ut/include/boost/ut.hpp:614:30: note: used as rewritten candidate for comparison of ‘std::basic_string_view<char>’ and ‘std::basic_string_view<char>’
../../external/boost_experimental/ut/include/boost/ut.hpp:616:12: error: cannot convert ‘<brace-enclosed initializer list>’ to ‘const bool’ in initialization
  616 |         }()} {}
      |            ^
make: *** [<builtin>: tests.o] Error 1

At first glance I'd say that the call to operator!=(std::string_view, std::string_view) in neq_ uses the overload of operator==(std::string_view, std::string_view defined just above, which of course doesn't return a bool. Adding a using std::operator== solves the issue.

Now as to why GCC 9 doesn't complain, I have no idea.

Expected Behavior

The ApprovalTests.cpp UT integration tests pass with the latest version of this project on macOS.

This combination passes:

• ApprovalTest.cpp: 63acddcf2d0833e5e757f801f3f3c821f110c581
• ut: 8004290

Actual Behavior

On macOS, with both g++9 homebrew and clang-9 MacPorts, using 515c080, the UT tests all fail with:

*****************************************************************************
*                                                                           *
* Welcome to Approval Tests.
*
* There seems to be a problem with your build configuration.
* We cannot find the test source file at:
*   unknown
*
* For details on how to fix this, please visit:
* https://github.com/approvals/ApprovalTests.cpp/blob/master/doc/TroubleshootingMisconfiguredBuild.md
*                                                                           *
*****************************************************************************

Steps to Reproduce the Problem

• Clone ApprovalTest.cpp: 63acddcf2d0833e5e757f801f3f3c821f110c581
• Copy ut header 515c080 over ApprovalTests.cpp/third_party/ut/include/boost/ut.hpp
• Build on Mac
• Run UT_Tests

Specifications

It's failing for me with:

CMAKE_VERSION = 3.16.2
CMAKE_GENERATOR = Unix Makefiles
CMAKE_SOURCE_DIR = /Users/clare/Documents/develop/ApprovalTests/ApprovalTests.cpp
CMAKE_CURRENT_SOURCE_DIR = /Users/clare/Documents/develop/ApprovalTests/ApprovalTests.cpp
CMAKE_CURRENT_BINARY_DIR = /Users/clare/Documents/develop/ApprovalTests/ApprovalTests.cpp/cmake-build-debug-g-9-brew
CMAKE_CXX_COMPILER = /usr/local/Cellar/gcc/9.2.0_2/bin/g++-9
CMAKE_CXX_COMPILER_ID = GNU
CMAKE_CXX_COMPILER_VERSION = 9.2.0
CMAKE_UNITY_BUILD = 

and

CMAKE_VERSION = 3.16.2
CMAKE_GENERATOR = Unix Makefiles
CMAKE_SOURCE_DIR = /Users/clare/Documents/develop/ApprovalTests/ApprovalTests.cpp
CMAKE_CURRENT_SOURCE_DIR = /Users/clare/Documents/develop/ApprovalTests/ApprovalTests.cpp
CMAKE_CURRENT_BINARY_DIR = /Users/clare/Documents/develop/ApprovalTests/ApprovalTests.cpp/cmake-build-debug-clang-mp-90-brew
CMAKE_CXX_COMPILER = /opt/local/bin/clang++-mp-9.0
CMAKE_CXX_COMPILER_ID = Clang
CMAKE_CXX_COMPILER_VERSION = 9.0.1
CMAKE_UNITY_BUILD = 

Likely cause

There are not many differences between 8004290 and 515c080

I suspect that the cause is this, as in both compiler cases, __APPLE__ is defined:

@@ -211,10 +211,10 @@ namespace reflection {
 class source_location {
  public:
   [[nodiscard]] static constexpr auto current(
-#if (__GNUC__ >= 9 or __clang_major__ >= 9)
+#if ((__GNUC__ >= 9 or __clang_major__ >= 9) and not defined(__APPLE__))
       const char* file = __builtin_FILE(), int line = __builtin_LINE()
 #else
-      const char* file = {}, int line = {}
+      const char* file = "unknown", int line = {}
 #endif
           ) noexcept {
     source_location sl{};

Expected Behavior

If this executes:

"unexcepted throw"_test = [] {
    throw std::runtime_error("unexpected");
};

I would expect that the test would fail.

Actual Behavior

The test doesn't fail, and silently passed. All of the rest of the expect() in that test are ignored.

Steps to Reproduce the Problem

1. Run the above code

Specifications

• Version: master
• Platform: Fedora 31
• Subsystem: ???

Expected Behavior

To receive no warnings when compiling.

Actual Behavior

I received the following warnings using GCC 9.1.1.

_deps/ut-src/include/boost/ut.hpp: In function ‘constexpr auto boost::ut::v1_1_7::operators::terse::operator==(const T&, const boost::ut::v1_1_7::operators::terse::detail::value_location<typename T::value_type>&)’:
_deps/ut-src/include/boost/ut.hpp:1974:11: warning: typedef ‘using type = using eq_t = class boost::ut::v1_1_7::detail::eq_<T, boost::ut::v1_1_7::operators::terse::detail::value_location<typename T::value_type> >’ locally defined but not used [-Wunused-local-typedefs]
 1974 |     using type = eq_t;
      |           ^~~~
_deps/ut-src/include/boost/ut.hpp: In function ‘constexpr auto boost::ut::v1_1_7::operators::terse::operator==(const boost::ut::v1_1_7::operators::terse::detail::value_location<typename T::value_type>&, const T&)’:
_deps/ut-src/include/boost/ut.hpp:1987:11: warning: typedef ‘using type = using eq_t = class boost::ut::v1_1_7::detail::eq_<boost::ut::v1_1_7::operators::terse::detail::value_location<typename T::value_type>, T>’ locally defined but not used [-Wunused-local-typedefs]
 1987 |     using type = eq_t;
      |           ^~~~
_deps/ut-src/include/boost/ut.hpp: In function ‘constexpr auto boost::ut::v1_1_7::operators::terse::operator!=(const T&, const boost::ut::v1_1_7::operators::terse::detail::value_location<typename T::value_type>&)’:
_deps/ut-src/include/boost/ut.hpp:2000:11: warning: typedef ‘using type = using neq_t = class boost::ut::v1_1_7::detail::neq_<T, boost::ut::v1_1_7::operators::terse::detail::value_location<typename T::value_type> >’ locally defined but not used [-Wunused-local-typedefs]
 2000 |     using type = neq_t;
      |           ^~~~
_deps/ut-src/include/boost/ut.hpp: In function ‘constexpr auto boost::ut::v1_1_7::operators::terse::operator!=(const boost::ut::v1_1_7::operators::terse::detail::value_location<typename T::value_type>&, const T&)’:
_deps/ut-src/include/boost/ut.hpp:2013:11: warning: typedef ‘using type = using neq_t = class boost::ut::v1_1_7::detail::neq_<boost::ut::v1_1_7::operators::terse::detail::value_location<typename T::value_type>, T>’ locally defined but not used [-Wunused-local-typedefs]
 2013 |     using type = neq_t;
      |           ^~~~
_deps/ut-src/include/boost/ut.hpp: In function ‘constexpr auto boost::ut::v1_1_7::operators::terse::operator>(const T&, const boost::ut::v1_1_7::operators::terse::detail::value_location<typename T::value_type>&)’:
_deps/ut-src/include/boost/ut.hpp:2026:11: warning: typedef ‘using type = using gt_t = class boost::ut::v1_1_7::detail::gt_<T, boost::ut::v1_1_7::operators::terse::detail::value_location<typename T::value_type> >’ locally defined but not used [-Wunused-local-typedefs]
 2026 |     using type = gt_t;
      |           ^~~~
_deps/ut-src/include/boost/ut.hpp: In function ‘constexpr auto boost::ut::v1_1_7::operators::terse::operator>(const boost::ut::v1_1_7::operators::terse::detail::value_location<typename T::value_type>&, const T&)’:
_deps/ut-src/include/boost/ut.hpp:2039:11: warning: typedef ‘using type = using gt_t = class boost::ut::v1_1_7::detail::gt_<boost::ut::v1_1_7::operators::terse::detail::value_location<typename T::value_type>, T>’ locally defined but not used [-Wunused-local-typedefs]
 2039 |     using type = gt_t;
      |           ^~~~
_deps/ut-src/include/boost/ut.hpp: In function ‘constexpr auto boost::ut::v1_1_7::operators::terse::operator>=(const T&, const boost::ut::v1_1_7::operators::terse::detail::value_location<typename T::value_type>&)’:
_deps/ut-src/include/boost/ut.hpp:2052:11: warning: typedef ‘using type = using ge_t = class boost::ut::v1_1_7::detail::ge_<T, boost::ut::v1_1_7::operators::terse::detail::value_location<typename T::value_type> >’ locally defined but not used [-Wunused-local-typedefs]
 2052 |     using type = ge_t;
      |           ^~~~
_deps/ut-src/include/boost/ut.hpp: In function ‘constexpr auto boost::ut::v1_1_7::operators::terse::operator>=(const boost::ut::v1_1_7::operators::terse::detail::value_location<typename T::value_type>&, const T&)’:
_deps/ut-src/include/boost/ut.hpp:2065:11: warning: typedef ‘using type = using ge_t = class boost::ut::v1_1_7::detail::ge_<boost::ut::v1_1_7::operators::terse::detail::value_location<typename T::value_type>, T>’ locally defined but not used [-Wunused-local-typedefs]
 2065 |     using type = ge_t;
      |           ^~~~
_deps/ut-src/include/boost/ut.hpp: In function ‘constexpr auto boost::ut::v1_1_7::operators::terse::operator<(const T&, const boost::ut::v1_1_7::operators::terse::detail::value_location<typename T::value_type>&)’:
_deps/ut-src/include/boost/ut.hpp:2078:11: warning: typedef ‘using type = using lt_t = class boost::ut::v1_1_7::detail::lt_<T, boost::ut::v1_1_7::operators::terse::detail::value_location<typename T::value_type> >’ locally defined but not used [-Wunused-local-typedefs]
 2078 |     using type = lt_t;
      |           ^~~~
_deps/ut-src/include/boost/ut.hpp: In function ‘constexpr auto boost::ut::v1_1_7::operators::terse::operator<(const boost::ut::v1_1_7::operators::terse::detail::value_location<typename T::value_type>&, const T&)’:
_deps/ut-src/include/boost/ut.hpp:2091:11: warning: typedef ‘using type = using lt_t = class boost::ut::v1_1_7::detail::lt_<boost::ut::v1_1_7::operators::terse::detail::value_location<typename T::value_type>, T>’ locally defined but not used [-Wunused-local-typedefs]
 2091 |     using type = lt_t;
      |           ^~~~
_deps/ut-src/include/boost/ut.hpp: In function ‘constexpr auto boost::ut::v1_1_7::operators::terse::operator<=(const T&, const boost::ut::v1_1_7::operators::terse::detail::value_location<typename T::value_type>&)’:
_deps/ut-src/include/boost/ut.hpp:2104:11: warning: typedef ‘using type = using le_t = class boost::ut::v1_1_7::detail::le_<T, boost::ut::v1_1_7::operators::terse::detail::value_location<typename T::value_type> >’ locally defined but not used [-Wunused-local-typedefs]
 2104 |     using type = le_t;
      |           ^~~~
_deps/ut-src/include/boost/ut.hpp: In function ‘constexpr auto boost::ut::v1_1_7::operators::terse::operator<=(const boost::ut::v1_1_7::operators::terse::detail::value_location<typename T::value_type>&, const T&)’:
_deps/ut-src/include/boost/ut.hpp:2117:11: warning: typedef ‘using type = using le_t = class boost::ut::v1_1_7::detail::le_<boost::ut::v1_1_7::operators::terse::detail::value_location<typename T::value_type>, T>’ locally defined but not used [-Wunused-local-typedefs]
 2117 |     using type = le_t;
      |           ^~~~
_deps/ut-src/include/boost/ut.hpp: In function ‘constexpr auto boost::ut::v1_1_7::operators::terse::operator&&(const TLhs&, const TRhs&)’:
_deps/ut-src/include/boost/ut.hpp:2130:11: warning: typedef ‘using type = using and_t = class boost::ut::v1_1_7::detail::and_<typename TLhs::type, typename TRhs::type>’ locally defined but not used [-Wunused-local-typedefs]
 2130 |     using type = and_t;
      |           ^~~~
_deps/ut-src/include/boost/ut.hpp: In function ‘constexpr auto boost::ut::v1_1_7::operators::terse::operator||(const TLhs&, const TRhs&)’:
_deps/ut-src/include/boost/ut.hpp:2143:11: warning: typedef ‘using type = using or_t = class boost::ut::v1_1_7::detail::or_<typename TLhs::type, typename TRhs::type>’ locally defined but not used [-Wunused-local-typedefs]
 2143 |     using type = or_t;
      |           ^~~~
_deps/ut-src/include/boost/ut.hpp: In function ‘constexpr auto boost::ut::v1_1_7::operators::terse::operator!(const T&)’:
_deps/ut-src/include/boost/ut.hpp:2154:11: warning: typedef ‘using type = using not_t = class boost::ut::v1_1_7::detail::not_<typename T::type>’ locally defined but not used [-Wunused-local-typedefs]
 2154 |     using type = not_t;
      |           ^~~~

Steps to Reproduce the Problem

It seems like these warnings can occur by just using a very basic test case.
For instance, expect(true);.
This does not occur for all of my test executables, only simple ones.

Specifications

• Version: 1.1.7
• Platform: CentOS 7-1908
• Subsystem: devtoolset-9, GCC 9.1.1, CMake 3.17.0, Ninja 1.10.0, CCache 3.7.9

Expected Behavior

no warnings

Actual Behavior

WarningErrorEntry(path='/Users/clausklein/Workspace/cpp/ut/include/boost/ut.hpp', lineno='154', severity='warning', message='multiple declarations in a single statement reduces readability [readability-isolate-declaration]', column='3')
WarningErrorEntry(path='/Users/clausklein/Workspace/cpp/ut/include/boost/ut.hpp', lineno='261', severity='warning', message='do not declare C-style arrays, use std::array<> instead [modernize-avoid-c-arrays]', column='19')
WarningErrorEntry(path='/Users/clausklein/Workspace/cpp/ut/include/boost/ut.hpp', lineno='263', severity='warning', message='multiple declarations in a single statement reduces readability [readability-isolate-declaration]', column='3')
WarningErrorEntry(path='/Users/clausklein/Workspace/cpp/ut/include/boost/ut.hpp', lineno='285', severity='warning', message='do not declare C-style arrays, use std::array<> instead [modernize-avoid-c-arrays]', column='19')
WarningErrorEntry(path='/Users/clausklein/Workspace/cpp/ut/include/boost/ut.hpp', lineno='288', severity='warning', message='statement should be inside braces [readability-braces-around-statements]', column='25')
WarningErrorEntry(path='/Users/clausklein/Workspace/cpp/ut/include/boost/ut.hpp', lineno='298', severity='warning', message='do not declare C-style arrays, use std::array<> instead [modernize-avoid-c-arrays]', column='19')
WarningErrorEntry(path='/Users/clausklein/Workspace/cpp/ut/include/boost/ut.hpp', lineno='300', severity='warning', message='statement should be inside braces [readability-braces-around-statements]', column='25')
WarningErrorEntry(path='/Users/clausklein/Workspace/cpp/ut/include/boost/ut.hpp', lineno='517', severity='warning', message="function 'what' should be marked [[nodiscard]] [modernize-use-nodiscard]", column='3')
WarningErrorEntry(path='/Users/clausklein/Workspace/cpp/ut/include/boost/ut.hpp', lineno='540', severity='warning', message='do not implicitly decay an array into a pointer; consider using gsl::array_view or an explicit cast instead [hicpp-no-array-decay]', column='10')
WarningErrorEntry(path='/Users/clausklein/Workspace/cpp/ut/include/boost/ut.hpp', lineno='546', severity='warning', message="return type 'const type_<TOther>' is 'const'-qualified at the top level, which may reduce code readability without improving const correctness [readability-const-return-type]", column='17')
WarningErrorEntry(path='/Users/clausklein/Workspace/cpp/ut/include/boost/ut.hpp', lineno='929', severity='warning', message='do not declare C-style arrays, use std::array<> instead [modernize-avoid-c-arrays]', column='11')
WarningErrorEntry(path='/Users/clausklein/Workspace/cpp/ut/include/boost/ut.hpp', lineno='1314', severity='warning', message="do not use 'else' after 'return' [readability-else-after-return]", column='7')
WarningErrorEntry(path='/Users/clausklein/Workspace/cpp/ut/include/boost/ut.hpp', lineno='1329', severity='warning', message="throwing an exception whose type 'events::fatal_assertion' is not derived from 'std::exception' [hicpp-exception-baseclass]", column='11')
WarningErrorEntry(path='/Users/clausklein/Workspace/cpp/ut/include/boost/ut.hpp', lineno='1428', severity='warning', message="method 'operator=' can be made const [readability-make-member-function-const]", column='18')
WarningErrorEntry(path='/Users/clausklein/Workspace/cpp/ut/include/boost/ut.hpp', lineno='1454', severity='warning', message='do not implicitly decay an array into a pointer; consider using gsl::array_view or an explicit cast instead [hicpp-no-array-decay]', column='26')
WarningErrorEntry(path='/Users/clausklein/Workspace/cpp/ut/test/ft/test_suite_2.cpp', lineno='14', severity='warning', message="throwing an exception whose type 'int' is not derived from 'std::exception' [hicpp-exception-baseclass]", column='62')
WarningErrorEntry(path='/Users/clausklein/Workspace/cpp/ut/benchmark/test.cpp', lineno='10', severity='warning', message="function 'main' exceeds recommended size/complexity thresholds [readability-function-size]", column='5')
WarningErrorEntry(path='/Users/clausklein/Workspace/cpp/ut/include/boost/ut.hpp', lineno='546', severity='warning', message="return type 'const type_<int>' is 'const'-qualified at the top level, which may reduce code readability without improving const correctness [readability-const-return-type]", column='17')
WarningErrorEntry(path='/Users/clausklein/Workspace/cpp/ut/include/boost/ut.hpp', lineno='1446', severity='warning', message='do not implicitly decay an array into a pointer; consider using gsl::array_view or an explicit cast instead [hicpp-no-array-decay]', column='28')
WarningErrorEntry(path='/Users/clausklein/Workspace/cpp/ut/include/boost/ut.hpp', lineno='1563', severity='warning', message='do not implicitly decay an array into a pointer; consider using gsl::array_view or an explicit cast instead [hicpp-no-array-decay]', column='25')
WarningErrorEntry(path='/Users/clausklein/Workspace/cpp/ut/include/boost/ut.hpp', lineno='2204', severity='warning', message='do not implicitly decay an array into a pointer; consider using gsl::array_view or an explicit cast instead [hicpp-no-array-decay]', column='44')
WarningErrorEntry(path='/Users/clausklein/Workspace/cpp/ut/test/ut/ut.cpp', lineno='71', severity='warning', message='use emplace_back instead of push_back [modernize-use-emplace]', column='25')
WarningErrorEntry(path='/Users/clausklein/Workspace/cpp/ut/test/ut/ut.cpp', lineno='73', severity='warning', message='use emplace_back instead of push_back [modernize-use-emplace]', column='25')
WarningErrorEntry(path='/Users/clausklein/Workspace/cpp/ut/test/ut/ut.cpp', lineno='75', severity='warning', message='use emplace_back instead of push_back [modernize-use-emplace]', column='25')
WarningErrorEntry(path='/Users/clausklein/Workspace/cpp/ut/test/ut/ut.cpp', lineno='96', severity='warning', message="throwing an exception whose type 'ut::events::fatal_assertion' is not derived from 'std::exception' [hicpp-exception-baseclass]", column='11')
WarningErrorEntry(path='/Users/clausklein/Workspace/cpp/ut/test/ut/ut.cpp', lineno='130', severity='warning', message="throwing an exception whose type 'int' is not derived from 'std::exception' [hicpp-exception-baseclass]", column='37')
WarningErrorEntry(path='/Users/clausklein/Workspace/cpp/ut/test/ut/ut.cpp', lineno='238', severity='warning', message="function 'main' exceeds recommended size/complexity thresholds [readability-function-size]", column='5')
WarningErrorEntry(path='/Users/clausklein/Workspace/cpp/ut/test/ut/ut.cpp', lineno='314', severity='warning', message="the 'empty' method should be used to check for emptiness instead of comparing to an empty object [readability-container-size-empty]", column='19')
WarningErrorEntry(path='/Users/clausklein/Workspace/cpp/ut/test/ut/ut.cpp', lineno='479', severity='warning', message="'auto old_cout' can be declared as 'auto *old_cout' [readability-qualified-auto]", column='7')
WarningErrorEntry(path='/Users/clausklein/Workspace/cpp/ut/test/ut/ut.cpp', lineno='480', severity='warning', message="'auto old_cerr' can be declared as 'auto *old_cerr' [readability-qualified-auto]", column='7')
WarningErrorEntry(path='/Users/clausklein/Workspace/cpp/ut/test/ut/ut.cpp', lineno='1006', severity='warning', message="the 'empty' method should be used to check for emptiness instead of comparing to an empty object [readability-container-size-empty]", column='14')
WarningErrorEntry(path='/Users/clausklein/Workspace/cpp/ut/test/ut/ut.cpp', lineno='1108', severity='warning', message="throwing an exception whose type 'int' is not derived from 'std::exception' [hicpp-exception-baseclass]", column='34')
WarningErrorEntry(path='/Users/clausklein/Workspace/cpp/ut/test/ut/ut.cpp', lineno='1109', severity='warning', message="throwing an exception whose type 'int' is not derived from 'std::exception' [hicpp-exception-baseclass]", column='54')
WarningErrorEntry(path='/Users/clausklein/Workspace/cpp/ut/test/ut/ut.cpp', lineno='1113', severity='warning', message="throwing an exception whose type 'int' is not derived from 'std::exception' [hicpp-exception-baseclass]", column='35')
WarningErrorEntry(path='/Users/clausklein/Workspace/cpp/ut/test/ut/ut.cpp', lineno='1150', severity='warning', message='statement should be inside braces [readability-braces-around-statements]', column='28')
WarningErrorEntry(path='/Users/clausklein/Workspace/cpp/ut/test/ut/ut.cpp', lineno='1150', severity='warning', message="throwing an exception whose type 'int' is not derived from 'std::exception' [hicpp-exception-baseclass]", column='35')
WarningErrorEntry(path='/Users/clausklein/Workspace/cpp/ut/test/ut/ut.cpp', lineno='1161', severity='warning', message="throwing an exception whose type 'events::exception' is not derived from 'std::exception' [hicpp-exception-baseclass]", column='15')
WarningErrorEntry(path='/Users/clausklein/Workspace/cpp/ut/test/ut/ut.cpp', lineno='1164', severity='warning', message="throwing an exception whose type 'int' is not derived from 'std::exception' [hicpp-exception-baseclass]", column='41')
WarningErrorEntry(path='/Users/clausklein/Workspace/cpp/ut/test/ut/ut.cpp', lineno='1605', severity='warning', message="function 'get' should be marked [[nodiscard]] [modernize-use-nodiscard]", column='7')

Steps to Reproduce the Problem

1. merge #421
2. run-clang-tidy.py -p build -header-filter='.*' -checks='-cert-*' test
3. see https://github.com/ClausKlein/ut/runs/1973383718?check_suite_focus=true
4. and https://github.com/boost-ext/ut/blob/d15ad0e4945d187d4500d4c191cda69262f4f884/.github/workflows/macos.yml

Specifications

• Version: 1.1.8
• Platform: OSX
• Subsystem: Clang

Expected Behavior

To execute the test and the assert inside.

I don't know how to co_await inside a test lambda, without being a coroutine.

Actual Behavior

All tests passed (0 asserts in 1 tests)

Steps to Reproduce the Problem

"hello coro"_test = []() -> ::cppcoro::task<> {
    ::boost::ut::log << "Not printed, because never reaches here.";
    auto task_lambda = []() -> ::cppcoro::task<int> { co_return 43; }();
    int coro_43 = co_await task_lambda;
    expect(42_i == coro_43);
};

Specifications

• Version: master branch at commit 4799a77
• Platform: Ubuntu 18.04
• Subsystem:

There are a number of cases when you want unit tests to run earlier than the program exits. E.g. an application may want to run and report failing unit tests on startup. While this is doable with a custom runner, it should be fairly straightforward to add a method to replicate the default runner's destructor behavior. Something that effectively allows:

    int main(int argc, char **argv)
    {
        boost::ut::cfg<>.run();                            // (1)
        boost::ut::cfg<>.on(boost::ut::events::summary{}); // (2) currently not accessible
        ...

Possible APIs could be a bool run_and_report() method to replicate the current destructor functionality or more flexible a void report_summary() to just do (2).

Nested tests

Expected Behavior

A file test.cpp contains

#include "ut.hpp"

int main() {
  using namespace boost::ut;

  test("set") = []{
    test("one") = []{ expect(1==1); };
    test("two") = []{ expect(2==2); };
  };
}

Compilation and execution is

$ g++ -std=c++20 -Wall -Wextra -pedantic   -c -o test.o test.cpp
$ g++ -std=c++20 -Wall -Wextra -pedantic -o test test.o
$ ./test
All tests passed (2 asserts in 2 tests)

Actual Behavior

All other things being equal,

$ ./test
All tests passed (2 asserts in 1 tests)

Steps to Reproduce the Problem

Project Directory Structure

./ut-test/
  |
  ├── ut.hpp
  |
  └── test.cpp

See Expected Behavior.

Specifications

OS: bento/ubuntu-20.04
Compiler: gcc version 10.0.1 (Ubuntu 10-20200411-0ubuntu1)
          g++ -> x86_64-linux-gnu-gcc-10

Further Implications

In BDD syntax, the standard output is undesirable. For example, something like

#include "ut.hpp"

int main() {
  using namespace boost::ut;

  feature("sort") = []{
    scenario("small arrays") = [] {
      given("trivial arrays") = [] {
        when("the array is empty") = [] { ... };
        when("the array is of size 1") = [] { ... };
      };
      given("array of size 2") = { ... };
      given("array of size 3") = { ... };
      ⋮  
    };
    scenario("significant arrays") = [] {
      given("array of size 20") = { ... }
      ⋮  
    };
    ⋮  
    scenario("mega arrays") = [] {
      ...
    };
  };
}

can result in the output

$ ./test
All tests passed (23192 asserts in 1 tests)

Suggestion

Add an optional parameter to the test function that will cause it to be counted as a test:

... auto test = [](const auto name, const bool isTest) { ...

Then, one could apply it as follows

#include "ut.hpp"

int main() {
  using namespace boost::ut;

  test("set") = []{
    test("one", true) = []{ expect(1==1); };
    test("two", true) = []{ expect(2==2); };
  };

  feature("sort") = []{
    scenario("small arrays") = [] {
      given("trivial arrays") = [] {
        when("the array is empty", true) = [] { ... };
        when("the array is of size 1", true) = [] { ... };
        ⋮  
      };
      ⋮  
    };
    ⋮  
  };
}

Employ a more elegant solution or update the tutorial if my understanding is off.

Thanks!

Expected Behavior

Something similar to
BOOST_REQUIRE_EXCEPTION(expression, exception_type, predicate);

In Boost.Test (at least) there is a useful opportunity to inspect exception object itself.
BOOST_REQUIRE_EXCEPTION

Actual Behavior

It seems now ut can not simulate this behaviour by overriding runner/reporter/... In the case of

expect(throws<std::runtime_error>([] { throw std::runtime_error{""}; }))

by the time runner::on(ut::events::assertion<TExpr>) fires, a decision on the status of test validation has already been made, without the ability to somehow intervene either before or after for completion.

As a workaround I just embed a 'predicate function' to ut::throws and ut::detail::throws_ as overriding variants.

template<class T>
void func_1(T const & e)
{
  boost::ut::expect(e.code() == ex_code_1);
}
...
"test1"_test = [] {

      /* some 'when' precondition */
      expect(throws<some_exception>(
          [] { raising_func(); },
          func_1) ); // fails at func_1 body line if raising_func throws with ex_code_2
      /* some other 'when' precondition */
      expect(throws<some_exception>(
          [] { raising_func(); },
          func_2) ); // passes if raising_func throws with ex_code_2
      expect((throws<some_exception>([] { throw some_exception(); })) >> fatal); // passes
      expect((throws<some_exception>([] { raising_func(); })) >> fatal); // passes if raising any "some_exception" (or fail at this particular line)
      expect((throws([] { throw 1; })) >> fatal); // as always!
}

And my question is: Will similar(somewhat) functionality be introduced to ut sometime?

Any plans on adding an expect_death test? One feature that I need badly is the ability to test for std::terminate() and continue on, and I am not sure how to best approach that as throwing from a std::terminate() handler is not an option when a function is marked as noexcept.

Expected Behavior

That it's easy to search text, and see the whole structure, of the docs on the front page:

https://github.com/boost-experimental/ut

Actual Behavior

• Most sections on the front page are collapsed, and some require two levels of expansion of triangles to see the content
• Searching in the browser (Ctrl+F) doesn't find text in collapsed sections
• I've seen someone who is more familiar with UT than me still click on 2 or 3 triangles before finding the code example they wanted to show me

Specifications

• Testing in both Firefox and Chrome (Windows 10)

Suggestions

• Perhaps provide a button at the top that opens all the triangles all the way down
• Or just remove the triangles and collapsing altogether

Expected Behavior

Parallel Runner example should build without error.

Actual Behavior

The parallel_runner fails to build with GCC 9.1.1 (devtoolset-9) on RHEL 7 and with Mingw64 (GCC 9.2) on Windows 10.
The <execution> header pulls in the exact same missing include on both platforms:

/opt/rh/devtoolset-9/root/usr/include/c++/9/pstl/parallel_backend_tbb.h:19:10: fatal error: tbb/blocked_range.h: No such file or directory
  19 | #include <tbb/blocked_range.h>
     |        ^~~~~~~~~~~~~~~~~~~~~

Steps to Reproduce the Problem

Build the parallel_runner test on RHEL 7 using devtoolset-9 with C++20 enabled.
Or
Build the parallel_runner test on Mingw64 (GCC 9.2) with C++20 enabled.

Specifications

• Version: 1.1.6
• Platform: RHEL 7 / Windows 10 (MSYS2)
• Toolchain: devtoolset-9 / Mingw64

Hi @krzysztof-jusiak - I have a patch for this, ready to submit as a PR... I thought it would be helpful to log the behaviour here to provide a more detailed description.

Expected Behavior

When there is an unexpected exception caught during the running of tests, the program should exit with a non-zero exit code.

Actual Behavior

When there is an unexpected exception caught during the running of tests, and no other assertion-type failures, there is a console message written out, but the program exits with exit code 0 - so ctest and CI systems see it as a passing test.

This is particularly important when using ctest --output-on-failure . ... ctest thinks the test passed, so it does not displaying any console output at all, and so the user sees only 100% success...

(This is the underlying cause of approvals/ApprovalTests.cpp#87.)

Steps to Reproduce the Problem

1. Edit the example program example/exception.cpp to have this content:

#include <boost/ut.hpp>
#include <stdexcept>

int result() {
  throw std::runtime_error("no!");
}

int main() {
  using namespace boost::ut;

  "exceptions"_test = [] {
    expect(result() == 1_i);
  };
}

2. Run the exception program
3. Note the exit code 0 in the console output:

.../ut-claremacrae/cmake-build-spaces/cmake-build-debug-gcc9-brew/example/exception
Running "exceptions"...
  Unexpected exception with message:
no!
FAILED

===============================================================================
tests:   1 | 1 failed
asserts: 0 | 0 passed | 0 failed


Process finished with exit code 0

Specifications

• Version: 46a4f2e
• Platform: macOS Catalina
• Subsystem: ?

Observed on multiple different compilers and generators.

What do you think about making a [Boost-ext].UT package available in ConanCenter?
I would like to able to handle the dependency through Conan, and having it available in ConanCenter seems the easiest way to do so.

Their instructions for doing so are available here.
I found an example recipe for a header-only library on ConanCenter here.

This

#define BOOST_UT_FORWARD
#include <boost/ut.hpp>
using namespace boost::ut;
suite some_tests = []{
    "x"_test = []{
        log << "Hello";
    };
};

does not compile. It does without #define BOOST_UT_FORWARD.

Expected Behavior

ut should provide user-defined literals and type aliases for all fundamental types.

Actual Behavior

Currently there are user defined literals and type aliases for char, unsigned char, short, unsigned short, int, unsigned, long, unsigned long, long long, float, double, long double (and with slightly different behavior bool).
But there's no such thing for signed char or unsigned long long.

I would even love to have user defined literals for the integer types from <cstdint>, e.g. std::int64_t or std::intmax_t, and the two types std::size_t and std::ptrdiff_t from <cstddef>. But I don't know ut well enough to know whether adding them makes sense.

Steps to Reproduce the Problem

#include <tuple>
#include "ut.hpp"

int main()
{
    // === user-defined literals
    using namespace boost::ut::literals;

    std::ignore = 0_c;      // char

    std::ignore = 0_s;      // signed short
    std::ignore = 0_i;      // signed int
    std::ignore = 0_l;      // signed long
    std::ignore = 0_ll;     // signed long long

    std::ignore = 0_uc;     // unsigned char
    std::ignore = 0_us;     // unsigned short
    std::ignore = 0_u;      // unsigned int
    std::ignore = 0_ul;     // unsigned long

    std::ignore = 0._f;     // float
    std::ignore = 0._d;     // double
    std::ignore = 0._ld;    // long double

    std::ignore = "everything is true"_b;

    // these two don't exist:
    //std::ignore = 0_sc;     // signed char
    //std::ignore = 0_ull;    // unsigned long long


    // ==== type aliases
    using namespace boost::ut;

    std::ignore = _c(0);     // char

    std::ignore = _s(0);     // signed short
    std::ignore = _i(0);     // signed int
    std::ignore = _l(0);     // signed long
    std::ignore = _ll(0);    // signed long long

    std::ignore = _uc(0);    // unsigned char
    std::ignore = _us(0);    // unsigned short
    std::ignore = _u(0);     // unsigned int
    std::ignore = _ul(0);    // unsigned long

    std::ignore = _f(0);     // float
    std::ignore = _d(0);     // double
    std::ignore = _ld(0);    // long double

    std::ignore = _b(0);     // bool

    // these two don't exist:
    //std::ignore = _sc(0);    // signed char
    //std::ignore = _ull(0);   // unsigned long long
}

Specifications

• Version: HEAD (1.8.1)
• Platform: all
• Subsystem: all

Several CMake options are likely to cause name collisions with similarly named variables, and variables are put into the CMake cache without consideration for parent projects.

Variables such as BUILD_TESTS are usually prefixed with the project's name to avoid potential name collisions.
An example for [Boost].µT is UT_BUILD_TESTS.

Projects incorporating [Boost].µT may wish to not have certain [Boost].µT-specific variables shown in the cache, like the BUILD_TESTS option.
Best practice is to use a non-cache variable as this leaves the decision up to parent projects whether to make those variables available in the cache or not.
A common idiom used for testing is to automatically include the tests if the project is the top-level project or if the appropriate variable is enabled.
The example below demonstrates this behavior.

if(TEST_UT OR CMAKE_SOURCE_DIR STREQUAL PROJECT_SOURCE_DIR)
  add_subdirectory(test)
endif()

There is no call to the option() command, i.e.
option(TEST_UT "Enable building the tests" yes), as this populates the variable in the CMake cache.

Is there anything in place to allow me to do
expect( x.has_value() and that % x.value() == 10 );
in such a way that if the first part fails, the second isn't evaluated?

Expected Behavior

When I execute:

std::string msg = "The answer is: 42\n";
if (auto strm = std::ofstream("test.txt")) {
    strm << msg;
}

I would expect the following:

The answer is: 42

Actual Behavior

I actually get:

{T, h, e,  , a, n, s, w, e, r,  , i, s, :,  , 4, 2, }

If I comment out the unit test header, the above code works as expected.

Steps to Reproduce the Problem

1. Run the example above

Specifications

• Version: master
• Platform: Fedora 31
• Subsystem: ???

Expected Behavior

I'd like to configure a parallel runner for a multi-translation unit program. Just took the example parallel_runner.cpp, fit into my main.cpp and expected the project to just build.

Actual Behavior

Tons of linker errors

Steps to Reproduce the Problem

1. Take parallel_runner.cpp
2. Create another test.cpp with tests
3. Compile and try to link them together
4. Observe the messages like the following:

[build] : && /usr/bin/c++ -g  CMakeFiles/aoc.dir/main.cpp.o CMakeFiles/aoc.dir/test2.cpp.o -o aoc  -lpthread  -lssl  -lcrypto  -lgmp  -ltbb && :
[build] /usr/bin/ld: /usr/bin/ld: DWARF error: could not find variable specification at offset befe
[build] CMakeFiles/aoc.dir/test2.cpp.o:(.bss._ZN5boost3ext2ut6v1_1_83cfgINS2_8overrideEJEEE[_ZN5boost3ext2ut6v1_1_83cfgINS2_8overrideEJEEE]+0x0): multiple definition of `boost::ext::ut::v1_1_8::cfg<boost::ext::ut::v1_1_8::override>'; /usr/bin/ld: DWARF error: could not find variable specification at offset c578
[build] CMakeFiles/aoc.dir/main.cpp.o:(.bss+0x0): first defined here
[build] collect2: error: ld returned 1 exit status

Specifications

• Version: v1.1.8
• Platform: Arch Linux x64
• Subsystem:

Specifications

• Version: gcc 10.0.1 (20200327)
• Platform: Linux - Centos 7

Error Message

../example/tmp.cpp:19:12: error: 'boost::ut::v1_1_7::detail::eq_<boost::ut::v1_1_7::detail::integral_constant<42>, int>' is not a valid type for a template non-type parameter because it is not structural
19 | expect(constant<42_i == i> and type == type and
| ^~~~~~~~~~~~~~~~~~~
In file included from ../example/tmp.cpp:8:
../include/boost/ut.hpp:738:14: note: 'boost::ut::v1_1_7::detail::eq_<boost::ut::v1_1_7::detail::integral_constant<42>, int>::lhs_' is not public
738 | const TLhs lhs_{};
| ^~~~

en.cppreference.com states, for non-type template parameters
a literal class type with the following properties:

• all base classes and non-static data members are public and non-mutable and
• the types of all bases classes and non-static data members are structural types or (possibly multi-dimensional) array thereof.

As a workaround, I've commented the ifdefs for Constant in line 2190

Expected Behavior

1. boost.ut target seems to no longer be created

That the target boost.ut is still provided, to link against, when this project is obtained via add_subdirectory(), FetchContent() or similar...

2. include file no longer found: #include <boost/ut.hpp>

Example code - this is how I obtain the boost.ut project - UtVersion is master:

https://github.com/claremacrae/ApprovalTests.cpp.CMakeSamples/blob/main/dev_approvals_fetch_content/CMakeLists.txt#L130-L133

FetchContent_Declare(boost.ut
        GIT_REPOSITORY https://github.com/boost-ext/ut.git
        GIT_TAG ${UtVersion})
FetchContent_MakeAvailable(boost.ut)

The repo goes on to use the target boost.ut.

This directory and GitHub action exists to help the ApprovalTests.cpp project have early detection of any changes in test frameworks that might affect our users, when we do our next release.

This has been working for some months, and was still working on the 24th of Feb.

Actual Behavior

The next build, on 27th Feb at 01:10 GMT gave these errors

1. in clang builds:

Fetching boost.ut...
CMake Error at CMakeLists.txt:137 (target_compile_options):
  Cannot specify compile options for target "boost.ut" which is not built by
  this project.

2. in gcc builds:

Building CXX object approvaltests.cpp_build/tests/UT_Tests/CMakeFiles/UT_Tests.dir/UTApprovalTestTests.cpp.o
In file included from /home/runner/work/ApprovalTests.cpp.CMakeSamples/ApprovalTests.cpp.CMakeSamples/ApprovalTests.cpp/ApprovalTests/ApprovalTests.hpp:68,
                 from /home/runner/work/ApprovalTests.cpp.CMakeSamples/ApprovalTests.cpp.CMakeSamples/ApprovalTests.cpp/tests/UT_Tests/UTApprovalTestTests.cpp:3:
/home/runner/work/ApprovalTests.cpp.CMakeSamples/ApprovalTests.cpp.CMakeSamples/ApprovalTests.cpp/ApprovalTests/../ApprovalTests/integrations/ut/UTApprovals.h:14:10: fatal error: boost/ut.hpp: No such file or directory
   14 | #include <boost/ut.hpp>
      |          ^~~~~~~~~~~~~~
compilation terminated.
approvaltests.cpp_build/tests/UT_Tests/CMakeFiles/UT_Tests.dir/build.make:81: recipe for target 'approvaltests.cpp_build/tests/UT_Tests/CMakeFiles/UT_Tests.dir/UTApprovalTestTests.cpp.o' failed

Steps to Reproduce the Problem

Run this on a unix box with a recent version of either linux or gcc installed:

git clone https://github.com/claremacrae/ApprovalTests.cpp.CMakeSamples
cd ApprovalTests.cpp.CMakeSamples
git clone https://github.com/approvals/ApprovalTests.cpp.git
cd dev_approvals_fetch_content
./build.sh

Wait until all the dependencies have been cloned, and then one of the errors will occur - depending on whether you are using gcc or clang.

Specifications

The failure was detected in a GitHub Actions build. The failing builds can be seen here:

https://github.com/claremacrae/ApprovalTests.cpp.CMakeSamples/actions/runs/604457152

• Linux Clang 6, 8, 9, and 10
• Linux gcc 9 and 10
• macOS Xcode 10.3, 11.7 and 12.

It's likely that the ones that passed are skipping the building of boost.ut

Would it be possible to support creating microbenchmarks? It is not uncommon for test frameworks to incorporate microbenchmark support. Below are a couple of frameworks I'm aware of which closely tie microbenchmarking and unit testing.

1. Catch2 implements microbenchmarks -> https://github.com/catchorg/Catch2/blob/master/docs/benchmarks.md#top
2. Google Benchmark (related to Google Test) is a simple, macro-based solution -> https://github.com/google/benchmark
3. The go programming language implements microbenchmarking as a part of its built-in testing framework -> https://golang.org/pkg/testing/

I'm not aware of any libraries for benchmarking C++ code that are macro-free, and I figure I'm not the only one that would have a similar interest in macro-free microbenchmarking.

For this to work with ExternalProject_Add, it really should have an install target.