Verbose boost test logging to stdout about pytest-cpp HOT 6 CLOSED

Hi @falloutphil,

You mean that the tests don't fail, but you would like to see more output from the test run? Currently we only show output if an error happens, otherwise we don't show any output at all (as normal pytest runs usually do).

What you want is possible to support however, we would need to introduce a new --cpp-verbose flag or something like that, which also outputs the stdout from calling the executable under test. Then you would be able to pass -s and let the output show in the terminal during the run.

I probably won't tackle this anytime soon, but would love to review and merge a PR to that effect. 👍

from pytest-cpp.

Thanks @nicoddemus

Yes - I think it would be useful to see check-level output as well as test-case level output.

So in each (for example) BOOST_AUTO_TEST_CASE I have has a series of BOOST_TEST_WARN, BOOST_TEST, and BOOST_TEST_REQUIRE conditions within them. Each of these is actually testing the result of a single outcome and is referred to as a "check" by Boost. At test case is a collection of one or more checks.

At the moment, in pytest, we only see the output of a check, if a check fails. We don't get a list of successful checks within a test case. This is exactly the same if we run the cpp executable with no arguments.

I'd like to see a list of checks performed that passed as well as those that failed - exactly like if I pass "-l all" into the cpp executable.

No promises, but if I can find the time I'll look at doing a PR!

from pytest-cpp.

We can probably get away with just printing the full output from each test, and using -P to print the captured output for even passing tests.

No promises, but if I can find the time I'll look at doing a PR!

OK, sounds good, thanks!

from pytest-cpp.

@nicoddemus @falloutphil what about adding he stdout of the tests in a log.info? something like this for gtest:

        try:
            log.info(
                subprocess.check_output(
                    args, stderr=subprocess.STDOUT, universal_newlines=True
                )
            )
        except subprocess.CalledProcessError as e:
            log.info(e.output)

so you can toggle whether you get the logs or not using the --log-cli-level=INFO

from pytest-cpp.

so we got a side discussion on a PR implementing my comment above, I'd like to summarize that here so we can design a solution that suites us all :)

@nicoddemus :
I'm not sure I'm a fan of hijacking logging.info like that: pytest-cpp is often used to run other kinds of tests (like normal python tests) alongside C++ tests.

How about the original idea, of just printing the output of the underlying executable to stdout? That's how other plugins do this, notably pytester.

@The-Compiler

FWIW I don't actively use this plugin, but I'd probably vote for a separate report section, similar to what e.g. pytest-qt does.

@nicoddemus

is mentioning Item.add_report_section. This will be displayed automatically by pytest if a test fails.

We could add the full test output like this: item.add_report_section("call", "c++", output).

me

but that wouldnt solve #41 as the output would be shown only in case of error

I'll explore add_report_section idea as I am not understanding it completely :)

from pytest-cpp.

it was a great idea :)

from pytest-cpp.