pytest-dev / pytest-cpp Goto Github PK

View Code? Open in Web Editor NEW

114.0 114.0 25.0 465 KB

Use pytest's runner to discover and execute C++ tests

License: MIT License

Python 2.62% C++ 97.30% C 0.08%

c-plus-plus pytest

pytest-cpp's People

Contributors

Stargazers

Watchers

pytest-cpp's Issues

Error when using gtest arguments

Hi,

I'm trying to run gtest binaries with pytes-cpp and I can't seem to find a way to use --gtest_filter properly. If I use only --gtest_repeat=x for example, it works fine, but if I try to use --gtest_filter I get

def run_test(self, executable, test_id, test_args=(), harness=None):
        harness = harness or []
        output = ""
        xml_filename = self._get_temp_xml_filename()
        args = harness + [
            executable,
            "--gtest_filter=" + test_id,
            "--gtest_output=xml:%s" % xml_filename,
        ]
        args.extend(test_args)
    
        try:
            output = subprocess.check_output(
                args, stderr=subprocess.STDOUT, universal_newlines=True
            )
        except subprocess.CalledProcessError as e:
            output = e.output
            if e.returncode != 1:
                msg = (
                    "Internal Error: calling {executable} "
                    "for test {test_id} failed (returncode={returncode}):\n"
                    "{output}"
                )
                failure = GoogleTestFailure(
                    msg.format(
                        executable=executable,
                        test_id=test_id,
                        output=e.output,
                        returncode=e.returncode,
                    )
                )
    
                return [failure], output
    
        results = self._parse_xml(xml_filename)
        os.remove(xml_filename)
        for (executed_test_id, failures, skipped) in results:
            if executed_test_id == test_id:
                if failures:
                    return [GoogleTestFailure(x) for x in failures], output
                elif skipped:
                    pytest.skip("\n".join(skipped))
                else:
                    return None, output
    
        msg = "Internal Error: could not find test " "{test_id} in results:\n{results}"
>       results_list = "\n".join(x for (x, f) in results)

/home/oci/.local/lib/python3.6/site-packages/pytest_cpp/google.py:106: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

.0 = <list_iterator object at 0x7f45e1d26c18>

>   results_list = "\n".join(x for (x, f) in results)
E   ValueError: too many values to unpack (expected 2)`

in lib/python3.6/site-packages/pytest_cpp/google.py:106: ValueError

If I run the binary itself with the filters I am using, it works; so I don't think the filter themselves are the issue (though I thought so at the beginning).
Any ideas?

marking cpp test targets.

Is there any way to mark tests (whether through ini, filename or otherwise) so that py.test -m=system_test or -m=unittest works with the tests?

Type-Parametrized Gtests failure

When using pytest-cpp-0.3.1 runner against type-parametrized google-test executable I always get the following failure:

    MatrixTest/0.  # TypeParam = Matrix<float>Equal ______________________________
    Internal Error: could not find test MatrixTest/0.  # TypeParam = Matrix<float>Equal in results:
    unknown file:0: C++ failure

My guess is that the parser code used to discover gtests is not robust to type-parametrized tests.
A fix for this would be highly appreciated.

It can be reproduced with gtest-1.7.0

Thanks,
joar

Display more than one failure using GoogleTest

When using Google::Test, we can have more than one failure in a test that uses EXPECT_* macros for assertion.

repr_failure should call superclass on unknown exceptions

Google: Support expected failures

Hi, I need to treat also xfail for google test, for example I write 'xfail' in the gtest name and I would that pytest cpp mark the test as xfail, so pytest can run it as a real pytest test marked xfail.
Is it possible?
Thanks

Add support for Catch testing framework

Catch is a very popular unit testing library. Is it possible support may be added? Catch test executables support listing all test names; filtering tests; outputing in an easily parsable format (JUnit). Just following what the Google Test support is doing, it seems to support everything necessary.

Verbose boost test logging to stdout

Hi,

This is a very useful plugin! There's only one thing I haven't been able to get working.

When I execute a Boost Test from the command line I like to see the output of each check, by passing the following parameters

./my_test -l all

You get output like:

my_test.cpp(20): info: check val->getStringValue() == test_string has passed

Or perhaps more importantly output from BOOST_WARN is completely swallowed, eg:

./my_test.cpp(15): warning: in "MyBoostTests": condition sp.foo() == "Bar" is not satisfied ["Blah" != "Bar"]

I assumed I'd be able to get this in pytest using something like:

pytest -v -ra -o cpp_arguments='-l all'

But the output is still at the unit test level not the BOOST_TEST level:

my_test::my_test PASSED

How can I increase the output to the terminal?

Thanks!

Catch2: escape special characters in test names

When specifying which tests to run from the command line, Catch2 treats some characters specially. For example, * is a wildcard character, ~ negates a test spec, etc.

However, these literal characters can also appear in test names. In order to match a specific test name that contains one or more special characters, they need to be escaped with a preceding backslash ("\").

You can also match test names with special characters by escaping them with a backslash ("\"), e.g. a test named "Do A, then B" is matched by "Do A, then B" test spec. Backslash also escapes itself.

(reference)

Currently, pytest-cpp reports a test failure for any tests with special characters in their name, with the following error message:

Internal Error: could not find test <test_id> in results:
:0: C++ failure

I believe this issue may be resolved by escaping any special characters in the test_id before invoking the test executable.

I'm happy to work on resolving this -- just want to make sure a fix would be welcome before making the effort to submit a PR.

Fix test output when writing to xmlfiles

Return proper TerminalRepr instance from repr_failure.

Key Error after upgrade to 2.1.0

After the upgrade to 2.1.0 I observe the following error:

    def _parse_xml(self, xml_filename):
        root = ElementTree.parse(xml_filename)
        result = []
        for test_suite in root.findall("testsuite"):
            test_suite_name = test_suite.attrib["name"]
            for test_case in test_suite.findall("testcase"):
                test_name = test_case.attrib["name"]
                failures = []
                failure_elements = test_case.findall("failure")
                for failure_elem in failure_elements:
                    failures.append(failure_elem.text)
                skippeds = []
>               if test_case.attrib["result"] == "skipped":
E               KeyError: 'result'

test_env\lib\site-packages\pytest_cpp\google.py:125: KeyError

It looks like replacing test_case.attrib.get("result", None) with test_case.attrib["result"] is the issue: 2a2f125

Include ini file option to configure file mask used to discover tests

Boost: Support expected failures

Expected failures as xfail.

http://www.boost.org/doc/libs/1_56_0_b1/libs/test/doc/html/utf/user-guide/test-organization/expected-failures.html

Incompatibility between pytest 6.x and pytest-cpp==2.1.2

The newest release pytest-cpp==2.1.2 seems to be incompatible with the pytest 6.x releases:

Steps to reproduce:

Create and activate a new virtualenv
pip install 'pytest<7' pytest-cpp==2.1.2
In an empty directory, run pytest

Expected behavior:

Pytest runs, without executing any tests.

Observed behavior:

The following exception is raised:

Traceback (most recent call last):
  File "/home/dgresch/venv/bin/pytest", line 8, in <module>
    sys.exit(console_main())
  File "/home/dgresch/venv/lib/python3.8/site-packages/_pytest/config/__init__.py", line 185, in console_main
    code = main()
  File "/home/dgresch/venv/lib/python3.8/site-packages/_pytest/config/__init__.py", line 143, in main
    config = _prepareconfig(args, plugins)
  File "/home/dgresch/venv/lib/python3.8/site-packages/_pytest/config/__init__.py", line 318, in _prepareconfig
    config = pluginmanager.hook.pytest_cmdline_parse(
  File "/home/dgresch/venv/lib/python3.8/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/home/dgresch/venv/lib/python3.8/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/home/dgresch/venv/lib/python3.8/site-packages/pluggy/_callers.py", line 55, in _multicall
    gen.send(outcome)
  File "/home/dgresch/venv/lib/python3.8/site-packages/_pytest/helpconfig.py", line 100, in pytest_cmdline_parse
    config: Config = outcome.get_result()
  File "/home/dgresch/venv/lib/python3.8/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/home/dgresch/venv/lib/python3.8/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/home/dgresch/venv/lib/python3.8/site-packages/_pytest/config/__init__.py", line 1003, in pytest_cmdline_parse
    self.parse(args)
  File "/home/dgresch/venv/lib/python3.8/site-packages/_pytest/config/__init__.py", line 1283, in parse
    self._preparse(args, addopts=addopts)
  File "/home/dgresch/venv/lib/python3.8/site-packages/_pytest/config/__init__.py", line 1172, in _preparse
    self.pluginmanager.load_setuptools_entrypoints("pytest11")
  File "/home/dgresch/venv/lib/python3.8/site-packages/pluggy/_manager.py", line 288, in load_setuptools_entrypoints
    self.register(plugin, name=ep.name)
  File "/home/dgresch/venv/lib/python3.8/site-packages/_pytest/config/__init__.py", line 436, in register
    ret: Optional[str] = super().register(plugin, name)
  File "/home/dgresch/venv/lib/python3.8/site-packages/pluggy/_manager.py", line 114, in register
    self._verify_hook(hook, hookimpl)
  File "/home/dgresch/venv/lib/python3.8/site-packages/pluggy/_manager.py", line 232, in _verify_hook
    raise PluginValidationError(
pluggy._manager.PluginValidationError: Plugin 'cpp' for hook 'pytest_collect_file'
hookimpl definition: pytest_collect_file(parent, file_path)
Argument(s) {'file_path'} are declared in the hookimpl but can not be found in the hookspec

Additional observations:

The exception can be avoided by either downgrading pytest-cpp to 2.1.1, or upgrading pytest to 7.x.

Add support of prefix-ish parameter to run executable

Hi (and Happy New Year)

I'm trying to use pytest-cpp to test multiplatform project.
One of my build target is Windows under Linux (using MinGW as compiler). It is also possible to use Linux to run Windows tests with wine.
I did try to specify cpp_harness parameter but it works only when the list of tests are known and to get that list from a gtest execurtable it is needed to run it first. And here I have an isssue, cpp_harness is not in use in 'list_tests' function and the function is trying to run executable as is.
From other hand I think it is not correct to use the same parameter as cpp_haness to get list of tests and to run each of test.

Maybe it will be usefull for other projects where an executable cannot be run directly and only via some kind of emulator layer.

Skipping invalid test suites

Hi,
Now pytest-cpp just skip executable if it's not a valid test suite. Sometimes it leads to a situation when we just skip some tests without any notification just because they fail on start. So we get clean test report though we have a serious issue.
Is it possible to catch invalid test suites and add them to the xml report?

No skipped message is included in result.xml file

In gtest version 1.11.0, now the GTEST_SKIP macro can contain a skipped message that is propagated to the output file, but pytest-cpp is just marking the test as skipped. Here is the doc: https://google.github.io/googletest/advanced.html#skipping-test-execution

Gtest output:

<?xml version="1.0" encoding="UTF-8"?>
<testsuites tests="1" failures="0" disabled="0" skipped="1" errors="0" time="0.081" timestamp="2021-12-02T17:30:51.978" random_seed="715" name="AllTests">
  <testsuite name="pv_policy_example" tests="1" failures="0" disabled="0" skipped="1" errors="0" time="0.08" timestamp="2021-12-02T17:30:51.979">
    <testcase name="policy_config_mpls_service_config" status="run" result="skipped" time="0.08" timestamp="2021-12-02T17:30:51.979" classname="pv_policy_example">
      <skipped message="../../../test/unit/tests/example/policy.cpp:2079&#x0A; This common test is skipped for Box."><![CDATA[../../../test/unit/tests/example/policy.cpp:2079
This common test is skipped for Box.]]></skipped>
<properties>
<property name="seed" value="715"/>
</properties>
    </testcase>
  </testsuite>
</testsuites>

pytest-cpp output:

<?xml version="1.0" encoding="utf-8"?>
<testsuites>
  <testsuite errors="0" failures="0" hostname="provision-sdk" name="pytest" skipped="0" tests="1" time="1.543" timestamp="2021-12-02T17:08:31.020895">
    <testcase classname="pvunittest" file="pvunittest" line="0" name="pv_policy_example.policy_config_mpls_service_config" time="0.114">
      <properties>
        <property name="test_id" value="pv_policy_example.policy_config_mpls_service_config"/>
        <property name="seed" value="64910"/>
      </properties>
    </testcase>
  </testsuite>
</testsuites>

Collect error when using xdist and temporary .pyc file is deleted

Rare error that might happen when using xdist:

../../../../../miniconda/envs/_petrolib20_examples-linux64/lib/python2.7/site-packages/pytest_cpp/plugin.py:16: in pytest_collect_file
    is_executable = os.stat(str(path)).st_mode & stat.S_IXUSR
E   OSError: [Errno 2] No such file or directory: '/home/jenkins/Work/Jenkins/workspace/souring-master-ix_two_way-linux64/etk/petrolib20_examples/source/python/petrolib20_examples/tests/__pycache__/test_duplicate_filenames.cpython-27-PYTEST.pyc.21746'

The problem appears to be that temporary pyc files which are generated by pytest's assertion rewrite are being collected by pytest and forwarded to the hooks. By the time the hook executes, the file is gone already.

cc @gqmelo

Ignore python test files maybe?

pytest version 4.0.0, pytest-cpp-1.1.0

I like to make my test files executable sometimes, so that I can run them directly. Unfortunately this interacts with pytest-cpp in a noticeably annoying way:

$ cat test_sample.py
#!/usr/bin/env pytest
import pytest

def test_thing():
    pass

$ time pytest test_sample.py

[snip]

real    0m1.026s
user    0m0.902s
sys     0m0.121s

$ chmod +x test_sample.py

$ time pytest test_sample.py

[snip]

real    0m3.152s
user    0m2.762s
sys     0m0.389s

Every time pytest-cpp sees an executable file with a test-like name, it tries to execute it twice (separately checking for boost/google tests). If the file is a python script that imports pytest, the whole thing ends up taking 3 times as much time as usual just for initialization.

I don't know the details of pytest plugin architecture, so I'm not sure which is the best way to fix it (if any, maybe I should just not do that thing, as the joke goes): make this plugin explicitly skip *.py files or maybe pytest itself shouldn't give the files it successfully imported and registered to any further plugins?

boost: output xml (log and report) to a file instead of stdout/stderr

Add code coverage

Add type-checking with mypy

Type checking would catch issues like the one in #84.

Getting some warnings with pytest 7.0

After installing the plugin and running the tests, I had the following warnings printed out after the run:

../../../../usr/local/lib/python3.6/dist-packages/_pytest/nodes.py:148
../../../../usr/local/lib/python3.6/dist-packages/_pytest/nodes.py:148
../../../../usr/local/lib/python3.6/dist-packages/_pytest/nodes.py:148
../../../../usr/local/lib/python3.6/dist-packages/_pytest/nodes.py:148
../../../../usr/local/lib/python3.6/dist-packages/_pytest/nodes.py:148
  /usr/local/lib/python3.6/dist-packages/_pytest/nodes.py:148: PytestDeprecationWarning: <class 'pytest_cpp.plugin.CppFile'> is not using a cooperative constructor and only takes {'parent', 'facade', 'fspath', 'arguments'}.
  See https://docs.pytest.org/en/stable/deprecations.html#constructors-of-custom-pytest-node-subclasses-should-take-kwargs for more details.
    f"{self} is not using a cooperative constructor and only takes {set(known_kw)}.\n"

../../../../usr/local/lib/python3.6/dist-packages/pytest_cpp/plugin.py:98
../../../../usr/local/lib/python3.6/dist-packages/pytest_cpp/plugin.py:98
../../../../usr/local/lib/python3.6/dist-packages/pytest_cpp/plugin.py:98
../../../../usr/local/lib/python3.6/dist-packages/pytest_cpp/plugin.py:98
../../../../usr/local/lib/python3.6/dist-packages/pytest_cpp/plugin.py:98
  /usr/local/lib/python3.6/dist-packages/pytest_cpp/plugin.py:98: PytestRemovedIn8Warning: The (fspath: py.path.local) argument to CppFile is deprecated. Please use the (path: pathlib.Path) argument instead.
  See https://docs.pytest.org/en/latest/deprecations.html#fspath-argument-for-node-constructors-replaced-with-pathlib-path
    pytest.File.__init__(self, fspath, parent)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

I'm using pytest-cpp 2.1.1 with Python 3.6.9. I've filtered the deprecation warnings out so it's not a huge issue but would be good to fix.

Check if files are executable before forwarding them to the facades

Use os.stat(filename).st_mode & stat.S_IXUSR to speed that up, including doing that before fnmatch.

Add boost::test support

C++ tests not collected due to '*_test' mask on Windows

Hello, I am a beginner so I probably just misunderstood how to use this package, however it seems that my installation has trouble collecting tests from C++ files. Here's my situation:

I have both pytest and pytest-cpp installed (both through pip).
I have a folder containing:
'generator_demo_test.py' that has two test functions, each with its own assert statement
'generator_demo_test.cpp' that #includes gtest.h and has two TEST() functions (but no main() function, if that makes a difference)

I compiled the cpp file with gtest and now have demo_test.exe in the folder with the .py and .cpp files. Then I open a terminal in that folder and type py.test --collect-only and I get the following output:

=============test session starts =============
platform win32 -- Python 3.7.7, pytest-5.4.1, py-1.8.1, pluggy-0.13.1
rootdir: C:\Users\myuser\Documents\Code\10_Local\demo
plugins: allure-pytest-2.8.10, cpp-1.2.0, TyphoonTest-1.11.1
collected 2 items
<Module generator_demo_test.py>
  <Function test_vout>
  <Function test_samples>
=============no tests ran in 0.14s ==========

So the c++ tests are ignored.

However, if I type py.test --collect-only generator_demo_test.exe, then the output is:

================================================= test session starts =================================================
platform win32 -- Python 3.7.7, pytest-5.4.1, py-1.8.1, pluggy-0.13.1
rootdir: C:\Users\myuser\Documents\Code\10_Local\demo
plugins: allure-pytest-2.8.10, cpp-1.2.0, TyphoonTest-1.11.1
collected 2 items
<CppFile oldgenerator_demo_test.exe>
  <CppItem Addition_TEST_SUITE.AdderTest_SIMPLE>
  <CppItem Addition_TEST_SUITE.AdderTest_COMPLEX>

================================================== warnings summary ===================================================
c:\users\myuser\appdata\local\programs\python\python37\lib\site-packages\pytest_cpp\plugin.py:41
  c:\users\myuser\appdata\local\programs\python\python37\lib\site-packages\pytest_cpp\plugin.py:41: PytestDeprecationWarning: direct construction of CppFile has been deprecated, please use CppFile.from_parent
    return CppFile(path, parent, facade_class(), test_args)

c:\users\myuser\appdata\local\programs\python\python37\lib\site-packages\pytest_cpp\plugin.py:67
c:\users\myuser\appdata\local\programs\python\python37\lib\site-packages\pytest_cpp\plugin.py:67
  c:\users\myuser\appdata\local\programs\python\python37\lib\site-packages\pytest_cpp\plugin.py:67: PytestDeprecationWarning: direct construction of CppItem has been deprecated, please use CppItem.from_parent
    yield CppItem(test_id, self, self.facade, self._arguments)

-- Docs: https://docs.pytest.org/en/latest/warnings.html
================================================= 3 warnings in 0.11s =================================================

Then it does find the C++ stuff. (Also, any ideas about those warnings?)

So I am not sure how to make it collect both py tests and c++ tests just by running py.test. As a disclaimer, I did not configure anything after installing pytest-cpp with pip. I saw about the cpp_files setting in the ReadMe but it is not clear to me where that configuration file is, and besides it says the default is already *_test executables.

Add a way to use valgrind, catchsegv or similar tool

Tools like valgrind and catchsegv are used in this way

valgrind --tool=memcheck gtest-binary

I think a way of using something like that would be to add a pytest_addoption on the ini like

[pytest]
harness = valgrind --tool=memcheck

thoughts? recommendations?

Issues if a test exits before xml is created

If for any reason a google test exits with exit(1) or before the test ends, code will still try to read the xml file, but since it doesn't exists a FileNotFound exception is raised.

https://github.com/pytest-dev/pytest-cpp/blob/master/src/pytest_cpp/google.py#L89
https://github.com/pytest-dev/pytest-cpp/blob/master/src/pytest_cpp/google.py#L106

Add support for Catch2 v3

As far as I can tell, v3 binaries don't work with the current Catch2 support, because --list-test-names-only command line option is missing.

pytest-dev / pytest-cpp Goto Github PK

pytest-cpp's People

Contributors

Stargazers

Watchers

Forkers

pytest-cpp's Issues

Recommend Projects

Recommend Topics

Recommend Org