Reporting integration about ocs-ci HOT 13 CLOSED

I have a WIP using the pytest-reportportal plugin. This allows us to use the --reportportal CLI option to post our results to the configured report portal instance. This configuration is done in pytest.ini at the moment, but we most likely want to determine a more secure place to pull our configuration from instead of having it in plain text.

Example execution:

pytest tests/test_pytest.py --reportportal --rp-launch MyLaunchName --rp-launch-desc "An example launch description"

What this will look like in report portal:

Top level launch:

Test cases for each test file executed:

Test case per function/method level test in the file:

Details on test cases including logs for errors/failures:

One caveat here is that we are stuck installing the reportportal-client dependency from the master branch of their github repo because they haven't published an actual release to pypi with code that this working version of pytest-reportportal depends on. This is pretty easy to do and will look like this in our setup.py:

    install_requires=[
        ...
        'reportportal-client @ git+https://github.com/reportportal/client-Python.git@master',
        'pytest-reportportal==1.0.5',
    ],

This solution also requires us to update pip to support this syntax so I will be adding the following to our getting started documentation, after creating the virtual environment:

pip install --upgrade pip

If this approach is sufficient for us I can get clean this up and add to our documentation to support it. If there is some functionality we might be missing here let's talk about it and figure out what needs to be done to meet our needs.

from ocs-ci.

@clacroix12 assigning this to you? Ideally we want the pypi method to import the module and use it?

from ocs-ci.

@petr-balogh do you have links to what RHV has done on this front?

from ocs-ci.

@clacroix12 in rhevm we used something like this: https://github.com/tareqalayan/reportportal_cli this is just fork from CNV team. It's executed after the execution of pytest so not part of run of pytest itself. Like publisher in the jenkins job from xunit file and you can add the logs and so on. How CNV is calling it is implemented here

from ocs-ci.

@clacroix12 Nice!, I think we can push them to release a version in pypi and until then we can use the master branch like you suggested.

from ocs-ci.

Initial take at this is here: ecabeba

from ocs-ci.

This is great @clacroix12

from ocs-ci.

#149

from ocs-ci.

@clacroix12 Speaking about linking to polarion test cases: Assuming that we are going to report the test run resutls via junit xml report (as generated by pytest), it seems to me that we need to find a way for a polarion test case will be referenced there.

There are few options which pytest provides here, like:

• https://docs.pytest.org/en/latest/usage.html#record-property
• https://docs.pytest.org/en/latest/usage.html#record-xml-attribute
• https://docs.pytest.org/en/latest/usage.html#record-testsuite-property

This means we could just use the fixture in the test code, or establish a convention along with ytest_collection_modifyitems hook for test markers for the same purpose.

What would you prefer? If I recall right, some of these fixtures were implemented by cloud forms qe team to get their test results importable to polarion.

from ocs-ci.

@mbukatov I think the record property is the cleanest solution for us. Specifically implementing it with a custom marker for the Polarion ID. Once it is a property on the Test Case we can do whatever we want with it from the reporting side, correct?

# content of conftest.py

def pytest_collection_modifyitems(session, config, items):
    for item in items:
        for marker in item.iter_markers(name="polarion_id"):
            polarion_id = marker.args[0]
            item.user_properties.append(("polarion_id", polarion_id))

# content of test_function.py
import pytest


@pytest.mark.polarion_id(1234)
def test_function():
    assert True

from ocs-ci.

@clacroix12 I agree using marker (as shown in your example) is the best approach. It also allow us to switch from property to xml attribute (or vice versa) without changing any test if CI updates requires it later.

The question whether to use properties (as you notes in the example) or record-xml attribute boils down to the needs of our CI pipeline and consumers of the junit xml file used there. So you, @petr-balogh , @dahorak and other CI people probably need to reach a consensus here.

from ocs-ci.

#243

from ocs-ci.

Closing as #149 and #243 are complete. Any subsequent issues or enhancements around reporting can be raised in their own issues.

from ocs-ci.