xfail support failed with pytest 5.4 about pytest-check HOT 7 CLOSED

Example code exhibiting the problem:

import pytest_check as check
import pytest

@pytest.mark.xfail()
def test_fail_check():
    check.equal(1, 2)

Note: I did verify that the code works with pytest 5.3.5.
It was the pytest 5.4.0 update that caused the failure

from pytest-check.

The code in question is probably the pytest_runtest_makereport() hook:

@pytest.hookimpl(hookwrapper=True, trylast=True)
def pytest_runtest_makereport(item, call):
    outcome = yield
    report = outcome.get_result()
    evalxfail = getattr(item, "_evalxfail", None)
    failures = check_methods.get_failures()
    check_methods.clear_failures()

    if failures:
        if evalxfail and evalxfail.wasvalid() and evalxfail.istrue():
            report.outcome = "skipped"
            report.wasxfail = evalxfail.getexplanation()
        elif outcome._result.longreprtext.startswith("[XPASS(strict)]"):
            report.outcome = "skipped"
            report.wasxfail = "\n".join(failures)
        else:
            summary = "Failed Checks: {}".format(len(failures))
            longrepr = ["\n".join(failures)]
            longrepr.append("-" * 60)
            longrepr.append(summary)
            report.longrepr = "\n".join(longrepr)
            report.outcome = "failed"

This is where I'm setting failue, xfail, etc.
My guess is the format of the _evalxfail object may have changed.

from pytest-check.

Ok. well, now that I pinpoint where I'm abusing the pytest internals manipulate the results, I'm not surprised that relying on _evalxfail and _result eventually broke.
This will take some head scratching to solve.

May be prudent to simplify this plugin and state that xfail isn't supported.

from pytest-check.

this is definitely something that I think should be exposed by pytest to plugins, but I'm not quite sure how at the moment 🤔

here's an UGLY workaround (which really just restores the access this plugin was doing before)

try:
    evalxfail = item._evalxfail
except AttributeError:
    from _pytest.skipping import evalxfail_key
    evalxfail = item._store[evalxfail_key]

from pytest-check.

thank you!

from pytest-check.

I'm not sure if this is reliable but you may be able to loop through the report object's keyword dict and look for the xfail key. The keyword should show up if xfail is added programmatically or with pytest.mark.xfail. This also would eliminate the use of protected objects.

from pytest-check.

I like this idea. I'll try it sometime soon-ish.
For now, I'll close this with the workaround in place

from pytest-check.