[🐛 Bug]: allure-reporter - setting a custom message in matcher options makes the test to be reported as broken instead of failed about webdriverio HOT 5 OPEN

@pfyod

Failed assertions should always report test as failed.

Sounds good. We need more details. How is the reporter supposed to guess that this stacktrace is an assert error and not a regular error?
Keep in mind that your solution should work with any assert library, even if we don't have invented it yet 😊

from webdriverio.

@BorisOsipov I guess the problem is indeed hard to solve if you want to support any third party assert library (maybe some sort of extension point that allows the user to override getTestStatus?).

My point is that it should probably work in an "obvious" manner for first-party library combination of expect-webdriverio + @wdio/allure-reporter, or be somehow mentioned in docs.

from webdriverio.

Thanks for reporting!

We greatly appreciate any contributions that help resolve the bug. While we understand that active contributors have their own priorities, we kindly request your assistance if you rely on this bug being fixed. We encourage you to take a look at our contribution guidelines or join our friendly Discord development server, where you can ask any questions you may have. Thank you for your support, and cheers!

from webdriverio.

Probably the best solution is to port https://github.com/allure-framework/allure-js/blob/dc06dbc09c64a256a573a43dbb7cedf15f27fb2b/packages/allure-js-commons/src/sdk/utils.ts#L5 to reporter.

from webdriverio.

@BorisOsipov: I think that there's actually some sort of bug (or convention that does not work that great in this scenario) somewhere. Only the first line of error message ends up inside test.error.message that gets passed to getTestStatus (the rest is inside test.error.stack only), and since custom message gets prepended to the original expect result with a newline, the actual whole message never gets inspected by getTestStatus. Would get "solved" if one reorders if (test.error.message) { and if (test.error.stack) {, but I do not know further implications of that.

from webdriverio.