Proposal: remove strict and always fail on undefined/pending steps about common HOT 11 CLOSED

Coming back to this. I think this should be removed because of the poor practice it lets you walk into. When i first came onto cucumber-js, the builds looked like this. One of the first things I wanted to do when I joined was get the test suite to be entirely green (and using --strict).

I don't think the ability to have "passing" tests while they are incomplete is a practice we want to have as the default. I would be okay with making strict the default and allowing users some way to go --not-strict but I think that needs to be exception instead of the default behavior.

I don't think incomplete features should ever be committed to the main branch. They can be left on a feature branch or use tags to avoid running them. I don't see the use in running them as if an earlier step breaks, a developer might fix and not realize the feature is incomplete until later.

from common.

Could you dig in the google group archives and/or github to find the original discussion? Probably 2008-2009 I'd guess

from common.

@sebrose please wheigh in

from common.

We don't use --strict, because we need a way of turning tests yellow without failing the entire build, so we are overloading pending and turning --strict off.

To explain our use case: we are checking routing to downstream HTTP services and don't 100% care if the service responds with 4xx or 5xx, because we don't control the downstream services, but we still would like to know (e.g., so that if people come to us and say "why isn't routing to service X working", we can say, "because service X is returning Y"). Therefore, we have step definitions that say "response code should not be" that return pending if they find a match (rather than raising an error).

We obviously would prefer that there was a clean, supported way to indicate a warning (as opposed to a fatal error), but as that doesn't seem to be supported in cucumber at this time, we use pending, and therefore require the ability to turn off --strict so that our builds don't fail.

from common.

At the moment Cucumber-CPP runs the old Cucumber-TCK, but without implementing all features. This means that we have to use the default non-strict mode or we'd have to add tags to Cucumber-TCK.

Despite this need till we move away from it, I agree that it promotes bad practice so my 👍 to @charlierudolph's proposal of --not-strict.

from common.

Cucumber-JVM already have the ability to both turn on and turn off the boolean options, so to be consistent with the naming there, --no-strict rather then --not-strict should be used if reverting the default behaviour to fail on undefined or pending steps.

from common.

The next version of cucumber-js is going to have strict as the default with a CLI option for --no-strict. Thus I'm going to close this. Please reopen if anyone wants to discuss further

from common.

@charlierudolph how about another ticket on this repo that documents what you did, with checkboxes for all the other implementations so we can keep track of which ones are working 'to spec'?

from common.

Does that belong here @mattwynne? Wouldn't it be more useful to create issues on each repo instead?

from common.

Let's keep all consolidation tickets here. Less work, easier to keep track of. We can use a (new) consolidation tag to group them.

from common.

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

from common.