Allowing when steps to follow then steps? about pytest-bdd HOT 12 CLOSED

We're currently using behave as a way of scripting selenium with a library of standard steps. Conventions are great, but I don't think it's desirable for a library to enforce those conventions.

The way pytest-bdd passes state through fixtures is very pleasant to work with. Being a little more permissive would make it much easier to adopt, at least for those of us used to other cucumber-alikes.

from pytest-bdd.

I'm not using my gherkin tests in a pure BDD-fashion, instead I'm using them as a way to script selenium as I find that cucumber style tests do a fairly good job of recording intent. We do this using a library of generic steps along with fixtures that setup our application. This means we don't have to write step implementations for specific scenarios very often.

Take a scenario that what's to test that the page title is updated as you navigate around a 'single page' app. When testing a change it's useful to make assertions both before an after the action that changes it:

Feature: Title
    Scenario: Title updated correctly on page changes
        When I visit "/"
        Then the title should be "Home"
        When I click the link to "/about"
        Then the title should be "About"
        When I go back
        Then the title should be "Home"

Removing the assertions before/after the title would make the test less robust.

This test is valid by the gherkin specification, "Cucumber doesn’t technically distinguish between these three kind of steps." It parses successfully in cucumber, behave and gherkin-parser:

$ ~/.gem/ruby/2.0.0/bin/cucumber title.feature 
Feature: Title

  Scenario: Title updated correctly on page changes # title.feature:2
    When I visit "/"                                # title.feature:3
    Then the title should be "Home"                 # title.feature:4
    When I click the link to "/about"               # title.feature:5
    Then the title should be "About"                # title.feature:6
    When I go back                                  # title.feature:7
    Then the title should be "Home"                 # title.feature:8

1 scenario (1 undefined)
6 steps (6 undefined)
0m0.007s

You can implement step definitions for undefined steps with these snippets:

When(/^I visit "([^"]*)"$/) do |arg1|
  pending # Write code here that turns the phrase above into concrete actions
end

Then(/^the title should be "([^"]*)"$/) do |arg1|
  pending # Write code here that turns the phrase above into concrete actions
end

When(/^I click the link to "([^"]*)"$/) do |arg1|
  pending # Write code here that turns the phrase above into concrete actions
end

When(/^I go back$/) do
  pending # Write code here that turns the phrase above into concrete actions
end

$ bin/behave title.feature 
Feature: Title # title.feature:1

  Scenario: Title updated correctly on page changes  # title.feature:2
    When I visit "/"                                 # None
    Then the title should be "Home"                  # None
    When I click the link to "/about"                # None
    Then the title should be "About"                 # None
    When I go back                                   # None
    Then the title should be "Home"                  # None


Failing scenarios:
  title.feature:2  Title updated correctly on page changes

0 features passed, 1 failed, 0 skipped
0 scenarios passed, 1 failed, 0 skipped
0 steps passed, 0 failed, 0 skipped, 6 undefined
Took 0m0.000s

You can implement step definitions for undefined steps with these snippets:

@when(u'I visit "/"')
def step_impl(context):
    raise NotImplementedError(u'STEP: When I visit "/"')

@then(u'the title should be "Home"')
def step_impl(context):
    raise NotImplementedError(u'STEP: Then the title should be "Home"')

@when(u'I click the link to "/about"')
def step_impl(context):
    raise NotImplementedError(u'STEP: When I click the link to "/about"')

@then(u'the title should be "About"')
def step_impl(context):
    raise NotImplementedError(u'STEP: Then the title should be "About"')

@when(u'I go back')
def step_impl(context):
    raise NotImplementedError(u'STEP: When I go back')

Only pytest-bdd gives an error:

$ bin/pytest-bdd generate title.feature 
Traceback (most recent call last):
  File "bin/pytest-bdd", line 9, in <module>
    load_entry_point('pytest-bdd==2.10.0', 'console_scripts', 'pytest-bdd')()
  File "/Users/lrowe/scratch/cuketest/lib/python3.4/site-packages/pytest_bdd/scripts.py", line 79, in main
    args.func(args)
  File "/Users/lrowe/scratch/cuketest/lib/python3.4/site-packages/pytest_bdd/scripts.py", line 49, in print_generated_code
    features, scenarios, steps = parse_feature_files(args.files)
  File "/Users/lrowe/scratch/cuketest/lib/python3.4/site-packages/pytest_bdd/generation.py", line 134, in parse_feature_files
    features = get_features(paths)
  File "/Users/lrowe/scratch/cuketest/lib/python3.4/site-packages/pytest_bdd/feature.py", line 171, in get_features
    feature = Feature.get_feature(base, name)
  File "/Users/lrowe/scratch/cuketest/lib/python3.4/site-packages/pytest_bdd/feature.py", line 301, in get_feature
    feature = Feature(base_path, filename, encoding=encoding)
  File "/Users/lrowe/scratch/cuketest/lib/python3.4/site-packages/pytest_bdd/feature.py", line 230, in __init__
    line_number, clean_line, filename)
pytest_bdd.feature.FeatureError: When steps must be the first or follow Given steps.
Line number: 5.
Line: When I click the link to "/about".
File: /Users/lrowe/scratch/cuketest/title.feature

This makes starting off with pytest-bdd unnecessarily frustrating!

from pytest-bdd.

hmm, your example is quite convincing :)
please create a PR to relax that restriction, try to use it - we'll discuss it then

from pytest-bdd.

complex scenario (e.g. a multi-page form.) should be split into smaller, understandable tests (scenarios)
Strict ordering helps to enforce and sanitize tests so you always have setup (given), action (when), assertion (then)
If behave or cucumber allow reorderding, then they allow bad quality tests to be created, in general

from pytest-bdd.

forgot to add: strict when then ordering also forces you to have one (grouped) action and one (grouped) assertion, which leads to better tests and not scripts

from pytest-bdd.

The thing is that bunch of whens can replace the setup for a certain step of your flow. In this case the assertion in Then section has nothing to do with conditions mentioned in the Given section, which is not a good test, but just a recorded script. Test has to assert the result of (preferably) one action in the certain setup.
We have an example in our project of multi-page wizard. There we abstracted the action "When I go to the page X" which does bunch of actions how to reach the desired page assuming everything works fine without assertions. After that you can call specific When action that you are asserting in Then section.

from pytest-bdd.

Speaking of a framework. The reason to implement pytest-bdd was our experience with fixtures and unhappiness about robotframework. We suffered quite a lot from scripts in our tests. I believe that this limitation has to be there because it is in the specification of the Gherkin language for a reason (given-when-then).
Speaking of a library. I guess we could think of a way where feature parser could be pluggable and passed via external configuration.
In this case you would have your own extended testing framework based on pytest-bdd library.
P.S. Have you also considered step implementation that will bring you to the right place in one When action? Are you saving on the test execution time doing all the asserts on the way? Or is it about porting the existing code? It is interesting to understand your motivation, maybe it is better to refactor tests after all? Technically it could be not too difficult, just by splitting your scripts into multiple scenarios with specific names. You'll get fine-grain specification and reporting.

from pytest-bdd.

@lrowe my impression is that you use those complex scenarios with the mix of when-then-when-then because the test setup is hard with your existing tools
pytest and pytest-bdd is born to make the setup easy, so you can and should split your scenario and reuse existing step definitions and also plain fixtures.
for example

Scenario: User edit form

    Given I have a user

    When I go to the user edit page
    And I fill username to "Some"
    And I submit the form

    Then I should see a success message
    And the user should have "Some" username 

    When I fill birthdate to "12-01-1981"
    And I submit the form

    Then I should see a success message
    And the user should have "12-01-1981" birthdate

should be split into:

Feature: Edit user details

    Background:
        Given I have a user

    Scenario: Edit username

        When I go to the user edit page
        And I fill username to "Some"
        And I submit the form

        Then I should see a success message
        And the user should have "Some" username 


    Scenario: Edit birthdate

        When I go to the user edit page
        And I fill birthdate to "12-01-1981"
        And I submit the form

        Then I should see a success message
        And the user should have "12-01-1981" birthdate

from pytest-bdd.

2.12.0 is out

from pytest-bdd.

@bubenkoff, @lrowe, by the way, where it's specified, that When can not be used after Then in Gherkin?

from pytest-bdd.

@AndrewPashkin it's not specified, but recommended https://github.com/cucumber/cucumber/wiki/Given-When-Then

from pytest-bdd.

This fix was not applied for Background. Works only for Scenario

Background: 
    When I visit "/"
    Then the title should be "Home"
    When I click the link to "/about"
    Then the title should be "About"

Fails

from pytest-bdd.