pytest-dev / pytest-bdd Goto Github PK

Step definitions are registered for specific keywords, so it would be useful to report which keyword was used in the failed lookup.

pytest_bdd.exceptions.StepDefinitionNotFoundError: Step definition is not found: "I should see an element with the css selector "table.sticky-area > tbody > tr"". Line 6 in scenario "Table" in the feature "/Users/lrowe/encoded/src/encoded/tests/features/experiments.feature

I'm probably missing something obvious, but is there a plain text reporting option that logs progress step by step to stdout?

Many Gherkin examples on the net indent And/But steps with an extra space:

Scenario: Indent my steps
    When I indent with 4 spaces
     And I indent with 5 spaces to line up

This breaks due to pytest-bdd using indentation of multiline steps: https://pytest-bdd.readthedocs.org/en/latest/#multiline-steps

It also makes feature files extremely sensitive to files with mixed tabs/spaces, such as the steps.feature in the pytest-bdd tests.

It would be nice if py.test_bdd provided a cli tool to automatically generate the necessary structures from .feature file(s). I think I would find it painful to have to reproduce the contents of .feature files in my tests.

For now it's totally unclear from pytest output from which line of feature you experience the fail. Make this better: Write the line number with some quote from feature file to help user understand where the problem is

On Python 2.7, we have six pegged to a version that apparently does not have tools needed by the latest pytest-bdd.

We will probably upgrade our six install; but nonetheless, pytest-bdd should probably declare a minimum requirement.

$ pip install --upgrade pytest-bdd

Downloading/unpacking pytest-bdd
  Downloading pytest-bdd-2.11.1.tar.gz (55kB): 55kB downloaded
  Running setup.py (path:/Users/dustin/.virtualenvs/napa/build/pytest-bdd/setup.py) egg_info for package pytest-bdd
    Traceback (most recent call last):
      File "<string>", line 17, in <module>
      File "/Users/dustin/.virtualenvs/napa/build/pytest-bdd/setup.py", line 11, in <module>
        import pytest_bdd
      File "pytest_bdd/__init__.py", line 6, in <module>
        from pytest_bdd.steps import given, when, then
      File "pytest_bdd/steps.py", line 43, in <module>
        from .feature import parse_line, force_encode
      File "pytest_bdd/feature.py", line 39, in <module>
        @six.python_2_unicode_compatible
    AttributeError: 'module' object has no attribute 'python_2_unicode_compatible'
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 17, in <module>

  File "/Users/dustin/.virtualenvs/napa/build/pytest-bdd/setup.py", line 11, in <module>

    import pytest_bdd

  File "pytest_bdd/__init__.py", line 6, in <module>

    from pytest_bdd.steps import given, when, then

  File "pytest_bdd/steps.py", line 43, in <module>

    from .feature import parse_line, force_encode

  File "pytest_bdd/feature.py", line 39, in <module>

    @six.python_2_unicode_compatible

AttributeError: 'module' object has no attribute 'python_2_unicode_compatible'

When the test is not implemented they are not executed without a warning.
I can imagine scenario like:

1. I have a feature file with scenario "This is my scenario"
2. I have implemented scenario with:

@scenario("my.feature", "This is my scenario")

1. I forgot to add dot to a feature file and I'm adding now.
2. If I forgot to fix implementation the scenario is lost silently.

It would be nice to have a warning/error (pytest command line option?) to say that some tests are not implemented (or maybe even that some steps are never used?).

Hi,
I'm starting to use pytest-bdd, which seems a very interesting plugin ! But I'm experiencing some troubles with the fixtures returned by the given steps.

I have the following feature :

Scenario: Logging in
  Given I'm a user
  When I go to the login page
  And I enter my credentials
  And I submit the form
  Then I should be logged-in
  And I should be redirected

Scenario: Logging in with non verified email
  Given I'm a user with a non verified email
  When I go to the login page
  And I enter my credentials
  And I submit the form
  Then I should be redirected to the email confirmation page

My issue is with the "I enter my credentials" step. This step require the user fixture which is returned by the previous given step. But as you can see, I have a different given step in each scenario, which both return an user.

I don't know how to implement this behavior with pytest-bdd, but I don't see why I couldn't. I managed to do it using a global variable rather than the fixture returned by the given steps, but it's not very elegant.

Due to unicode encoding mess, pip fails to install pytest-bdd on Windows. By default, Python 3.3 uses local Windows encoding settings (1252 instead of UTF-8) and 'curved' quotation marks in README.rst, line 513 trigger an error.

Could you please replace 'curved' quotation marks with apostrophes:

    ‘pytestbdd_feature_base_dir’ -> 'pytestbdd_feature_base_dir'

or add encoding parameter to setup.py, line 28:

    ...
    open(os.path.join(dirname, 'README.rst'), encoding="utf-8").read() + '\n' +
    ...

Thanks!

What kind of stuff can be done inside the function:

@scenario(...)
def test_001():
    # What kind of stuff can be done here?
    pass

To reproduce:

virtualenv bdd -p python3.4 && bdd/bin/pip install pytest-bdd && bdd/bin/pytest-bdd

Output:

Traceback (most recent call last):
  File "bdd/bin/pytest-bdd", line 9, in <module>
    load_entry_point('pytest-bdd==2.5.3', 'console_scripts', 'pytest-bdd')()
  File "/home/spinus/tmp/bdd/lib/python3.4/site-packages/pytest_bdd/scripts.py", line 78, in main
    args.func(args)
AttributeError: 'Namespace' object has no attribute 'func'

Also line 59 in setup.py:

] + [("Programming Language :: Python :: %s" % x) for x in "2.6 2.7 3.0 3.1 3.2 3.3".split()],

missing 3.4 but tox.ini has it.

You may already know this but the Travis CI build is timing out and causing the build status to read as "error".

This is at the bottom of the logs:

I'm sorry but your test run exceeded 50.0 minutes.

One possible solution is to split up your test run.

Really excited about this library by the way!

A step name implemented by the same named function throws a StepDefinitionNotFound error as the step definition is overwritten by the function itself once it is returned from the decorator.

Test case: https://github.com/lrowe/pytest-bdd/tree/same-function-name

My initial thought was to work around this by prefixing with '_step' the name set on the module by .steps.contribute_to_module and looked up by .scenario._find_step_function, but that ran into problems with argumented steps:

diff --git a/pytest_bdd/scenario.py b/pytest_bdd/scenario.py
index 9cf4a39..966c631 100644
--- a/pytest_bdd/scenario.py
+++ b/pytest_bdd/scenario.py
@@ -84,12 +84,12 @@ def _find_step_function(request, step, scenario, encoding):
     """
     name = step.name
     try:
-        return request.getfuncargvalue(force_encode(name, encoding))
+        return request.getfuncargvalue('__step_' + force_encode(name, encoding))
     except python.FixtureLookupError:
         try:
             name = find_argumented_step_fixture_name(name, request._fixturemanager, request)
             if name:
-                return request.getfuncargvalue(name)
+                return request.getfuncargvalue('__step_' + name)
             raise
         except python.FixtureLookupError:
             raise exceptions.StepDefinitionNotFoundError(
diff --git a/pytest_bdd/steps.py b/pytest_bdd/steps.py
index 07ef4e6..d698a51 100644
--- a/pytest_bdd/steps.py
+++ b/pytest_bdd/steps.py
@@ -237,7 +237,7 @@ def contribute_to_module(module, name, func):

     :return: New function copy contributed to the module
     """
-    name = force_encode(name)
+    name = '__step_' + force_encode(name)
     func = recreate_function(func, module=module)
     setattr(module, name, func)
     return func

Readthedocs complains
Running Sphinx v1.1.3
Sphinx Standard Error

Exception occurred:
File "/var/build/user_builds/pytest-bdd/checkouts/latest/docs/conf.py", line 51, in
version = pytest_bdd.version
AttributeError: 'module' object has no attribute 'version'
The full traceback has been saved in /tmp/sphinx-err-lGXJIV.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
Either send bugs to the mailing list at http://groups.google.com/group/sphinx-dev/,
or report them in the tracker at http://bitbucket.org/birkenfeld/sphinx/issues/. Thanks!

it looks like a good practice to add the version

Within https://github.com/olegpidsadnyi/pytest-bdd#reusing-fixtures the example provided shows a previously used code snippet: given('I have beautiful article', fixture='article')

I'm not understanding from what's shown how this is being 'reused with another name'. Could we also say given('I have beautiful article', fixture='banana') and how would affect things. I think more context would help. Thank you : )

I'd like to switch to pytest-bdd from behave but it does not appear from the documentation that pytest-bdd supports multiline text (or text blocks) within scenarios. For example, a hypothetical scenario written for behave:

Scenario: Creating a resource
  Given a resource with an id of "xyz"
  When I make a POST request to "/resource/xyz/things" with
  """
  {
    "name": "something"
  }
  """
  Then I should get a 201 status code

If pytest-bdd does support this, I was assuming I'd see it in the documentation. If it doesn't not support this then I would love for it to be added so I can switch. 😉

In publish_article.feature:

The step:

Given I'm an author user

is not implemented.

Is this an oversight? Is this supposed to come from a different feature?

it should put a "# coding: utf-8" in the beginning of the file.

Regex is powerful tool but sometimes this power is not needed.
Probably for most cases just simple parser could be used (idea stolen from behave framework) – https://pypi.python.org/pypi/parse could be used as supplement to regexes in case of matching scenarios.

In that case step definition could be like:

@Given("I have {number} items")
@Given("the temperature is {temperature:float}C")

It's easy to miss the assert statement, and we can do something about it

For now, we have just plain pytest: fixture 'Bla bla' not found

Find good license and add to the repo and update the readme

I tried the example in the README and it breaks. It thinks the argument to the original step is a fixture.

fixture 'start' not found

When you have a scenario like:

Scenario: Title
Given Xxx
When Yyy
Then Zzz

And a test file like:

@given("Yyy")
def myfunc(): 
  pass
#  other fixtures...

It passes silently. I would expect an error, because in a When statement I am referencing a Given statement.

Currently it's only possible to add parameter for a step or scenario via the testcode. Gherkin support the feature of "Scenario Outline" http://docs.behat.org/guides/1.gherkin.html#scenario-outlines add different datasets to the feature-file. Is there a way to implement this?

P.S.: If you need support I can help you.

As in the subject.

I have prepared a minimal case to demonstrate this issue:
https://github.com/ropez/pytest-bdd-issue

Having a conftest.py in a subdirectory, and the subdirectory also having a __init__.py, the conftest module is evaluated twice. This is an important issue, because any hooks added there, such as pytest_pyfunc_call, will be registered twice, and called twice for each test case. This is causing problems in our actual, much bigger project.

Removing pytest-bdd from the equation, for instance by commenting out the @scenario decorator to make it a simple test function, removes the problem. Also, running py.test inside the subdirectory, or deleting the __init__.py file, removes the problem.

Awful quiet in here.

When a test fails, there are an additional 70 lines of stacktrace from: C:\Program Files (x86)\Python\2.7\lib\site-packages\pytest_bdd\scenario.py

These lines of the stacktrace are irrelevant to my test failure. Is there a way of suppressing these lines from the stacktrace?

With Cucumber and Behave, scenario outline steps have their placeholders substituted with the real values from the current example before the step definition is looked up.

We have a library of steps with arguments implementing commonly used parts of the Splinter API. So an outlined scenario might look like this:

Feature: Generics
    Scenario Outline: Visit collection and page
        When I visit "/<item_type>/"
        When I click the link with text that contains "<link_text>"

    Examples: Collections
        | item_type                     | link_text                             |
        | antibody_lot                  | ENCAB000ACQ                           |

Using step definitions:

@when(parse('I visit "{url}"'))
def when_i_visit_url(browser, base_url, url):
    full_url = urljoin(base_url, url)
    browser.visit(full_url)

@when(parse('I click the link with text that contains "{text}"'))
def click_link_with_text_that_contains(browser, text):
    anchors = browser.find_link_by_partial_text(text)
    anchors.first.click()

However here , the url for When I visit "/<item_type>/" is the unsubstituted string: "/<item_type>/".

Tests are created using @scenario decorator and after that the function has to be written (with unique function name). So basically test name exist in 3 places, feature file, decorator and (kind of) in function name. Instead something like:

@scenario(....)
def _():
    pass

could be used to generate functions automatically searching for unique names.

Is this possible?

Can we make a class implement a scenario? Then common steps can be in a base class.

Your docs don't explicitly say one way or the other.

It doesn't work with Python 3.3... In the file "steps.py" there is a function "contribute_to_module" using "func.func_code". The "func_code" doesn't exists anymore in the Python 3.3...

To do things like:

Feature: Admin changing stuff

  Background:
    Given I am logged in as an administrator

  Scenario: Adding users
    When I act..

  Scenario: Changing passwords
    When I act...

  Scenario: Deleting stuff etc..
    When I act...

Would it not be nice to hide the regular expression code inside pytest-bdd to have parameterized steps similarly clean as in lettuce?
They do:

@step('I have the number (\d+)')

If you just leave off the @ symbol, it works as expected. The docs (wrongly) suggest that you can use the cucumber tag with that at symbol:
https://github.com/pytest-dev/pytest-bdd/blob/master/README.rst#organizing-your-scenarios

If I noticed this correctly, there is currently no support for Gherkin's background keywords. Is this a design decision, or is it just not implemented yet?

You can read up on background keywords here: http://docs.behat.org/guides/1.gherkin.html#backgrounds

Given a this directory structure and this feature file:

When I run pytest-bdd generate features/feature1.feature from within the tracepkg directory, I get the following output:

"""Minimum viable bdd feature tests."""

from pytest_bdd import (
    given,
    scenario,
    then,
    when,
)


@scenario('features/feature1.feature', 'Verify all bdd project components')
def test_verify_all_bdd_project_components():
    """Verify all bdd project components."""


@given('I'm a project admin')
def im_a_project_admin():
    """I'm a project admin."""


@then('I should see a features directory')
def i_should_see_a_features_directory():
    """I should see a features directory."""


@then('I should see a test directory')
def i_should_see_a_test_directory():
    """I should see a test directory."""


@then('I should see a test/bdd directory')
def i_should_see_a_testbdd_directory():
    """I should see a test/bdd directory."""


@when('I read the project's package directory')
def i_read_the_projects_package_directory():
    """I read the project's package directory."""

But I expected to see given, when, then, then, then and NOT given, then, then, then, when. Ideally, the steps would be ordered by gherkin step conventions.

There should be a way to escape a # or even better, if it is inside quotes or on a table it should be treated as data.

When a step is registered for both when and then it throws a StepTypeError because the first step registration decorator (first decorator runs last) overwrites the step registration of the other step registration. Similar to #120

Test case: https://github.com/lrowe/pytest-bdd/tree/same-step-name

Interestingly the builtin trace decorator which is registered for all of given/when/then does appear to work for When trace and Then trace: https://github.com/pytest-dev/pytest-bdd/blob/2.10.0/pytest_bdd/plugin.py#L24

That seems to be due to the function name being the same as the step name and the Given trace fixture being incorrectly loaded. Taking an example:

Feature: Trace
    Scenario: When trace
        When trace

from pytest_bdd import scenario


@scenario('trace.feature', 'When trace')
def test_when_trace():
    pass

Continuing from When trace throws an AttributeError when I would expect the test to pass:

$ .tox/py34/bin/py.test test_trace.py --tb native
========================================== test session starts ===========================================
platform darwin -- Python 3.4.2 -- py-1.4.28 -- pytest-2.7.1 -- /Users/lrowe/scratch/pytest-bdd/.tox/py34/bin/python3.4
cachedir: /Users/lrowe/scratch/pytest-bdd/.cache
rootdir: /Users/lrowe/scratch/pytest-bdd, inifile: tox.ini
plugins: bdd, cache, cov, pep8, xdist
collected 1 items 

test_trace.py::test_when_trace 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB set_trace (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
--Return--
> /Users/lrowe/scratch/pytest-bdd/pytest_bdd/plugin.py(29)trace()->None
-> pytest.set_trace()
(Pdb) c
FAILED

================================================ FAILURES ================================================
____________________________________________ test_when_trace _____________________________________________
Traceback (most recent call last):
  File "/Users/lrowe/scratch/pytest-bdd/test_trace.py", line 5, in test_when_trace
    def test_when_trace():
  File "/Users/lrowe/scratch/pytest-bdd/pytest_bdd/scenario.py", line 179, in _execute_scenario
    if step_func.step_type != step.type:
AttributeError: 'NoneType' object has no attribute 'step_type'
======================================== 1 failed in 1.24 seconds ========================================

Renaming the function def trace(): in plugin.py to def trace2(): exposes the underlying error reported here:

$ .tox/py34/bin/py.test test_trace.py --tb native
========================================== test session starts ===========================================
platform darwin -- Python 3.4.2 -- py-1.4.28 -- pytest-2.7.1 -- /Users/lrowe/scratch/pytest-bdd/.tox/py34/bin/python3.4
cachedir: /Users/lrowe/scratch/pytest-bdd/.cache
rootdir: /Users/lrowe/scratch/pytest-bdd, inifile: tox.ini
plugins: bdd, cache, cov, pep8, xdist
collected 1 items 

test_trace.py::test_when_trace FAILED

================================================ FAILURES ================================================
____________________________________________ test_when_trace _____________________________________________
Traceback (most recent call last):
  File "/Users/lrowe/scratch/pytest-bdd/test_trace.py", line 5, in test_when_trace
    def test_when_trace():
  File "/Users/lrowe/scratch/pytest-bdd/pytest_bdd/scenario.py", line 181, in _execute_scenario
    'Wrong step type "{0}" while "{1}" is expected.'.format(step_func.step_type, step.type)
pytest_bdd.exceptions.StepTypeError: Wrong step type "given" while "when" is expected.
======================================== 1 failed in 0.01 seconds ========================================

I'm not sure but probably pytest command is somehow not used (legacy?). Pytest is using now py.test as entrypoint.
I think pytest-bdd could use plugins to integrate with py.test (like other plugins do) and then to run the test only py.test command would be needed. I only assume that output plugin has to be changed to what pytest-bdd is using right now, but I didn't look is it hard or not, just an idea.

If it would be not easy/not needed change, still maybe would be better to follow pytest and use something like py.test-bdd?

It works fine if defined in the test file itself; but not if it is defined, say, in conftest.py.

Two things:

1. before scenario hook is not documented.
2. there is no after scenario hook :(

It would be nice to have full stack as in all "cucumber" frameworks, I mean: before/after, all, feature, scenario, step. What do you think?

pytest-bdd would be much more awesome if it would auto create scenario functions, so you wouldn't have to define empty functions just to make the test go. I think this either could be implemented as a test collector (also would make the test reporter more user friendly as it would show the feature file as the source of tests) or it could be just a invocation form of scenario function like:

scenario('feature/something.feature', auto=True)

While it is generally good practice to order steps Given/When/Then it can sometimes be expedient to check conditions part way through a complex scenario (e.g. a multi-page form.)

I think the ordering checks in feature.py should be relaxed.

Cucumber doesn't distinguish between step types internally and does not enforce ordering. With behave one can register for all keywords using the step() decorator and ordering is not enforced.

The documentation on https://github.com/pytest-dev/pytest-bdd#multiline-steps (https://github.com/pytest-dev/pytest-bdd/blame/master/README.rst#L317-L321) seems to be wrong:

@then('the text should be parsed with correct indentation')
def eat_cucumbers(i_have_text, text):
    assert i_have_text == text == """Some
Extra
Lines"""

I cannot see where the "Some\nExtra\nLines" is coming from. Probably the
assertion needs to be fixed / enhanced to handle / compare the indentation.

Also, while at it, the function should be renamed. It's not about cucumbers
in this section.. :)

This causes the entire scenario to be replayed rather than the parameterized steps.

Would it be possible to provide a hint to use parameterize?

Repeating full scenario phrase in feature file and then in @scenario decorator is a little cumbersome. As @bubenkoff said, that approach has few values over the implicit scenario detection.
Some of them:

• assertion in scenario function (executed after scenario is run)
• scenarios can be parametrized and use fixtures
  So removing @Scenario decorator won't work as we want to have those features.
  But maybe for simple feature files without a lot of requirements we can have a @scenarios('<feature_file>)` decorator which can generate all scenarios in specific feature file?

Then it could look like:

@scenarios("my.feature")
def _(fixture_for_all_scenarios_in_that_feature):
    pass

What do you think?

I had the same setup working in an existing project, but in a new one I'm working on from scrtach BDD style, I get this error message,

==================================== ERRORS ====================================
______________ ERROR collecting tests/functional/test_landing.py _______________
tests/functional/test_landing.py:5: in <module>
    @scenario('landing.feature', 'Landing')
../../.virtualenvs/flask-todomvc/lib/python2.7/site-packages/pytest_bdd/scenario.py:266: in scenario
    'Scenario "{0}" in feature "{1}" is not found.'.format(scenario_name, feature_name)
E   ScenarioNotFound: Scenario "Landing" in feature "landing.feature" is not found.

You can see that the landing.feature file was found, though not the particular scenario, which is written like this (in features/landing.feature):

Scenario: Landing
  When I go to the home page
  Then I should see "Welcome to Home"

and in test_landing.py, I have:

# -*- coding: utf-8 -*-
from pytest_bdd import scenario


@scenario('landing.feature', 'Landing')
def test_landing_page():
    pass