Preconditions

1. Magento 2.3.0
2. MFTF 2.3.13

Steps to reproduce

1. Include parameterized="false" in a page declaration where there is no parameter in URL.
   

Expected result

1. Test will be generated without issue.

Actual result

1. Parameter resolution fails, because Parser thinks URL ist parameterized, when there is no parameter in URL.

It does work with a fresh magento dev version. It seems like it can't find the webshop url and perform an assertion.

Preconditions

1. Ubuntu 7.10
2. Selenium 3.9
3. Magento 2.2-master. Installed the testing framework manually.

Steps to reproduce

1. Local Magento installation with nginx

Expected result

1. The default functional tests should run

Actual result

In IsEqual.php line 140:

Expected HTTP Status Code: 200 (OK). Actual Status Code: 404 (Not Found)
Failed asserting that 404 matches expected 200.

run [-o|--override OVERRIDE] [-e|--ext EXT] [--report] [--html [HTML]] [--xml [XML]] [--tap [TAP]] [--json [JSON]] [--colors] [--no-colors] [--silent] [--steps] [-d|--debug] [--coverage [COVERAGE]] [--coverage-html [COVERAGE-HTML]] [--coverage-xml [COVERAGE-XML]] [--coverage-text [COVERAGE-TEXT]] [--coverage-crap4j [COVERAGE-CRAP4J]] [--coverage-phpunit [COVERAGE-PHPUNIT]] [--no-exit] [-g|--group GROUP] [-s|--skip SKIP] [-x|--skip-group SKIP-GROUP] [--env ENV] [-f|--fail-fast] [--no-rebuild] [--] [] []

PHP Fatal error: Uncaught RuntimeException: Command Did Not Finish Properly in /home/jeroen/public_html/magento2/dev/tests/acceptance/vendor/codeception/codeception/src/Codeception/Subscriber/ErrorHandler.php:101
Stack trace:
#0 [internal function]: Codeception\Subscriber\ErrorHandler->shutdownHandler()
#1 {main}
thrown in /home/jeroen/public_html/magento2/dev/tests/acceptance/vendor/codeception/codeception/src/Codeception/Subscriber/ErrorHandler.php on line 101

Preconditions

1. Any supported Magento version
2. Any MFTF version

Steps to reproduce

Run the vendor/bin/mftf run:test, vendor/bin/mftf run:group or vendor/bin/mftf run:failed command with failing tests

Expected result

The command should return a non-zero exit code as errors occured

Actual result

The command returns an exit code of zero

This is especially relevant in a CI environment where many job setups depend on the exit code of the run commands. In this case, we had a job showing as green even though the MFTF tests in the job failed.
This also blocks command chaining in the CLI, e.g. vendor/bin/mftf run:test FailingTest >> test.log && echo "Test success" will always print out "Test success".

command.php assumes that if PHP_BINARY is set that it is always the PHP cli bin. When FPM is used, the bin will be php-fpm which isn't compatible with the PHP cli.

Preconditions

1. Environment must be using PHP-FPM for PHP

Steps to reproduce

1. Run any test that uses <magentoCli/>

Expected result

1. The test executes the command correctly and proceeds.

Actual result

1. The test will fail with a 403 error upon execution of the <magentoCli/> command.

I'm just going through this documentation but I can't get it to work.

I think I got everything set-up right, as for example vendor/bin/robo example works fine. However, when I run vendor/bin/codecept run the first tests works but then I just get errors. Running it with -f gives me the following additional information for AdminCreateCmsPageTestCest

1) AdminCreateCmsPageTestCest: Admin create cms page test
 Test  tests/functional/Magento/FunctionalTest/_generated/AdminCreateCmsPageTestCest.php:AdminCreateCmsPageTest
 Step  Fill field "#username","admin"
 Fail  Field by name, label, CSS or XPath element with '#username' was not found.

Scenario Steps:

 4. $I->amOnPage("admin/admin/auth/logout/") at tests/functional/Magento/FunctionalTest/_generated/AdminCreateCmsPageTestCest.php:33
 3. $I->saveScreenshot() at tests/functional/Magento/FunctionalTest/_generated/AdminCreateCmsPageTestCest.php:42
 2. $I->fillField("#username","admin") at tests/functional/Magento/FunctionalTest/_generated/AdminCreateCmsPageTestCest.php:58
 1. $I->amOnPage("/backend/admin") at tests/functional/Magento/FunctionalTest/_generated/AdminCreateCmsPageTestCest.php:57

I have taken a look at the screenshot, and to my surprise what it showed is the standard dashboard for the admin, the page you get to after logging in. Seemingly it tried to log in but already was logged in? Perhaps it's on the same session as the previous test?

What have I done wrong?

General summary

We noticed on our team after 2.1.0 release that amOnPage action previously assumed the .url attribute from the page, but then after needed to be explicitly included (all our tests failed).

Steps to reproduce

1. Create test with amOnPage action:
   <amOnPage url="{{AdminLoginPage}}" stepKey="goToAdminLoginPage"/>
2. Generate tests by running: vendor/bin/robo generate:tests

Expected result

1. Warning message appears: Can not resolve argument on url attribute for amOnPage action. Step Key: "goToAdminLoginPage".

Actual result

1. Tests successfully generated and action amOnPage looks like this:
   $I->amOnPage("{{AdminLoginPage}}");

I used Developer experience issue template from magento/magento2 repository

Summary (*)

MFTF tests are trying to be similar to BDD-way of writing tests, known from younger e-commerce projects like https://github.com/Sylius/Sylius

https://github.com/Sylius/Sylius/blob/48729b86334ba78d8f2011b9743389024c334d56/features/checkout/paying_for_order/paying_with_paypal_during_checkout.feature#L1-L13

The real value of such tests is possibility to confront the Business Requirements with test results without any extra interpretation. The content is completely obvious:

    @ui
    Scenario: Successfully authorize payment
        Given I added product "PHP T-Shirt" to the cart
        And I have proceeded selecting "PayPal" payment method
        And the payment method "PayPal" requires authorization before capturing
        When I confirm my order with paypal payment
        And I sign in to PayPal and authorize successfully
        Then I should be notified that my payment has been completed
        And I should see the thank you page
        And the latest order should have a payment with state "completed"

Let's take a look at the easiest test in Magento Functional Tesitng Framework - AdminLoginTest:

--------------------------------------------------------------------------------
AdminLoginTestCest: Admin login test
Signature: Magento\AcceptanceTest\_default\Backend\AdminLoginTestCest:AdminLoginTest
Test: tests/functional/Magento/FunctionalTest/_generated/default/AdminLoginTestCest.php:AdminLoginTest
Scenario --
 I am on page "/strix/admin"
 I fill field "#username","strix"
 I fill field "#login","79XedTOBjPiHFtLt"
 I click ".actions .action-primary"
 I wait for page load 30
 I close admin notification 
 I see in current url "/strix/admin"
 PASSED 

Even there we can see too much "magic":

• What are the #username and #login fields? What is the difference between?
• Do we want to put our secrets in test results?
• What the heck is .actions .action-primary?!
• What is the literal of 30 in I wait for page load?
• Does it really matter if we close Admin Notification or not?
• What does it mean that I see /strix/admin in URL?

Proposed solution

In my humble opinion, that would be much more readable if we use friendly names for DOM elements. Example (parenthesis are grayed out, not visible when not in -vv mode):

 I am on Admin URL (Admin URL: "/strix/admin")
 I fill username field with "strix" (xpath: "#username")
 I fill password field with "79XedTOBjPiHFtLt" (xpath: "#login")
 I click Login button (xpath: ".actions .action-primary")
 I wait for page load (timeout: 30)
 I see Admin URL in current url (Admin URL: "/strix/admin")
 PASSED 

Such Friendly Label would be defined inside Section:

<sections xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:noNamespaceSchemaLocation="urn:magento:mftf:Page/etc/SectionObject.xsd">
    <section name="AdminLoginFormSection">
        <element name="username" type="input" selector="#username" label="Username Input"/>
        <element name="password" type="input" selector="#login" label="Password Input"/>
        <element name="signIn" type="button" selector=".actions .action-primary" timeout="30" label="[Sign In] button"/>
    </section>
</sections>

By design - Magento Functional Testing Framework should use as much Action Groups as it's possible - each time when we need to log in as Administrator:

<amOnPage url="{{_ENV.MAGENTO_BACKEND_NAME}}" stepKey="navigateToAdmin"/>
        <fillField userInput="{{_ENV.MAGENTO_ADMIN_USERNAME}}" selector="{{AdminLoginFormSection.username}}" stepKey="fillUsername"/>
        <fillField userInput="{{_ENV.MAGENTO_ADMIN_PASSWORD}}" selector="{{AdminLoginFormSection.password}}" stepKey="fillPassword"/>
        <click selector="{{AdminLoginFormSection.signIn}}" stepKey="clickLogin"/>

should be replaced with:

<actionGroup ref="LoginActionGroup" stepKey="login"/>

Amazing! 🎆

However ⚠️

What if we have more complicated action groups like... creating products:

<actionGroup ref="CreateCustomVariableActionGroup" stepKey="createCustomVariable" />

Sounds like... no business value for review.
What if instead of current implementation

<actionGroup name="CreateCustomVariableActionGroup">
    <amOnPage url="admin/admin/system_variable/new/" stepKey="goToNewCustomVarialePage" />
    <waitForPageLoad stepKey="waitForPageLoad" />
    <fillField selector="{{CustomVariableSection.variableCode}}" userInput="{{customVariable.code}}" stepKey="fillVariableCode" />
    <fillField selector="{{CustomVariableSection.variableName}}" userInput="{{customVariable.name}}" stepKey="fillVariableName" />
    <fillField selector="{{CustomVariableSection.variableHTML}}" userInput="{{customVariable.html}}" stepKey="fillVariableHtml" />
    <fillField selector="{{CustomVariableSection.variablePlain}}" userInput="{{customVariable.plain}}" stepKey="fillVariablePlain" />
    <click selector="{{CustomVariableSection.saveCustomVariable}}" stepKey="clickSaveVariable"/>
</actionGroup>

We use friendly label:

<actionGroup name="CreateCustomVariableActionGroup" label="Create Custom Variable with {{CustomVariableSection.variableCode}} code and {{CustomVariableSection.variableName}} as a name">

Everytime when you include Action Group - the steps included wouldn't be displayed directly, but we will show the Action Group label to user. Instead of having extra 5 steps each time when we log in as Administrator:

AdminCreateProductDuplicateUrlkeyTestCest: Admin create product duplicate urlkey test
Signature: Magento\AcceptanceTest\_default\Backend\AdminCreateProductDuplicateUrlkeyTestCest:AdminCreateProductDuplicateUrlkeyTest
Test: tests/functional/Magento/FunctionalTest/_generated/default/AdminCreateProductDuplicateUrlkeyTestCest.php:AdminCreateProductDuplicateUrlkeyTest
Scenario --
 I am going to create entity that has the stepKey: simpleProduct
 I am on page "/strix/admin"
 I wait for page load 
 I fill field "#username","strix"
 I fill field "#login","79XedTOBjPiHFtLt"
 I click ".actions .action-primary"
 I wait for page load 30
 I close admin notification 
 I am on page "/strix/catalog/product/index"
 I click ".action-toggle.primary.add"
 I save screenshot 
 FAIL 

 I am on page "admin/admin/auth/logout/"
 I am going to delete entity that has the createDataKey: simpleProduct

we would have only one:

 I am going to create entity that has the stepKey: simpleProduct
 I am logged in as Administrator (login: "strix", password: "xyz123")
 I am on page "/strix/catalog/product/index"
 I click ".action-toggle.primary.add"
 I save screenshot 

Isn't that better?

Preconditions

1. Any compatible Magento version
2. Any compatible MFTF version
3. Have your web server (Apache or Nginx) point to the <MAGE_ROOT_DIR>/pub as recommended in the DevDocs

Steps to reproduce

Run any test or suite with a magentoCLI step in test <before> or <after>

Expected result

CLI command is executed properly

Actual result

The command is not executed via shell exec but via POST call to the Magento instance

The problem is that the file to be executed, /dev/tests/acceptance/utils/command.php, is outside the /pub directory and as such usually not accessible.

To overcome this, there is additional configuration necessary on web server level (example ngnix configuration):

location /dev/tests/acceptance/ {
    root $MAGE_ROOT;

    location ~ ^/dev/tests/acceptance/utils/[^.]+\.php {
        fastcgi_pass   $FASTCGI_BACKEND;
        fastcgi_index  index.php;
        fastcgi_param  SCRIPT_FILENAME  $document_root$fastcgi_script_name;
        include        fastcgi_params;

        allow <local_ip>;
        deny all;
    }
}

This should probably either be documented in the setup guides for MFTF or rectified to use the shell command instead.

Currently you can only reference credentials (secret data) in fill field actions. Instead, it should be possible to access this data from magentoCLI calls as well

Preconditions

1. Create a credentials file and add values to it

Steps to reproduce

1. In a block (either in a test or in a suite) add <magentoCLI command="config:set some/secret/password {{_CREDS.mySecretPassword}}" stepKey="SetSecretPassword"/>
2. Attempt to generate the suite/test bin/mftf generate:suite MySuite

Expected result

1. Tests should be generated

Actual result

1. Error message:

In ActionMergeUtil.php line 112:

  You cannot reference secret data outside of fill field actions


generate:suite [-r|--remove] [--] <suites> (<suites>)...

Entity Data Object after http request might have array as the result property.
my case:

(
    [store_id] => 0
    [parent] => 2
    [name] => Category 1
    [url_key] => simple_category1
    [is_active] => 1
    [include_in_menu] => 1
    [is_anchor] => 1
    [description] => Lorem Ipsum Unity sit amet 1
    [display_mode] => PRODUCTS
    [default_sort_by] => position
    [available_sort_by] => position
    [use_config[available_sort_by]] => true
    [use_config[default_sort_by]] => true
    [use_config[filter_price_range]] => true
    [custom_use_parent_settings] => 0
    [custom_apply_to_products] => 0
    [return] => Array
        (
            [0] => /catalog/category/delete/id/17/
            [val] => 17
            [1] => 17
        )

)

This object is the result of method
setCreatedObject in
Magento/FunctionalTestingFramework/DataGenerator/Persist/DataPersistenceHandler.php:202

Preconditions

1. Write a regex with one or a few capture groups and try to get access to values

Steps to reproduce

1. Write a test with returnRegex e.g. ~\/catalog\/category\/delete\/id\/(?'val'\d*?)\/~
2. Try to get access to data stored in val capture group in other step key
3. e.g. $$stepKeyName.return[val]$$ or $$stepKeyName.return.val$$
4. see wrong result or array to string error

Expected result

1. one can get access to any embedded property of result

Actual result

1. key is casted as a string and the requested property is not found since in \Magento\FunctionalTestingFramework\DataGenerator\Objects\EntityDataObject::getDataByName
   it just lowercase it

Preconditions

1. Magento 2.3.0 dev
2. Manually install MFTF into dev folder (using magento2 2.2-develop latest)
3. Virtual Host setup with https and a self signed certificate (in my case it's https://magento22mftfcombine.test/)

Steps to reproduce

1. Run a test that uses the Magento API. For example, the checkout test uses the API to create a category and a product before it starts interacting with the UI.

vendor/bin/robo group checkout

Expected result

1. Both "checkout" test should run without any errors.

Actual result

I get this result in the cli:

`[Exec] Running ./vendor/bin/codecept run functional --verbose --steps --skip-group skip --group checkout
Codeception PHP Testing Framework v2.3.8
Powered by PHPUnit 6.5.6 by Sebastian Bergmann and contributors.
[Groups] checkout

Magento\FunctionalTestingFramework.functional Tests (2) ------------------
Modules: \Magento\FunctionalTestingFramework\Module\MagentoWebDriver, \Magento\FunctionalTestingFramework\Helper\Acceptance, \Magento\FunctionalTestingFramework\Helper\MagentoFakerData, \Magento\FunctionalTestingFramework\Module\MagentoRestDriver, PhpBrowser, \Magento\FunctionalTestingFramework\Module\MagentoSequence, Asserts

StorefrontCustomerCheckoutTestCest: Storefront customer checkout test
Signature: Magento\AcceptanceTest_generated\Backend\StorefrontCustomerCheckoutTestCest:StorefrontCustomerCheckoutTest
Test: tests/functional/Magento/FunctionalTest/_generated/StorefrontCustomerCheckoutTestCest.php:StorefrontCustomerCheckoutTest
Scenario --
I am going to create entity that has the stepKey: simplecategory
ERROR

StorefrontGuestCheckoutTestCest: Storefront guest checkout test
Signature: Magento\AcceptanceTest_generated\Backend\StorefrontGuestCheckoutTestCest:StorefrontGuestCheckoutTest
Test: tests/functional/Magento/FunctionalTest/_generated/StorefrontGuestCheckoutTestCest.php:StorefrontGuestCheckoutTest
Scenario --
I am going to create entity that has the stepKey: createCategory
ERROR

Time: 3.14 seconds, Memory: 18.00MB

There were 2 errors:

1. StorefrontCustomerCheckoutTestCest: Storefront customer checkout test
   Test tests/functional/Magento/FunctionalTest/_generated/StorefrontCustomerCheckoutTestCest.php:StorefrontCustomerCheckoutTest

[Magento\FunctionalTestingFramework\Exceptions\TestFrameworkException] SSL: certificate subject name 'localhost' does not match target host name 'magento22mftfcombine.test'

Scenario Steps:

1. // I am going to create entity that has the stepKey: simplecategory

#1 /var/www/html/magento22mftfcombine/dev/tests/acceptance/vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/Util/Protocol/CurlTransport.php:173
#2 /var/www/html/magento22mftfcombine/dev/tests/acceptance/vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/DataGenerator/Persist/Curl/WebapiExecutor.php:116
#3 /var/www/html/magento22mftfcombine/dev/tests/acceptance/vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/DataGenerator/Persist/Curl/WebapiExecutor.php:83
#4 /var/www/html/magento22mftfcombine/dev/tests/acceptance/vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/DataGenerator/Persist/Curl/WebapiExecutor.php:65
#5 /var/www/html/magento22mftfcombine/dev/tests/acceptance/vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/DataGenerator/Persist/CurlHandler.php:132
#6 /var/www/html/magento22mftfcombine/dev/tests/acceptance/vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/DataGenerator/Persist/DataPersistenceHandler.php:72
#7 /var/www/html/magento22mftfcombine/dev/tests/acceptance/tests/functional/Magento/FunctionalTest/_generated/StorefrontCustomerCheckoutTestCest.php:48
#8 Magento\AcceptanceTest_generated\Backend\StorefrontCustomerCheckoutTestCest->_before

2. StorefrontGuestCheckoutTestCest: Storefront guest checkout test
   Test tests/functional/Magento/FunctionalTest/_generated/StorefrontGuestCheckoutTestCest.php:StorefrontGuestCheckoutTest

[Magento\FunctionalTestingFramework\Exceptions\TestFrameworkException] SSL: certificate subject name 'localhost' does not match target host name 'magento22mftfcombine.test'

Scenario Steps:

1. // I am going to create entity that has the stepKey: createCategory

#1 /var/www/html/magento22mftfcombine/dev/tests/acceptance/vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/Util/Protocol/CurlTransport.php:173
#2 /var/www/html/magento22mftfcombine/dev/tests/acceptance/vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/DataGenerator/Persist/Curl/WebapiExecutor.php:116
#3 /var/www/html/magento22mftfcombine/dev/tests/acceptance/vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/DataGenerator/Persist/Curl/WebapiExecutor.php:83
#4 /var/www/html/magento22mftfcombine/dev/tests/acceptance/vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/DataGenerator/Persist/Curl/WebapiExecutor.php:65
#5 /var/www/html/magento22mftfcombine/dev/tests/acceptance/vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/DataGenerator/Persist/CurlHandler.php:132
#6 /var/www/html/magento22mftfcombine/dev/tests/acceptance/vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/DataGenerator/Persist/DataPersistenceHandler.php:72
#7 /var/www/html/magento22mftfcombine/dev/tests/acceptance/tests/functional/Magento/FunctionalTest/_generated/StorefrontGuestCheckoutTestCest.php:43
#8 Magento\AcceptanceTest_generated\Backend\StorefrontGuestCheckoutTestCest->_before

ERRORS!
Tests: 2, Assertions: 0, Errors: 2.
[Exec] Exit code 1 Time 3.293s
`
################################################################

I also get the same error in Magento 2.3

It might just be that I don't know how to properly configure MFTF, however I looked into the source code and there doesn't seem to be any way to give curl and parameter to make it ignore insercure certificates.

I found a way to make it work by going inside the core of MFTF, more specifically to the Magento\FunctionalTestingFramework\Util\Protocol\CurlTransport class and inside the write method adding CURLOPT_SSL_VERIFYHOST => false to the initial $options array. Of courrse this is not a good solution since I shouldn't be editing the core files, but in case this is a real issue, would this be a proper solution? Or am I completely off the mark here?

Preconditions

1. Magento 2.2.2
2. Manually install MFTF into dev folder (using magento2 2.2-develop latest)

Steps to reproduce

1. Setup Magento store(s) without having one with a store code "default"
2. Run tests as follows

$ vendor/bin/robo example -v

Expected result

1. POST request issued to /rest/V1/integration/admin/token

Actual result

1. POST request issued to /rest/default/V1/integration/admin/token, causes 404 error

172.18.0.6 - - [31/Jan/2018:05:55:37 +0000] "POST /rest/default/V1/integration/admin/token HTTP/1.1" 404 1883 "-" "Symfony BrowserKit"

Would be good to either define a different store code to use for the REST API requests, or just to omit it and let the Magento WebAPI's request processor handle the logic (it chooses the default store's code when omitted).

Currently you're required to run build:project than manually configure the .env file through another editor. This could be wrapped in an interactive robo command: http://robo.li/getting-started/#io

User flow:

vendor/bin/robo setup:env
Please specify your base URL: http://magento.test
Please specify your admin path [admin]:
Please specify your admin username: admin
Please specify your admin password: password123

Validation on the inputs on these steps would also help detect issues with the setup. The system could also do a quick validation upon the configuration's input to verify if the environment exists and is working correctly.

For Magento contribution we are expected to follow several code style rules defined by PHP Code Sniffer ruleset: https://github.com/magento/magento2/blob/c934ccc3bfa1a79b78052c61a5478eaf77de7029/dev/tests/static/testsuite/Magento/Test/Php/_files/phpmd/ruleset.xml

For Magento Functional Testing Framework the ruleset (https://github.com/magento/magento2-functional-testing-framework/blob/95aa636421933fe23338acb98089408ce7637a63/dev/tests/static/Magento/ruleset.xml) for code is different, inconsistent to the Magento one.

Expected result

1. Ruleset should be consistent across all Magento projects.

The most irritating rule for MFTF is...

Hello
I need help with <executeJS> action.
Is it possible to get value from executed JS via <executeJS> action and pass it in different action?

I need something like example attached in documentation $myVar = $I->executeJS('return $("#myField").val()');
I know about <grabTextFrom> action, but before grabbing value I need to process it.

Preconditions

1. MFTF 2.1.2
2. PHP 7.0.28

Steps to reproduce

1. Create a Page file/entity for the backend that references a variable in its URL, e.g. <page name="AdminProductEditPage" url="catalog/product/edit/id/{{id}}/" area="admin" module="Magento_Catalog" parameterized="true">
2. Inside a test make a reference to that page and pass it an entity that was created during the test. For example have this:

<createData entity="_defaultCategory" stepKey="createCategory"/>
<createData entity="_defaultProduct" stepKey="createSimpleStandaloneProduct">
       <requiredEntity createDataKey="createCategory"/>
</createData>

Then have this:

<amOnPage url="{{AdminProductEditPage.url($$createSimpleStandaloneProduct.id$$)}}" stepKey="goToProductEditPage1"/>

3. Generate the test that contains those steps.

Expected result

1. The step that uses the amOnPage action should look like this:

$I->amOnPage("/" . getenv("MAGENTO_BACKEND_NAME") . "/catalog/product/edit/id/" . $this->createSimpleStandaloneProduct->getCreatedDataByName('id') . "/");

Actual result

1. The step actually looks like this:

$I->amOnPage("/" . getenv("MAGENTO_BACKEND_NAME") . "/catalog/product/edit/id/$$createSimpleStandaloneProduct.id$$/");

This only happens on admin pages. Will open a pull request to fix this.

Preconditions

1. Magento 2.3.0

Steps to reproduce

1. Create a custom MFTF Test and use the action group ProductSetAdvancedPricing with argument price = "Fixed"
2. Run the test

Expected result

1. This should normally fill the advanced price form with a fixed discount amount.

Actual result

1. The test fails and does not find the selector [name='product[tier_price][0][percentage_value]']

In magento/module-catalog/Test/Mftf/ActionGroup/AdminProductActionGroup.xml line 205 the amount is inserted into the field with the selector {{AdminProductFormAdvancedPricingSection.productTierPricePercentageValuePriceInput('0')}}
But this field only exists, when the price argument is set to "Discount". As soon as this is "Fixed", the field is called product[tier_price][0][price].
So either the action group should be renamed to "ProductSetAdvancedPricingDiscountAmount" and the price removed or there should be a condition check which decides which field to fill.

As a test developer
I want a debug flag which I can provide during single test generation which prints each stepKey, action name and originating file
so that I can clearly see how MFTF is rendering resulting tests

AC:

1. Debug flag exists printing steps for a single test in order they would appear generated with information out to console:
   - stepKey name
   - action name
   - originating file

There is a need to have tooling that can output what data is defined and available to a test.
Currently, to avoid duplication of already defined data, a test writer must manually check what data files have been committed.
It would be useful to run a command that would return a list of data entity names so I can quickly see if the data I need already exists.
Additionally, as the available data increases, it would be useful for this tool to discriminate on data type.
It is likely that this tooling would be a new RoboFile command.

Expected result

1. A new command is added to robo (robo data:list --type )
2. Type is optional (if not provided, default to all data types returned)
3. User can specify the data type defined in xml that they want to see
4. When ran, a list is returned to stdout
5. The list contains only the name of the entity
6. The list is sorted alphabetically
7. This command is documented

#distributed-CD

Running command vendor/bin/mftf generate:tests gets error.

Preconditions

Magento version 2.3.0-beta32
Mftf version 2.3.11

Steps to reproduce

Run `vendor/bin/mftf generate:tests`

Expected result

The tests should been generated.

Actual result

Currently due an error in a third party extension (dotmailer/dotmailer-magento-extension#306) the command fails to generate the tests. This shouldn't happen imho.

If I manually disable the exception thrown in vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/Test/Util/ActionMergeUtil.php (Line 278) the command runs fine and all the other tests are generated.

Imho the command shouldn't fail if any test fails to be generated. I should instead print a warning or log the error and proceed with the next test.

Running command vendor/bin/mftf generate:tests gets error.

Preconditions

1. Magento version 2.3.0-beta23
2. Mftf version 2.3.8

Steps to reproduce

1. Run vendor/bin/mftf generate:tests

Expected result

1. Tests should been generated

Actual result

1. Error in console with message: `In Dom.php line 274:

More than one node matching the query: /tests/test[@name='StorefrontConfigurableProductBasicInfoTest']/before/actionGroup[@stepKey='createProduct']/argument`

Allow running tests for modules installed in Magento instance via vendor directory.

Preconditions

1. Install Magento 2.2.x or later
2. Install custom module using composer (so module is installed under MAGENTO_BP/vendor
3. Custom module contains tests under Test/Mftf
4. Mftf is installed and configured per DevDocs instructions

Steps to reproduce

1. cd MAGENTO_BP/dev/tests/acceptance
2. vendor/bin/robo generate:tests

Expected result

1. Tests from custom module are generated under MAGENTO_BP/dev/tests/acceptance/tests/functional/Magento/FunctionalTest/_generated/default

Actual result

1. Custom module is completely ignored as MAGENTO_BP/vendor is not parsed as a location for tests. Only the following paths are parsed:
   a. MAGENTO_BP/app/code/
   b. MAGENTO_BP/dev/tests/acceptance/vendor
   c. MAGENTO_BP/dev/tests/acceptance/tests/functional/Magento/FunctionalTest

Magento ships with core bundled extensions. These extensions also ship with their own MFTF tests. Conceivably a lot of them will require test credentials to run when interacting with outside services (vertex, klarna come to mind but I am sure most have an outside service component).

It is currently not clear how this should be handled.

• Are there common credentials that everyone can use?
• Is every one required to sign up on their own just so that the full test suite can pass?
• Something else?

Alternatively shipping tests which can't be run or fail is also not an ideal road to travel on.

When i tried to generate the tests using vendor/bin/mftf generate:tests, MFTF is generating only one test,that is "testName":"AdminResetCustomerPasswordTest.

Showing
Could not find ActionGroup by ref "AdminUpdateCustomerGroupByEmailActionGroup" in Element "before"
/var/www/html/magentoone/vendor/magento/module-inventory-admin-ui/Test/Mftf/Test/AdminOrderCreatedWithConfigurableProductWithSwatchAttrib
uteViaTheAdminTest.xml

Preconditions

PHP 7.2.15
selenium-server-standalone-3.141.59 and chrome driver
Browser is chrome Version 72.0.3626.119
Magento 2.3
Magento Functional Testing Framework version 2.3

Steps to reproduce

1. vendor/bin/mftf generate:tests

Expected result

1. Generate Tests Command Run

Actual result

1. DEPRECATION: Test AdminResetCustomerPasswordTest is missing required annotations.{"testName":"AdminResetCustomerPasswordTest","missingAnnotations":"title"}

In TestGenerator.php line 229:

Could not find ActionGroup by ref "AdminUpdateCustomerGroupByEmailActionGroup" in Element "before"
/var/www/html/magentoone/vendor/magento/module-inventory-admin-ui/Test/Mftf/Test/AdminOrderCreatedWithConfigurableProductWithSwatchAttrib
uteViaTheAdminTest.xml

UnselectOption parameterArray is not working on generation step

I have installed vanilla magento2.6 version. Open terminal, cd to <magento_root>/dev/tests/acceptance/ , run composer install and vendor/bin/robo build:project, changed settings in .env file to point to local magento2 installation. Then on running vendor/bin/robo generate:tests, I am getting error: Fetching version information from http://127.0.0.1/careshop2/magento_version
PHP Notice: Undefined index: SampleData in C:\xampp2\htdocs\careshop2\dev\tests\acceptance\vendor\magento\magento2-functional-testing-framework\src\Magento\FunctionalTestingFramework\Util\ModuleResolver.php on line 309

Notice: Undefined index: SampleData in C:\xampp2\htdocs\careshop2\dev\tests\acceptance\vendor\magento\magento2-functional-testing-framework\src\Magento\FunctionalTestingFramework\Util\ModuleResolver.php on line 309
[error] Could not resolve entity reference {{StorefrontCustomerOrderSection.isMyOrdersSection}} in Test "StorefrontRedirectToOrderHistory" in Test "StorefrontRedirectToOrderHistory"
ERROR: Undefined index: SampleData
in C:\xampp2\htdocs\careshop2\dev\tests\acceptance\vendor\magento\magento2-functional-testing-framework\src\Magento\FunctionalTestingFramework\Util\ModuleResolver.php:309

Preconditions

1. magento version 2.2.6
2. magento2-functional-testing-framework version 2.2.0

Steps to reproduce

1. Install magento 2.2.6
2. run terminal and cd to dev/tests/acceptance, Run composer install , command run completed successfully.
3. Run vendor/bin/robo build:project, command run completed successfully. Edit .env settings to point to magento2 with correct url, admin name, adminusername, admin password, Save file
4. Run selenium server on another terminal with chrome driver and selenium server jar both kept at same location. Output selenium server up and running on port 4444
5. Try to run vender/bin/robo generate:tests , it gives undefined index sampleData error

Expected result

1. Test should be generated successfully

Actual result

1. Got below error:
   Fetching version information from http://127.0.0.1/careshop2/magento_version
   PHP Notice: Undefined index: SampleData in C:\xampp2\htdocs\careshop2\dev\tests\acceptance\vendor\magento\magento2-functional-testing-framework\src\Magento\FunctionalTestingFramework\Util\ModuleResolver.php on line 309

Notice: Undefined index: SampleData in C:\xampp2\htdocs\careshop2\dev\tests\acceptance\vendor\magento\magento2-functional-testing-framework\src\Magento\FunctionalTestingFramework\Util\ModuleResolver.php on line 309
[error] Could not resolve entity reference {{StorefrontCustomerOrderSection.isMyOrdersSection}} in Test "StorefrontRedirectToOrderHistory" in Test "StorefrontRedirectToOrderHistory"
ERROR: Undefined index: SampleData
in C:\xampp2\htdocs\careshop2\dev\tests\acceptance\vendor\magento\magento2-functional-testing-framework\src\Magento\FunctionalTestingFramework\Util\ModuleResolver.php:309

This error is platform independent as I have tried in another system with ubuntu.
Also I tried not following step4 (from above steps I mentioned) and then to run generate:tests but still got same error.

I am using StorefrontPersistedCustomerLoginTest.xml as a template and trying the custom test StorefrontLoginRequiredFieldTest.xml by just sending empty string for email and password field and validating the error messages. When I try running the test, I am getting below error.

In TestObjectHandler.php line 81:

Test StorefrontLoginRequiredFieldTest not defined in xml.

I have attached the steps with screen shots to reproduce the issue. I want use the tests under various modules from {magento_root}/vendor/magento folder and use mftf command. Please guide me the correct way of setting .env file and create custom test.

Preconditions

1. Selenium - 3.141, Chrome version is 73x and chromedriver is ChromeDriver 73.0.3683.86
2. Magento 2.3.1 and MFTF 2.3.14
   Steps to reproduce issue.docx

Steps to reproduce

1. In the test (StorefrontPersistedCustomerLoginTest), passing empty string for email and password and validating the error message. The new test name is StorefrontLoginRequiredFieldTest.xml .
   I am creating the custom test cases under the folder /var/www/html//var/www/html/magtiretest/app/code/ISNEZTire/login-module/Test/Mftf
2. Created StorefrontCustomerSignInFormSection.xml in Section folder
3. Ran new test using the Command - vendor/bin/mftf run:test StorefrontLoginRequiredFieldTest --remove

Expected result

1. Test should pass

Actual result

1. [Logs, Errors]
   In TestObjectHandler.php line 81:

Test StorefrontLoginRequiredFieldTest not defined in xml.

We should allow user to configure browser type in .env file.
Currently it's locked down to chrome in dev/tests/functional/MFTF.suite.yml.

Preconditions

1. MFTF 2.1.2
2. PHP 7.1.15

Steps to reproduce

1. Use waitForElementChange action in a test <waitForElementChange selector="#selector" function="function(\WebDriverElement $el) {return $el->isDisplayed();}" stepKey="waitForElementChange"/>.
2. Generate tests vendor/bin/robo generate:tests.
3. Check generated Cest file.

Expected result

1. waitForElementChange method is generated without double quotes around closure function $I->waitForElementChange("#selector", function(\WebDriverElement $el) {return $el->isDisplayed();});.

Actual result

1. 1. waitForElementChange method is generated with double quotes around closure function $I->waitForElementChange("#selector", "function(\WebDriverElement $el) {return $el->isDisplayed();}");.

ActionGroup parameters should be able to contain a defaultValue that is filled by another argument. An example scenario:

<actionGroup name="CheckProductPricingOnStorefrontOrderReview">
    <arguments>
        <argument name="product"/>
        <argument type="string" name="priceInclTax"/>
        <argument type="string" name="priceExclTax"/>
        <argument type="string" name="subtotalExclTax" defaultValue="{{priceExclTax}}"/>
        <argument type="string" name="subtotalInclTax" defaultValue="{{priceInclTax}}"/>
    </arguments>

    <see stepKey="seeProductPriceExclTax" userInput="{{priceExclTax}}" selector="{{StorefrontCustomerOrderViewSection.productPriceExclTaxByName(product.name)}}"/>
    <see stepKey="seeProductPriceInclTax" userInput="{{priceInclTax}}" selector="{{StorefrontCustomerOrderViewSection.productPriceInclTaxByName(product.name)}}"/>

    <see stepKey="seeProductSubtotalExclTax" userInput="{{subtotalExclTax}}" selector="{{StorefrontCustomerOrderViewSection.productSubtotalExclTaxByName(product.name)}}"/>
    <see stepKey="seeProductSubtotalInclTax" userInput="{{subtotalInclTax}}" selector="{{StorefrontCustomerOrderViewSection.productSubtotalInclTaxByName(product.name)}}"/>
</actionGroup>

I am trying to build our own suite to run tests. I am testing using MFTF standalone per the directions here https://devdocs.magento.com/mftf/2.3/getting-started.html#set-up-a-standalone-mftf. When I try to generate a single test or our suite, I get errors from other (unrelated) tests.

See Slack thread for more info: https://magentocommeng.slack.com/archives/C82DDUSNP/p1542748941043300

Preconditions

1. Using standalone MFTF per https://devdocs.magento.com/mftf/2.3/getting-started.html#set-up-a-standalone-mftf
2. Have 2.3.0-beta32 installed
3. Have a custom suite (or at minimum custom test) in a custom module

Steps to reproduce

1. Run bin/mftf generate:suite MySuite or bin/mftf generate:test MySingleTest

Expected result

1. Suite or Test should generate (or throw an error only about specific tests)

Actual result

The following files all fail validation for different reasons. I only captured the actual errors from the first one. I manually deleted the files at each run to try to get the generation of just our suite to run. I also ran the generation with the debug flag (-d) to get this output

1. ee23/vendor/magento/module-multiple-wishlist/Test/Mftf/Section/StorefrontCreateMultipleWishlistSection.xml

In ActionMergeUtil.php line 278:

  ee23/vendor/magento/module-multiple-wishlist/Test/Mftf/Section/StorefrontCreateMultipleWishlistSection.xml

  Invalid Document

  Element 'element', attribute 'type': [facet 'enumeration'] The value 'image' is not an element of the set {'text', 'textarea', 'input', 'button', 'checkbox', 'radio', 'checkboxset', 'radioset', 'date', 'file', 'select', 'multiselect', 'wysiwyg', 'iframe', 'block'}.

  Line: 12

  Element 'element', attribute 'type': 'image' is not a valid value of the atomic type 'uiElementType'.

  Line: 12

  .

  Exception occurred parsing action at StepKey "ApiVersionNA"

generate:suite [-r|--remove] [--] <suites> (<suites>)...

2. ee23/vendor/magento/module-customer-custom-attributes/Test/Mftf/Section/CustomerAccountSection.xml
3. ee23/vendor/magento/module-theme/Test/Mftf/Section/StorefrontFooterSection.xml
4. ee23/vendor/magento/module-configurable-product/Test/Mftf/Test/ConfigurableProductPriceAdditionalStoreViewTest.xml
5. ee23/vendor/magento/module-catalog/Test/Mftf/Test/AdminAddImageToWYSIWYGProductTest.xml

If you attempt to run generate:tests before building the project and setting up the .env file with valid credentials a PHP error is thrown whilst trying to resolve the environment variables.

> /acceptance on MAGETWO-66176-column-configuration ⨯ vendor/bin/robo generate:tests                                                             09:41:33
PHP Notice:  Undefined index: MAGENTO_ADMIN_USERNAME in /dev/tests/acceptance/vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/Util/ModuleResolver.php on line 196

Notice: Undefined index: MAGENTO_ADMIN_USERNAME in /dev/tests/acceptance/vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/Util/ModuleResolver.php on line 196
PHP Notice:  Undefined index: MAGENTO_ADMIN_PASSWORD in /dev/build/bluefoot/dev/tests/acceptance/vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/Util/ModuleResolver.php on line 197

Notice: Undefined index: MAGENTO_ADMIN_PASSWORD in /dev/build/bluefoot/dev/tests/acceptance/vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/Util/ModuleResolver.php on line 197
 [error]  Could not resolve entity reference {{_ENV.MAGENTO_ADMIN_USERNAME}} in Test "LoginAsAdmin" in Cest "AdminLoginCest" 
ERROR: Undefined index: MAGENTO_ADMIN_PASSWORD 
in /dev/tests/acceptance/vendor/magento/magento2-functional-testing-framework/src/Magento/FunctionalTestingFramework/Util/ModuleResolver.php:197

Alternatively validation must present before a command runs which verifies project is ready for test generation and running. In the event of a failure a nice user friendly error can be returned to help prompt the user how to resolve the issue.
"Please run vendor/bin.robo build:project and configure your environment (.env) first."

I have installed Magento v2.2.2 with sample data, and then install "Magento functional testing framework" with respect to the Magento devdocs step, After that run test based on default test cases, I got 404 error in most of admin side test.

Preconditions

1. Browser version chrome 63
2. Selenium v2.44.0,
3. PHP 7.0.25-0ubuntu0.16.04.1
4. Magento 2.2.2 ,MFTF

Steps to reproduce

1. All the magento default test successfully passed by default magento

Expected result

1. I got 404 error in most of admin side test.

Actual result

1. [Logs, Errors]
   

If system window chrome "print" is opened, then we can not to do anything, even close Tab, and do not perform another action (go to the page, click ...)

Preconditions

1. Browser Chrome

Steps to reproduce

1. Create order on StoreFront
2. Click Print Receipt button
3. System window chrome "print" is opened

Expected result

1. <closeTab stepKey="close"/> - should close the tab

Actual result

1. [Logs, Errors]
   <closeTab stepKey="close"/> - does not close the tab

I am trying to group my tests in a suite to share entities created in <before/> but these entities are not being created properly.
Problem only happen if <createData/> has a <requiredEntity/> inside it.

Here is an example: (I tested it with default WYSIWYGDisabledSuite )

<suite name="WYSIWYGDisabledSuite">
        <before>
            <createData entity="SimpleSubCategory" stepKey="createCategory"/>
            <createData entity="SimpleProduct" stepKey="createSimpleProduct">
                <requiredEntity createDataKey="createCategory"/>
            </createData>
        </before>

        <after>
            <deleteData createDataKey="createSimpleProduct" stepKey="deleteProduct"/>
            <deleteData createDataKey="createCategory" stepKey="deleteCategory"/>
        </after>
        <include>
            <group name="WYSIWYGDisabled"/>
        </include>
        <exclude>
            <group name="skip"/>
        </exclude>
    </suite>

It generates following precondition code in Suite file:

       PersistedObjectHandler::getInstance()->createEntity(
                    "createSimpleProduct",
                    "suite",
                    "SimpleProduct"
                );

When I run this suit all of tests tests fail with following error:

INCOMPLETE: Error HTTP response code: 400 Response:{"message":"Error occurred during "custom_attributes" processing. The "category_ids" attribute code doesn't have a value set. Enter the value and try again."}

When I move my <before/> to test files, following precondition in generated for Simple product in test file:

     PersistedObjectHandler::getInstance()->createEntity(
			"createSimpleProduct",
			"hook",
			"SimpleProduct",
			["createCategory"],
			null
		);

and all tests execute successfully.

Preconditions

1. Magento version 2.3.0
2. Ubuntu 18.04

Expected result

1. Test should run successfully.

Actual result

1. All Tests fail

Description

Currently, each wait related action has their own specified default timeout stored on vendor/codeception/codeception/src/Codeception/Module/WebDriver.php. These values are different for each action. It would be better to have 1 global value that all of these actions read from and use as the default. Test writers could overwrite this default in their xml test steps

Acceptance Criteria

1. All wait related actions read their default timeout value from a single, global variable
2. All wait related actions can have the default value overridden by providing timeout attribute on the action level (from a test step or action group step)
3. Global value is set to 10 seconds by default

Currently, webdriver config can be set via $browser.yml file in _ENV directory
OR
in functional.suite.yml

However,
We should have only one canonical config file to set webdriver options.
To support this, we should remove the _ENV directory and any files therein.

Expected result

_ENV directory and files removed.
Webdriver configuration occurs only via functional.suite.yml

Given the following element and test action:

//In AdminLoginFormSection

<element name="paramelement" type="button" selector="{{var1}}" parameterized="true"/>
<click stepKey="1" selector="{{AdminLoginFormSection.paramelement('SomeString.$data.key$.after')}}"/>

The Generator actually outputs

$I->click("SomeString." . $data->getCreatedDataByName('key') . ".after");

We do not want to allow this kind of nested replacement, as this kind of replacement can't be replicated nested xml.data references (and is a door to recursive replacement we do not want to be open).

Expected result

ActionObject detects when a user is attempting to input $data.key$ references inside arguments for parameterized selectors/urls and throws a warning/dies.

Empty data (e.g. empty string) for primitive types are skipped during generation of persist operation data.

Preconditions

1. Magento 2.2.x installed

Steps to reproduce

1. Create store_info_config-meta.xml
2. Create StoreConfigData.xml
3. Run test AdminSetupStoreInformationTest.xml

Expected result

1. Test should pass successfully

Actual result

1. Test failed with error: Failed asserting that a string is empty.

assertEquals outputs actualResult with a prefixed "$" rather than actually parsing the value.

Preconditions

MFTF 2.2

Steps to reproduce

Following the example in the docs

<!-- Grab a value from the page using any grab action -->
<grabTextFrom selector="#elementId" stepKey="stepKeyOfGrab"/>

<!-- Ensure that the value we grabbed matches our expectation -->
<assertEquals message="This is an optional human readable hint that will be shown in the logs if this assert fails." stepKey="assertEquals1">
   <expectedResult type="string">Some String</expectedResult>
   <actualResult type="variable">A long assert string {$stepKeyOfGrab} with an embedded variable reference.</actualResult>
</assertEquals>

Expected result

$stepKeyOfGrab = $I->grabTextFrom("#elementId");
$I->assertEquals("Some String", 'A long assert string ' . $stepKeyOfGrab. ' with an embedded variable reference.', "This is an optional human readable hint that will be shown in the logs if this assert fails.");

Actual result

$stepKeyOfGrab = $I->grabTextFrom("#elementId");
$I->assertEquals("Some String", $A long assert string {$stepKeyOfGrab} with an embedded variable reference., "This is an optional human readable hint that will be shown in the logs if this assert fails.");

After attempt of adding PHPMD's to check our code some classes were refactored to reduce Cyclomatic Complexity, but some had a @supressWarnings tag added to them.

AC:

1. Reduce the Cyclomatic Complexity to <=10 in the following methods:

    - OperationDefinitionObjectHandler::initDataDefinitions()
    - OperationDataArrayResolver::resolveOperationDataArray()
   

Note:
Do not worry about TestGenerator.php, as we have a different issue to completely refactoring its use.

Steps to reproduce

1. Create an element in Section entity which will reflect for example <body /> or any <div /> tag
2. Select element type
   <element name="pageBody" type =" ?????? " selector="body" />

Expected result

1. There is a possibility to select block type

Actual result

1. Only this types exists which are not filling my requirements:

text, textarea, input, button, checkbox, radio, checkboxset, radioset, date, file, select, multiselect, wysiwyg, iframe

Preconditions

Install Magento 2.2.x or later
Mftf is installed and configured per DevDocs instructions (version 2.3.11)

Steps to reproduce

1. Create action group that is using parameterized selector with its argument in that selector. For example:

<actionGroups xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:mftf:Test/etc/actionGroupSchema.xsd"> <actionGroup name="AdminDeleteSharedCatalogActionGroup"> <arguments> <argument name="sharedCatalogName"/> </arguments> <click selector="{{AdminSharedCatalogsSection.select}}" stepKey="clickSelectButton"/> <click selector="{{AdminSharedCatalogsSection.delete('sharedCatalogName')}}" stepKey="clickDeleteAction"/> <waitForElementVisible selector="{{AdminConfirmationModalSection.ok}}" stepKey="waitForConfirmationModalToAppear"/> <click selector="{{AdminConfirmationModalSection.ok}}" stepKey="confirmDeleteSharedCatalog"/> <see selector="{{AdminMessagesSection.success}}" userInput="The shared catalog was deleted successfully." stepKey="seeSuccessMessage"/> </actionGroup> </actionGroups>

2. Create parametrized selector mentioned at first step in AdminSharedCatalogsSection:

   <element name="delete" type="button" selector="//div[text()='{{sharedCatalogName}}']/parent::td//following-sibling::td[@class='data-grid-actions-cell']//a[text()='Delete']" timeout="30" parameterized="true"/>

3. Use action group mentioned at first step in your test with argument value as entity key:

<amOnPage url="{{AdminSharedCatalogIndexPage.url}}" stepKey="goToAdminSharedCatalogIndexPage1"/> <actionGroup ref="AdminDeleteSharedCatalogActionGroup" stepKey="deleteSharedCatalog"> <argument name="sharedCatalogName" value="{{NewSharedCatalog.name}}"/> </actionGroup>

Expected result

The result in generated test should be the next:

$I->click("//div[text()='New Shared Catalog" . msq("NewSharedCatalog") . "']/parent::td//following-sibling::td[@class='data-grid-actions-cell']//a[text()='Delete']");

Actual result

The result in generated test is the next:

$I->click("{{AdminSharedCatalogsSection.delete('New Shared Catalog" . msq("NewSharedCatalog") . "')}}");

Preconditions

1. PHP 7.2
2. magento/project-enterprise-edition (2.3.1)
3. magento/magento2-functional-testing-framework (2.3.14)
4. installed extension with it's own MFTF suite

Steps to reproduce

1. vendor/bin/mftf build:project
2. vendor/bin/mftf generate:suite MyCustomSuite

Expected result

1. Only the tests in the suite and the action groups / sections / pages it is using should be generated

Actual result

1. All available tests are being generated

The expected result can be achieved using the following preconditions (default installation):
magento/project-enterprise-edition (2.3.0)
magento/magento2-functional-testing-framework (2.3.11)

Preconditions

This was verified on Magento 2.3-develop branch

Steps to reproduce

1. Create an action group similar to such:

<actionGroup name="CreateGroupedProduct">
    <createData entity="_defaultCategory" stepKey="createCategory"/>

    <createData entity="_defaultProduct" stepKey="createFirst">
        <requiredEntity createDataKey="createCategory"/>
    </createData>

    <createData entity="GroupedProduct" stepKey="createGroup">
        <requiredEntity createDataKey="createCategory"/>
    </createData>

    <!-- The Important Part -->
    <createData entity="ProductLinkSimple1" stepKey="createFirstLink">
        <field key="sku">$$createGroup.sku$$</field>
        <field key="linked_product_sku">$$createFirst.sku$$</field>
    </createData>
</actionGroup>

<!-- ... -->
<actionGroup ref="CreateGroupedProduct" stepKey="InGroup" />
<!-- ... -->

2. Generate Tests

Expected result

$createFirstLinkInGroupFields['sku'] = PersistedObjectHandler::getInstance()->retrieveEntityField('createGroupInGroup', 'sku', 'hook');
$createFirstLinkInGroupFields['linked_product_sku'] = PersistedObjectHandler::getInstance()->retrieveEntityField('createFirstInGroup', 'sku', 'hook');
$I->amGoingTo("create entity that has the stepKey: createFirstLinkInGroup");
PersistedObjectHandler::getInstance()->createEntity(
	"createFirstLinkInGroup",
	"hook",
	"ProductLinkSimple1",
	[],
	$createFirstLinkInGroupFields
);

Actual result

$createFirstLinkInGroupFields['sku'] = PersistedObjectHandler::getInstance()->retrieveEntityField('createGroup', 'sku', 'hook');
$createFirstLinkInGroupFields['linked_product_sku'] = PersistedObjectHandler::getInstance()->retrieveEntityField('createFirst', 'sku', 'hook');
$I->amGoingTo("create entity that has the stepKey: createFirstLinkInGroup");
PersistedObjectHandler::getInstance()->createEntity(
	"createFirstLinkInGroup",
	"hook",
	"ProductLinkSimple1",
	[],
	$createFirstLinkInGroupFields
);

Resulting in the error during run: [Magento\FunctionalTestingFramework\Exceptions\TestReferenceException] Entity with a CreateDataKey of createGroup could not be found

Currently, out XML parsing errors where the symbols "<" or ">" are used.
These symbols can be useful to assert with html elements.

Expected result

• Modify the XML Parser (?) so it allows the parsing of HTML in Test XML actions, such as but not limited too:
  ** seeInPageSource
  *** {{}}
  ** seeInSource
  *** {{}}
  ** dontSeeInSource
  *** {{}}### Preconditions

As a Tester I'd like the XML-to-PHP generator to create the PHP test files inside the Module directory that it's associated with.

AC:

1. Tests listed in a specific Module (ex "Sales") should be exported to that Module directory instead of the "_generated" directory.
2. Tests listed in a specific Module should use the Namespace for that Module instead of a generic Namespace.

Preconditions

1. Magento 2.2.2, MFTF

Steps to reproduce

1. Try to configure a custom admin domain (for example, admin.example.org)

Expected result

1. Configuration is available

Actual result

1. No applicable configuration appears to be documented

The old version of PHP Codesniffer, as well as old sniffs, are used for a static code check.

As a result, when adding declare(strict_types=1); causes sniffer to throw a warning on the PSR-2 standard.

As a workaround in #69 I had to add @codingStandardsIgnoreFile annotation. The old version of PHP Codesniffer has made also impossible to use @codingStandardsIgnoreStart and @codingStandardsIgnoreEnd at the beginning of the file.

Suggest updating "squizlabs/php_codesniffer": "1.5.3" to "squizlabs/php_codesniffer": "3.0.1" and use updated sniffs for Magento 2.