golemhq / golem Goto Github PK

While reading the code I found a file called selenium.py at golem/golem/selenium.py
, won't that cause an import conflict?

How can user set dependencies between multiple tests?

Example:
Test1->Test2 to TestN : In order to execute Test2 to TestN , Test1 should be executed successfully first as that is the primary/basic test for the module.

Hey @luciano-renzi ,

Have you started on this? I started integrating appium into 0.4.1 yesterday..
Hoping to have golem working with Android today, but I think there's some significant changes needed to browser.py to get the desired capabilities from a test case.

Did you want to create a feature branch on here that I can do PRs and people can pull down if they want to test/contribute?

Also, Have you got anywhere we can discuss feature implementation, or creating an issue is the best place?

Cheers,
Daniel

Unable to delete an Module(Folder) inside test folder from UI.
Even though Module(folder) is empty.

First of all met me say: Awesome job on this framework! I am really digging it.

I want to report some bugs I encountered moving from v0.2.1 -> v0.2.2 (new GUI layout)

1. The Suites settings wont show any suggestions anymore for drivers like eg. "chrome or chrome-headless"
2. Selected data like driver and test cases are not persisted anymore, pressing save generates the following error:

File "/home/xxxxxx/dev/python/venvs/doodle/lib/python3.5/site-packages/golem/gui/__init__.py", line 626, in save_suite
    environments = request.json['environments']
KeyError: 'environments'

Unable to rename an Suite/Module/Test from the web UI.

Where to add capabilities to accept trusted SSL certificates in Chrome and Firefox browsers?

After updating to [0.2.2] - 2017-10-16, am not able to add steps.

Error :

(index):260 Uncaught ReferenceError: testCase is not defined at HTMLButtonElement.onclick

OS name and version: Ubuntu 16.04
Python version: 3.5.2
Golem version: 0.4.1

Description:
When running a test suite, Golem appears to repeat messages logged to console. The number of times it repeats increases by one for each test (one message at a time, then 2 messages, then 3, etc.). I'm guessing this has to do with Python's logger propagation and handlers not getting cleaned up after each test (I could never quite get Python's logging module down).

Steps to Reproduce:

• Create a test in Golem
• Create a suite containing the same test 3 times (different tests should also be affected, but this makes it apparent)
• Run the test suite

Expected Output:
Golem outputs logs for each sequential test only once

Actual Sample Output:

(venv) ubuntu@localhost:~/golem$ python golem.py run Dashboards ExampleSuite
22:41:22 INFO Test execution started: OpenGoogle
22:41:22 INFO Browser: chrome-headless
22:41:22 INFO Environment: production
22:41:24 INFO Navigate to: 'https://www.google.com'
22:41:24 INFO Verify the element 'hplogo' is visible
22:41:24 INFO Test passed
22:41:24 INFO Test execution started: OpenGoogle
22:41:24 INFO Test execution started: OpenGoogle
22:41:24 INFO Browser: chrome-headless
22:41:24 INFO Browser: chrome-headless
22:41:24 INFO Environment: production
22:41:24 INFO Environment: production
22:41:27 INFO Navigate to: 'https://www.google.com'
22:41:27 INFO Navigate to: 'https://www.google.com'
22:41:27 INFO Verify the element 'hplogo' is visible
22:41:27 INFO Verify the element 'hplogo' is visible
22:41:27 INFO Test passed
22:41:27 INFO Test passed
22:41:27 INFO Test execution started: OpenGoogle
22:41:27 INFO Test execution started: OpenGoogle
22:41:27 INFO Test execution started: OpenGoogle
22:41:27 INFO Browser: chrome-headless
22:41:27 INFO Browser: chrome-headless
22:41:27 INFO Browser: chrome-headless
22:41:27 INFO Environment: production
22:41:27 INFO Environment: production
22:41:27 INFO Environment: production
22:41:29 INFO Navigate to: 'https://www.google.com'
22:41:29 INFO Navigate to: 'https://www.google.com'
22:41:29 INFO Navigate to: 'https://www.google.com'
22:41:30 INFO Verify the element 'hplogo' is visible
22:41:30 INFO Verify the element 'hplogo' is visible
22:41:30 INFO Verify the element 'hplogo' is visible
22:41:30 INFO Test passed
22:41:30 INFO Test passed
22:41:30 INFO Test passed

data file:
test_decimal
20.00
'20.00'
"20.00"

if any of the first 3 options are set in the data file, it gets converted to 20.0 when the test runs and can cause validation errors. Works correctly if I save the data in the gui as "20.00" which comes out as """20.00""" in the csv.

Working as intended? Running golem-framework-0.3.1

I created a project called billing and attempted to create a directory inside of it, but web ui fails.
I can see this traceback:

127.0.0.1 - - [02/Jan/2018 17:34:11] "POST /new_tree_element/ HTTP/1.1" 500 -
Traceback (most recent call last):
  File "/Users/ndefosse/.virtualenvs/golem/lib/python3.6/site-packages/flask/app.py", line 1997, in __call__
    return self.wsgi_app(environ, start_response)
  File "/Users/ndefosse/.virtualenvs/golem/lib/python3.6/site-packages/flask/app.py", line 1985, in wsgi_app
    response = self.handle_exception(e)
  File "/Users/ndefosse/.virtualenvs/golem/lib/python3.6/site-packages/flask/app.py", line 1540, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/ndefosse/.virtualenvs/golem/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise
    raise value
  File "/Users/ndefosse/.virtualenvs/golem/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/ndefosse/.virtualenvs/golem/lib/python3.6/site-packages/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/ndefosse/.virtualenvs/golem/lib/python3.6/site-packages/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/ndefosse/.virtualenvs/golem/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise
    raise value
  File "/Users/ndefosse/.virtualenvs/golem/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/ndefosse/.virtualenvs/golem/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/ndefosse/.virtualenvs/golem/lib/python3.6/site-packages/golem/gui/__init__.py", line 580, in new_tree_element
    element_name, dir_type='suites')
  File "/Users/ndefosse/.virtualenvs/golem/lib/python3.6/site-packages/golem/core/io_manager.py", line 34, in new_directory
    utils.create_new_directory(path=path, add_init=True)

Review code and installation

On creating an single test and executing by UI, on test failure the results are not captured in Reports for later reference.
Test Failure Console Output

Empty Reports

In the adding-tests.md docs, the teardown() method needs to have a 'data' parameter, otherwise it will have an error.

TODO: Change line 68 of adding-tests.md from def teardown(): to def teardown(data):

How can user can import test data from External Files(Excel/CSV) ?
This is especially helpful when test data is created and handed over by separate manual testers/BAs in the team.

New Actions

fail(message) : mark the test as failure, stop, jump to teardown
error(message) : add an error to the test, the test should continue, will be marked as error at the end
verify_url(url) : verify the current url
verify_url_contains(url_substring) : verify the current url contains the url_substring passed as argument
get_current_url() : returns the current url
tab management (open tab, close tab, get current tabs, switch to tab)
alerts (verify alert is present, verify alert text, close alert)
verify_window_title
verify_window_title_contains
scroll
IFrame (switch to IFrame)
get_element_text(element) : returns the text of an element

Current Actions Refactor

capture() : rename to screenshot()
verify_text_in_element(element, text) : rename to verify_element_text(element, text)

How can user define setup and teardown actions on suite level?

I want to filter the test results in the bottom table of the execution report by clicking on a row of the top table.

I want to filter the bottom table by the column values

Creating a test in the GUI

type clic
cursor moves to the X, does not give you a field to specify an element
click does give you the input.

When running a test using the environments config, the test sets and information doesn't show in the info window. Test executes fine, however the window stays open and you just have to click dismiss.

request to reopen the following issue:
#41

Please note the updated input after the commit with the fixes

Might be as designed, but if you select the wrong environment from the dropdown, you have no way to back out, dismiss button doesn't cancel the test(will raise as a separate issue)

There should be an option for suite, CLI and settings file to re-run failed tests

When the setup function of a test fails, the test should be marked as 'error' and stop the execution, It should not execute the test function and jump to teardown function if it is defined.

following folders are missing in /golem/gui/templates/:
page_builder/
project/
report/
test_builder/

~/Development/pythons/golem/lib/python3.6/site-packages/golem/gui/templates$ ll
total 56
drwxr-xr-x 2 daniel daniel 4096 Dec 13 15:24 ./
drwxr-xr-x 5 daniel daniel 4096 Dec 13 15:24 ../
-rw-r--r-- 1 daniel daniel 360 Dec 13 15:24 404.html
-rw-r--r-- 1 daniel daniel 6148 Dec 13 15:24 .DS_Store
-rw-r--r-- 1 daniel daniel 2228 Dec 13 15:24 environments.html
-rw-r--r-- 1 daniel daniel 1353 Dec 13 15:24 index.html
-rw-r--r-- 1 daniel daniel 8873 Dec 13 15:24 layout.html
-rw-r--r-- 1 daniel daniel 1129 Dec 13 15:24 login.html
-rw-r--r-- 1 daniel daniel 323 Dec 13 15:24 not_permission.html
-rw-r--r-- 1 daniel daniel 1917 Dec 13 15:24 settings.html
-rw-r--r-- 1 daniel daniel 3917 Dec 13 15:24 suite.html

Traceback (most recent call last):
  File "/home/daniel/Development/pythons/golem/lib/python3.6/site-packages/Flask-0.12.2-py3.6.egg/flask/app.py", line 1997, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/daniel/Development/pythons/golem/lib/python3.6/site-packages/Flask-0.12.2-py3.6.egg/flask/app.py", line 1985, in wsgi_app
    response = self.handle_exception(e)
  File "/home/daniel/Development/pythons/golem/lib/python3.6/site-packages/Flask-0.12.2-py3.6.egg/flask/app.py", line 1540, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/home/daniel/Development/pythons/golem/lib/python3.6/site-packages/Flask-0.12.2-py3.6.egg/flask/_compat.py", line 33, in reraise
    raise value
  File "/home/daniel/Development/pythons/golem/lib/python3.6/site-packages/Flask-0.12.2-py3.6.egg/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/daniel/Development/pythons/golem/lib/python3.6/site-packages/Flask-0.12.2-py3.6.egg/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/daniel/Development/pythons/golem/lib/python3.6/site-packages/Flask-0.12.2-py3.6.egg/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/daniel/Development/pythons/golem/lib/python3.6/site-packages/Flask-0.12.2-py3.6.egg/flask/_compat.py", line 33, in reraise
    raise value
  File "/home/daniel/Development/pythons/golem/lib/python3.6/site-packages/Flask-0.12.2-py3.6.egg/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/daniel/Development/pythons/golem/lib/python3.6/site-packages/Flask-0.12.2-py3.6.egg/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/daniel/Development/pythons/golem/lib/python3.6/site-packages/flask_login/utils.py", line 261, in decorated_view
    return func(*args, **kwargs)
  File "/home/daniel/Development/pythons/golem/lib/python3.6/site-packages/golem/gui/__init__.py", line 132, in project_suites
    return render_template('project/project_suites.html', project=project)
  File "/home/daniel/Development/pythons/golem/lib/python3.6/site-packages/Flask-0.12.2-py3.6.egg/flask/templating.py", line 133, in render_template
    return _render(ctx.app.jinja_env.get_or_select_template(template_name_or_list),
  File "/home/daniel/Development/pythons/golem/lib/python3.6/site-packages/jinja2/environment.py", line 869, in get_or_select_template
    return self.get_template(template_name_or_list, parent, globals)
  File "/home/daniel/Development/pythons/golem/lib/python3.6/site-packages/jinja2/environment.py", line 830, in get_template
    return self._load_template(name, self.make_globals(globals))
  File "/home/daniel/Development/pythons/golem/lib/python3.6/site-packages/jinja2/environment.py", line 804, in _load_template
    template = self.loader.load(self, name, globals)
  File "/home/daniel/Development/pythons/golem/lib/python3.6/site-packages/jinja2/loaders.py", line 113, in load
    source, filename, uptodate = self.get_source(environment, name)
  File "/home/daniel/Development/pythons/golem/lib/python3.6/site-packages/Flask-0.12.2-py3.6.egg/flask/templating.py", line 57, in get_source
    return self._get_source_fast(environment, template)
  File "/home/daniel/Development/pythons/golem/lib/python3.6/site-packages/Flask-0.12.2-py3.6.egg/flask/templating.py", line 85, in _get_source_fast
    raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: project/project_suites.html

You get the data values in both the element field and the value field. Should have the page elements in the element field.

In this page : http://golem-framework.readthedocs.io/en/latest/adding-tests.html
The sample test in not valid : validate_article_title.py
def teardown() should be replaced with def teardown(data)

http://golem-framework.readthedocs.io/en/latest/_images/interactive-console.png

I think it should be from golem.browser import get_browser()

Cannot scrolls the rest results model

With the current latest source of (22-10-2017) I am experiencing the following GUI bug.

The field indicators when adding a test step that requires multiple arguments is displayed in the incorrect order within the gui. This results in a issue where once calling the test step the function is called with the wrong order of arguments.

Example:
The following test step function:

end up being presented as (TOP: correct order for function call, BOTTOM: incorrect initial suggestions, the bug)

The result of the above:

python golem.py run test_project test --environment Local -i
Traceback (most recent call last):
File "golem.py", line 15, in
execute_from_command_line(os.getcwd())
File "/Users/daniel/Development/pythons/golem/lib/python3.6/site-packages/golem/main.py", line 27, in execute_from_command_line
commands.run(args.main_action, test_execution, args)
File "/Users/daniel/Development/pythons/golem/lib/python3.6/site-packages/golem/commands/init.py", line 51, in run
cmd.run(tex, args)
File "/Users/daniel/Development/pythons/golem/lib/python3.6/site-packages/golem/commands/base.py", line 99, in run
test=test_execution.test)
File "/Users/daniel/Development/pythons/golem/lib/python3.6/site-packages/golem/test_runner/start_execution.py", line 167, in run_test_or_suite
run_test(test_execution.root_path, test_execution.project,
NameError: name 'run_test' is not defined

Unable to run any tests/suites after physically moving tests through windows explorer from one suite to another or outside any folder ,directly inside test folder.

Following files aren't loading correctly..

http://0.0.0.0:5000/static/css/font-awesome/font-awesome.min.css
http://0.0.0.0:5000/static/js/chartJs/Chart.min.js
http://0.0.0.0:5000/static/js/chartJs/Chart.min.js

chartJs looks like its because the file names are wrong..

So if I want help with completion in my IDE using golem, I need:

from golem import actions
from ..pages import MerchantFormPage1, MerchantFormPage2

and

actions.click(something)

This allows my IDE to help me with the actions. But the golem GUI doesn't strip the actions part from the statements, so I'm stuck with the gui showing actions.click

This might be less of an issue if I could insert steps into a test without having to use my IDE to insert them because the GUI only supports appending. PS, just discovered drag and drop in the GUI (completely by accident) so insertion is a thing.

Currently the test can be 'passed' or 'failed'

The 'fail' status should be differentiated in two: 'failure' and 'error'

If an assert action fails the test should be marked as 'failure'

Any other error or exception should mark the test as 'error'

Environment:
OS:Win7
Python: 3.6.3

Steps:
1).Installed Python 3.6.3 in Windows 7
2).Installed Golem by pip install golem-framework.
3).Created the test directory : golem-admin C:\Golem
4).Downloaded Chrome Browser Driver 2.33 and placed inside Drivers folder.
5).Started the web Module:
cd C:\Golem
python golem.py gui
6).Got the error on command prompt as attached screenshot.

However this is working fine in Python 3.4.4 version.

The ones in the code base aren't really PageObjects so I can't import them and instantiate them, and I don't see how to get the page objects from this?

>>> locals().keys()
dict_keys(['__name__', '__doc__', '__package__', '__loader__', '__spec__', '__file__', '__cached__', '__builtins__', 'time', 'uuid', 'os', 'importlib', 'string', 'rand', 'selenium', 'Keys', 'ActionChains', 'EC', 'WebDriverWait', 'requests', 'TextNotPresent', 'ElementNotFound', 'get_browser', 'browser', 'execution', '_run_wait_hook', '_capture_or_add_step', 'assert_contains', 'assert_equals', 'assert_false', 'assert_true', 'capture', 'clear', 'click', 'close', 'debug', 'get', 'mouse_hover', 'navigate', 'press_key', 'random', 'refresh_page', 'select_by_index', 'select_by_text', 'select_by_value', 'send_keys', 'set_window_size', 'step', 'store', 'verify_exists', 'verify_is_enabled', 'verify_is_not_enabled', 'verify_is_not_selected', 'verify_is_not_visible', 'verify_is_selected', 'verify_is_visible', 'verify_not_exists', 'verify_selected_option', 'verify_text', 'verify_text_in_element', 'wait', 'wait_for_element_not_exist', 'wait_for_element_not_visible', 'wait_for_element_enabled', 'wait_for_element_visible', 'http_get', 'http_post', 'verify_response_status_code', 'vars_copy', 'console_exit', 'code', 'readline', 'exit'])

How can user dynamically filter tests based on some labels or tags like 'smoke','regression', 'fast','slow' from high level folder instead of physically dividing tests by creating separate folders like regression/smoke or fast/slow.

This is like saying execute all tests with 'fast' tag irrespective of any folder hierarchy in which they actually reside.

How framework supports hard vs. soft assertions inside an test.

Terminology
Hard Assertions:Test execution stops as soon as assertion failure found.
Soft Assertions:Test execution continues even if assertion failure found.

On having any python syntax error like missing parentheses in the page object method, unable to correct it.
Always getting "builtins.SyntaxError" page from multiple locations.Tried:
1).Directly from error page, could not find a way back to Original Page to correct the syntax error.
2).On opening browser again, and navigating to Page object code, still opening same error page.
3).Navigating from test to Page object code.

Suggested Approach: There should be a way to go back from error page to Page object code to correct it.

Placing An Existing Test Inside a newly Created Module( a folder from Windows Explorer) loses the test.
Env:
Windows 7/8
Python: 3.4.4
Golem: 0.2.2

Steps:
1).Create an Test called 'Test1'directly inside tests folder.
2).Create a folder named 'Module1' from windows explorer inside tests folder and cut paste the 'Test1' inside it.
3).By GUI, try to access 'Test1' by clicking on it by expanding 'Module1' folder and clicking on 'Test1'.
4).Will get the error as attached.

Note: Did not find a way to add a Module from UI so added folder from windows explorer.

Add information to install from pip

First of all, thanks so much for this open source project!

So when I run tests on chrome/chrome-headless, it sometimes skips commands.

    ...
    banner.check_tab(banner.browse_now_button, "First tab")
    # Check third tab
    click(banner.third_tab)
    banner.check_tab(banner.services_button, "Third tab")
    # Check second tab and modal when clicking button on this tab
    click(banner.second_tab)
    banner.check_tab(banner.submit_listing_button, "Second tab")
    ...

it will skip the 6th line of code and go straight to checking an element.
banner.check_tab is a function in a 'page'.

Here I highlight the execution_console.log lines that are important

6:40:06.400 INFO Take screenshot First tab
16:40:07.423 INFO Verify the element 'Browse now button' is visible
16:40:07.465 INFO Click Third tab banner
16:40:07.510 INFO Take screenshot Third tab
16:40:12.825 INFO Verify the element 'Services button' is visible
16:40:12.836 INFO Take screenshot error
16:40:13.385 ERROR An error ocurred:
Traceback (most recent call last):
File "/home/Glaydur/environments/golem_test/lib/python3.5/site-packages/golem/test_runner/test_runner.py", line 124, in run_test
test_module.test(execution.data)
File "/home/Glaydur/dev/[redacted]/projects/[redacted]/tests/banner.py", line 27, in test
banner.check_tab(banner.services_button, "Third tab")
File "/home/Glaydur/dev/[redacted]/projects/[redacted]/pages/banner.py", line 14, in check_tab
actions.verify_is_visible(element)
File "/home/Glaydur/environments/golem_test/lib/python3.5/site-packages/golem/actions.py", line 330, in verify_is_visible
raise Exception('Element is not visible')
Exception: Element is not visible

This doesn't happen all the time (just every time I change the test code). I find that if I run the test in firefox first then in chrome/chrome-headless it would work fine (until I change the test code).

There seems to be no equivalent to driver.getCurrentUrl()

Installed it in a virtualenv then i tried to create the folder and i got this.
python 3.5.3

(venv) ➜  golem-framework golem-admin createdirectory test
zsh: command not found: golem-admin

So i brute forced installed it as sudo and then i got this

Traceback (most recent call last):
  File "/usr/local/bin/golem-admin", line 7, in <module>
    from golem.bin.golem_admin import main
  File "/usr/local/lib/python2.7/dist-packages/golem/bin/golem_admin.py", line 6, in <module>
    from .. import commands
  File "/usr/local/lib/python2.7/dist-packages/golem/commands/__init__.py", line 1, in <module>
    from .base import (CommandException, RunCommand,
  File "/usr/local/lib/python2.7/dist-packages/golem/commands/base.py", line 3, in <module>
    from golem.core import (utils,
  File "/usr/local/lib/python2.7/dist-packages/golem/core/__init__.py", line 37
    raise Exception(msg) from None
                            ^

Does the guide need to be updated?
if so, let me know so i can help

When running a single testcase manually via the GUI a http error 400 is generated.
This behavior started after upgrading from v0.2.1 to v0.2.2 of the framework

Error when executing:
127.0.0.1 - - [21/Oct/2017 17:26:15] "POST /run_test_case/ HTTP/1.1" 400 -

Executing via command line is working properly.

In order to publish & display test results on CI tools dashboard , need to have reports in Junit xml format to be importable.

Reported as bug now, please see my comments regarding the issue:
#51

Experienced on the current source version.

Steps to reproduce:
1 Use the latest source version of GOLEM
2. create a new test case via the GUI
3. Fill test data with regular string types and press save
HeaderA, HeaderB, HeaderC,
Avalue, Bvalue,Cvalue
4. reload the test case, value error will be displayed

Additional remark regarding the CSV file, i notice in many cases the order of headers is changed when saving the file, It could be that the example of above is saved as: HeaderB, HeaderA,HeaderC.
is this behavior intended ?

In case user needs to pass data in more complex data structures like lists,dictionary,tuples etc, how can it be defined & passed as parameters inside functions(actions)?

Example:
List of standard common labels across application pages , to be verified so the list of those labels/objects can be passed around as arguments to functions to verify .