comtihon / catcher Goto Github PK
View Code? Open in Web Editor NEWmicroservices and data pipelines end-to-end test tool
License: Apache License 2.0
microservices and data pipelines end-to-end test tool
License: Apache License 2.0
It is better to have global functions with configurable params.
Like random_int(1,10)
instead of RANDOM_INT
Need to add the same regex_replace filter, as Ansible has, as Jinja's replace doesn't support regexp.
add ability to set max time test can run for every test.
fail test if it is running longer.
run steps output should be differentiated from the main steps output.
Also there should be a possibility to hide them.
variables:
<variables block>
steps:
<steps block>
finally:
<steps block> with additional param (run: always/fail/pass)
from/to timestamp
and others
After looking at the docs or at the catcher modules I do not see support for rabbitmq.
Is there a plan for rabbitmq support ? Are there guide lines to create a new module ?
there should be timing for every step. F.e.
postgres:
action:
query: <some_query>
the output will be step postgres OK/fail in 4 seconds
there should be a step to run an ansible playbook.
Probably with cloning it from git before (or use git, as another step)
ignored: true
variables: ...
steps: ...
We should be able to ignore the test. Then test will be skipped.
Ignored should also support check-equal (to ignore tests for env var/inventory)
For full E2E test frontend also needs to be triggered.
Need to make selenium step?
Need to provide a working example on FE + BE E2E testing.
In case of CI without tty it is unreadable.
need to be able to execute simple steps:
sh:
run: "git clone my {{repo}}"
variables:
phone_num: "{{ random(phone_number) }}"
steps:
...
instead of:
steps:
- echo: {from: '{{ random("phone_number") }}', register: {phone_num: '{{ OUTPUT }}'}}
f.e. 1586995200000
should be translated to 2020-04-16
.
Current approach will throw ValueError: year 52259 is out of range
error.
We need to delete this TS by 1000 to receive real value.
In case some steps need to be run only locally. On some conditional steps, based on the other steps result
Need to revisit str vs int variables.
Is it possible to check them?
F.e.
if list [1,2,3,4] contains 2 or list ['1', '2', '3','4'] contains '2'
For now only one http session is allowed.
It prevents test to run multiple users tests.
http step needs to have a named session param to specify which session should be used for the request.
Add ability to skip step based on a logical condition like in Ansible
F.e.:
steps:
- docker:
start:
image: 'jamesdbloom/mockserver'
ports:
'1080/tcp': 8000
when:
equals: {the: '{{ my_http_service_url }}', is: 'localhost'}
Dependabot couldn't authenticate with https://pypi.python.org/simple/.
You can provide authentication details in your Dependabot dashboard by clicking into the account menu (in the top right) and selecting 'Config variables'.
In test conf object add concurrency group:
variables:
foo: bar
config:
concurrency:
enabled: true
group: teamA
steps:
....
If config.concurrency.enabled is true - test can be run concurrent.
If config.concurrency.group is true - no tests of the same group can be run at once.
Should use multiprocessing and create pool with CPU + 1
Test results in json/html should be concurrent safe.
I encountered a problem when using the --format json and wait (for) module:
For example, if I run following test with --format json:
steps:
- wait:
name: 'Waiting for postgres'
seconds: 30
for:
postgres:
request:
conf: '{{ postgres_conf }}'
query: 'select 1 as test;'
I get this error message:
Traceback (most recent call last):
File "/usr/local/bin/catcher", line 33, in <module>
sys.exit(load_entry_point('catcher==1.32.3', 'console_scripts', 'catcher')())
File "/usr/local/lib/python3.7/site-packages/catcher-1.32.3-py3.7.egg/catcher/__main__.py", line 50, in main
result = run_tests(path, arguments)
File "/usr/local/lib/python3.7/site-packages/catcher-1.32.3-py3.7.egg/catcher/__main__.py", line 81, in run_tests
return runner.run_tests(output=output)
File "/usr/local/lib/python3.7/site-packages/catcher-1.32.3-py3.7.egg/catcher/core/runner.py", line 70, in run_tests
logger.log_storage.write_report(join(self.path, 'reports'))
File "/usr/local/lib/python3.7/site-packages/catcher-1.32.3-py3.7.egg/catcher/modules/log_storage.py", line 79, in write_report
formatter.format(path, self._data)
File "/usr/local/lib/python3.7/site-packages/catcher-1.32.3-py3.7.egg/catcher/modules/formatter.py", line 23, in format
json.dump(data, fp)
File "/usr/local/lib/python3.7/json/__init__.py", line 179, in dump
for chunk in iterable:
File "/usr/local/lib/python3.7/json/encoder.py", line 429, in _iterencode
yield from _iterencode_list(o, _current_indent_level)
File "/usr/local/lib/python3.7/json/encoder.py", line 325, in _iterencode_list
yield from chunks
File "/usr/local/lib/python3.7/json/encoder.py", line 405, in _iterencode_dict
yield from chunks
File "/usr/local/lib/python3.7/json/encoder.py", line 325, in _iterencode_list
yield from chunks
File "/usr/local/lib/python3.7/json/encoder.py", line 405, in _iterencode_dict
yield from chunks
File "/usr/local/lib/python3.7/json/encoder.py", line 405, in _iterencode_dict
yield from chunks
File "/usr/local/lib/python3.7/json/encoder.py", line 405, in _iterencode_dict
yield from chunks
[Previous line repeated 1 more time]
File "/usr/local/lib/python3.7/json/encoder.py", line 325, in _iterencode_list
yield from chunks
File "/usr/local/lib/python3.7/json/encoder.py", line 405, in _iterencode_dict
yield from chunks
File "/usr/local/lib/python3.7/json/encoder.py", line 405, in _iterencode_dict
yield from chunks
File "/usr/local/lib/python3.7/json/encoder.py", line 438, in _iterencode
o = _default(o)
File "/usr/local/lib/python3.7/json/encoder.py", line 179, in default
raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type function is not JSON serializable
If I run test without --format json all is correct.
But if I run simple wait module with --format json:
steps:
- wait:
seconds: 5
Everything works well.
I did a little research and here's what I found out.
The error data object
File "/usr/local/lib/python3.7/site-packages/catcher-1.32.3-py3.7.egg/catcher/modules/formatter.py", line 23, in format json.dump(data, fp)
looks like this:
[{
'end_time': '2020-11-10 13:32:50',
'start_time': '2020-11-10 13:32:49',
'file': 'tests/001/main.yml',
'type': 'test',
'output': [{
'time': '2020-11-10 13:32:49',
'step': {
'wait': {
'name': 'Waiting for postgres',
'seconds': 30,
'for': {
'postgres': {
'request': {
'conf': '{{ postgres_conf }}',
'query': 'select 1 as test;'
},
'_get_actions': < function StepFactory.ensure_body_dict. < locals > .get_actions_fun at 0x7faf85cb30e0 > ,
'_get_action': < function StepFactory.ensure_body_dict. < locals > .get_action_fun at 0x7faf85cb3170 > ,
'_path': '/opt/catcher'
}
}
}
},
'variables': {
'PATH': '/root/.sdkman/candidates/kotlin/current/bin:/usr/lib/jvm/java-1.8-openjdk/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin',
'HOSTNAME': 'aksenk',
'LANG': 'C.UTF-8',
'GPG_KEY': '0D96DF4D4110E5C43FBFB17F2D347EA6AA65421D',
'PYTHON_VERSION': '3.7.9',
'PYTHON_PIP_VERSION': '20.2.3',
'PYTHON_GET_PIP_URL': 'https://github.com/pypa/get-pip/raw/fa7dc83944936bf09a0e4cb5d5ec852c0d256599/get-pip.py',
'PYTHON_GET_PIP_SHA256': '6e0bb0a2c2533361d7f297ed547237caf1b7507f197835974c0dd7eba998c53c',
'JAVA_HOME': '/usr/lib/jvm/java-1.8-openjdk',
'HOME': '/root',
'postgres_conf': {
'dbname': 'db',
'user': 'postgres',
'host': 'localhost',
'password': 'password'
},
'INVENTORY': 'ci',
'INVENTORY_FILE': 'inventory/ci.yml',
'CURRENT_DIR': '/opt/catcher',
'RESOURCES_DIR': '/opt/catcher/resources',
'TEST_NAME': 'tests/001/main.yml'
},
'nested': 0
}, {
'time': '2020-11-10 13:32:50',
'data': 'Step Waiting for postgres [0s]\x1b[32m OK\x1b[0m',
'level': 'info'
}, {
'time': '2020-11-10 13:32:50',
'step': {
'wait': {
'name': 'Waiting for postgres',
'seconds': 30,
'for': {
'postgres': {
'request': {
'conf': '{{ postgres_conf }}',
'query': 'select 1 as test;'
},
'_get_actions': < function StepFactory.ensure_body_dict. < locals > .get_actions_fun at 0x7faf85cb30e0 > ,
'_get_action': < function StepFactory.ensure_body_dict. < locals > .get_action_fun at 0x7faf85cb3170 > ,
'_path': '/opt/catcher'
}
}
}
},
'variables': {
'PATH': '/root/.sdkman/candidates/kotlin/current/bin:/usr/lib/jvm/java-1.8-openjdk/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin',
'HOSTNAME': 'aksenk',
'LANG': 'C.UTF-8',
'GPG_KEY': '0D96DF4D4110E5C43FBFB17F2D347EA6AA65421D',
'PYTHON_VERSION': '3.7.9',
'PYTHON_PIP_VERSION': '20.2.3',
'PYTHON_GET_PIP_URL': 'https://github.com/pypa/get-pip/raw/fa7dc83944936bf09a0e4cb5d5ec852c0d256599/get-pip.py',
'PYTHON_GET_PIP_SHA256': '6e0bb0a2c2533361d7f297ed547237caf1b7507f197835974c0dd7eba998c53c',
'JAVA_HOME': '/usr/lib/jvm/java-1.8-openjdk',
'HOME': '/root',
'postgres_conf': {
'dbname': 'db',
'user': 'postgres',
'host': 'localhost',
'password': 'password'
},
'INVENTORY': 'ci',
'INVENTORY_FILE': 'inventory/ci.yml',
'CURRENT_DIR': '/opt/catcher',
'RESOURCES_DIR': '/opt/catcher/resources',
'TEST_NAME': 'tests/001/main.yml'
},
'nested': 0,
'success': True,
'output': None
}, {
'time': '2020-11-10 13:32:50',
'data': 'Test main\x1b[32m passed.\x1b[0m',
'level': 'info'
}],
'status': 'OK',
'comment': None
}]
I think error in this section:
'_get_actions': < function StepFactory.ensure_body_dict. < locals > .get_actions_fun at 0x7faf85cb30e0 > ,
'_get_action': < function StepFactory.ensure_body_dict. < locals > .get_action_fun at 0x7faf85cb3170 > ,
Investigate a way to store credentials encrypted, as Ansible does.
This is my yaml
steps:
# Status endpoint
- http: {get: {url: 'http://localhost:5000/api/v2/status'}}
# Cluster health endpoint
- http: {get: {url: 'http://localhost:5000/api/v2/clusterHealth'}}
# indexAndType endpoint
- http:
post:
url: 'http://localhost:5000/api/v2/indexAndType'
body: {email: '[email protected]'}
I checked the yaml and it is valid and I was following the format from the docs but I still get this error WARNING:catcher:Test catcher_test.yml failed: too many values to unpack (expected 1)
on the last step and it doesn't run because of it
Web api needs catcher cli to run step-by-step. With ability to run from some (perhaps failed) step and see variables input/output for every step.
Using catcher-1.36.2
Let's try simple test:
variables:
test_data: [ "NOT_PRESENT", "NOT_PRESENT", "PRESENT" ]
steps:
- echo:
name: 'echo test items'
from: 'Items are {{ test_data }}'
- loop:
foreach:
in: test_data
do:
- echo:
name: 'echo test item'
from: 'Item is {{ ITEM }}'
register: {found: '{{ ITEM == "PRESENT" }}'}
- echo:
name: 'echo flag'
from: 'Found flag is {{ found }}'
- check:
name: 'truefalse test'
skip_if: '{{ not found }}'
equals: {the: '{{ found }}', is: True}
"check" step must NOT fail test in any case
Also, first step in loop prints there some garbage: INFO:catcher:Item is t
In case there is a requirements.txt in the resource folder - catcher should install these requirements before running tests
Does http module support basic auth feature.
I was looking for a way to do basic authentication with REST api.
I am not able to see a way to do it in docs.
The other option would be to write an external step that creates the Base64 encoding and then use that in the http step for the headers
workaround: include env
var in each inventory and name it as inventory name.
Need a tutorial for creating a jvm based step.
How to pack and launch it (ideally with gradle/maven set up).
Maybe add catcher generate -l java
cmd for generating skeleton for a module
after catcher_modules 4.x.x documentation and examples needs to be updated.
would be good to have a testrail integration.
to think (need a server as well)
It would be nice to have an intellij idea plugin.
Refactor filters & make them dynamically loadable with ability to add custom filters by just loading them.
Need to find a workaround for older python versions.
In case of multiple tests to run I need a nice summary at the output end:
Just like there is support for kafka and postgres, I was wondering if there is support for ES because we use those heavily in our microservices and were wondering if that would be possible, because writing ES request bodies with queries and aggreagations if writing them as a simple http request in catcher can be very cumbersome so it would be nice to have something like
elasticsearch:
search:
index: indexName
query:
bool: {{filter_query_body}}
I could try sending a PR for this if you don't have the time to implement this. Also I took a look at the codebase but I can't understand how the actions for kafka and postgres are implemented.
cc @comtihon
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.