pytest-dev / pytest-testinfra Goto Github PK
View Code? Open in Web Editor NEWTestinfra test your infrastructures
Home Page: https://testinfra.readthedocs.io
License: Apache License 2.0
Testinfra test your infrastructures
Home Page: https://testinfra.readthedocs.io
License: Apache License 2.0
Hi,
is it possible to specify a inventory file relative to some tests?
For example i have the following structure:
├── hosts
│ ├── databases
│ │ ├── inventory.ini
│ │ ├── mysql_test.py
│ └── webservers
│ ├── apache_test.py
│ ├── inventory.ini
I can run tests for a specific directory by using testinfra --connection=ansible --ansible-inventory hosts/webservers/inventory hosts/webservers
. Thats fine.
But i want to run tests on all directories in one run. So it would be great if the inventory would be detected relative to the test files. So i would just call testinfra --connection=ansible hosts
I tried around with relative parameters, but i can not get it working:
testinfra --connection=ansible --ansible-inventory inventory.ini
I always get this error report, which seams to be the error when no inventory file is found.:
[gw0] darwin -- Python 2.7.10 /usr/bin/python
testinfra_backend = <object object at 0x1046ad0d0>
@pytest.fixture(scope="module")
def f(testinfra_backend):
> return testinfra_backend.get_module(cls.__name__)
E AttributeError: 'object' object has no attribute 'get_module'
/Library/Python/2.7/site-packages/testinfra/modules/base.py:77: AttributeError
Maybe there is a possibility to do it via an ansible.cfg, a marker or a special notation in the --ansible-inventory switch, or something like this?
Hope you have some ideas on this.
Ps: Great Lib, i really love it!
For some backends (eg docker backend) it would be nice to have setup and teardown phases. For now container creation and removal processes are held in run method, which is not optimal.
It would be preferable to have setup process during which docker container could be created and tear down process during which its removal could be executed. It would really quicken testing.
Hi,
is there a possibility to filter ansible hosts by host pattern? I know the way to filter by name via --hosts=web*
, but this doesn't work for groups in ansible, only for hostnames. For example I can run a ad hoc command like ansible -m ping -i inventory groupname
to just contact hosts in the group. This is a pretty cool feature of ansible.
Something like this would be great: https://github.com/jlaska/pytest-ansible#parameterizing-with-pytestmarkansible. A marker to limit a test to a specific host pattern. So you can have a big test suite, but apply a test for mysql stuff on database hosts only.
What do you think, is this possible? Maybe even now and I don't get it?
Here is the performance I see for 2 simple tests on different backends.
It seems that ansible is much, much, much slower !
Admittedly ansible needs more time to update its inventory, but the task themselves are slow as if it were connecting to the machine at every single test.
Here is some sample output:
1/ test with IP provided
2/ test with ansible
<!> Also output is not the same for some unknown reason
$ testinfra [email protected],[email protected],[email protected],[email protected],[email protected] main.py
================================================= test session starts =================================================
platform linux2 -- Python 2.7.6, pytest-2.8.5, py-1.4.31, pluggy-0.3.1
rootdir: /home/emayssat/workspaces/emayssat/devtools-vpc/devops/aws/release/tests, inifile:
plugins: testinfra-1.0.0.0a21
collected 10 items
main.py ..........
============================================== 10 passed in 7.62 seconds ==============================================
(aws)~/workspaces/emayssat/devtools-vpc/devops/aws/release/tests
emayssat@xps13 $ make execute_tests
AWS_PROFILE=7009 testinfra -v --connection=ansible --ansible-inventory=inventories/ec2_7009 --hosts=key_Release001 main.py
================================================= test session starts =================================================
platform linux2 -- Python 2.7.6, pytest-2.8.5, py-1.4.31, pluggy-0.3.1 -- /home/emayssat/.virtualenvs/aws/bin/python
cachedir: .cache
rootdir: /home/emayssat/workspaces/emayssat/devtools-vpc/devops/aws/release/tests, inifile:
plugins: testinfra-1.0.0.0a21
collected 10 items
main.py::test_passwd_file[ansible://52.8.21.124] PASSED
main.py::test_surrogate_file[ansible://52.8.21.124] PASSED
main.py::test_passwd_file[ansible://54.67.33.158] PASSED
main.py::test_surrogate_file[ansible://54.67.33.158] PASSED
main.py::test_passwd_file[ansible://52.9.29.73] PASSED
main.py::test_surrogate_file[ansible://52.9.29.73] PASSED
main.py::test_passwd_file[ansible://54.67.6.175] PASSED
main.py::test_surrogate_file[ansible://54.67.6.175] PASSED
main.py::test_passwd_file[ansible://54.67.81.102] PASSED
main.py::test_surrogate_file[ansible://54.67.81.102] PASSED
============================================= 10 passed in 82.07 seconds ==============================================
(aws)~/workspaces/emayssat/devtools-vpc/devops/aws/release/tests
emayssat@xps13 $ make execute_tests
AWS_PROFILE=7009 testinfra -v --connection=ansible --ansible-inventory=inventories/ec2_7009 --hosts=key_Release001 main.py
================================================= test session starts =================================================
platform linux2 -- Python 2.7.6, pytest-2.8.5, py-1.4.31, pluggy-0.3.1 -- /home/emayssat/.virtualenvs/aws/bin/python
cachedir: .cache
rootdir: /home/emayssat/workspaces/emayssat/devtools-vpc/devops/aws/release/tests, inifile:
plugins: testinfra-1.0.0.0a21
collected 10 items
main.py::test_passwd_file[ansible://52.8.21.124] PASSED
main.py::test_surrogate_file[ansible://52.8.21.124] PASSED
main.py::test_passwd_file[ansible://54.67.33.158] PASSED
main.py::test_surrogate_file[ansible://54.67.33.158] PASSED
main.py::test_passwd_file[ansible://52.9.29.73] PASSED
main.py::test_surrogate_file[ansible://52.9.29.73] PASSED
main.py::test_passwd_file[ansible://54.67.6.175] PASSED
main.py::test_surrogate_file[ansible://54.67.6.175] PASSED
main.py::test_passwd_file[ansible://54.67.81.102] PASSED
main.py::test_surrogate_file[ansible://54.67.81.102] PASSED
============================================= 10 passed in 47.62 seconds ==============================================
(aws)~/workspaces/emayssat/devtools-vpc/devops/aws/release/tests
emayssat@xps13 $ make execute_tests
AWS_PROFILE=7009 testinfra -v --connection=ansible --ansible-inventory=inventories/ec2_7009 --hosts=key_Release001 main.py
================================================= test session starts =================================================
platform linux2 -- Python 2.7.6, pytest-2.8.5, py-1.4.31, pluggy-0.3.1 -- /home/emayssat/.virtualenvs/aws/bin/python
cachedir: .cache
rootdir: /home/emayssat/workspaces/emayssat/devtools-vpc/devops/aws/release/tests, inifile:
plugins: testinfra-1.0.0.0a21
collected 10 items
main.py::test_passwd_file[ansible://52.8.21.124] PASSED
main.py::test_surrogate_file[ansible://52.8.21.124] PASSED
main.py::test_passwd_file[ansible://54.67.33.158] PASSED
main.py::test_surrogate_file[ansible://54.67.33.158] PASSED
main.py::test_passwd_file[ansible://52.9.29.73] PASSED
main.py::test_surrogate_file[ansible://52.9.29.73] PASSED
main.py::test_passwd_file[ansible://54.67.6.175] PASSED
main.py::test_surrogate_file[ansible://54.67.6.175] PASSED
main.py::test_passwd_file[ansible://54.67.81.102] PASSED
main.py::test_surrogate_file[ansible://54.67.81.102] PASSED
============================================= 10 passed in 54.28 seconds ==============================================
(aws)~/workspaces/emayssat/devtools-vpc/devops/aws/release/tests
This may be usefull for CI / build environment
When I run testinfra, I specify the test markers on the command line to run only a subsets of the tests.
Now I used to have --platform aws
as a command line option
and use pytest,mark,platform('aws') as a marker.
Now I are passing a configuration file to testinfra and therefore not specifying the --platform option anymore (since its value is in the config file).
Now, when I run testinfra without markers the AWS tests are run, when I use markers the test aren't run!
Is there a way to specify "platform('aws')" in the marker logic.
It seems that markers should just be simple words to works
Example command line:
AWS_PROFILE=3742 AWS_REGION=us-west-1 testinfra --capture=fd --connection=paramiko --sudo [email protected] -m "common or kafka or zookeeper" --maxfail 998 --tb=long --configuration-file configs/15_at_pre_deployment.yml test_suite/
In short, I would like to add platform('aws') in the -m option.
Is there a way to do so?
Thanks,
To build my inventory on ansible, I use
AWS_PROFILE='my_aws_profile' ansible -i inventories/ec2.py .....
To deploy on another AWS account, I simply change the AWS_PROFILE variable.
Now, does your ansible backend recognize environment variables?
At the present time, it doesn't seem to be the case.
In other words, your solution works only for the default AWS account.
Emmanuel
In Ansible 2.x the Python API has changed, the relevant docs are here.
A check is made in L21-27 of ansible.py to try and import ansible.runner
and an variable HAS_ANSIBLE
is set to True or False as a result.
try:
import ansible.inventory
import ansible.runner
except ImportError:
HAS_ANSIBLE = False
else:
HAS_ANSIBLE = True
The module ansible.runner
has been removed from Ansible 2.x (a PITA I know).
Later a check is made on this variable, if it is set to False the program bails out with an error 'You must install ansible package to use the ansible backend'.
Please make it possible to run with Python 3.5
I have use case where i will be creating new machines and setting up different users for access. So is there any way i can provide different username and passwords using testinfra.
AWS_PROFILE=7009 REGION=us-west-2 testinfra --capture=fd --connection=paramiko --sudo --hosts= -m "common or safeview or master" --maxfail 999 --tb=long --configuration-file configs/700995487849/Release/001e/config.yml --deployment-id default test_suite/
A command like the one above generates a lot of output when the problem is only due to the --host parameter is not set correctly. A simple validation check on the input parameter could make the troubleshooting easier and more user friendly.
Therefore I recommend you display a simple error instead of running every test and reporting their failure.
testinfra -v --connection=ansible --ansible-inventory=inventories/ec2.py --hosts=key_Release001 main.py
The command above seems to work except
1/ My entire inventory is dynamic
2/ SSH agent configuration is working and tested
3/ How can I specify the user to ssh on the remote box?
In my ansible setup, I do it with a directive in the playbook (remote_user). My local user is emayssat, my remote user is ubuntu.
Maybe should I use ansible.cfg? Other?
I would be nice to implement custom backend outside of testinfra.
I'm thinking about an implementation like
testinfra --connection=custom_module.CustomBackendClass
testinfra will just dynamically import the external module here.
Hello,
As a python aficionado, I am seriously considering using testinfra to test my production Ansible/AWS deployments. I noticed that the latest version of testinfra is an alpha release. What does that mean?
I understand to date you only support linux platforms and not Windows nor OS X.
Not a problem for us as we run everything on linux (ubuntu 14.04 at the moment).
Input/advice appreciated.
Emmanuel
Having issues using Interface
.
[jodewey:~/git … e-systems/deploy/metapod/vagrant_lite] CIA-232(+5/-5)+ 1 ± pip freeze | grep testinfra
testinfra==1.0.0.0a17
__________________________________________ test_has_internal_vip[ansible://mcp-rhel-7] ___________________________________________
Interface = <interface None>
def test_has_internal_vip(Interface):
> interface = Interface('eth1')
E TypeError: 'LinuxInterface' object is not callable
This is a feature request.
I'd like to be able to use a (dynamic) ansible inventory in combination with --hosts
to determine the target hosts, but just use an ssh connection instead of the ansible connection backend to actually perform the tests against the target hosts (mainly because performing the tests using ansible is much, much slower).
A more generic solution to this would be the ability to specify a custom generic *.py file that provides the hosts to test_infra (the --hosts parameter would be passed to this file). This way, everyone can write their own integrations (very similar to how ansible does it).
Is it a bug in testinfra ? i can see it goes to minion correctly as salt-minion -ldebug
telling me, but i don't know why it says there is a missing argument
[INFO ] User root Executing command cmd.run_all with jid 20150803202851787339
[DEBUG ] Command details {'tgt_type': 'glob', 'jid': '20150803202851787339', 'tgt': 'minion01', 'ret': '', 'user': 'root', 'arg': ['uname -s'], 'fun': 'cmd.run_all'}
[INFO ] Starting a new job with PID 4805
[INFO ] Executing command 'uname -s' in directory '/root'
[DEBUG ] stdout: Linux
[INFO ] Returning information for job: 20150803202851787339
[DEBUG ] Re-using SAuth for ('/etc/salt/pki/minion', 'minion01', 'tcp://172.28.128.10:4506')
======================================================================= ERRORS =======================================================================
________________________________ ERROR at setup of test_saltstack_formula_standard[coreutils-viewer-formula-minion01] ________________________________
SystemInfo = <testinfra.modules.systeminfo.SystemInfo object at 0x2679650>
@pytest.fixture(scope="session")
def f(SystemInfo):
> if SystemInfo.type == "linux":
/usr/lib/python2.7/site-packages/testinfra/modules/file.py:167:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python2.7/site-packages/testinfra/modules/systeminfo.py:82: in type
return self.sysinfo["type"]
/usr/lib/python2.7/site-packages/testinfra/modules/systeminfo.py:33: in sysinfo
self._sysinfo = self.get_system_info()
/usr/lib/python2.7/site-packages/testinfra/modules/systeminfo.py:43: in get_system_info
sysinfo["type"] = self.check_output("uname -s").lower()
/usr/lib/python2.7/site-packages/testinfra/modules/base.py:26: in run
return get_backend().run(command, *args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <testinfra.backend.salt.SaltBackend object at 0x2679b90>, command = 'uname -s', args = ()
out = {'pid': 4738, 'retcode': 0, 'stderr': '', 'stdout': 'Linux'}
def run(self, command, *args):
if self.sudo:
command = "sudo " + command
command = self.quote(command, *args)
out = self.run_salt("cmd.run_all", [command])
return base.CommandResult(
> out['retcode'], out['stdout'], out['stderr'], command)
E TypeError: __init__() takes exactly 6 arguments (5 given)
/usr/lib/python2.7/site-packages/testinfra/backend/salt.py:53: TypeError
============================================================== 1 error in 0.16 seconds ===============================================================
Here is the salt command result if it helps you in fixing this:
[root@master ~]# salt minion01 cmd.run_all 'uname -s'
minion01:
----------
pid:
5443
retcode:
0
stderr:
stdout:
Linux
When both testinfra and pytest-ansible are installed, testinfra fails with the following output:
$ testinfra --connection=ansible --sudo --ansible-inventory=inventory test_proxy.py
Traceback (most recent call last):
File "/home/ivan/cde/devops/devops-env/bin/testinfra", line 11, in <module>
sys.exit(main())
File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/testinfra/main.py", line 100, in main
return pytest.main()
File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/config.py", line 38, in main
config = _prepareconfig(args, plugins)
File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/config.py", line 117, in _prepareconfig
pluginmanager=pluginmanager, args=args)
File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 724, in __call__
return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 338, in _hookexec
return self._inner_hookexec(hook, methods, kwargs)
File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 333, in <lambda>
_MultiCall(methods, kwargs, hook.spec_opts).execute()
File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 595, in execute
return _wrapped_call(hook_impl.function(*args), self.execute)
File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 249, in _wrapped_call
wrap_controller.send(call_outcome)
File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/helpconfig.py", line 28, in pytest_cmdline_parse
config = outcome.get_result()
File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 279, in get_result
_reraise(*ex) # noqa
File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 264, in __init__
self.result = func()
File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 596, in execute
res = hook_impl.function(*args)
File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/config.py", line 852, in pytest_cmdline_parse
self.parse(args)
File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/config.py", line 957, in parse
self._preparse(args)
File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/config.py", line 922, in _preparse
self.known_args_namespace = ns = self._parser.parse_known_args(args)
File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/config.py", line 490, in parse_known_args
return self.parse_known_and_unknown_args(args)[0]
File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/config.py", line 496, in parse_known_and_unknown_args
optparser = self._getparser()
File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/config.py", line 475, in _getparser
arggroup.add_argument(*n, **a)
File "/usr/lib64/python2.7/argparse.py", line 1308, in add_argument
return self._add_action(action)
File "/usr/lib64/python2.7/argparse.py", line 1509, in _add_action
action = super(_ArgumentGroup, self)._add_action(action)
File "/usr/lib64/python2.7/argparse.py", line 1322, in _add_action
self._check_conflict(action)
File "/usr/lib64/python2.7/argparse.py", line 1460, in _check_conflict
conflict_handler(action, confl_optionals)
File "/usr/lib64/python2.7/argparse.py", line 1467, in _handle_conflict_error
raise ArgumentError(action, message % conflict_string)
argparse.ArgumentError: argument --ansible-inventory: conflicting option string(s): --ansible-inventory
Removing --ansible-inventory
from the command line does not change the output. Uninstalling pytest-ansible and running testinfra fixes the issue, but leaves my tests which use it broken.
Contents of test_proxy.py
:
# Proxy servers must have nginx installed, running and enabled on boot
def test_nginx(Package, Service):
assert Package('nginx').is_installed
assert Service('nginx').is_running
assert Service('nginx').is_enabled
Environment:
Python 2.7.11
pip 7.1.2
ansible 1.9.4
configured module search path = None
virtualenv 13.1.2
testinfra 1.0.1
pytest-ansible 1.3.1
pytest 2.8.4
Hi,
trying to run a simple test via ansible:
testinfra --connection=ansible --ansible-inventory=/etc/ansible/hosts --hosts=test -v pakete.py
if run locally it works.
if run via ansible I always get following error:
RuntimeError: host unreachable: {'msg': u'SSH Error: data could not be sent to the remote host. Make sure this host can be reached over ssh', 'unreachable': True, 'changed': False}
if i try to run ansible on this host i get no error:
ansible test -m ping
XXXX | SUCCESS => {
"changed": false,
"ping": "pong"
}
ansible 2.0.1.0
config file = /etc/ansible/ansible.cfg
configured module search path = /etc/ansible/roles/
This is py.test version 2.5.1, imported from /usr/lib/python2.7/dist-packages/pytest.pyc
setuptools registered plugins:
testinfra-1.0.2 at /usr/local/lib/python2.7/dist-packages/testinfra/plugin.pyc
Thanks :-)
Is there a way to access ansible inventory inside the test itself?
I am putting together a set of tests that run across multiple environments. I would like some of the tests to be explicit about the domain or URL it is testing. This information is stored in ansible's inventory. The test below feels wrong (hostname -d
).
@pytest.mark.parametrize('host, scheme', [('api.pub.{}', 'https'), ('api.loc1.{}', 'http'),
])
def test_keystone_service_list(Command, host, scheme):
domain = Command('hostname -d').stdout
cmd = openstack('service list',
password='keystone_admin_password',
auth_host=host.format(domain),
auth_host_scheme=scheme)
out = Command.check_output(cmd)
assert re.search(r'image', out)
Ideally, I'd like to run this same test across multiple environments, swapping out the inventory.
@pytest.mark.parametrize('host, scheme', [
('api.pub.{}.format(VAR_FROM_INVENTORY)', 'https'),
('api.loc1.{}.format(VAR_FROM_INVENTORY)', 'http'),
])
...
Test setup with vagrant is kind of weak. Seems docker-compose can do the work (build & launch multiple containers).
On debian jessie and for legacy service (ie providing sysv script in /etc/init.d/). systemctl is-enabled doesn't work properly.
# systemctl is-enabled ntp; echo $?
Failed to get unit file state for ntp.service: No such file or directory
1
See:
Why can I not use the same filename for my test when in different directory?
I organized my tests based on groups and become more and more specific all the way to my host.
So reusing the same file name
BTW, I love this software ;-)
AWS_PROFILE=7009 testinfra --connection=paramiko [email protected] -m "common or pnr or proxy or aws" --maxfail 1 test_suite
================================================= test session starts =================================================
platform linux2 -- Python 2.7.6, pytest-2.8.5, py-1.4.31, pluggy-0.3.1
rootdir: /home/emayssat/workspaces/emayssat/devtools_ops276_vpc_pnr_image_update/devops/aws/release/testinfra, inifile: pytest.ini
plugins: testinfra-1.0.0.0a21
collecting 37 items / 1 errors
======================================================= ERRORS ========================================================
_________________________________ ERROR collecting test_suite/safeview/test_monit.py __________________________________
import file mismatch:
imported module 'test_monit' has this __file__ attribute:
/home/emayssat/workspaces/emayssat/devtools_ops276_vpc_pnr_image_update/devops/aws/release/testinfra/test_suite/common/test_monit.py
which is not the same as the test file we want to collect:
/home/emayssat/workspaces/emayssat/devtools_ops276_vpc_pnr_image_update/devops/aws/release/testinfra/test_suite/safeview/test_monit.py
HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules
I like the -k argument for testinfra.
I can run test based on the name of the test.
Nevertheless, I would like to be able to filter-out or select tests based on something else than the test name. It seems that the extra_keyword_matches is what I want.
Unfortunately, there is no doc on this.
Info in the docs would be appreciated instead of having to look at the source.
Best,
What is the level of interest for receiving pull requests to introduce new modules? For instance I have written a couple of modules for verifying Python and Ruby modules (e.g. is_installed
and version checking), and I am wondering if it would be desirable to include these directly in TestInfra or if they would be better served in an extension package?
Workaround http://pastebin.com/TqGfqVZ3
OHAI.
I have the current setup:a
* code #guess what
* code/playbooks #again, guess what.
* code/playbooks/group_vars/all # file I need facts from in my test cases
* code/tests # guess again
I use the following command to run my tests:
testinfra --connection ansible -n 5 --ssh-config ~/.ssh/config --ansible-inventory tests/hosts.txt
Now in one of my test cases I need a value of these group_vars file. It's rather large. And using ConfigParser to somehow get it's contents when really it's just ansible facts sounds wrong. Can this facts in there be somehow available from Ansible('setup')['ansible_facts']
?
Thanks in advance, this thing really helps!
I have a custom test script made with python , there are imported libraries and lots of code, also i am using salt client api inside my script, I am guessing when my script comes to a line like this it freeze or something caller = salt.client.Caller()
here is a simple script you can test with it:
#test_myinfra.py
import salt.client
def test_passwd_file(File):
passwd = File("/etc/passwd")
assert passwd.contains("root")
assert passwd.user == "root"
assert passwd.group == "root"
assert passwd.mode == 644
caller = salt.client.Caller()
state_data = []
assert type(state_data) == dict
i am using the following command
py.test test_myinfra.py --connection=salt --hosts=minion_name -v
and here is a screenshot to enusre that if i moved my script to minion and executed it by typing py.test test_myinfra.py
it works without problems , so the currently problem is when i use --connection=salt --hosts=minion_name
It will be great to make salt minion matching working with testinfra:
testinfra --connection=salt --hosts="web*"
testinfra --connection=salt --hosts="[email protected]/24"
etc
I wrote a proof of concept of an external package that contains testinfra based fixtures.
https://github.com/philpep/testinfra_echo
Need to document this. Also need to document custom fixture in module or in conftest file (eg inside the test suite).
We have some basic test that can run in travis. But we also have integration test that can run using docker and vagrant inside popular OS like debian, ubuntu, fedora, centos.
Theses tests can be easily run from the developper side (just need vagrant and docker), and my jenkins instance can run theses tests as well: https://jenkins.philpep.org/job/testinfra/
We need to write more of theses integration tests, test more backends and document the test procedure.
Right now the test fail when running on OS X since SystemInfo.type == "darwin"
is supported by the File
module.
I resolved this in #20 by doing a check for SystemInfo.type.endswith("bsd") or SystemInfo.type == "darwin"
. Hopefully it would be that easy to get OS X working in the other modules.
I have a pytest.ini
file that defines additional arguments to be passed to pytest-ansible. It makes testinfra
crash with the following output:
% testinfra -p no:pytest-ansible --connection=ansible --sudo --ansible-inventory=inventory --hosts=dev-proxy test_proxy.py
usage: testinfra [options] [file_or_dir] [file_or_dir] [...]
testinfra: error: unrecognized arguments: --ansible-host-pattern test_proxy.py
inifile: /home/ivan/cde/devops/ansible/pytest.ini
rootdir: /home/ivan/cde/devops/ansible
Contents of pytest.ini
:
[pytest]
addopts = --ansible-host-pattern localhost
My current workaround is to move the pytest.ini
file before running testinfra
with -no:pytest-ansible
.
I have installed testinfra-1.0.0.0a13 version successfully, but when i run sample test.. it gives me following error....
fixture 'File' not found
available fixtures: tmpdir_factory, pytestconfig, cache, recwarn, monkeypatch, record_xml_property, capfd, capsys, tmpdir
use 'py.test --fixtures [testpath]' for help on them.
platform linux2 -- Python 2.7.10, pytest-2.8.2, py-1.4.30, pluggy-0.3.1 -- /usr/bin/python2.7
Can some one help me...
I'd be nice to have a Process module
I am using testinfra extensively.
This is an awesome tool also for debugging.
Now I am deploying in europe while my desktop is in the california (west USA).
I am interested in using testinfra for support (i.e. quickly find out what the issue is)
But there is significant latency between EU and US, so if you run a lot of tests, it is going to take a lot of time.
I already use the multi-process testinfra (i.e. pytest-xdist, or testinfra -n 5), which somewhat address that issue. To go even further, I would like to disable/enable tests based on the result of others.
Is this possible?
The idea is to run only a small set of tests, if one fails, I run another series of test to troubleshoot even further the issue. That would be a killer feature if not implemented yet!
Best,
I'll be great to have a helper to read ansible variables.
I suggest:
>>> Ansible.get("myvar")
[{'my': 'complex'}, {'value': True}]
Also add a note on the docs to explain that testinfra loads ansible variable like ansible does (using the python API). The relevant environment variable that alter ansible variable loading (vars plugins) must be set when calling testinfra.
Hi,
Been wondering how we can achieve to test dockerfile instead of Docker Image like latest version of rspec or docker-unit?
This would be faster and more efficient in some ways I think.
Any clues maybe?
Howdy,
I just created a simple test to check for a few services running:
def test_passwd_file(File):
passwd = File("/etc/passwd")
assert passwd.contains("root")
assert passwd.user == "root"
def test_nimsoft(Service):
nimsoft = Service("nimbus")
assert nimsoft.is_running
def test_uxauthd(Service):
uxauthd = Service("uxauthd")
assert uxauthd.is_running
def test_sshd(Service):
sshd = Service("sshd")
assert sshd.is_running
If I attempt to check a host via ssh, I'm told the services are failed. But if I copy the file to the remote host, install testinfra, and run it locally, everything passes:
$ testinfra --connection=ssh [email protected] --sudo test_integrations.py
======================= test session starts =
platform linux -- Python 3.5.0+, pytest-2.8.3, py-1.4.31, pluggy-0.3.1
rootdir: /home/xian/work/validater, inifile:
plugins: testinfra-1.0.0.0a16
collected 4 items
test_integrations.py .FFF
================== FAILURES =======
_____________________________ test_nimsoft[ssh://evprappbpwl01.domain.name] ____
Service = <service None>
def test_nimsoft(Service):
nimsoft = Service("nimbus")
> assert nimsoft.is_running
E assert <service nimbus>.is_running
test_integrations.py:8: AssertionError
____________________ test_uxauthd[ssh://evprappbpwl01.domain.name]
Service = <service None>
def test_uxauthd(Service):
uxauthd = Service("uxauthd")
> assert uxauthd.is_running
E assert <service uxauthd>.is_running
test_integrations.py:12: AssertionError
_______________________ test_sshd[ssh://evprappbpwl01.domain.name]
Service = <service None>
def test_sshd(Service):
sshd = Service("sshd")
> assert sshd.is_running
E assert <service ssh>.is_running
test_integrations.py:16: AssertionError
========================= 3 failed, 1 passed in 2.51 seconds
Versus running it on the host
[hcladmin@evprappbpwl01 ~]$ testinfra --sudo test_integrations.py
=========== test session starts =============================
platform linux2 -- Python 2.6.6, pytest-2.8.3, py-1.4.31, pluggy-0.3.1
rootdir: /home/hcladmin, inifile:
plugins: testinfra-1.0.0.0a16
collected 4 items
test_integrations.py ....
======================= 4 passed in 0.78 seconds ===========
Thanks!
Hello I recently tried to use testinfra on Windows it did work. Would you be open to pull request to support windows? And if so are open do you have any opinions on what i would need to do to start? I have a decent python background and another developer willing to help.
This will allow to write something like
file = File("/var/log/messages")
savepoint = file.get_savepoint() # get a pointer to the end of the file
assert savepoint.content == ""
Command.check_output("logger hello world")
assert "logger: hello world" in savepoint.content # just match the content from the savepoint to the end of the file
Warning: Handle file rotation or file truncated (how ?)
There is apparently an issue with the File fixture when using it on directories
The issue exists for File.is_directory and File.exists
Here is my output
remote# file /etc/menlo/conf/certificates/SurfCrew/active
/etc/menlo/conf/certificates/SurfCrew/active: directory
But in test
certificate_dir = words[6]
d = File(certificate_dir)
> assert d.is_directory
E assert <file /etc/menlo/conf/certificates/SurfCrew/active>.is_directory
Consider this simple test:
def test_foo(Ansible):
Ansible("debug", "msg={{ foo }}")
WIth ansible v1 this produce an error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "testinfra/modules/ansible.py", line 48, in __call__
module_name, module_args, check=check, **kwargs)
File "testinfra/backend/ansible.py", line 60, in run_ansible
**kwargs)
File "testinfra/utils/ansible_runner.py", line 199, in run
host, module_name, module_args, **kwargs)
File "testinfra/utils/ansible_runner.py", line 85, in run
**kwargs).run()["contacted"][host]
KeyError: 'localhost'
Expected result is like ansible v2: {'msg': u"'foo' is undefined", 'failed': True}
I am getting the following error during test
salt_checker.py::test_saltstack_formula_standard[coreutils-viewer-formula-minion_001] ERROR
======================================================================= ERRORS =======================================================================
_______________________________ ERROR at setup of test_saltstack_formula_standard[coreutils-viewer-formula-minion_001] _______________________________
SystemInfo = <testinfra.modules.systeminfo.SystemInfo object at 0x284dc50>
@pytest.fixture(scope="session")
def f(SystemInfo):
> if SystemInfo.type == "linux":
/usr/lib/python2.6/site-packages/testinfra/modules/file.py:167:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/python2.6/site-packages/testinfra/modules/systeminfo.py:82: in type
return self.sysinfo["type"]
/usr/lib/python2.6/site-packages/testinfra/modules/systeminfo.py:33: in sysinfo
self._sysinfo = self.get_system_info()
/usr/lib/python2.6/site-packages/testinfra/modules/systeminfo.py:43: in get_system_info
sysinfo["type"] = self.check_output("uname -s").lower()
/usr/lib/python2.6/site-packages/testinfra/modules/base.py:26: in run
return get_backend().run(command, *args, **kwargs)
/usr/lib/python2.6/site-packages/testinfra/backend/salt.py:51: in run
out = self.run_salt("cmd.run_all", [command])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <testinfra.backend.salt.SaltBackend object at 0x284dc90>, func = 'cmd.run_all', args = ['uname -s']
def run_salt(self, func, args=None):
> return self.client.cmd(self.host, func, args or [])[self.host]
E KeyError: u'minion_001'
/usr/lib/python2.6/site-packages/testinfra/backend/salt.py:56: KeyError
while investigating this problem, i found that minions are returning Minion did not return. [No response]
I am suggesting to make testinfra
return the same if that is possible, instead of raising KeyError: u'minion_001'
so it would be easier to debug any further issues, what do you think ?
I would like to thank you for this great tool, i just saw it today and i am really impressed.
I have a question which i couldn't found it in the documentation. currently i have a py.test script and i would like to have a testinfra functionality inside my current script, for example:
I have added package is installed and service is running/enabled to my script, but when i run my script with py.test:
1- it couldn't found Service or Package.
2- Also i noticed that testinfra doesn't show in plugins section when i run the test.
3- I have a custom pytest.ini so i can add a custom option to use it when i run py.test , which is not working with testinfra
so if i could use testinfra as a plugin so i can make use of its functionality like saltstack integration and also use all the current custom options i have at py.test it would be nice.
Hi, how do we run the tests in test
? When i tried to run them using py.test
, I got errors such as:
fixture 'Command' not found
I understand the issue. Shouldn't we be importing these classes?
When running testinfra --hosts=testinfra --ssh-config=.vagrant/ssh-config testinfra/* --junit-xml junit.xml
I get the error UserWarning: Unknown ssh-rsa host key for [127.0.0.1]:2222: 987c6ae262c2748e47b9c20c4694485a
in junit.xml:
<?xml version="1.0" encoding="utf-8"?>
<testsuite errors="0" failures="0" name="pytest" skips="0" tests="2" time="1.006">
<testcase classname="test" file="test.py" line="0" name="test_hostname_file[testinfra]" time="0.887089014053">
<system-err>/home/muffl0n/IdeaProjects/ansible-testinfra/venv/local/lib/python2.7/site-packages/paramiko/client.py:580: UserWarning: Unknown ssh-rsa host key for [127.0.0.1]:2222: 987c6ae262c2748e47b9c20c4694485a
(key.get_name(), hostname, hexlify(key.get_fingerprint())))
</system-err>
</testcase>
<testcase classname="test" file="test.py" line="4" name="test_dummy_file[testinfra]" time="0.105942964554"/>
</testsuite>
The output is only shown in the file junit.xml
not in stdout:
======================================== test session starts =========================================
platform linux2 -- Python 2.7.9, pytest-2.8.2, py-1.4.30, pluggy-0.3.1
rootdir: /home/muffl0n/IdeaProjects/ansible-testinfra/testinfra, inifile:
plugins: testinfra-1.0.0.0a12
collected 2 items
testinfra/test.py ..
------------- generated xml file: /home/muffl0n/IdeaProjects/ansible-testinfra/junit.xml -------------
====================================== 2 passed in 1.00 seconds =====================================
.vagrant/ssh-config:
Host testinfra
HostName 127.0.0.1
User vagrant
Port 2222
UserKnownHostsFile /dev/null
StrictHostKeyChecking no
PasswordAuthentication no
IdentityFile /home/muffl0n/.vagrant.d/insecure_private_key
IdentitiesOnly yes
LogLevel FATAL
So it seems like StrictHostKeyChecking
is ignored when calling paramiko.
I installed testinfra from git with pip install 'git+https://github.com/philpep/testinfra@master#egg=testinfra'
Other dependencies:
argparse==1.2.1
ecdsa==0.13
paramiko==1.15.3
py==1.4.30
pycrypto==2.6.1
pytest==2.8.2
six==1.10.0
testinfra==1.0.0.0a12
wsgiref==0.1.2
I have set up an proof of concept for testing role provision with Ansible and Vagrant here: https://github.com/muffl0n/ansible-testinfra. You can see the error when running that poc.
Ex:
testinfra --ansible-inventory=/my/inventory --connection=ansible --hosts=host1,host2 tests
>>> Ansible("file", "path=/etc/passwd")
{u'group': u'root', u'uid': 0, u'changed': False, 'invocation': {'module_name': 'file', 'module_complex_args': {}, 'module_args': u'path=/etc/passwd'}, u'state': u'file', u'gid': 0, u'mode': u'0644', u'owner': u'root', u'path': u'/etc/passwd', u'size': 2590}
This can be done with ansible.runner.Runner
python API
There's an issue with SystemInfo module on CentOS/RedHat 6.
By default there is no lsb_release command available and you have to install it (redhat-lsb-core-4.1-27.el7.centos.1.i686).
INFO:testinfra:RUN CommandResult(command="sudo /bin/sh -c 'uname -s'", exit_status=0, stdout='Linux\n', stderr='')
INFO:testinfra:RUN CommandResult(command="sudo /bin/sh -c 'lsb_release -a'", exit_status=127, stdout='', stderr='/bin/sh: lsb_release: command not found\n')
INFO:testinfra:RUN CommandResult(command="sudo /bin/sh -c 'cat /etc/os-release'", exit_status=1, stdout='', stderr='cat: /etc/os-release: No such file or directory\n')
I guess this should fix the issue (testinfra/modules/systeminfo.py):
@@ -53,7 +53,7 @@ class SystemInfo(InstanceModule):
elif key == "codename":
sysinfo["codename"] = value
else:
- os_release = self.run("cat /etc/os-release")
+ os_release = self.run("cat /etc/os-release || cat /etc/redhat-release")
Cheers,
Stefan!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.