I am getting the following error during test

salt_checker.py::test_saltstack_formula_standard[coreutils-viewer-formula-minion_001] ERROR

======================================================================= ERRORS =======================================================================
_______________________________ ERROR at setup of test_saltstack_formula_standard[coreutils-viewer-formula-minion_001] _______________________________

SystemInfo = <testinfra.modules.systeminfo.SystemInfo object at 0x284dc50>

    @pytest.fixture(scope="session")
    def f(SystemInfo):
>       if SystemInfo.type == "linux":

/usr/lib/python2.6/site-packages/testinfra/modules/file.py:167: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib/python2.6/site-packages/testinfra/modules/systeminfo.py:82: in type
    return self.sysinfo["type"]
/usr/lib/python2.6/site-packages/testinfra/modules/systeminfo.py:33: in sysinfo
    self._sysinfo = self.get_system_info()
/usr/lib/python2.6/site-packages/testinfra/modules/systeminfo.py:43: in get_system_info
    sysinfo["type"] = self.check_output("uname -s").lower()
/usr/lib/python2.6/site-packages/testinfra/modules/base.py:26: in run
    return get_backend().run(command, *args, **kwargs)
/usr/lib/python2.6/site-packages/testinfra/backend/salt.py:51: in run
    out = self.run_salt("cmd.run_all", [command])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <testinfra.backend.salt.SaltBackend object at 0x284dc90>, func = 'cmd.run_all', args = ['uname -s']

    def run_salt(self, func, args=None):
>       return self.client.cmd(self.host, func, args or [])[self.host]
E       KeyError: u'minion_001'

/usr/lib/python2.6/site-packages/testinfra/backend/salt.py:56: KeyError

while investigating this problem, i found that minions are returning Minion did not return. [No response]

I am suggesting to make testinfra return the same if that is possible, instead of raising KeyError: u'minion_001' so it would be easier to debug any further issues, what do you think ?

When both testinfra and pytest-ansible are installed, testinfra fails with the following output:

$ testinfra --connection=ansible --sudo --ansible-inventory=inventory test_proxy.py
Traceback (most recent call last):
  File "/home/ivan/cde/devops/devops-env/bin/testinfra", line 11, in <module>
    sys.exit(main())
  File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/testinfra/main.py", line 100, in main
    return pytest.main()
  File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/config.py", line 38, in main
    config = _prepareconfig(args, plugins)
  File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/config.py", line 117, in _prepareconfig
    pluginmanager=pluginmanager, args=args)
  File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 724, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 338, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 333, in <lambda>
    _MultiCall(methods, kwargs, hook.spec_opts).execute()
  File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 595, in execute
    return _wrapped_call(hook_impl.function(*args), self.execute)
  File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 249, in _wrapped_call
    wrap_controller.send(call_outcome)
  File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/helpconfig.py", line 28, in pytest_cmdline_parse
    config = outcome.get_result()
  File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 279, in get_result
    _reraise(*ex)  # noqa
  File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 264, in __init__
    self.result = func()
  File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 596, in execute
    res = hook_impl.function(*args)
  File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/config.py", line 852, in pytest_cmdline_parse
    self.parse(args)
  File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/config.py", line 957, in parse
    self._preparse(args)
  File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/config.py", line 922, in _preparse
    self.known_args_namespace = ns = self._parser.parse_known_args(args)
  File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/config.py", line 490, in parse_known_args
    return self.parse_known_and_unknown_args(args)[0]
  File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/config.py", line 496, in parse_known_and_unknown_args
    optparser = self._getparser()
  File "/home/ivan/cde/devops/devops-env/lib/python2.7/site-packages/_pytest/config.py", line 475, in _getparser
    arggroup.add_argument(*n, **a)
  File "/usr/lib64/python2.7/argparse.py", line 1308, in add_argument
    return self._add_action(action)
  File "/usr/lib64/python2.7/argparse.py", line 1509, in _add_action
    action = super(_ArgumentGroup, self)._add_action(action)
  File "/usr/lib64/python2.7/argparse.py", line 1322, in _add_action
    self._check_conflict(action)
  File "/usr/lib64/python2.7/argparse.py", line 1460, in _check_conflict
    conflict_handler(action, confl_optionals)
  File "/usr/lib64/python2.7/argparse.py", line 1467, in _handle_conflict_error
    raise ArgumentError(action, message % conflict_string)
argparse.ArgumentError: argument --ansible-inventory: conflicting option string(s): --ansible-inventory

Removing --ansible-inventory from the command line does not change the output. Uninstalling pytest-ansible and running testinfra fixes the issue, but leaves my tests which use it broken.

Contents of test_proxy.py:

# Proxy servers must have nginx installed, running and enabled on boot
def test_nginx(Package, Service):
    assert Package('nginx').is_installed
    assert Service('nginx').is_running
    assert Service('nginx').is_enabled

Environment:
Python 2.7.11
pip 7.1.2
ansible 1.9.4
configured module search path = None
virtualenv 13.1.2
testinfra 1.0.1
pytest-ansible 1.3.1
pytest 2.8.4

OHAI.
I have the current setup:a

* code #guess what
* code/playbooks #again, guess what.
* code/playbooks/group_vars/all # file I need facts from in my test cases
* code/tests # guess again

I use the following command to run my tests:

testinfra --connection ansible -n 5 --ssh-config ~/.ssh/config --ansible-inventory tests/hosts.txt

Now in one of my test cases I need a value of these group_vars file. It's rather large. And using ConfigParser to somehow get it's contents when really it's just ansible facts sounds wrong. Can this facts in there be somehow available from Ansible('setup')['ansible_facts'] ?

Thanks in advance, this thing really helps!

I have a pytest.ini file that defines additional arguments to be passed to pytest-ansible. It makes testinfra crash with the following output:

% testinfra -p no:pytest-ansible --connection=ansible --sudo --ansible-inventory=inventory --hosts=dev-proxy test_proxy.py
usage: testinfra [options] [file_or_dir] [file_or_dir] [...]
testinfra: error: unrecognized arguments: --ansible-host-pattern test_proxy.py
  inifile: /home/ivan/cde/devops/ansible/pytest.ini
  rootdir: /home/ivan/cde/devops/ansible

Contents of pytest.ini:

[pytest]
addopts = --ansible-host-pattern localhost

My current workaround is to move the pytest.ini file before running testinfra with -no:pytest-ansible.

I'll be great to have a helper to read ansible variables.

I suggest:

>>> Ansible.get("myvar")
[{'my': 'complex'}, {'value': True}]

Also add a note on the docs to explain that testinfra loads ansible variable like ansible does (using the python API). The relevant environment variable that alter ansible variable loading (vars plugins) must be set when calling testinfra.

This will allow to write something like

file = File("/var/log/messages")
savepoint = file.get_savepoint()  # get a pointer to the end of the file
assert savepoint.content == ""
Command.check_output("logger hello world")
assert "logger: hello world" in savepoint.content # just match the content from the savepoint to the end of the file

Warning: Handle file rotation or file truncated (how ?)

On debian jessie and for legacy service (ie providing sysv script in /etc/init.d/). systemctl is-enabled doesn't work properly.

# systemctl is-enabled ntp; echo $?
Failed to get unit file state for ntp.service: No such file or directory
1

See:

• https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=760616
• https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=705254
• ansible/ansible-modules-core#1905

I wrote a proof of concept of an external package that contains testinfra based fixtures.

https://github.com/philpep/testinfra_echo

Need to document this. Also need to document custom fixture in module or in conftest file (eg inside the test suite).

Hi,

is it possible to specify a inventory file relative to some tests?

For example i have the following structure:

├── hosts
│   ├── databases
│   │   ├── inventory.ini
│   │   ├── mysql_test.py
│   └── webservers
│       ├── apache_test.py
│       ├── inventory.ini

I can run tests for a specific directory by using testinfra --connection=ansible --ansible-inventory hosts/webservers/inventory hosts/webservers. Thats fine.

But i want to run tests on all directories in one run. So it would be great if the inventory would be detected relative to the test files. So i would just call testinfra --connection=ansible hosts

I tried around with relative parameters, but i can not get it working:
testinfra --connection=ansible --ansible-inventory inventory.ini

I always get this error report, which seams to be the error when no inventory file is found.:

[gw0] darwin -- Python 2.7.10 /usr/bin/python
testinfra_backend = <object object at 0x1046ad0d0>

    @pytest.fixture(scope="module")
    def f(testinfra_backend):
>       return testinfra_backend.get_module(cls.__name__)
E       AttributeError: 'object' object has no attribute 'get_module'

/Library/Python/2.7/site-packages/testinfra/modules/base.py:77: AttributeError

Maybe there is a possibility to do it via an ansible.cfg, a marker or a special notation in the --ansible-inventory switch, or something like this?

Hope you have some ideas on this.

Ps: Great Lib, i really love it!

Hi,

trying to run a simple test via ansible:
testinfra --connection=ansible --ansible-inventory=/etc/ansible/hosts --hosts=test -v pakete.py
if run locally it works.
if run via ansible I always get following error:
RuntimeError: host unreachable: {'msg': u'SSH Error: data could not be sent to the remote host. Make sure this host can be reached over ssh', 'unreachable': True, 'changed': False}
if i try to run ansible on this host i get no error:
ansible test -m ping
XXXX | SUCCESS => {
"changed": false,
"ping": "pong"
}

ansible 2.0.1.0
config file = /etc/ansible/ansible.cfg
configured module search path = /etc/ansible/roles/

This is py.test version 2.5.1, imported from /usr/lib/python2.7/dist-packages/pytest.pyc
setuptools registered plugins:
testinfra-1.0.2 at /usr/local/lib/python2.7/dist-packages/testinfra/plugin.pyc

Thanks :-)

I have installed testinfra-1.0.0.0a13 version successfully, but when i run sample test.. it gives me following error....

fixture 'File' not found
available fixtures: tmpdir_factory, pytestconfig, cache, recwarn, monkeypatch, record_xml_property, capfd, capsys, tmpdir
use 'py.test --fixtures [testpath]' for help on them.

platform linux2 -- Python 2.7.10, pytest-2.8.2, py-1.4.30, pluggy-0.3.1 -- /usr/bin/python2.7

Can some one help me...

I am using testinfra extensively.
This is an awesome tool also for debugging.

Now I am deploying in europe while my desktop is in the california (west USA).
I am interested in using testinfra for support (i.e. quickly find out what the issue is)
But there is significant latency between EU and US, so if you run a lot of tests, it is going to take a lot of time.

I already use the multi-process testinfra (i.e. pytest-xdist, or testinfra -n 5), which somewhat address that issue. To go even further, I would like to disable/enable tests based on the result of others.
Is this possible?

The idea is to run only a small set of tests, if one fails, I run another series of test to troubleshoot even further the issue. That would be a killer feature if not implemented yet!

Best,

There is apparently an issue with the File fixture when using it on directories
The issue exists for File.is_directory and File.exists
Here is my output

remote# file /etc/menlo/conf/certificates/SurfCrew/active
/etc/menlo/conf/certificates/SurfCrew/active: directory

But in test

certificate_dir = words[6]
 d = File(certificate_dir)
>       assert d.is_directory
E       assert <file /etc/menlo/conf/certificates/SurfCrew/active>.is_directory

What is the level of interest for receiving pull requests to introduce new modules? For instance I have written a couple of modules for verifying Python and Ruby modules (e.g. is_installed and version checking), and I am wondering if it would be desirable to include these directly in TestInfra or if they would be better served in an extension package?

When I run testinfra, I specify the test markers on the command line to run only a subsets of the tests.

Now I used to have --platform aws
as a command line option
and use pytest,mark,platform('aws') as a marker.

Now I are passing a configuration file to testinfra and therefore not specifying the --platform option anymore (since its value is in the config file).

Now, when I run testinfra without markers the AWS tests are run, when I use markers the test aren't run!
Is there a way to specify "platform('aws')" in the marker logic.
It seems that markers should just be simple words to works

Example command line:
AWS_PROFILE=3742 AWS_REGION=us-west-1 testinfra --capture=fd --connection=paramiko --sudo [email protected] -m "common or kafka or zookeeper" --maxfail 998 --tb=long --configuration-file configs/15_at_pre_deployment.yml test_suite/

In short, I would like to add platform('aws') in the -m option.
Is there a way to do so?

Thanks,

Hi,

is there a possibility to filter ansible hosts by host pattern? I know the way to filter by name via --hosts=web*, but this doesn't work for groups in ansible, only for hostnames. For example I can run a ad hoc command like ansible -m ping -i inventory groupname to just contact hosts in the group. This is a pretty cool feature of ansible.

Something like this would be great: https://github.com/jlaska/pytest-ansible#parameterizing-with-pytestmarkansible. A marker to limit a test to a specific host pattern. So you can have a big test suite, but apply a test for mysql stuff on database hosts only.

What do you think, is this possible? Maybe even now and I don't get it?

AWS_PROFILE=7009 REGION=us-west-2 testinfra --capture=fd --connection=paramiko --sudo --hosts= -m "common or safeview or master" --maxfail 999 --tb=long --configuration-file configs/700995487849/Release/001e/config.yml --deployment-id default test_suite/

A command like the one above generates a lot of output when the problem is only due to the --host parameter is not set correctly. A simple validation check on the input parameter could make the troubleshooting easier and more user friendly.

Therefore I recommend you display a simple error instead of running every test and reporting their failure.

Why can I not use the same filename for my test when in different directory?
I organized my tests based on groups and become more and more specific all the way to my host.
So reusing the same file name

BTW, I love this software ;-)

AWS_PROFILE=7009 testinfra  --connection=paramiko [email protected]  -m "common or pnr or proxy or aws" --maxfail 1 test_suite
================================================= test session starts =================================================
platform linux2 -- Python 2.7.6, pytest-2.8.5, py-1.4.31, pluggy-0.3.1
rootdir: /home/emayssat/workspaces/emayssat/devtools_ops276_vpc_pnr_image_update/devops/aws/release/testinfra, inifile: pytest.ini
plugins: testinfra-1.0.0.0a21
collecting 37 items / 1 errors
======================================================= ERRORS ========================================================
_________________________________ ERROR collecting test_suite/safeview/test_monit.py __________________________________
import file mismatch:
imported module 'test_monit' has this __file__ attribute:
  /home/emayssat/workspaces/emayssat/devtools_ops276_vpc_pnr_image_update/devops/aws/release/testinfra/test_suite/common/test_monit.py
which is not the same as the test file we want to collect:
  /home/emayssat/workspaces/emayssat/devtools_ops276_vpc_pnr_image_update/devops/aws/release/testinfra/test_suite/safeview/test_monit.py
HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules

I have use case where i will be creating new machines and setting up different users for access. So is there any way i can provide different username and passwords using testinfra.

I would be nice to implement custom backend outside of testinfra.

I'm thinking about an implementation like

testinfra --connection=custom_module.CustomBackendClass

testinfra will just dynamically import the external module here.

There's an issue with SystemInfo module on CentOS/RedHat 6.
By default there is no lsb_release command available and you have to install it (redhat-lsb-core-4.1-27.el7.centos.1.i686).

INFO:testinfra:RUN CommandResult(command="sudo /bin/sh -c 'uname -s'", exit_status=0, stdout='Linux\n', stderr='')
INFO:testinfra:RUN CommandResult(command="sudo /bin/sh -c 'lsb_release -a'", exit_status=127, stdout='', stderr='/bin/sh: lsb_release: command not found\n')
INFO:testinfra:RUN CommandResult(command="sudo /bin/sh -c 'cat /etc/os-release'", exit_status=1, stdout='', stderr='cat: /etc/os-release: No such file or directory\n')

I guess this should fix the issue (testinfra/modules/systeminfo.py):

@@ -53,7 +53,7 @@ class SystemInfo(InstanceModule):
                     elif key == "codename":
                         sysinfo["codename"] = value
             else:
-                os_release = self.run("cat /etc/os-release")
+                os_release = self.run("cat /etc/os-release || cat /etc/redhat-release")

Cheers,
Stefan!

Right now the test fail when running on OS X since SystemInfo.type == "darwin" is supported by the File module.

I resolved this in #20 by doing a check for SystemInfo.type.endswith("bsd") or SystemInfo.type == "darwin". Hopefully it would be that easy to get OS X working in the other modules.

Hello,

As a python aficionado, I am seriously considering using testinfra to test my production Ansible/AWS deployments. I noticed that the latest version of testinfra is an alpha release. What does that mean?

I understand to date you only support linux platforms and not Windows nor OS X.
Not a problem for us as we run everything on linux (ubuntu 14.04 at the moment).

Input/advice appreciated.

Best,

Emmanuel

Having issues using Interface.

[jodewey:~/git … e-systems/deploy/metapod/vagrant_lite] CIA-232(+5/-5)+ 1 ± pip freeze | grep testinfra
testinfra==1.0.0.0a17

__________________________________________ test_has_internal_vip[ansible://mcp-rhel-7] ___________________________________________

Interface = <interface None>

    def test_has_internal_vip(Interface):
>       interface = Interface('eth1')
E       TypeError: 'LinuxInterface' object is not callable

This may be usefull for CI / build environment

It will be great to make salt minion matching working with testinfra:

testinfra --connection=salt --hosts="web*"
testinfra --connection=salt --hosts="[email protected]/24"
etc

When running testinfra --hosts=testinfra --ssh-config=.vagrant/ssh-config testinfra/* --junit-xml junit.xml I get the error UserWarning: Unknown ssh-rsa host key for [127.0.0.1]:2222: 987c6ae262c2748e47b9c20c4694485a in junit.xml:

<?xml version="1.0" encoding="utf-8"?>
<testsuite errors="0" failures="0" name="pytest" skips="0" tests="2" time="1.006">
    <testcase classname="test" file="test.py" line="0" name="test_hostname_file[testinfra]" time="0.887089014053">
        <system-err>/home/muffl0n/IdeaProjects/ansible-testinfra/venv/local/lib/python2.7/site-packages/paramiko/client.py:580: UserWarning: Unknown ssh-rsa host key for [127.0.0.1]:2222: 987c6ae262c2748e47b9c20c4694485a
            (key.get_name(), hostname, hexlify(key.get_fingerprint())))
        </system-err>
    </testcase>
    <testcase classname="test" file="test.py" line="4" name="test_dummy_file[testinfra]" time="0.105942964554"/>
</testsuite>

The output is only shown in the file junit.xml not in stdout:

======================================== test session starts =========================================
platform linux2 -- Python 2.7.9, pytest-2.8.2, py-1.4.30, pluggy-0.3.1
rootdir: /home/muffl0n/IdeaProjects/ansible-testinfra/testinfra, inifile: 
plugins: testinfra-1.0.0.0a12
collected 2 items 

testinfra/test.py ..

------------- generated xml file: /home/muffl0n/IdeaProjects/ansible-testinfra/junit.xml -------------
====================================== 2 passed in 1.00 seconds =====================================

.vagrant/ssh-config:

Host testinfra
  HostName 127.0.0.1
  User vagrant
  Port 2222
  UserKnownHostsFile /dev/null
  StrictHostKeyChecking no
  PasswordAuthentication no
  IdentityFile /home/muffl0n/.vagrant.d/insecure_private_key
  IdentitiesOnly yes
  LogLevel FATAL

So it seems like StrictHostKeyChecking is ignored when calling paramiko.

I installed testinfra from git with pip install 'git+https://github.com/philpep/testinfra@master#egg=testinfra'
Other dependencies:

argparse==1.2.1
ecdsa==0.13
paramiko==1.15.3
py==1.4.30
pycrypto==2.6.1
pytest==2.8.2
six==1.10.0
testinfra==1.0.0.0a12
wsgiref==0.1.2

I have set up an proof of concept for testing role provision with Ansible and Vagrant here: https://github.com/muffl0n/ansible-testinfra. You can see the error when running that poc.

Hi,
Been wondering how we can achieve to test dockerfile instead of Docker Image like latest version of rspec or docker-unit?

This would be faster and more efficient in some ways I think.

Any clues maybe?

I would like to thank you for this great tool, i just saw it today and i am really impressed.

I have a question which i couldn't found it in the documentation. currently i have a py.test script and i would like to have a testinfra functionality inside my current script, for example:

I have added package is installed and service is running/enabled to my script, but when i run my script with py.test:
1- it couldn't found Service or Package.
2- Also i noticed that testinfra doesn't show in plugins section when i run the test.
3- I have a custom pytest.ini so i can add a custom option to use it when i run py.test , which is not working with testinfra

so if i could use testinfra as a plugin so i can make use of its functionality like saltstack integration and also use all the current custom options i have at py.test it would be nice.

Workaround http://pastebin.com/TqGfqVZ3

For some backends (eg docker backend) it would be nice to have setup and teardown phases. For now container creation and removal processes are held in run method, which is not optimal.
It would be preferable to have setup process during which docker container could be created and tear down process during which its removal could be executed. It would really quicken testing.

Consider this simple test:

def test_foo(Ansible):
    Ansible("debug", "msg={{ foo }}")

WIth ansible v1 this produce an error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "testinfra/modules/ansible.py", line 48, in __call__
    module_name, module_args, check=check, **kwargs)
  File "testinfra/backend/ansible.py", line 60, in run_ansible
    **kwargs)
  File "testinfra/utils/ansible_runner.py", line 199, in run
    host, module_name, module_args, **kwargs)
  File "testinfra/utils/ansible_runner.py", line 85, in run
    **kwargs).run()["contacted"][host]
KeyError: 'localhost'

Expected result is like ansible v2: {'msg': u"'foo' is undefined", 'failed': True}

I like the -k argument for testinfra.
I can run test based on the name of the test.
Nevertheless, I would like to be able to filter-out or select tests based on something else than the test name. It seems that the extra_keyword_matches is what I want.
Unfortunately, there is no doc on this.
Info in the docs would be appreciated instead of having to look at the source.

Best,

Is there a way to access ansible inventory inside the test itself?

I am putting together a set of tests that run across multiple environments. I would like some of the tests to be explicit about the domain or URL it is testing. This information is stored in ansible's inventory. The test below feels wrong (hostname -d).

@pytest.mark.parametrize('host, scheme', [('api.pub.{}', 'https'), ('api.loc1.{}', 'http'),
])
def test_keystone_service_list(Command, host, scheme):
    domain = Command('hostname -d').stdout
    cmd = openstack('service list',
    password='keystone_admin_password',
    auth_host=host.format(domain),
    auth_host_scheme=scheme)
    out = Command.check_output(cmd)

    assert re.search(r'image', out)

Ideally, I'd like to run this same test across multiple environments, swapping out the inventory.

@pytest.mark.parametrize('host, scheme', [
    ('api.pub.{}.format(VAR_FROM_INVENTORY)', 'https'),
    ('api.loc1.{}.format(VAR_FROM_INVENTORY)', 'http'),
])
...

I'd be nice to have a Process module

• process lookup (by pid, name, user etc)
• user (integration with user module)
• pid
• cpu/mem usage
• process arguments
• sockets (integration with socket module)

We have some basic test that can run in travis. But we also have integration test that can run using docker and vagrant inside popular OS like debian, ubuntu, fedora, centos.

Theses tests can be easily run from the developper side (just need vagrant and docker), and my jenkins instance can run theses tests as well: https://jenkins.philpep.org/job/testinfra/

We need to write more of theses integration tests, test more backends and document the test procedure.

Hello I recently tried to use testinfra on Windows it did work. Would you be open to pull request to support windows? And if so are open do you have any opinions on what i would need to do to start? I have a decent python background and another developer willing to help.

To build my inventory on ansible, I use

AWS_PROFILE='my_aws_profile' ansible -i inventories/ec2.py .....

To deploy on another AWS account, I simply change the AWS_PROFILE variable.
Now, does your ansible backend recognize environment variables?
At the present time, it doesn't seem to be the case.
In other words, your solution works only for the default AWS account.

Regards,

Emmanuel

Ex:

testinfra --ansible-inventory=/my/inventory --connection=ansible --hosts=host1,host2 tests

>>> Ansible("file", "path=/etc/passwd")
{u'group': u'root', u'uid': 0, u'changed': False, 'invocation': {'module_name': 'file', 'module_complex_args': {}, 'module_args': u'path=/etc/passwd'}, u'state': u'file', u'gid': 0, u'mode': u'0644', u'owner': u'root', u'path': u'/etc/passwd', u'size': 2590}

This can be done with ansible.runner.Runner python API

Test setup with vagrant is kind of weak. Seems docker-compose can do the work (build & launch multiple containers).

In Ansible 2.x the Python API has changed, the relevant docs are here.

A check is made in L21-27 of ansible.py to try and import ansible.runner and an variable HAS_ANSIBLE is set to True or False as a result.

try:
    import ansible.inventory
    import ansible.runner
except ImportError:
    HAS_ANSIBLE = False
else:
    HAS_ANSIBLE = True

The module ansible.runner has been removed from Ansible 2.x (a PITA I know).

Later a check is made on this variable, if it is set to False the program bails out with an error 'You must install ansible package to use the ansible backend'.

Please make it possible to run with Python 3.5

Here is the performance I see for 2 simple tests on different backends.
It seems that ansible is much, much, much slower !
Admittedly ansible needs more time to update its inventory, but the task themselves are slow as if it were connecting to the machine at every single test.

Here is some sample output:
1/ test with IP provided
2/ test with ansible

<!> Also output is not the same for some unknown reason

$ testinfra [email protected],[email protected],[email protected],[email protected],[email protected] main.py
================================================= test session starts =================================================
platform linux2 -- Python 2.7.6, pytest-2.8.5, py-1.4.31, pluggy-0.3.1
rootdir: /home/emayssat/workspaces/emayssat/devtools-vpc/devops/aws/release/tests, inifile:
plugins: testinfra-1.0.0.0a21
collected 10 items

main.py ..........

============================================== 10 passed in 7.62 seconds ==============================================
(aws)~/workspaces/emayssat/devtools-vpc/devops/aws/release/tests
emayssat@xps13 $ make execute_tests
AWS_PROFILE=7009 testinfra -v --connection=ansible --ansible-inventory=inventories/ec2_7009 --hosts=key_Release001 main.py
================================================= test session starts =================================================
platform linux2 -- Python 2.7.6, pytest-2.8.5, py-1.4.31, pluggy-0.3.1 -- /home/emayssat/.virtualenvs/aws/bin/python
cachedir: .cache
rootdir: /home/emayssat/workspaces/emayssat/devtools-vpc/devops/aws/release/tests, inifile:
plugins: testinfra-1.0.0.0a21
collected 10 items

main.py::test_passwd_file[ansible://52.8.21.124] PASSED
main.py::test_surrogate_file[ansible://52.8.21.124] PASSED
main.py::test_passwd_file[ansible://54.67.33.158] PASSED
main.py::test_surrogate_file[ansible://54.67.33.158] PASSED
main.py::test_passwd_file[ansible://52.9.29.73] PASSED
main.py::test_surrogate_file[ansible://52.9.29.73] PASSED
main.py::test_passwd_file[ansible://54.67.6.175] PASSED
main.py::test_surrogate_file[ansible://54.67.6.175] PASSED
main.py::test_passwd_file[ansible://54.67.81.102] PASSED
main.py::test_surrogate_file[ansible://54.67.81.102] PASSED

============================================= 10 passed in 82.07 seconds ==============================================
(aws)~/workspaces/emayssat/devtools-vpc/devops/aws/release/tests
emayssat@xps13 $ make execute_tests
AWS_PROFILE=7009 testinfra -v --connection=ansible --ansible-inventory=inventories/ec2_7009 --hosts=key_Release001 main.py
================================================= test session starts =================================================
platform linux2 -- Python 2.7.6, pytest-2.8.5, py-1.4.31, pluggy-0.3.1 -- /home/emayssat/.virtualenvs/aws/bin/python
cachedir: .cache
rootdir: /home/emayssat/workspaces/emayssat/devtools-vpc/devops/aws/release/tests, inifile:
plugins: testinfra-1.0.0.0a21
collected 10 items

main.py::test_passwd_file[ansible://52.8.21.124] PASSED
main.py::test_surrogate_file[ansible://52.8.21.124] PASSED
main.py::test_passwd_file[ansible://54.67.33.158] PASSED
main.py::test_surrogate_file[ansible://54.67.33.158] PASSED
main.py::test_passwd_file[ansible://52.9.29.73] PASSED
main.py::test_surrogate_file[ansible://52.9.29.73] PASSED
main.py::test_passwd_file[ansible://54.67.6.175] PASSED
main.py::test_surrogate_file[ansible://54.67.6.175] PASSED
main.py::test_passwd_file[ansible://54.67.81.102] PASSED
main.py::test_surrogate_file[ansible://54.67.81.102] PASSED

============================================= 10 passed in 47.62 seconds ==============================================
(aws)~/workspaces/emayssat/devtools-vpc/devops/aws/release/tests
emayssat@xps13 $ make execute_tests
AWS_PROFILE=7009 testinfra -v --connection=ansible --ansible-inventory=inventories/ec2_7009 --hosts=key_Release001 main.py
================================================= test session starts =================================================
platform linux2 -- Python 2.7.6, pytest-2.8.5, py-1.4.31, pluggy-0.3.1 -- /home/emayssat/.virtualenvs/aws/bin/python
cachedir: .cache
rootdir: /home/emayssat/workspaces/emayssat/devtools-vpc/devops/aws/release/tests, inifile:
plugins: testinfra-1.0.0.0a21
collected 10 items

main.py::test_passwd_file[ansible://52.8.21.124] PASSED
main.py::test_surrogate_file[ansible://52.8.21.124] PASSED
main.py::test_passwd_file[ansible://54.67.33.158] PASSED
main.py::test_surrogate_file[ansible://54.67.33.158] PASSED
main.py::test_passwd_file[ansible://52.9.29.73] PASSED
main.py::test_surrogate_file[ansible://52.9.29.73] PASSED
main.py::test_passwd_file[ansible://54.67.6.175] PASSED
main.py::test_surrogate_file[ansible://54.67.6.175] PASSED
main.py::test_passwd_file[ansible://54.67.81.102] PASSED
main.py::test_surrogate_file[ansible://54.67.81.102] PASSED

============================================= 10 passed in 54.28 seconds ==============================================
(aws)~/workspaces/emayssat/devtools-vpc/devops/aws/release/tests

Is it a bug in testinfra ? i can see it goes to minion correctly as salt-minion -ldebug telling me, but i don't know why it says there is a missing argument

[INFO    ] User root Executing command cmd.run_all with jid 20150803202851787339
[DEBUG   ] Command details {'tgt_type': 'glob', 'jid': '20150803202851787339', 'tgt': 'minion01', 'ret': '', 'user': 'root', 'arg': ['uname -s'], 'fun': 'cmd.run_all'}
[INFO    ] Starting a new job with PID 4805
[INFO    ] Executing command 'uname -s' in directory '/root'
[DEBUG   ] stdout: Linux
[INFO    ] Returning information for job: 20150803202851787339
[DEBUG   ] Re-using SAuth for ('/etc/salt/pki/minion', 'minion01', 'tcp://172.28.128.10:4506')

======================================================================= ERRORS =======================================================================
________________________________ ERROR at setup of test_saltstack_formula_standard[coreutils-viewer-formula-minion01] ________________________________

SystemInfo = <testinfra.modules.systeminfo.SystemInfo object at 0x2679650>

    @pytest.fixture(scope="session")
    def f(SystemInfo):
>       if SystemInfo.type == "linux":

/usr/lib/python2.7/site-packages/testinfra/modules/file.py:167: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib/python2.7/site-packages/testinfra/modules/systeminfo.py:82: in type
    return self.sysinfo["type"]
/usr/lib/python2.7/site-packages/testinfra/modules/systeminfo.py:33: in sysinfo
    self._sysinfo = self.get_system_info()
/usr/lib/python2.7/site-packages/testinfra/modules/systeminfo.py:43: in get_system_info
    sysinfo["type"] = self.check_output("uname -s").lower()
/usr/lib/python2.7/site-packages/testinfra/modules/base.py:26: in run
    return get_backend().run(command, *args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <testinfra.backend.salt.SaltBackend object at 0x2679b90>, command = 'uname -s', args = ()
out = {'pid': 4738, 'retcode': 0, 'stderr': '', 'stdout': 'Linux'}

    def run(self, command, *args):
        if self.sudo:
            command = "sudo " + command
        command = self.quote(command, *args)
        out = self.run_salt("cmd.run_all", [command])
        return base.CommandResult(
>           out['retcode'], out['stdout'], out['stderr'], command)
E       TypeError: __init__() takes exactly 6 arguments (5 given)

/usr/lib/python2.7/site-packages/testinfra/backend/salt.py:53: TypeError
============================================================== 1 error in 0.16 seconds ===============================================================

Here is the salt command result if it helps you in fixing this:

[root@master ~]# salt minion01 cmd.run_all 'uname -s'
minion01:
    ----------
    pid:
        5443
    retcode:
        0
    stderr:
    stdout:
        Linux

Hi, how do we run the tests in test? When i tried to run them using py.test, I got errors such as:

 fixture 'Command' not found

I understand the issue. Shouldn't we be importing these classes?

testinfra -v --connection=ansible --ansible-inventory=inventories/ec2.py --hosts=key_Release001 main.py

The command above seems to work except
1/ My entire inventory is dynamic
2/ SSH agent configuration is working and tested
3/ How can I specify the user to ssh on the remote box?
In my ansible setup, I do it with a directive in the playbook (remote_user). My local user is emayssat, my remote user is ubuntu.

Maybe should I use ansible.cfg? Other?

Howdy,

I just created a simple test to check for a few services running:

def test_passwd_file(File):
  passwd = File("/etc/passwd")
  assert passwd.contains("root")
  assert passwd.user == "root"

def test_nimsoft(Service):
  nimsoft = Service("nimbus")
  assert nimsoft.is_running

def test_uxauthd(Service):
  uxauthd = Service("uxauthd")
  assert uxauthd.is_running

def test_sshd(Service):
  sshd = Service("sshd")
  assert sshd.is_running

If I attempt to check a host via ssh, I'm told the services are failed. But if I copy the file to the remote host, install testinfra, and run it locally, everything passes:

$ testinfra --connection=ssh [email protected] --sudo test_integrations.py 
======================= test session starts =
platform linux -- Python 3.5.0+, pytest-2.8.3, py-1.4.31, pluggy-0.3.1
rootdir: /home/xian/work/validater, inifile: 
plugins: testinfra-1.0.0.0a16
collected 4 items 

test_integrations.py .FFF

================== FAILURES =======
_____________________________ test_nimsoft[ssh://evprappbpwl01.domain.name] ____

Service = <service None>

    def test_nimsoft(Service):
      nimsoft = Service("nimbus")
>     assert nimsoft.is_running
E     assert <service nimbus>.is_running

test_integrations.py:8: AssertionError
____________________ test_uxauthd[ssh://evprappbpwl01.domain.name] 

Service = <service None>

    def test_uxauthd(Service):
      uxauthd = Service("uxauthd")
>     assert uxauthd.is_running
E     assert <service uxauthd>.is_running

test_integrations.py:12: AssertionError
_______________________ test_sshd[ssh://evprappbpwl01.domain.name] 

Service = <service None>

    def test_sshd(Service):
      sshd = Service("sshd")
>     assert sshd.is_running
E     assert <service ssh>.is_running

test_integrations.py:16: AssertionError
========================= 3 failed, 1 passed in 2.51 seconds 

Versus running it on the host

[hcladmin@evprappbpwl01 ~]$ testinfra --sudo test_integrations.py
=========== test session starts =============================
platform linux2 -- Python 2.6.6, pytest-2.8.3, py-1.4.31, pluggy-0.3.1
rootdir: /home/hcladmin, inifile:
plugins: testinfra-1.0.0.0a16
collected 4 items

test_integrations.py ....

======================= 4 passed in 0.78 seconds ===========

Thanks!

This is a feature request.

I'd like to be able to use a (dynamic) ansible inventory in combination with --hosts to determine the target hosts, but just use an ssh connection instead of the ansible connection backend to actually perform the tests against the target hosts (mainly because performing the tests using ansible is much, much slower).

A more generic solution to this would be the ability to specify a custom generic *.py file that provides the hosts to test_infra (the --hosts parameter would be passed to this file). This way, everyone can write their own integrations (very similar to how ansible does it).

I have a custom test script made with python , there are imported libraries and lots of code, also i am using salt client api inside my script, I am guessing when my script comes to a line like this it freeze or something caller = salt.client.Caller()

here is a simple script you can test with it:

#test_myinfra.py
import salt.client
def test_passwd_file(File):
    passwd = File("/etc/passwd")
    assert passwd.contains("root")
    assert passwd.user == "root"
    assert passwd.group == "root"
    assert passwd.mode == 644
    caller = salt.client.Caller()
    state_data = []
    assert type(state_data) == dict

i am using the following command

py.test test_myinfra.py --connection=salt --hosts=minion_name -v

and here is a screenshot to enusre that if i moved my script to minion and executed it by typing py.test test_myinfra.py it works without problems , so the currently problem is when i use --connection=salt --hosts=minion_name