Code Monkey home page Code Monkey logo

rookcheck's Introduction

rookcheck

rookcheck is a testing platform for rook.io. The intention is to provide developers with a way to simulate various environments and scenarios that may occur within them.

For example, rookcheck can perform tests such as adding new nodes to your kubernetes cluster and ensuring that they are correctly enrolled and handled by rook.io/ceph.

Additionally rookcheck can handle disaster testing such as kernel panics, physically removed nodes, and so forth.

Because a test may need to interact with the underlying hardware the unit tests will set up and configure the nodes, distros, kubernetes, and rook itself. These are then exposed to the test writer to interact with further or to verify the environment.

Read the full documentation.

Quickstart

Install requirements:

sudo zypper in python-pip
sudo pip install tox
sudo zypper in $(tox -qq -e bindep -- -b)
sudo systemctl start docker
sudo usermod -aG docker $USER

Configure:

cp configuration.env .env
vim .env # Make any changes needed

Run tests:

tox -e py38

Documentation Status

rookcheck's People

Contributors

brunoleon avatar colder-is-better avatar jhesketh avatar shaas avatar swiftgist avatar toabctl avatar

Stargazers

 avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

rookcheck's Issues

kernel-default installation failed with latest openSUSE 15.1 OpenStack image

I used the openSUSE Leap 15.1 OpenStack image from https://software.opensuse.org/distributions/leap#JeOS-ports
Error is:

fatal: [42723ebb-worker-1]: FAILED! => {"changed": false, "cmd": ["/usr/bin/zypper", "--quiet", "--non-interactive", "--xmlout", "--non-interactive", "--gpg-auto-import-keys", "install", "--type", "package", "--auto-agree-with-licenses", "--no-recommends", "--", "+kernel-default"], "msg": "Zypper run command failed with return code 4.", "rc": 4, "stderr": "", "stderr_lines": [], "stdout": "<?xml version='1.0'?>\n<stream>\n<prompt id=\"1\">\n<description>Problem: kernel-default-base-4.12.14-lp151.28.48.1.x86_64 conflicts with kernel-default = 4.12.14-lp151.28.48 provided by kernel-default-4.12.14-lp151.28.48.1.x86_64\n Solution 1: Following actions will be done:\n  downgrade of kernel-default-base-4.12.14-lp151.28.48.1.x86_64 to kernel-default-base-4.12.14-lp151.28.44.1.x86_64\n  deinstallation of kernel-default-base-4.12.14-lp151.28.48.1.x86_64\n Solution 2: do not install kernel-default-4.12.14-lp151.28.48.1.x86_64\n</description>\n<text>Choose from above solutions by number or cancel</text>\n<option value=\"1\" desc=\"Choose solution 1\"/>\n<option value=\"2\" desc=\"Choose solution 2\"/>\n<option default=\"1\" value=\"c\" desc=\"Choose no solution and cancel.\"/>\n<option value=\"d\" desc=\"Toggle show detailed conflict information.\"/>\n</prompt>\n</stream>\n", "stdout_lines": ["<?xml version='1.0'?>", "<stream>", "<prompt id=\"1\">", "<description>Problem: kernel-default-base-4.12.14-lp151.28.48.1.x86_64 conflicts with kernel-default = 4.12.14-lp151.28.48 provided by kernel-default-4.12.14-lp151.28.48.1.x86_64", " Solution 1: Following actions will be done:", "  downgrade of kernel-default-base-4.12.14-lp151.28.48.1.x86_64 to kernel-default-base-4.12.14-lp151.28.44.1.x86_64", "  deinstallation of kernel-default-base-4.12.14-lp151.28.48.1.x86_64", " Solution 2: do not install kernel-default-4.12.14-lp151.28.48.1.x86_64", "</description>", "<text>Choose from above solutions by number or cancel</text>", "<option value=\"1\" desc=\"Choose solution 1\"/>", "<option value=\"2\" desc=\"Choose solution 2\"/>", "<option default=\"1\" value=\"c\" desc=\"Choose no solution and cancel.\"/>", "<option value=\"d\" desc=\"Toggle show detailed conflict information.\"/>", "</prompt>", "</stream>"]}

Possible race condition when creating the ansible inventory directory structure

16:28:48              tests.lib.hardware.hardware_base    INFO : Removing 192.168.128.38 from known-hosts if exists.
16:28:48              tests.lib.hardware.hardware_base    INFO : deleted current ansible inventory dir /tmp/rookcheck/rookcheck-kyr-7881/inventory
16:28:48              tests.lib.hardware.hardware_base    INFO : Inventory path: /tmp/rookcheck/rookcheck-kyr-7881/inventory
16:28:48              tests.lib.hardware.hardware_base    INFO : deleted current ansible inventory dir /tmp/rookcheck/rookcheck-kyr-7881/inventory
Exception in thread Thread-6:
Traceback (most recent call last):
  File "/usr/lib64/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/lib64/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/kyr/suse/rookcheck/tests/lib/hardware/libvirt.py", line 360, in _boot_node
    self.node_add(node)
  File "/home/kyr/suse/rookcheck/tests/lib/hardware/hardware_base.py", line 108, in node_add
    self._ansible_create_inventory()
  File "/home/kyr/suse/rookcheck/tests/lib/hardware/hardware_base.py", line 157, in _ansible_create_inventory
    os.makedirs(group_vars_all_dir)
  File "/usr/lib64/python3.8/os.py", line 213, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File "/usr/lib64/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/rookcheck/rookcheck-kyr-7881/inventory/group_vars'

Image files are always downloaded multiple times.

Rookcheck does not cache big image files while trying to download them, it is waste of traffic and time.
For example, given we have this log:

Downloading image from http://download.suse.de/install/SLE-15-SP2-JeOS-GM/SLES15-SP2-JeOS.x86_64-15.2-OpenStack-Cloud-GM.qcow2

Before actually download it would be great to check digest of the file if possible:

curl -I http://download.suse.de/install/SLE-15-SP2-JeOS-GM/SLES15-SP2-JeOS.x86_64-15.2-OpenStack-Cloud-GM.qcow2
HTTP/1.1 302 Found
date: Tue, 20 Oct 2020 22:50:05 GMT
server: Apache/2.4.43 (Linux/SUSE)
x-mirrorbrain-mirror: dist.suse.de
x-mirrorbrain-realm: prefix
link: <http://download.suse.de/install/SLE-15-SP2-JeOS-GM/SLES15-SP2-JeOS.x86_64-15.2-OpenStack-Cloud-GM.qcow2.meta4>; rel=describedby; type="application/metalink4+xml"
link: <http://dist.suse.de/install/SLE-15-SP2-JeOS-GM/SLES15-SP2-JeOS.x86_64-15.2-OpenStack-Cloud-GM.qcow2>; rel=duplicate; pri=1; geo=de
link: <http://dist.nue.suse.com/install/SLE-15-SP2-JeOS-GM/SLES15-SP2-JeOS.x86_64-15.2-OpenStack-Cloud-GM.qcow2>; rel=duplicate; pri=2; geo=de
link: <http://mirror.suse.cz/install/SLE-15-SP2-JeOS-GM/SLES15-SP2-JeOS.x86_64-15.2-OpenStack-Cloud-GM.qcow2>; rel=duplicate; pri=3; geo=cz
link: <http://ibs-mirror.prv.suse.net/install/SLE-15-SP2-JeOS-GM/SLES15-SP2-JeOS.x86_64-15.2-OpenStack-Cloud-GM.qcow2>; rel=duplicate; pri=4; geo=us
link: <http://mirror.suse.asia/dist/install/SLE-15-SP2-JeOS-GM/SLES15-SP2-JeOS.x86_64-15.2-OpenStack-Cloud-GM.qcow2>; rel=duplicate; pri=5; geo=cn
digest: MD5=53HuoyARHkQAPsvn/pYCPg==
digest: SHA=sf/H3cUKiQsuN/n9xR5swIsu3Q4=
digest: SHA-256=vx5V125wORLGjB81i+qhDXSuXhHKJmLC4Ye89NNbgW8=
location: http://dist.suse.de/install/SLE-15-SP2-JeOS-GM/SLES15-SP2-JeOS.x86_64-15.2-OpenStack-Cloud-GM.qcow2
content-type: text/html; charset=iso-8859-1

Any from md5, sha, and sha-256 can be used.

Just for illustration:

> sha256sum /tmp/rookcheck/rookcheck-kyr-596a/SLES15-SP2-JeOS.x86_64-15.2-OpenStack-Cloud-GM.qcow2
bf1e55d76e703912c68c1f358beaa10d74ae5e11ca2662c2e187bcf4d35b816f  /tmp/rookcheck/rookcheck-kyr-596a/SLES15-SP2-JeOS.x86_64-15.2-OpenStack-Cloud-GM.qcow2

and

echo vx5V125wORLGjB81i+qhDXSuXhHKJmLC4Ye89NNbgW8= | base64 -d - | xxd -c 32 -ps
bf1e55d76e703912c68c1f358beaa10d74ae5e11ca2662c2e187bcf4d35b816f

test_basic fails

Using https://download.opensuse.org/distribution/leap/15.1/jeos/openSUSE-Leap-15.1-JeOS.x86_64-OpenStack-Cloud.qcow2 and the latest master commit 5c75ddb2b105edaeab786c01f82bca1aa45c8a5c, the command

tox -e py37 tests/test_basic.py

errors with

E               subprocess.CalledProcessError: Command '/home/rookcheck/tmp/rookcheck/rookcheck-rookcheck-a652/kubectl --kubeconfig /home/rookcheck/tmp/rookcheck/rookcheck-rookcheck-a652/kubeconfig --namespace rook-ceph exec -t "rook-direct-mount-66b7c68959-bhq9t" -- bash -c "$(cat <<'EOF'
E               
E                       # Create the directory
E                       mkdir /tmp/registry
E              
E                       # Detect the mon endpoints and the user secret for the connection
E                       mon_endpoints=$(grep mon_host /etc/ceph/ceph.conf | awk '{print $3}')
E                       my_secret=$(grep key /etc/ceph/keyring | awk '{print $3}')
E               
E                       # Mount the filesystem
E                       mount -t ceph -o mds_namespace=myfs,name=admin,secret=$my_secret             $mon_endpoints:/ /tmp/registry
E               
E                       # See your mounted filesystem
E                       df -h /tmp/registry
E               
E                       echo "Hello Rook" > /tmp/registry/hello
E                       umount /tmp/registry
E                       rmdir /tmp/registry
E     
E               EOF
E               )"' returned non-zero exit status 1.

tests/lib/common.py:171: CalledProcessError


----------------------------------- Captured log call ------------------------------------
11:35:08 kubectl apply -f /home/rookcheck/tmp/rookchec    INFO : deployment.apps/rook-direct-mount created
11:35:13 kubectl --namespace rook-ceph get pod -l app=    INFO : rook-direct-mount-66b7c68959-bhq9t
11:35:14 kubectl --namespace rook-ceph exec -t "rook-d WARNING : mount error 2 = No such file or directory
11:35:14 kubectl --namespace rook-ceph exec -t "rook-d    INFO : Filesystem      Size  Used Avail Use% Mounted on
11:35:14 kubectl --namespace rook-ceph exec -t "rook-d    INFO : overlay          10G  4.1
G  5.9G  42% /
11:35:14 kubectl --namespace rook-ceph exec -t "rook-d WARNING : umount: /tmp/registry: not mounted
11:35:14 kubectl --namespace rook-ceph exec -t "rook-d WARNING : rmdir: failed to remove '/tmp/registry': Directory not empty
11:35:14 kubectl --namespace rook-ceph exec -t "rook-d WARNING : command terminated with exit code 1

I'll see if I can learn anything since I can reproduce this.

Fix total number of workers

As a follow up to #104.

By tainting the master node we allow it to be used as a worker. This is how quorum is reached for the previous implementation. Therefore the minimum could be 1 master and 2 workers still.

Ideally I'd like to keep the amount of resource usage to a minimum. So should we:

  1. Revert to the 1 master (tainted) + 2 workers by default*
  2. Require 1 master, 3 workers, and stop tainting the master
  3. Leave it as is effectively having 4 workers
  • We could also add an option to taint the master.

Rookcheck want more packages than needed

~/suse/rookcheck> which git
/usr/bin/git
~/suse/rookcheck> zypper wp $(which git)
Command 'what-provides' is replaced by 'search --provides --match-exact'.
See 'help search' for all available options.
Loading repository data...
Reading installed packages...

S | Name     | Summary        | Type
--+----------+----------------+--------
i | git-core | Core git tools | package

~/suse/rookcheck> git --version
git version 2.26.2
~/suse/rookcheck> tox -qq -e bindep -- -b
git

But it still want to install something.

tmp directory not created

With the latest addition of TMPDIR having /tmp appended, the directory doesn't exist and the build fails.

E               subprocess.CalledProcessError: Command 'PATH=/home/rookcheck/tmp/rookcheck/rookcheck-rookcheck-3478/rook_build/go/bin:$PATH GOPATH=/home/rookcheck/tmp/rookcheck/rookcheck-rookcheck-3478/rook_build TMPDIR=/home/rookcheck/tmp/rookcheck/tmp make --directory='/home/rookcheck/tmp/rookcheck/rookcheck-rookcheck-3478/rook_build/src/github.com/rook/rook' -j BUILD_REGISTRY='rook-build' IMAGES='ceph' build' returned non-zero exit status 2.

tests/lib/common.py:173: CalledProcessError

In my case, /home/rookcheck/tmp/rookcheck does get created, but /home/rookcheck/tmp/rookcheck/tmp does not. If I create the directory manually and rerun tox -e py37 tests/test_basic.py, then the test completes successfully.

Rookcheck want libcloud auth module when I want libvirt

I have clone rookcheck repo.
Copied the environment file, and modified it so LIBVIRT variables are uncomment:

# For automatic inclusion, copy this file to ".env". Otherwise, after making
# a copy of this file, make sure to source it into your session (eg
# `source my.env`). You may also need to `export` each of the variables.

# Configuration is provided with dynaconf (https://dynaconf.readthedocs.io).
# You can override any settings found in config/ by exporting them into your
# shell environment. See dynaconf documentation for full details.
# The variables must be prefixed with ROOKCHECK.

# NOTE: This prefix may not change unless you also change what variables are
#       passed via tox.ini.
ENVVAR_PREFIX_FOR_DYNACONF=ROOKCHECK

###############################
# OpenStack provider settings #
###############################
# # Make sure to source your openrc_v3.
# ROOKCHECK_HARDWARE_PROVIDER='OPENSTACK'
# ROOKCHECK_OS_VERIFY_SSL_CERT=FALSE
# ROOKCHECK_OS_EXTERNAL_NETWORK=floating

#############################
# libvirt provider settings #
#############################
ROOKCHECK_LIBVIRT_CONNECTION="qemu:///system"
ROOKCHECK_LIBVIRT_NETWORK_RANGE="192.168.124.0/24"
ROOKCHECK_LIBVIRT_IMAGE="openSUSE-15.2.qcow2"
ROOKCHECK_LIBVIRT_VM_MEMORY="8"

I set the provider environment: HARDWARE_PROVIDER='LIBVIRT'

But when it came to run the test I got this:

tox -e py36 -- tests/test_basic.py
py36 installed: ansible==2.9.12,ansible-lint==4.3.2,appdirs==1.4.4,attrs==20.1.0,bcrypt==3.2.0,boto3==1.14.48,botocore==1.17.48,cachetools==4.1.1,certifi==2020.6.20,cffi==1.14.2,chardet==3.0.4,click==7.1.2,colorama==0.4.3,commonmark==0.9.1,cryptography==3.0,dataclasses==0.7,decorator==4.4.2,docutils==0.15.2,dogpile.cache==1.0.2,dynaconf==2.2.3,flake8==3.8.3,google-auth==1.20.1,idna==2.10,importlib-metadata==1.7.0,importlib-resources==3.0.0,iniconfig==1.0.1,iso8601==0.1.12,Jinja2==2.11.2,jmespath==0.10.0,jsonpatch==1.26,jsonpointer==2.0,keystoneauth1==4.2.1,kubernetes==11.0.0,libvirt-python==6.6.0,MarkupSafe==1.1.1,mccabe==0.6.1,more-itertools==8.4.0,munch==2.5.0,mypy==0.782,mypy-extensions==0.4.3,netaddr==0.8.0,netifaces==0.10.9,oauthlib==3.1.0,openstacksdk==0.48.0,os-service-types==1.7.0,packaging==20.4,paramiko==2.7.1,pbr==5.4.5,pluggy==0.13.1,py==1.9.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pycodestyle==2.6.0,pycparser==2.20,pyflakes==2.2.0,Pygments==2.6.1,PyNaCl==1.4.0,pyparsing==2.4.7,pytest==6.0.1,python-box==3.4.6,python-dateutil==2.8.1,python-dotenv==0.14.0,PyYAML==5.3.1,requests==2.24.0,requests-oauthlib==1.3.0,requestsexceptions==1.4.0,rich==6.0.0,rsa==4.6,ruamel.yaml==0.16.10,ruamel.yaml.clib==0.2.0,s3transfer==0.3.3,six==1.15.0,stevedore==3.2.0,toml==0.10.1,typed-ast==1.4.1,typing-extensions==3.7.4.3,urllib3==1.25.10,websocket-client==0.57.0,wget==3.2,zipp==3.1.0
py36 run-test-pre: PYTHONHASHSEED='718839036'
py36 run-test: commands[0] | py.test -s tests/test_basic.py
================================================================ test session starts =================================================================
platform linux -- Python 3.6.10, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
cachedir: .tox/py36/.pytest_cache
rootdir: /home/kyr/suse/rookcheck, configfile: tox.ini
collected 8 items                                                                                                                                    

tests/test_basic.py::test_deploy_filesystem 
------------------------------------------------------------------- live log setup -------------------------------------------------------------------
15:47:23 ssh-agent -a /tmp/rookcheck/rookcheck-kyr-33f    INFO : SSH_AUTH_SOCK=/tmp/rookcheck/rookcheck-kyr-33f6/ssh-agent.sock; export SSH_AUTH_SOCK;
15:47:23 ssh-agent -a /tmp/rookcheck/rookcheck-kyr-33f    INFO : SSH_AGENT_PID=11655; export SSH_AGENT_PID;
15:47:23 ssh-agent -a /tmp/rookcheck/rookcheck-kyr-33f    INFO : echo Agent pid 11655;
15:47:23                                          root    INFO : Adding ssh-key to agent
15:47:23                           tests.lib.workspace    INFO : Workspace rookcheck-kyr-33f6 set up at /tmp/rookcheck/rookcheck-kyr-33f6
15:47:23                           tests.lib.workspace    INFO : public key ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNZNan85dK6vRg99v//NH7ed/XK9v3l9Z621N7P3httJaZ2BE+1bH4A3PDaxG59pbQjG4y1XS5hnklO7ov2Dt1ffaMqB+5jV041uFHdl+ybc0S4P/KyF4LxZah3yr43VjGPXAsm1tqOHFuLXtowvQtkdVYMGtZWGF+yhyjjtmQxpCLyiHxQNXaAlItdFsO2jlD7zdhx/gdXjz6qlWUWEp2tbT/1USZhOqfe73tajOgI6YQp/UlHzzKOy2nPTooG7z+HiDizIvQQVR1f41t+A2V5nHPEa9kg+PrH4MjHOKXI6UqySkONvpfeRNXUCPLJ6xOD+CARC8hqYt0n/xJlb41
15:47:23                           tests.lib.workspace    INFO : private key /tmp/rookcheck/rookcheck-kyr-33f6/private.key
ERROR
tests/test_basic.py::test_file_creation ERROR
tests/test_basic.py::test_service_mons ERROR
tests/test_basic.py::test_services ERROR
tests/test_basic.py::test_service_rbd ERROR
tests/test_basic.py::test_osd_number ERROR
tests/test_basic.py::test_add_node SKIPPED
tests/test_basic.py::test_add_storage ERROR
----------------------------------------------------------------- live log teardown ------------------------------------------------------------------
15:47:23                                  ssh-agent -k    INFO : unset SSH_AUTH_SOCK;

15:47:23                                  ssh-agent -k    INFO : unset SSH_AGENT_PID;
15:47:23                                  ssh-agent -k    INFO : echo Agent pid 11655 killed;
15:47:23                           tests.lib.workspace    INFO : Removing workspace /tmp/rookcheck/rookcheck-kyr-33f6 from disk


======================================================================= ERRORS =======================================================================
______________________________________________________ ERROR at setup of test_deploy_filesystem ______________________________________________________

workspace = <tests.lib.workspace.Workspace object at 0x7f09f5b5d908>

    @pytest.fixture(scope="module")
    def rook_cluster(workspace):
>       with Hardware(workspace) as hardware:

tests/conftest.py:105: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/lib/hardware/openstack_sdk.py:150: in __init__
    super().__init__(workspace)
tests/lib/hardware/hardware_base.py:44: in __init__
    self._conn = self.get_connection()
tests/lib/hardware/openstack_sdk.py:181: in get_connection
    return openstack.connect()
.tox/py36/lib/python3.6/site-packages/openstack/__init__.py:63: in connect
    options=options, **kwargs)
.tox/py36/lib/python3.6/site-packages/openstack/config/__init__.py:36: in get_cloud_region
    return config.get_one(options=parsed_options, **kwargs)
.tox/py36/lib/python3.6/site-packages/openstack/config/loader.py:1111: in get_one
    auth_plugin = loader.load_from_options(**config['auth'])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <keystoneauth1.loading._plugins.identity.generic.Password object at 0x7f09f5b5da20>, kwargs = {}, missing_required = [<Opt: auth-url>]

    def load_from_options(self, **kwargs):
        """Create a plugin from the arguments retrieved from get_options.
    
        A client can override this function to do argument validation or to
        handle differences between the registered options and what is required
        to create the plugin.
        """
        missing_required = [o for o in self.get_options()
                            if o.required and kwargs.get(o.dest) is None]
    
        if missing_required:
>           raise exceptions.MissingRequiredOptions(missing_required)
E           keystoneauth1.exceptions.auth_plugins.MissingRequiredOptions: Auth plugin requires parameters which were not given: auth_url

.tox/py36/lib/python3.6/site-packages/keystoneauth1/loading/base.py:162: MissingRequiredOptions
----------------------------------------------------------------- Captured log setup -----------------------------------------------------------------
15:47:23 ssh-agent -a /tmp/rookcheck/rookcheck-kyr-33f    INFO : SSH_AUTH_SOCK=/tmp/rookcheck/rookcheck-kyr-33f6/ssh-agent.sock; export SSH_AUTH_SOCK;
15:47:23 ssh-agent -a /tmp/rookcheck/rookcheck-kyr-33f    INFO : SSH_AGENT_PID=11655; export SSH_AGENT_PID;
15:47:23 ssh-agent -a /tmp/rookcheck/rookcheck-kyr-33f    INFO : echo Agent pid 11655;
15:47:23                                          root    INFO : Adding ssh-key to agent
15:47:23                           tests.lib.workspace    INFO : Workspace rookcheck-kyr-33f6 set up at /tmp/rookcheck/rookcheck-kyr-33f6
15:47:23                           tests.lib.workspace    INFO : public key ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNZNan85dK6vRg99v//NH7ed/XK9v3l9Z621N7P3httJaZ2BE+1bH4A3PDaxG59pbQjG4y1XS5hnklO7ov2Dt1ffaMqB+5jV041uFHdl+ybc0S4P/KyF4LxZah3yr43VjGPXAsm1tqOHFuLXtowvQtkdVYMGtZWGF+yhyjjtmQxpCLyiHxQNXaAlItdFsO2jlD7zdhx/gdXjz6qlWUWEp2tbT/1USZhOqfe73tajOgI6YQp/UlHzzKOy2nPTooG7z+HiDizIvQQVR1f41t+A2V5nHPEa9kg+PrH4MjHOKXI6UqySkONvpfeRNXUCPLJ6xOD+CARC8hqYt0n/xJlb41
15:47:23                           tests.lib.workspace    INFO : private key /tmp/rookcheck/rookcheck-kyr-33f6/private.key
________________________________________________________ ERROR at setup of test_file_creation ________________________________________________________

workspace = <tests.lib.workspace.Workspace object at 0x7f09f5b5d908>

    @pytest.fixture(scope="module")
    def rook_cluster(workspace):
>       with Hardware(workspace) as hardware:

tests/conftest.py:105: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/lib/hardware/openstack_sdk.py:150: in __init__
    super().__init__(workspace)
tests/lib/hardware/hardware_base.py:44: in __init__
    self._conn = self.get_connection()
tests/lib/hardware/openstack_sdk.py:181: in get_connection
    return openstack.connect()
.tox/py36/lib/python3.6/site-packages/openstack/__init__.py:63: in connect
    options=options, **kwargs)
.tox/py36/lib/python3.6/site-packages/openstack/config/__init__.py:36: in get_cloud_region
    return config.get_one(options=parsed_options, **kwargs)
.tox/py36/lib/python3.6/site-packages/openstack/config/loader.py:1111: in get_one
    auth_plugin = loader.load_from_options(**config['auth'])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <keystoneauth1.loading._plugins.identity.generic.Password object at 0x7f09f5b5da20>, kwargs = {}, missing_required = [<Opt: auth-url>]

    def load_from_options(self, **kwargs):
        """Create a plugin from the arguments retrieved from get_options.
    
        A client can override this function to do argument validation or to
        handle differences between the registered options and what is required
        to create the plugin.
        """
        missing_required = [o for o in self.get_options()
                            if o.required and kwargs.get(o.dest) is None]
    
        if missing_required:
>           raise exceptions.MissingRequiredOptions(missing_required)
E           keystoneauth1.exceptions.auth_plugins.MissingRequiredOptions: Auth plugin requires parameters which were not given: auth_url

.tox/py36/lib/python3.6/site-packages/keystoneauth1/loading/base.py:162: MissingRequiredOptions
________________________________________________________ ERROR at setup of test_service_mons _________________________________________________________

workspace = <tests.lib.workspace.Workspace object at 0x7f09f5b5d908>

    @pytest.fixture(scope="module")
    def rook_cluster(workspace):
>       with Hardware(workspace) as hardware:

tests/conftest.py:105: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/lib/hardware/openstack_sdk.py:150: in __init__
    super().__init__(workspace)
tests/lib/hardware/hardware_base.py:44: in __init__
    self._conn = self.get_connection()
tests/lib/hardware/openstack_sdk.py:181: in get_connection
    return openstack.connect()
.tox/py36/lib/python3.6/site-packages/openstack/__init__.py:63: in connect
    options=options, **kwargs)
.tox/py36/lib/python3.6/site-packages/openstack/config/__init__.py:36: in get_cloud_region
    return config.get_one(options=parsed_options, **kwargs)
.tox/py36/lib/python3.6/site-packages/openstack/config/loader.py:1111: in get_one
    auth_plugin = loader.load_from_options(**config['auth'])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <keystoneauth1.loading._plugins.identity.generic.Password object at 0x7f09f5b5da20>, kwargs = {}, missing_required = [<Opt: auth-url>]

    def load_from_options(self, **kwargs):
        """Create a plugin from the arguments retrieved from get_options.
    
        A client can override this function to do argument validation or to
        handle differences between the registered options and what is required
        to create the plugin.
        """
        missing_required = [o for o in self.get_options()
                            if o.required and kwargs.get(o.dest) is None]
    
        if missing_required:
>           raise exceptions.MissingRequiredOptions(missing_required)
E           keystoneauth1.exceptions.auth_plugins.MissingRequiredOptions: Auth plugin requires parameters which were not given: auth_url

.tox/py36/lib/python3.6/site-packages/keystoneauth1/loading/base.py:162: MissingRequiredOptions
__________________________________________________________ ERROR at setup of test_services ___________________________________________________________

workspace = <tests.lib.workspace.Workspace object at 0x7f09f5b5d908>

    @pytest.fixture(scope="module")
    def rook_cluster(workspace):
>       with Hardware(workspace) as hardware:

tests/conftest.py:105: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/lib/hardware/openstack_sdk.py:150: in __init__
    super().__init__(workspace)
tests/lib/hardware/hardware_base.py:44: in __init__
    self._conn = self.get_connection()
tests/lib/hardware/openstack_sdk.py:181: in get_connection
    return openstack.connect()
.tox/py36/lib/python3.6/site-packages/openstack/__init__.py:63: in connect
    options=options, **kwargs)
.tox/py36/lib/python3.6/site-packages/openstack/config/__init__.py:36: in get_cloud_region
    return config.get_one(options=parsed_options, **kwargs)
.tox/py36/lib/python3.6/site-packages/openstack/config/loader.py:1111: in get_one
    auth_plugin = loader.load_from_options(**config['auth'])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <keystoneauth1.loading._plugins.identity.generic.Password object at 0x7f09f5b5da20>, kwargs = {}, missing_required = [<Opt: auth-url>]

    def load_from_options(self, **kwargs):
        """Create a plugin from the arguments retrieved from get_options.
    
        A client can override this function to do argument validation or to
        handle differences between the registered options and what is required
        to create the plugin.
        """
        missing_required = [o for o in self.get_options()
                            if o.required and kwargs.get(o.dest) is None]
    
        if missing_required:
>           raise exceptions.MissingRequiredOptions(missing_required)
E           keystoneauth1.exceptions.auth_plugins.MissingRequiredOptions: Auth plugin requires parameters which were not given: auth_url

.tox/py36/lib/python3.6/site-packages/keystoneauth1/loading/base.py:162: MissingRequiredOptions
_________________________________________________________ ERROR at setup of test_service_rbd _________________________________________________________

workspace = <tests.lib.workspace.Workspace object at 0x7f09f5b5d908>

    @pytest.fixture(scope="module")
    def rook_cluster(workspace):
>       with Hardware(workspace) as hardware:

tests/conftest.py:105: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/lib/hardware/openstack_sdk.py:150: in __init__
    super().__init__(workspace)
tests/lib/hardware/hardware_base.py:44: in __init__
    self._conn = self.get_connection()
tests/lib/hardware/openstack_sdk.py:181: in get_connection
    return openstack.connect()
.tox/py36/lib/python3.6/site-packages/openstack/__init__.py:63: in connect
    options=options, **kwargs)
.tox/py36/lib/python3.6/site-packages/openstack/config/__init__.py:36: in get_cloud_region
    return config.get_one(options=parsed_options, **kwargs)
.tox/py36/lib/python3.6/site-packages/openstack/config/loader.py:1111: in get_one
    auth_plugin = loader.load_from_options(**config['auth'])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <keystoneauth1.loading._plugins.identity.generic.Password object at 0x7f09f5b5da20>, kwargs = {}, missing_required = [<Opt: auth-url>]

    def load_from_options(self, **kwargs):
        """Create a plugin from the arguments retrieved from get_options.
    
        A client can override this function to do argument validation or to
        handle differences between the registered options and what is required
        to create the plugin.
        """
        missing_required = [o for o in self.get_options()
                            if o.required and kwargs.get(o.dest) is None]
    
        if missing_required:
>           raise exceptions.MissingRequiredOptions(missing_required)
E           keystoneauth1.exceptions.auth_plugins.MissingRequiredOptions: Auth plugin requires parameters which were not given: auth_url

.tox/py36/lib/python3.6/site-packages/keystoneauth1/loading/base.py:162: MissingRequiredOptions
_________________________________________________________ ERROR at setup of test_osd_number __________________________________________________________

workspace = <tests.lib.workspace.Workspace object at 0x7f09f5b5d908>

    @pytest.fixture(scope="module")
    def rook_cluster(workspace):
>       with Hardware(workspace) as hardware:

tests/conftest.py:105: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/lib/hardware/openstack_sdk.py:150: in __init__
    super().__init__(workspace)
tests/lib/hardware/hardware_base.py:44: in __init__
    self._conn = self.get_connection()
tests/lib/hardware/openstack_sdk.py:181: in get_connection
    return openstack.connect()
.tox/py36/lib/python3.6/site-packages/openstack/__init__.py:63: in connect
    options=options, **kwargs)
.tox/py36/lib/python3.6/site-packages/openstack/config/__init__.py:36: in get_cloud_region
    return config.get_one(options=parsed_options, **kwargs)
.tox/py36/lib/python3.6/site-packages/openstack/config/loader.py:1111: in get_one
    auth_plugin = loader.load_from_options(**config['auth'])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <keystoneauth1.loading._plugins.identity.generic.Password object at 0x7f09f5b5da20>, kwargs = {}, missing_required = [<Opt: auth-url>]

    def load_from_options(self, **kwargs):
        """Create a plugin from the arguments retrieved from get_options.
    
        A client can override this function to do argument validation or to
        handle differences between the registered options and what is required
        to create the plugin.
        """
        missing_required = [o for o in self.get_options()
                            if o.required and kwargs.get(o.dest) is None]
    
        if missing_required:
>           raise exceptions.MissingRequiredOptions(missing_required)
E           keystoneauth1.exceptions.auth_plugins.MissingRequiredOptions: Auth plugin requires parameters which were not given: auth_url

.tox/py36/lib/python3.6/site-packages/keystoneauth1/loading/base.py:162: MissingRequiredOptions
_________________________________________________________ ERROR at setup of test_add_storage _________________________________________________________

workspace = <tests.lib.workspace.Workspace object at 0x7f09f5b5d908>

    @pytest.fixture(scope="module")
    def rook_cluster(workspace):
>       with Hardware(workspace) as hardware:

tests/conftest.py:105: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/lib/hardware/openstack_sdk.py:150: in __init__
    super().__init__(workspace)
tests/lib/hardware/hardware_base.py:44: in __init__
    self._conn = self.get_connection()
tests/lib/hardware/openstack_sdk.py:181: in get_connection
    return openstack.connect()
.tox/py36/lib/python3.6/site-packages/openstack/__init__.py:63: in connect
    options=options, **kwargs)
.tox/py36/lib/python3.6/site-packages/openstack/config/__init__.py:36: in get_cloud_region
    return config.get_one(options=parsed_options, **kwargs)
.tox/py36/lib/python3.6/site-packages/openstack/config/loader.py:1111: in get_one
    auth_plugin = loader.load_from_options(**config['auth'])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <keystoneauth1.loading._plugins.identity.generic.Password object at 0x7f09f5b5da20>, kwargs = {}, missing_required = [<Opt: auth-url>]

    def load_from_options(self, **kwargs):
        """Create a plugin from the arguments retrieved from get_options.
    
        A client can override this function to do argument validation or to
        handle differences between the registered options and what is required
        to create the plugin.
        """
        missing_required = [o for o in self.get_options()
                            if o.required and kwargs.get(o.dest) is None]
    
        if missing_required:
>           raise exceptions.MissingRequiredOptions(missing_required)
E           keystoneauth1.exceptions.auth_plugins.MissingRequiredOptions: Auth plugin requires parameters which were not given: auth_url

.tox/py36/lib/python3.6/site-packages/keystoneauth1/loading/base.py:162: MissingRequiredOptions
============================================================== short test summary info ===============================================================
ERROR tests/test_basic.py::test_deploy_filesystem - keystoneauth1.exceptions.auth_plugins.MissingRequiredOptions: Auth plugin requires parameters w...
ERROR tests/test_basic.py::test_file_creation - keystoneauth1.exceptions.auth_plugins.MissingRequiredOptions: Auth plugin requires parameters which...
ERROR tests/test_basic.py::test_service_mons - keystoneauth1.exceptions.auth_plugins.MissingRequiredOptions: Auth plugin requires parameters which ...
ERROR tests/test_basic.py::test_services - keystoneauth1.exceptions.auth_plugins.MissingRequiredOptions: Auth plugin requires parameters which were...
ERROR tests/test_basic.py::test_service_rbd - keystoneauth1.exceptions.auth_plugins.MissingRequiredOptions: Auth plugin requires parameters which w...
ERROR tests/test_basic.py::test_osd_number - keystoneauth1.exceptions.auth_plugins.MissingRequiredOptions: Auth plugin requires parameters which we...
ERROR tests/test_basic.py::test_add_storage - keystoneauth1.exceptions.auth_plugins.MissingRequiredOptions: Auth plugin requires parameters which w...
============================================================ 1 skipped, 7 errors in 0.47s ============================================================
ERROR: InvocationError for command /home/kyr/suse/rookcheck/.tox/py36/bin/py.test -s tests/test_basic.py (exited with code 1)
______________________________________________________________________ summary _______________________________________________________________________
ERROR:   py36: commands failed

libvirt test-requirements.txt

I have had to add libvirt-python to the test-requirements.txt to successfully run tox -e py37 tests/test_basic.py.

Basing the test-requirements on the hardware_provider in settings.toml seems like the most desirable option.

Rather than recreate the tox environment, I did change the testenv command to update any missing or outdated dependencies

 [testenv]
-deps = -r{toxinidir}/test-requirements.txt
+deps = -Ur{toxinidir}/test-requirements.txt

I do think the -U option would prevent future issues for developers that are extending RookCheck. With the -U, updating test-requirements.txt is sufficient and rerunning the tox command is straightforward.

Libvirt duplication key error

tests/test_basic.py::test_deploy_filesystem 
------------------------------------------------------------------- live log setup -------------------------------------------------------------------
16:18:45 ssh-agent -a /tmp/rookcheck/rookcheck-kyr-e90    INFO : SSH_AUTH_SOCK=/tmp/rookcheck/rookcheck-kyr-e901/ssh-agent.sock; export SSH_AUTH_SOCK;
16:18:45 ssh-agent -a /tmp/rookcheck/rookcheck-kyr-e90    INFO : SSH_AGENT_PID=12981; export SSH_AGENT_PID;
16:18:45 ssh-agent -a /tmp/rookcheck/rookcheck-kyr-e90    INFO : echo Agent pid 12981;
16:18:45                                          root    INFO : Adding ssh-key to agent
16:18:45                           tests.lib.workspace    INFO : Workspace rookcheck-kyr-e901 set up at /tmp/rookcheck/rookcheck-kyr-e901
16:18:45                           tests.lib.workspace    INFO : public key ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCjbDWCXWoBD/V2syIP1KJhiWZczPb/gnAzywYUy+Ahm1e5EUeqzmkiETNnOEy7fmBSWj5ih2pyyyb5mVd34R/cp3Cvb5q3qIwk3/yCbPSjqC7ziYiJ4LCJcccZxwcxCyZjcGBsGTGwW2V2yIgoSX92TRImd1zFMiJNyHTyWyt4SLf+ZBTe5BytRa9BsFdOeNGN/Sw3SLCabYsmnorwvFXZ1NEySiltj87BAUZCwnq5Ukjk3LYdHAkmnI3BlikA6quq3+qV3+hC61xdOaeU5PIC35RQvrx9dHMJYYp+XVHVaWhz1vdsvzWyuT3AiZpwUWKcZI9VvmgouzXHi64xpr9T
16:18:45                           tests.lib.workspace    INFO : private key /tmp/rookcheck/rookcheck-kyr-e901/private.key
16:18:45              tests.lib.hardware.hardware_base    INFO : hardware <tests.lib.hardware.libvirt.Hardware object at 0x7f2f12c386a0>: Using rookcheck-kyr-e901
16:18:45                    tests.lib.hardware.libvirt    INFO : created network 192.168.124.0/26 as rookcheck-kyr-e901
16:18:45                    tests.lib.hardware.libvirt    INFO : Got libvirt network rookcheck-kyr-e901
16:18:48          tests.lib.kubernetes.kubernetes_base    INFO : kube init on hardware <tests.lib.hardware.libvirt.Hardware object at 0x7f2f12c386a0>
16:18:48                           tests.lib.rook.base    INFO : rook init on <tests.lib.hardware.libvirt.Hardware object at 0x7f2f12c386a0>
16:18:48              tests.lib.hardware.hardware_base    INFO : boot nodes
16:18:48              tests.lib.hardware.hardware_base    INFO : creating a new node for hardware <tests.lib.hardware.libvirt.Hardware object at 0x7f2f12c386a0>
16:18:48              tests.lib.hardware.hardware_base    INFO : creating a new node for hardware <tests.lib.hardware.libvirt.Hardware object at 0x7f2f12c386a0>
16:18:48              tests.lib.hardware.hardware_base    INFO : creating a new node for hardware <tests.lib.hardware.libvirt.Hardware object at 0x7f2f12c386a0>
16:18:48              tests.lib.hardware.hardware_base    INFO : creating a new node for hardware <tests.lib.hardware.libvirt.Hardware object at 0x7f2f12c386a0>
16:18:48 qemu-img create -f qcow2 -F qcow2 -o backing_    INFO : Formatting '/tmp/rookcheck/rookcheck-kyr-e901/rookcheck-kyr-e901-master-0-snapshot.qcow2', fmt=qcow2 size=10737418240 backing_file=/tmp/rookcheck/rookcheck-kyr-e901/openSUSE-Leap-15.2-JeOS.x86_64-OpenStack-Cloud.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16
16:18:48 qemu-img create -f qcow2 -F qcow2 -o backing_    INFO : Formatting '/tmp/rookcheck/rookcheck-kyr-e901/rookcheck-kyr-e901-worker-0-snapshot.qcow2', fmt=qcow2 size=10737418240 backing_file=/tmp/rookcheck/rookcheck-kyr-e901/openSUSE-Leap-15.2-JeOS.x86_64-OpenStack-Cloud.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16
16:18:48                    tests.lib.hardware.libvirt    INFO : node rookcheck-kyr-e901-master-0: created qcow2 backing file under/tmp/rookcheck/rookcheck-kyr-e901/rookcheck-kyr-e901-master-0-snapshot.qcow2
16:18:48                    tests.lib.hardware.libvirt    INFO : node rookcheck-kyr-e901-worker-0: created qcow2 backing file under/tmp/rookcheck/rookcheck-kyr-e901/rookcheck-kyr-e901-worker-0-snapshot.qcow2
16:18:48 qemu-img create -f qcow2 -F qcow2 -o backing_    INFO : Formatting '/tmp/rookcheck/rookcheck-kyr-e901/rookcheck-kyr-e901-worker-1-snapshot.qcow2', fmt=qcow2 size=10737418240 backing_file=/tmp/rookcheck/rookcheck-kyr-e901/openSUSE-Leap-15.2-JeOS.x86_64-OpenStack-Cloud.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16
16:18:48 qemu-img create -f qcow2 -F qcow2 -o backing_    INFO : Formatting '/tmp/rookcheck/rookcheck-kyr-e901/rookcheck-kyr-e901-worker-2-snapshot.qcow2', fmt=qcow2 size=10737418240 backing_file=/tmp/rookcheck/rookcheck-kyr-e901/openSUSE-Leap-15.2-JeOS.x86_64-OpenStack-Cloud.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16
16:18:48                    tests.lib.hardware.libvirt    INFO : node rookcheck-kyr-e901-worker-1: created qcow2 backing file under/tmp/rookcheck/rookcheck-kyr-e901/rookcheck-kyr-e901-worker-1-snapshot.qcow2
16:18:48                    tests.lib.hardware.libvirt    INFO : node rookcheck-kyr-e901-worker-2: created qcow2 backing file under/tmp/rookcheck/rookcheck-kyr-e901/rookcheck-kyr-e901-worker-2-snapshot.qcow2
16:18:48                    tests.lib.hardware.libvirt    INFO : node rookcheck-kyr-e901-worker-0: booting with image /tmp/rookcheck/rookcheck-kyr-e901/rookcheck-kyr-e901-worker-0-snapshot.qcow2
16:18:48                    tests.lib.hardware.libvirt    INFO : node rookcheck-kyr-e901-master-0: booting with image /tmp/rookcheck/rookcheck-kyr-e901/rookcheck-kyr-e901-master-0-snapshot.qcow2
16:18:48                    tests.lib.hardware.libvirt    INFO : node rookcheck-kyr-e901-worker-1: booting with image /tmp/rookcheck/rookcheck-kyr-e901/rookcheck-kyr-e901-worker-1-snapshot.qcow2
libvirt:  error : internal error: Duplicate key
16:18:48                    tests.lib.hardware.libvirt    INFO : node rookcheck-kyr-e901-worker-2: booting with image /tmp/rookcheck/rookcheck-kyr-e901/rookcheck-kyr-e901-worker-2-snapshot.qcow2
Exception in thread Thread-6:
Traceback (most recent call last):
  File "/usr/lib64/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib64/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/home/kyr/suse/rookcheck/tests/lib/hardware/libvirt.py", line 400, in _boot_node
    node = self.node_create(name, role, tags)
  File "/home/kyr/suse/rookcheck/tests/lib/hardware/libvirt.py", line 396, in node_create
    node.boot()
  File "/home/kyr/suse/rookcheck/tests/lib/hardware/libvirt.py", line 75, in boot
    self._dom = self._conn.defineXML(xml)
  File "/home/kyr/suse/rookcheck/.tox/py36/lib64/python3.6/site-packages/libvirt.py", line 4049, in defineXML
    if ret is None:raise libvirtError('virDomainDefineXML() failed', conn=self)
libvirt.libvirtError: internal error: Duplicate key

Create utils to get useful logs and information

Create utils to get useful logs and information out of a running deploying.

These could be used by the job itself to verify states, or by the CLI util for a developer to do debugging etc.

Skuba package update breaks when kernel-default-base is present and kernel-default used

Jun 24 10:08:56 rookcheck-bleon-a1ee-master-0 skuba-update[4861]:
Jun 24 10:08:57 rookcheck-bleon-a1ee-master-0 kubelet[4801]: E0624 10:08:57.546374    4801 kubelet.go:2183] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: Missing CNI default network 
Jun 24 10:08:57 rookcheck-bleon-a1ee-master-0 skuba-update[4861]: Loading repository data...
Jun 24 10:08:57 rookcheck-bleon-a1ee-master-0 skuba-update[4861]: Reading installed packages...
Jun 24 10:08:57 rookcheck-bleon-a1ee-master-0 skuba-update[4861]: Resolving package dependencies...
Jun 24 10:08:57 rookcheck-bleon-a1ee-master-0 skuba-update[4861]: Problem: patch:SUSE-SLE-Module-Basesystem-15-SP1-2020-1599-1.noarch conflicts with kernel-default-base.x86_64 < 4.12.14-197.45.1 provided by kernel-default-base-4.12.14-197.37.1.x86_64
Jun 24 10:08:57 rookcheck-bleon-a1ee-master-0 skuba-update[4861]:  Solution 1: deinstallation of kernel-default-4.12.14-197.45.1.x86_64
Jun 24 10:08:57 rookcheck-bleon-a1ee-master-0 skuba-update[4861]:  Solution 2: deinstallation of kernel-default-base-4.12.14-197.37.1.x86_64
Jun 24 10:08:57 rookcheck-bleon-a1ee-master-0 skuba-update[4861]:  Solution 3: do not install patch:SUSE-SLE-Module-Basesystem-15-SP1-2020-1599-1.noarch
Jun 24 10:08:57 rookcheck-bleon-a1ee-master-0 skuba-update[4861]: Choose from above solutions by number or cancel [1/2/3/c/d/?] (c): c
Jun 24 10:08:57 rookcheck-bleon-a1ee-master-0 skuba-update[4861]: 2020-06-24 10:08:47 [skuba-update] running 'zypper --version'
Jun 24 10:08:57 rookcheck-bleon-a1ee-master-0 skuba-update[4861]: 2020-06-24 10:08:47 [skuba-update] running 'zypper --userdata skuba-update ref -s'
Jun 24 10:08:57 rookcheck-bleon-a1ee-master-0 skuba-update[4861]: 2020-06-24 10:08:50 [skuba-update] running 'zypper --userdata skuba-update --non-interactive --non-interactive-include-reboot-patches patch'
Jun 24 10:08:57 rookcheck-bleon-a1ee-master-0 skuba-update[4861]: 2020-06-24 10:08:56 [skuba-update] running 'zypper --userdata skuba-update --non-interactive --non-interactive-include-reboot-patches patch'
Jun 24 10:08:57 rookcheck-bleon-a1ee-master-0 skuba-update[4861]: Traceback (most recent call last):
Jun 24 10:08:57 rookcheck-bleon-a1ee-master-0 skuba-update[4861]:   File "/usr/sbin/skuba-update", line 11, in <module>
Jun 24 10:08:57 rookcheck-bleon-a1ee-master-0 skuba-update[4861]:     load_entry_point('skuba-update==1.3.5', 'console_scripts', 'skuba-update')()
Jun 24 10:08:57 rookcheck-bleon-a1ee-master-0 skuba-update[4861]:   File "/usr/lib/python3.6/site-packages/skuba_update/skuba_update.py", line 80, in main
Jun 24 10:08:57 rookcheck-bleon-a1ee-master-0 skuba-update[4861]:     code = update()
Jun 24 10:08:57 rookcheck-bleon-a1ee-master-0 skuba-update[4861]:   File "/usr/lib/python3.6/site-packages/skuba_update/skuba_update.py", line 123, in update
Jun 24 10:08:57 rookcheck-bleon-a1ee-master-0 skuba-update[4861]:     code = run_zypper_patch()
Jun 24 10:08:57 rookcheck-bleon-a1ee-master-0 skuba-update[4861]:   File "/usr/lib/python3.6/site-packages/skuba_update/skuba_update.py", line 331, in run_zypper_patch
Jun 24 10:08:57 rookcheck-bleon-a1ee-master-0 skuba-update[4861]:     'patch'
Jun 24 10:08:57 rookcheck-bleon-a1ee-master-0 skuba-update[4861]:   File "/usr/lib/python3.6/site-packages/skuba_update/skuba_update.py", line 318, in run_zypper_command
Jun 24 10:08:57 rookcheck-bleon-a1ee-master-0 skuba-update[4861]:     raise Exception('"{0}" failed'.format(' '.join(zypperCommand)))
Jun 24 10:08:57 rookcheck-bleon-a1ee-master-0 skuba-update[4861]: Exception: "zypper --userdata skuba-update --non-interactive --non-interactive-include-reboot-patches patch" failed
Jun 24 10:08:57 rookcheck-bleon-a1ee-master-0 systemd[1]: skuba-update.service: Main process exited, code=exited, status=1/FAILURE
Jun 24 10:08:57 rookcheck-bleon-a1ee-master-0 systemd[1]: Failed to start Update the system.
-- Subject: Unit skuba-update.service has failed

Exception: One or more hosts failed when trying to execute the basic tests

With commit 8500257, I try to run:

tox -epy38 -- tests/test_basic.py

After some time (~5-10 min), I get:

TASK [Download kubeconfig] ***************************************************************************************
changed: [tom-rookci-618b03be_master_0]
100% [........................................................................] 43499520 / 43499520Command `taint nodes --all node-role.kubernetes.io/master-`
 failed
STDOUT:
node/tom-rookci-618b03be-master-0 untainted

STDERR:
taint "node-role.kubernetes.io/master" not found
taint "node-role.kubernetes.io/master" not found

Re-joining rook build thread

PLAY [Upload rook image] *****************************************************************************************
[WARNING]: Failure using method (v2_runner_on_start) in callback plugin (<tests.lib.hardware.ResultCallback
object at 0x7f68c843ab50>): 'show_per_host_start'

TASK [Copy Rook Ceph image to cluster nodes] *********************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: If you are using a module and expect the file to exist on the
 remote, see the remote_src option
fatal: [tom-rookci-618b03be_worker_0]: FAILED! => {"changed": false, "msg": "Could not find or access '/tmp/tom-rookci-618b03be_6decwpm1/rook_build/rook-ceph.
tar.gz' on the Ansible Controller.\nIf you are using a module and expect the file to exist on the remote, see the remote_src option"}
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: If you are using a module and expect the file to exist on the
 remote, see the remote_src option
fatal: [tom-rookci-618b03be_worker_1]: FAILED! => {"changed": false, "msg": "Could not find or access '/tmp/tom-rookci-618b03be_6decwpm1/rook_build/rook-ceph.
tar.gz' on the Ansible Controller.\nIf you are using a module and expect the file to exist on the remote, see the remote_src option"}
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: If you are using a module and expect the file to exist on the
 remote, see the remote_src option
fatal: [tom-rookci-618b03be_master_0]: FAILED! => {"changed": false, "msg": "Could not find or access '/tmp/tom-rookci-618b03be_6decwpm1/rook_build/rook-ceph.
tar.gz' on the Ansible Controller.\nIf you are using a module and expect the file to exist on the remote, see the remote_src option"}
rook destroy
<tests.lib.rook.RookCluster object at 0x7f68c9079910>
kube destroy
<tests.lib.kubernetes.VanillaKubernetes object at 0x7f68c90d1160>
destroy nodes
<tests.lib.hardware.Hardware object at 0x7f68c9121070>
E

===================================================== ERRORS =====================================================
______________________________________ ERROR at setup of test_file_creation ______________________________________
tests/lib/rook.py:178: Exception
================================================ warnings summary ================================================
tests/test_basic.py: 52 tests with warnings
  /home/tom/devel/smoke_rook2/.tox/py38/lib/python3.8/site-packages/urllib3/connectionpool.py:979: InsecureRequestWarning: Unverified HTTPS request is being made to host 'engcloud.prv.suse.net'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
    warnings.warn(

tests/test_basic.py::test_file_creation
  /tmp/tom-rookci-618b03be_6decwpm1/mitogen-0.2.9/mitogen/utils.py:60: DeprecationWarning: sys.getcheckinterval() and sys.setcheckinterval() are deprecated.  Use sys.setswitchinterval() instead.
    sys.setcheckinterval(100000)

-- Docs: https://docs.pytest.org/en/latest/warnings.html
============================================ short test summary info =============================================
ERROR tests/test_basic.py::test_file_creation - Exception: One or more hosts failed
=================================== 53 warnings, 1 error in 675.23s (0:11:15) ====================================
ERROR: InvocationError for command /home/tom/devel/smoke_rook2/.tox/py38/bin/py.test --capture=sys -s tests/test_basic.py (exited with code 1)
____________________________________________________ summary _____________________________________________________
ERROR:   py38: commands failed

Handle failures in node boot thread

Because we are booting nodes in threads we need to check that they were actually successful. If an exception is raised we aren't handling it currently and might not notice if one of the nodes booted (in openstack provider)

Add SES support

We need to add support for SES deployment using rook.

Should not build rook but us prebuilt images.
Deploy using RPM based yaml files (rook-k8s-yaml)

Implement CaaSP support

Using the nodes that are brought up with the current libcloud provider, boot SLES nodes and install CaaSP via Skuba.

Move from docker to podman

Currently docker is used to build the rook images and then uploaded to the receiving cluster. We should use podman instead.
m
(This is only for the building of images. The cluster may still run docker).

libvirt & k8s upstream fails with: Unable to connect to the server: dial tcp 192.168.128.52:6443: i/o timeout

When trying to deploy with:

source examples/libvirt-k8s-upstream.env
tox -epy38 -- --pdb tests/test_basic.py

I get the following error. This happened now 4x in a row to me.

16:16:02        kubectl --namespace rook-ceph get pods    INFO : rook-ceph-operator-599765ff49-h8nrv   0/1     ContainerCreating   0          11s                                                                                                                                                                   [109/1225]
16:16:02        kubectl --namespace rook-ceph get pods    INFO : rook-ceph-tools-877c4d966-vlcd6       0/1     ContainerCreating   0          0s                                                                                                                                                                              
16:16:17        kubectl --namespace rook-ceph get pods    INFO : NAME                                  READY   STATUS              RESTARTS   AGE                                                                                                                                                                             
16:16:17        kubectl --namespace rook-ceph get pods    INFO : rook-ceph-operator-599765ff49-h8nrv   0/1     ContainerCreating   0          26s                                                                                                                                                                             
16:16:17        kubectl --namespace rook-ceph get pods    INFO : rook-ceph-tools-877c4d966-vlcd6       0/1     ContainerCreating   0          15s                                                                                                                                                                             
16:17:02        kubectl --namespace rook-ceph get pods WARNING : Unable to connect to the server: dial tcp 192.168.128.52:6443: i/o timeout                                                                                                                                                                                   
16:17:02                           tests.lib.rook.base    INFO : rook destroy on <tests.lib.hardware.libvirt.Hardware object at 0x7f73549800d0>                                                                                                                                                                               
16:17:02          tests.lib.kubernetes.kubernetes_base    INFO : kube destroy on hardware <tests.lib.hardware.libvirt.Hardware object at 0x7f73549800d0>                                                                                                                                                                      
16:17:02              tests.lib.hardware.hardware_base    INFO : Remove all nodes from Hardware                                                                                                                                                                                                                               
16:17:02              tests.lib.hardware.hardware_base    INFO : removing node rookcheck-tom-fa05-master-0 from hardware <tests.lib.hardware.libvirt.Hardware object at 0x7f73549800d0>                                                                                                                                       
16:17:02              tests.lib.hardware.hardware_base    INFO : deleted current ansible inventory dir /tmp/rookcheck/rookcheck-tom-fa05/inventory                                                                                                                                                                            
16:17:02              tests.lib.hardware.hardware_base    INFO : Inventory path: /tmp/rookcheck/rookcheck-tom-fa05/inventory                                                                                                                                                                                                  
16:17:02              tests.lib.hardware.hardware_base    INFO : removing node rookcheck-tom-fa05-worker-0 from hardware <tests.lib.hardware.libvirt.Hardware object at 0x7f73549800d0>                                                                                                                                       
16:17:02              tests.lib.hardware.hardware_base    INFO : deleted current ansible inventory dir /tmp/rookcheck/rookcheck-tom-fa05/inventory                                                                                                                                                                            
16:17:02              tests.lib.hardware.hardware_base    INFO : Inventory path: /tmp/rookcheck/rookcheck-tom-fa05/inventory                                                                                                                                                                                                  
16:17:03                    tests.lib.hardware.libvirt    INFO : Deleted disk rookcheck-tom-fa05-worker-0-volume-eltra at path /tmp/rookcheck/rookcheck-tom-fa05/rookcheck-tom-fa05-worker-0-volume-eltra.qcow2                                                                                                               
16:17:03              tests.lib.hardware.hardware_base    INFO : removing node rookcheck-tom-fa05-worker-1 from hardware <tests.lib.hardware.libvirt.Hardware object at 0x7f73549800d0>                                                                                                                                       
16:17:03              tests.lib.hardware.hardware_base    INFO : deleted current ansible inventory dir /tmp/rookcheck/rookcheck-tom-fa05/inventory                                                                                                                                                                            
16:17:03              tests.lib.hardware.hardware_base    INFO : Inventory path: /tmp/rookcheck/rookcheck-tom-fa05/inventory                                                                                                                                                                                                  
16:17:03                    tests.lib.hardware.libvirt    INFO : Deleted disk rookcheck-tom-fa05-worker-1-volume-vkzpb at path /tmp/rookcheck/rookcheck-tom-fa05/rookcheck-tom-fa05-worker-1-volume-vkzpb.qcow2                                                                                                               
16:17:03              tests.lib.hardware.hardware_base    INFO : removing node rookcheck-tom-fa05-worker-2 from hardware <tests.lib.hardware.libvirt.Hardware object at 0x7f73549800d0>                                                                                                                                       
16:17:03              tests.lib.hardware.hardware_base    INFO : deleted current ansible inventory dir /tmp/rookcheck/rookcheck-tom-fa05/inventory                                                                                                                                                                            
16:17:03              tests.lib.hardware.hardware_base    INFO : Inventory path: /tmp/rookcheck/rookcheck-tom-fa05/inventory                                                                                                                                                                                                  
16:17:04                    tests.lib.hardware.libvirt    INFO : Deleted disk rookcheck-tom-fa05-worker-2-volume-rehcw at path /tmp/rookcheck/rookcheck-tom-fa05/rookcheck-tom-fa05-worker-2-volume-rehcw.qcow2                                                                                                               
16:17:04                    tests.lib.hardware.libvirt    INFO : network rookcheck-tom-fa05 destroyed                                                                                                                                                                                                                         
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
                                                                                                                                                                                                                                                                                                                              
workspace = <tests.lib.workspace.Workspace object at 0x7f73549957f0>                                                                                                                                                                                                                                                          
                                                                                                                                                                                                                                                                                                                              
    @pytest.fixture(scope="module")                                                                                                                                                                                                                                                                                           
    def rook_cluster(workspace):                                                                                                                                                                                                                                                                                              
        with Hardware(workspace) as hardware:                                                                                                                                                                                                                                                                                 
            with Kubernetes(workspace, hardware) as kubernetes:                                                                                                                                                                                                                                                               
                with RookCluster(workspace, kubernetes) as rook_cluster:                                                                                                                                                                                                                                                      
                    if settings.as_bool('_USE_THREADS'):                                                                                                                                                                                                                                                                      
                        logger.info("Starting rook build in a thread")                                                                                                                                                                                                                                                        
                        build_thread = threading.Thread(                                                                                                                                                                                                                                                                      
                            target=rook_cluster.build())                                                                                                                                                                                                                                                                      
                        build_thread.start()                                                                                                                                                                                                                                                                                  
                                                                                                                                                                                                                                                                                                                              
                    # build rook thread                                                                                                                                                                                                                                                                                       
                    hardware.boot_nodes(masters=settings.NUMBER_MASTERS,                                                                                                                                                                                                                                                      
                                        workers=settings.NUMBER_WORKERS)                                                                                                                                                                                                                                                      
                    hardware.prepare_nodes()                                                                                                                                                                                                                                                                                  
                    kubernetes.bootstrap()                                                                                                                     
                    kubernetes.install_kubernetes()                                                                                                            
                                                                                                                                                               
                    if settings.as_bool('_USE_THREADS'):                                                                                                       
                        logger.info("Re-joining rook build thread")                                                                                            
                        build_thread.join()                                                                                                                    
                    else:                                                                                                                                      
                        rook_cluster.build()                                                                                                                   
                                                                                                                                                               
                    # NOTE(jhesketh): The upload is very slow.. may want to                                                                                    
                    #                 consider how to do this in a thread too but                                                                              
                    #                 is more complex with ansible.                                                                                            
                    rook_cluster.preinstall()                                                                                                                  
>                   rook_cluster.install()                                                                                                                     
                                                                                                                                                               
/home/tom/devel/suse/rookcheck/tests/conftest.py:213:                                                                                                          
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/home/tom/devel/suse/rookcheck/tests/lib/rook/base.py:85: in install                                                                                           
    common.wait_for_result(                                                                                                                                    
/home/tom/devel/suse/rookcheck/tests/lib/common.py:55: in wait_for_result                                                                                      
    out = func(*args, **kwargs)                                                                                                                                
/home/tom/devel/suse/rookcheck/tests/lib/kubernetes/kubernetes_base.py:98: in kubectl                                                                          
    return common.execute(                                                                                                                                     
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
                                                                                                                                                               
command = '/tmp/rookcheck/rookcheck-tom-fa05/bin/kubectl --kubeconfig /tmp/rookcheck/rookcheck-tom-fa05/kubeconfig --namespace rook-ceph get pods', capture = True, check = True, log_stdout = True, log_stderr = True, env = None, logger_name = 'kubectl --namespace rook-ceph get pods'

Give options to debug failures in job setup

Currently when a job fails during the setup phase, the test is marked as failed and the infrastructure destroyed. We should provide a flag to dump the user into an environment to test it.

Should the quickstart suggest to install system tox using pip?

In the quick start we can observer the next steps.

sudo zypper in python-pip
sudo pip install tox
sudo zypper in $(tox -qq -e bindep -- -b)
sudo systemctl start docker
sudo usermod -aG docker $USER

From my point of view we should not ask a user to install untested tox version to the system area.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.