gluster / distaf Goto Github PK

Automation Test Framework For Distributed Systems

License: GNU General Public License v2.0

Shell 3.48% Python 96.52%

distaf's Introduction

DiSTAF - Di'stributed Systems Test Automation Framework

DiSTAF is a test automation framework for distributed systems. It is used for test automation of glusterfs and its related projects. This framework is written with modularity in mind. So many parts of it can be modified for the liking of the project, without affecting other parts. DiSTAF can be used to test projects which runs on physical machines, virtual machines or even containers. DiSTAF requires remote machines (or containers) to be reachable by IP address (or FQDN). On Linux systems it requires sshd to be running in the remote systems with bash environment as well.

About the name

DiSTAF (or distaf) is short for Di'stributed Systems Test Automation Framework. 'distaff' is a tool used in spinning, which is designed to hold unspun fibres together keeping them untangled and thus easing the process of spinning.This framework is also trying to do just that, keeping the machines untangled and easing the process of writing and executing the test cases. Hence the name DiSTAF (distaf).

Architecture of the Framework

Terminologies used

Management node - The node from which this test suite is executed. This node is responsible for orchestration of test automation.

Test machines - These include all the systems that participate in the tests. This can be physical machines, VMs or containers.

To run distaf, passwordless ssh should be setup from management node to all the test machines. The management node connects to test machines using rpyc zero-deploy, which internally makes use of ssh tunneling protocol for establishing and maintaining the secure connections. The connection is kept open for the entire duration of the tests. All the synchronous commands run by the test cases, uses this connection to run them. For asynchronous calls, a new connection is opened. This connection will be closed when async command returns.

DiSTAF uses python-unittest for running tests and generating the results.

When a test run is started, DiSTAF first reads a config file, which is in yaml format. The config file will have information about servers and clients DiSTAF can connect to. DiSTAF establishes a ssh connection to each of the servers and clients, and maintains the connection until the end of the test run. All the remote commands, bash or python will go through this connection. DiSTAF provides two APIs to run commands on the servers or clients synchronously and asynchronously. For more information about distaf APIs, please refer HOWTO.

Test case philosophy

DiSTAF has two modes of running. The Global Mode and Non-global Mode. There is a configuration variable in config.yml global_mode to toggle between them. The idea here is that each test case should be independent of the volume type and access protocol used to mount the volume.

When the distaf is started in the non-global mode, it runs each test case against all the volume type and mount protocol combinations. This means a single test case will run many times and each time a different volume and mount combination is used. Each test case will have it's own metadata in yaml format in test case docstring. For more information about the fields and values of test case metadata (test case config), please refer to HOWTO.

When distaf is started in global mode, each test case is run only once. The volume type and mount protocol specified in the config.yml is used for each test case. This is helpful if a test case needs to run against a particular type of volume, to run some checks.

Few things to take care before running test case in DiSTAF.

Setting up and provisioning the test machines. This needs to be handled before running distaf tests.
Updating the config.yml and setting up password-less ssh from management node to test machines.
Keeping the test machines in the same state if a test case fails. Since distaf does not manage the bringing up and maintaining the test machine, this should be handled outside distaf as well.

distaf's People

Contributors

Stargazers

Watchers

Forkers

amit05 mnomkar2008 lalatendumohanty nixpanic loadtheaccumulator msvbhat apekhsha aloganat anoop20 neharawat shwetha-h-p savithabs ericdoug gridl vdas-redhat

distaf's Issues

All source files should have a license boilerplate

Please add a license and copyright notice in all of the source files.

Add software version check to test metedata

Because tests and code are split over multiple repositories, it would be good to have an option in the test metadata that can be used to only run the test if the version of the components is high enough. (Or run the tests, but do not include the result in the final PASS/FAIL result?)

Practical example:
I would like to add a test to distaf that tests for different mount options. The change in Gluster might not be merged or backported to all branches yet. It would be bad for distaf to fail for tests that are not expected to pass for certain versions.

code reorg

get_config_data() in lib/util.py could be more elegant. Instead of a series of if/else statements, iterating a list of possible keys (along side the regular assignments) would be better IMO.

Two questions

I have two questions need to inquiry:
1, when I perform the test case, prompt yaml is not found, yaml is what? How is generated?
2, util.py document, testcase, test_list, test_seq, test_mount, globl_configs, global_mode, tc value fill out there is no limit? Can you see you how to fill in?

Supress the AssertionError in printing the result

Right now whenever there is a failure in each test case, the result says FAIL and throws exception. That exception looks bit ugly and should be suppressed.

Package the management node in a docker container

Packaging the management node in a docker container will help users quickly started with running the tests and contribution.

Update README and create HOWTO

There should be guide to write and execute new test case. Please add proper documentation in the source code itself.

help

Distaf is a distributed testing framework, but it can be done? What are the advantages? Whether can describe for me? I saw a test tool blockade yesterday, I know it can be done? But I see the distaf, I don't know what can it do?

No /var/log/tests/ at test nodes (gluster server) to verify the test result

I installed and ran distaf against two gluster server and its result is passed it seems but I could not see any gluster related info at the test nodes (gluster server).

I did ssh-copy-id for user fractalio which is not root user.

Do I need to do ssh-copy-id for root user ?

How to verify that the testcase has been really executed on the test nodes ?

Let me know, if you need any further info.

config.sh file content:

export MGMT_NODE="mgmt_node"

export NODES="192.168.1.244"

export CLIENTS="client0 client1"

export PEERS="192.168.1.245"

export GM_NODES="gm_master0 gm_master1"

export GS_NODES="gs_slave0 gs_slave1"

development1@development1-ME051 ~/Documents/distaf $ sudo python main.py

test_gluster_basic_test (main.gluster_tests) ... ok

Ran 1 test in 0.001s

development1@development1-ME051 ~/Documents/distaf $ tail -f /var/log/tests/distaf_tests.log

2015-02-20 13:49:49,254 INFO gluster_basic_test Testing gluster volume create and mounting
^C

peer_status throws exception when there is a single node in the cluster

When there is a single node in the cluster, the peer validation fails because peer status fails.

Add the option to tag test cases, and run (or skip) certain tags

One of the QA guys reported a bug against Gluster/NFS. The problem was reproducible with quite a time-consuming test:

mount -t nfs storage.example.com:/volume /mnt
cd /mnt
if [ ! -e linux-3.19.tar.gz ]
then
    curl https://cdn.kernel.org/pub/linux/kernel/v3.x/linux-3.19.tar.gz > linux-3.19.tar.gz
fi

# abort on error
set -e

for i in {1..1000}
do
    tar -xvf linux-3.19.tar.gz
    sleep 180
    rm -rf linux-3.19
done

I would like to write a test-case for this, but it should not execute by default. It would be nice to have an option to mark a test-case with "tags: long-running" or something in the doc string so that tests like this can easily be skipped.

Running unmodified distaf fails with Python stack in geo_rep/test_geo_basic.py

When I run distaf from the main source directory, I get the following Python stack:

# ./main.py -c example.conf.yml 
Traceback (most recent call last):
  File "./main.py", line 117, in <module>
    main()
  File "./main.py", line 100, in main
    collect_tests()
  File "./main.py", line 36, in collect_tests
    __import__(iname.replace("/", "."))
  File "/opt/distaf/tests_d/geo_rep/test_geo_basic.py", line 4, in <module>
    @testcase("changelog-test-create")
  File "/opt/distaf/distaf/util.py", line 49, in decorator
    tc_config = get_testcase_config(func.__doc__)
  File "/opt/distaf/distaf/config_parser.py", line 33, in get_testcase_config
    config_dict = yaml.load(config_string)
  File "/usr/lib64/python2.7/site-packages/yaml/__init__.py", line 69, in load
    loader = Loader(stream)
  File "/usr/lib64/python2.7/site-packages/yaml/loader.py", line 34, in __init__
    Reader.__init__(self, stream)
  File "/usr/lib64/python2.7/site-packages/yaml/reader.py", line 85, in __init__
    self.determine_encoding()
  File "/usr/lib64/python2.7/site-packages/yaml/reader.py", line 124, in determine_encoding
    self.update_raw()
  File "/usr/lib64/python2.7/site-packages/yaml/reader.py", line 178, in update_raw
    data = self.stream.read(size)
AttributeError: 'NoneType' object has no attribute 'read'

Removing the tests_d/geo_rep directory makes it possible to run the tests.

Better way to handle exceptions in each test case

Right now whenever there is a exception in any test case it is caught and only name of the exception is thrown. But this is not very friendly for debugging. So either don't catch the exception or find a way to print the entire stack trace in a better way.