To ensure that our cloud platform continues to meet our quality standards over time, we use a set of acceptance tests to validate various aspects of our offering:

• Features work as documented.
• Response times meet our expectations.
• Regressions are avoided.

These tests are run regularly against our public infrastructure as well as our internal test environment where upgrades are staged prior to rollout.

⚠️ Running these tests yourself may incur unexpected costs and may result in data loss if run against a production account with live systems. Therefore, we strongly advise you to use a separate account for these tests.

ℹ︎ Note that you need at least Python 3.6.

To install the tests, you have to clone the repository:

git clone [email protected]:cloudscale-ch/acceptance-tests.git

Now, every time you want to run the tests in a new shell, use the following command first:

source acceptance-tests/pre-flight

You will be automatically switched to the acceptance-tests directory, ready to run the tests as outlined below.

To run all tests, run py.test as follows:

py.test .

To only run a specific test, run py.test as follows:

py.test . -k <test-name>

By default, all tests are run against the default image, most tests are run against a set of common images, and some tests are run against all images provided by cloudscale.ch.

To run all tests against a specific image, use this image as the default:

py.test --default-image ubuntu-20.04 --default-image-only

Note that our default image is Debian 10. If you pick a different default image your results may differ.

Custom images can be used as the default image by specifying their slug, along with a username that can be used to connect via SSH. Note that custom images are less likely to pass all tests without prior modification, as the acceptance tests mainly focus on our common images.

py.test --default-image custom:alpine --default-image-only --username alpine

By default, tests are run against a randomly selected zone.

Alternatively, you can specify the zone to run the tests against:

py.test --zone rma1
py.test --zone lpg1

During test development, it can be useful to manually connect to hosts created by the tests. In this case it is necessary to explicitly specify your own SSH key, since tests connect to hosts using temporary SSH keys only:

py.test --ssh-key ~/.ssh/id_rsa.pub

Sometimes it is useful to run a specific test multiple times in a row:

py.test --count=10 test_floating_ip.py

During execution, the acceptance tests generate a detailed log in the events directory (one file per test-run). Each line in such an event log is a structured JSON object.

Using a custom command, you can create a human-readable output of this log:

invoke pretty-print --file events/<file>

You can include filters as well:

invoke pretty-print --file events/<file> --regex outcome=failed

Or, during test execution, you can follow the log in a separate terminal window while it is being written. This will tail all the event logs that are currently being written. No need to specify a single file.

invoke tail

During normal operation, all resources created by the acceptance tests are automatically cleaned up. However, if the process receives a SIGKILL signal, or if it crashes, there may be resources left afterwards.

If you want to be sure, you can clean up all resources created by any acceptance test using the cleanup command:

invoke cleanup

All resources created by acceptance tests receive a unique tag, based on a securely hashed version of the API token, so using this command should be reasonably safe. However, we still strongly advise you to use a separate account for these tests as a precaution.

In order to review tests and to be able to develop multiple tests in parallel, they should be developed in a separate Git branch:

git branch <your_branch_name>

If you write a test with the image fixture, it will be called with the default image. This default image can be changed using the --default-image command line parameter.

If you want to ensure that a test runs against all common images, use the image fixture and include all_images in the name of your test:

def test_all_images_have_a_hosts_file(create_server, image):
    server = create_server(image=image)

If you use common_images in the name of your test, only common images will be tested:

def test_common_images_have_a_hosts_file(create_server, image):
    server = create_server(image=image)

git add <new_or_changed_files>
git commit

git push origin <your_branch_name>

To create a pull request follow the link that will be displayed upon pushing a branch.