When a TestCase is run which contains a large number of tests, should any of those tests fail they are printed along with passed tests in what could be a very long list. When the number of tests becomes quite large, for example over 100, one must carefully scan every entry to see what the failed tests are. This is tedious and can result in tests being missed.

When any tests fail in a TestCase containing multiple tests, aggregate and display the failed tests together as the final portion of the output. Do not change any of the existing output, just add a section at the bottom which groups only the failed tests together.

When kuttl parses a TestStep file, there is currently no validation performed to ensure basic things such as:

1. a field is specified but unrecognized / indentation errors
2. a required field is missing
3. a referenced file in a field cannot be located

We need to add some of these basic validation checks to ensure authors of these kuttl tests don't waste time chasing simple problems.

What would you like to be added:

An easy, declarative way to check that output from the result of a command, script, or CLI:

1. equals what I define (i.e., it has nothing more than what I state and looks precisely what I define)
2. contains what I define (i.e., it may have more than what I state and it may be found anywhere in the output)

Why is this needed:

We need a way to test things like kyverno apply and kyverno jp reliably outside the confines of unit tests and from an end user perspective.

Kyverno's version of kuttl needs several enhancements to make it eminently usable for Kyverno's test needs. This issue captures all of the enhancements needed not present in upstream and which we intend to add here.

• Wait for Namespace deletion (#6)
• Support for individual test case disabling (#5)
• Support for simple declaration of manifest application failure (#17)
• Support for directory recursion (#16)
• Show full path to test cases in output (#4)
• #25
• #24
• #26
• #22
• #23

When tests are performed serially as they are in Kyverno's case, no tests should fail. For any test to fail indicates a problem somewhere. Continuing to test other tests may often be undesirable as it wastes time and processing in order for a user/contributor to know what may be broken and requires remediation.

Add a parameter to a TestCase file to allow the user to indicate that testing should not proceed upon the first test failure.

What would you like to be added:

A way to make an assertion that the contents of a field which is an array of objects or a list:

1. has the defined portion listed somewhere/anywhere
2. has the defined portion listed in the prescribed order

Why is this needed:

Kubernetes has loads of resources where arrays are components. In some cases, lists are unordered (like environment variables or Policy Report results) and in others the ordering is important (like initContainers). We need a way to specify how the assertion should be performed which entails stating "this entry must be present in the collection" versus "the collection must have all these entries and they must be in this exact order".