According to KEP-05:

• Test setup and tear down methods (is currently coupled to steps).

Based on KEP-05

A generic Kubernetes client (currently this is contained inside of pkg/test/harness.go)

What happened:

There is a race condition when creating namespaces in Kubernetes that if you create a Pod in the namespace before the ServiceAccount exists, it will fail.

We should resolve this for the tests by waiting for the default ServiceAccount to exist in created namespaces before continuing on to the test steps to prevent errors like:

            case.go:102: pods "influxdb-querier" is forbidden: error looking up service account kudo-test-lenient-lab/default: serviceaccount "default" not found

https://github.com/kubernetes/community/tree/master/sig-testing

Should we have a kuttl plugin? seems useful.

According to KEP-05:

Additionally, Operator-specific helper methods will be provided in the library that can be used by library users.

We will also endeavor to stay close to the API used by the kudo-cassandra-operator testing library to ensure that migration of existing tests is easy.

According to KEP-05:

The cmd package should provide conveniences for running arbitrary commands (i.e. shelling out) from Go code in integration tests.

Current functionality:

• Providing an Exec function that returns exit status, stdout and stderr

package cmd
func Exec(
  command string,
  arguments []string,
  environmentVariables map[string]string,
) (
  exitStatus int,
  stdout *bytes.Buffer,
  stderr *bytes.Buffer,
  error error,
)

[ ] go run cmd/kubectl-kuttl/main.go test pkg/test/test_data/
[] CI setup to run tests and ^^ tests
[] update makefile for tests and integration-test
[] circle ci
[] goreleaser release
[] .git support for templates and PR defaults

this is a feature request, for the kuttl API to provide a means to obtain the Pod log when a test step executes a Pod, the API would let you get back the Pod log via the API call to execute a Test Step.

Our use case includes running custom container images within a Test Step, those images will need to produce log output in a particular format (e.g. v1alpha2 in our case), then we can aggregate or do other processing on the test log output, our test logs from the pods will show various kinds of feedback useful for the end-user (e.g. suggestions or error descriptions).

It would be worth added a retry to commands... this would retry to the command upon failure. Controls might include:

• number of retries
• amount of time to retry
• timeout for retry
• freq of retries

Add ability to have before each and after each conditions defined in TestSuite to be applied / asserted before and after each test

StartKUDO bool json:"startKUDO"` should be removed

The CLI should function without a kuttl-test.yaml as long as all required details are provided

for instance k kuttl test --start-control-plane pkg/test/test_data/ should run without a file

According to KEP-05:

• Cluster and test environment provisioning (this is currently coupled to Harness in harness.go).

When the BG process is started KUTTL moves forward with other commands (and tests).. however the commands could need the BG process to be ready to success. We do want to the process in the background however we need to know when it is live.

An example use case:

commands:
  - command: ./bin/kubectl-kudo init --crd-only	  - command: ./bin/kubectl-kudo init --crd-only
  - command: ./bin/manager
    background: true
  - command: ./bin/kubectl-kudo install --skip-instance ./repository/cassandra/3.11/operator/	  

The kudo install command will fail if manager isn't ready. We need manager to be "ready" before the install command.

There should be a shared testing library that provides all necessary facilities for complex KUDO-based operators to have extensive and thorough integration tests.

Take a look at the kudo-cassandra-operator testing facilities to get a better understanding to how this differs from the current KUDO test harness, and provides functionality that is still not clear how to implement in the KUDO test harness, including a single verbose and useful test run output, useful for development and CI.

There's a private protokep for this in Google Drive.

According to KEP-05:

The kubectl package should provide conveniences for interacting with a Kubernetes cluster as one would do via kubectl from the shell.

package kubectl
func Exec(
  namespaceName string,
  containerName string,
  command string,
  argvuments []string,
  environmentVariables map[string]string,
) error

The newly added technique to start a background process is working in testing but leaves the process running after exit. It isn't clear yet why... theories:

1. the process exits too quickly for the kill to take place
2. the process is a grandchild (not a child) which go doesn't kill (frankly we may need to support killing all child and grandchild processes anyway.

for ref: https://medium.com/@felixge/killing-a-child-process-and-all-of-its-children-in-go-54079af94773

Asserts today are easy for YAML or K8s objects... however there are users wanting to do a regex match against pod logs. Is there a way to make this easier?

or at a min, allow a command used for assertion that can be repeat for each test.

perhaps there is something more here... pull and logging a pod log to test might be useful.

RunCommands in kubernetes.go returns background processes which we ignore for kubectl commands and steps. However there should be an error or warning and that command should not leak. (which it currently does)

for a bad kuttl-test.yaml for with kuttl v0.1.0 the kuttl cli panics... please don't

to duplicate... change apiVersion: kudo.dev/v1beta1 to apiVersion: kuttl.dev/v1beta1

we should probably also switch to kuttl.dev

According to KEP-05:

• Assertion methods (needs better documentation for library usage).

We should add a new setting to the commands struct called script that accepts a bash script as a string that we run. When running the script, we can set the $NAMESPACE and $KUBECONFIG variables. This will allow users to more easily write scripts inline when necessary.

apiVersion: kudo.dev/v1alpha1
kind: TestStep
commands:
- script: until ! kubectl get pod --namespace $NAMESPACE my-pod; do sleep 1; done

The best part about data-driven tests is we can easily perform fuzz testing and create generative tests around parameters. @jbarrick-mesosphere wdyt?

We need a contribs doc...

in it we need to make sure devs are aware that we are dependent on kubebuilder... along with other dependents and requirements to get started.

What would you like to be added:

In the test harness, you can match objects without specifying a name, for example:

apiVersion: v1
kind: Pod
metadata:
  labels:
    app: nginx
status:
  state: READY

It should be possible to specify how many objects match.

Why is this needed:

This can be used to verify that a certain number of pods are ready or a certain number of plan executions exist.

Since KUTTL has it's own domain, we should update the CRDs to be kuttl.dev/v1 (or v1beta1)

Need a way to define an assertion that is true for every test in the testsuite

we will need to be clear if this is every teststep or every test... perhaps configurable.

User should be able to define a yaml assertion that is tested for each test and not need to repeat that for every test configuration. This is part of the spirit of the composability KEP.

What happened:

We currently use a very naive loop for implementing the test timeout:

• https://github.com/kudobuilder/kudo/blob/master/pkg/test/step.go#L318

The timeout parameter currently means the number of times the check is run and not the number of seconds. Instead a Context should be used to enforce the timeout.

Perhaps it could be useful to define command aliases for a testsuite for long pathed commands that are repeated.

example:

commands:
- command: ../../../bin/dispatch -n dispatch-tasks serviceaccount create multi-namespace

currently it is possible to specify a CRD dir in TestSuite after which there is a CRD wait.
It is also documented that a CRD could be the result of a TestStep with a pattern to follow to "wait" via an Assert.

It would be good to support a way to waiting other that TestSuite. It would be good to support CRD via a command exec from TestSuite.

If possible we should:

1. define a publicly exposed wait function for CRDs which were created (perhaps this is only supported via API ... it would be great to support declaratively
2. provide a way in TestSuite to define CRD via a command. so a k kudo init --crd-only could be used to define the CRD (which would wait)
3. We should support the CRD command in a TestStep as well.

kudobuilder/kudo#1420 (comment)

A great strategy for "ensuring" a controller or resources is to have a step 0 assert of that instance running
see kudobuilder/kudo#1473
test/e2e/cli-install-uninstall/00-assert.yaml

apiVersion: v1
kind: Pod
metadata:
  labels:
    app: kudo-manager
    control-plane: controller-manager
  namespace: kudo-system
status:
   phase: Running 

Assuming you may want this as a precondition on all tests... it would be great to assert this at the end of TestSuite setup prior to tests.

The kuttl cli overrides the testsuite file... with the exception of the test env. (this is a require to fix that).

If the kuttl-test.yaml has a configuration for startControlPlane: true and the cli is started with --start-kind.... the cli override should start kind.

Perhaps we should have an override that allows to point at a cluster (currently it is the default that is selected if nothing is provided).

Perhaps pull kudo docs and modify to be generic

This code was pulled from kudo... where there was a runKUDO function that injected kudo with the kudo flag.

We need a way to declare a controller to add... or a lifecycle hook

The k8s package should provide conveniences for interacting with Kubernetes objects.

• Partial CRUD functionality for namespaces
• CRUD functionality for all useful Kubernetes objects
• Current known use-cases:
• Creating and deleting namespaces during test runs

This should likely include WaitFor functions

package k8s
func Init(kubectlOptions *kubectl.KubectlOptions) error
func CreateNamespace(namespaceName string) error
func DeleteNamespace(namespaceName string) error

currently kubebuilder with etcd is required and needs to be documented as such

seeing projects is commonly looking for a configmap for a teststep
example

apiVersion: kudo.dev/v1alpha1
kind: TestStep
commands:
- command: 'kubectl create configmap example-cm --from-file=../../jobs/tensorflow.py -n tensorflow-testing'

looks like a need for create, from file, from values with a namespace

There are two issues with the built-in kind cluster functionality (that are likely related):

1. If a cluster has no containers running but has not been deleted (e.g., kind delete cluster was not ran and --skip-cluster-delete was set), kuttl will still attempt to use it and hang forever.
2. When the kind cluster is already running, kuttl will not detect it and attempt to start a new cluster.

These may be regressions related to the kind version bump from 0.5.1 to 0.7.0 as upstream kind changed a lot of the API and how kind clusters are managed.

• I have a TestStep that calls kubectl, I'm wanting the TestStep to use a namespace for running that kubectl command...is this possible? so far, I have to add the --namespace onto the kubectl command string itself to have it run within that namespace.

apiVersion: kudo.dev/v1alpha1
kind: TestStep
delete:
- apiVersion: v1
  kind: ConfigMap
  name: my-static-test
  namespace: rq
kubectl:
- create configmap my-static-test --from-file=bundle=/tmp/bundle.zip --namespace=rq

When kuttl runs commands specified in a TestStep or TestSuite, they can run forever as we use exec.Command and not exec.CommandContext.

https://github.com/kudobuilder/kuttl/blob/master/pkg/test/utils/kubernetes.go#L929

kuttl.io / kuttl.dev either.. both... it would be nice to have this in place prior to April 1

KUDO needs to use this new dependency

• needs to be added for kudo test
• inject kudo manager into the tests (with designed approach)

What happened:
While working on a test that includes a webhook, I noticed that timeout parameter is not used when the test step fails in the apply-phase:

    harness.go:66: going to run test suite with timeout of 3000000 seconds for each step
    --- FAIL: kudo/harness (0.02s)
        --- FAIL: kudo/harness/terminal-failed-job (7.67s)
            logger.go:37: 14:51:05 | terminal-failed-job | Ignoring failjob-operator as it does not match file name regexp: ^(\d+)-([^.]+)(.yaml)?$
            logger.go:37: 14:51:05 | terminal-failed-job | Ignoring job-timeout-operator as it does not match file name regexp: ^(\d+)-([^.]+)(.yaml)?$
            logger.go:37: 14:51:05 | terminal-failed-job | Creating namespace: kudo-test-heroic-dodo
            logger.go:37: 14:51:05 | terminal-failed-job/0-install | starting test step 0-install
            logger.go:37: 14:51:05 | terminal-failed-job/0-install | running command:  {kubectl kudo install --instance failjob-operator-instance ./failjob-operator %!s(bool=true) %!s(bool=false) %!s(bool=false)}
            logger.go:37: 14:51:12 | terminal-failed-job/0-install | Error: failed to install instance failjob-operator-instance: installing Instance: Internal error occurred: failed calling webhook "instance-admission.kudo.dev": Post https://kudo-controller-manager-service.kudo-system.svc:443/admit-kudo-dev-v1beta1-instance?timeout=30s: dial tcp 10.103.201.124:443: connect: connection refused

            logger.go:37: 14:51:12 | terminal-failed-job/0-install | operator.kudo.dev/v1beta1/failjob-operator created
                operatorversion.kudo.dev/v1beta1/failjob-operator-0.1.0 created

            case.go:156: exit status 255
            case.go:159: failed in step 0-install

The code suggests that the timeout is only respected during the assertion phase of the step. I often use the timeout to "pause" the execution of the failing step and look into the cluster (using kubectl). A number of k8s admission controllers might reject applied resources so it might be beneficial to extend timeout to the whole step and not just the assertion phase.

What you expected to happen:
The timeout should be applied to the whole step.

How to reproduce it (as minimally and precisely as possible):
Using any step command that fails (e.g. false)

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):
• KUTTL version (use kubectl kuttl version):
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

Currently, the test harness polls for object state when tests are running which offers opportunity for races (and is likely more inefficient). For example, to test delete steps in KUDO properly, we need to be able verify that a Job is created and is completed before it gets deleted, without racing KUDO, if we poll, there is a chance we can miss the Job before KUDO deletes it.

KUTTL currently always creates namespaces for each test step, but some users may want to run their tests in a specific namespace or target an existing deployment. This would also be useful for running kudo test in less privileged environments where it is unable to create namespaces.

It is possible today to target static namespaces by specifying them on any commands or manifests, but it would be useful to add a --namespace flag to kudo test that controls the namespace used.

For example, if --namespace is set on kudo test, instead of creating namespaces for each test step and using the created namespace as the default for each resource, kudo test will set the default namespace to the --namespace flag and use that instead.

What happened:

Currently, the test harness panics if it cannot find etcd and the kube-api-server and start-control-plane is set. We should handle this more nicely. Either we download them for the user or bail out with a nice error.

For presentation mentioned on #38

possibly this script: https://github.com/kubernetes/kubernetes/blob/master/test/cmd/apps.sh

What would you like to be added:

Add a gracePeriod or timeout to the error teststep type.

Why is this needed:

I have a controller that migrates one type of object to another. I want to be able to specify that after some time, my original object no longer exists. Currently I cannot do that as it may take some amount of time for the controller to pick up the original object, create the derived object, and remove the original. Currently, the error teststep would fail immediately as the original object will likely still exist.