I think most of the activity we're hearing about is on EKS and the QS may not be renewed for 1.5 in favor of TF. If e2e supported EKS then I believe we'd see heavy use.

provision nodes to attempt joining the cluster on EKS and this way remove the need of doing manually.

It was working on 0.11 scripts...

During execution of terraform plan I got following errors :

Error: Error fetching Availability Zones: UnauthorizedOperation: You are not authorized to perform this operation.
	status code: 403, request id: 17536565-dcb6-49c3-ab83-1f8f29a5f64f

  on providers.tf line 18, in data "aws_availability_zones" "available":
  18: data "aws_availability_zones" "available" {



Error: Error finding Route 53 Hosted Zone: AccessDenied: User: arn:aws:iam::598531485887:user/vault.publiccloud.qa.suse.de is not authorized to perform: route53:ListHostedZones
	status code: 403, request id: ee80b0aa-b77c-4fcc-bf5f-aaa2f9407d8a

  on modules/eks/worker-iam.tf line 2, in data "aws_route53_zone" "selected":
   2: data "aws_route53_zone" "selected" {



Error: Error fetching Availability Zones: UnauthorizedOperation: You are not authorized to perform this operation.
	status code: 403, request id: b16c1421-3af5-4802-be52-5bc6f3f93990

  on modules/network/vpc.tf line 20, in data "aws_availability_zones" "available":
  20: data "aws_availability_zones" "available" {}

so looks like credentials which I was using does not have proper permissions but I failed to find any document mentioning which exactly

On EKS deployments, it would be nice to have a default storageclass named persistent, instead of gp2. That would bring consistency with the documentation, and allows to easily reuse existing automation without specific cases for EKS.

I've encountered 2 race conditions recently that I believe have been caused by terraform's management of the aws-auth configmap here

In a recent CI-initiated terraform destroy, the following error was encountered:

...
module.eks.aws_security_group.aws-node: Still destroying... [id=sg-069b01ce27e2bffd3, 30m1s elapsed]
Error: Error deleting security group: DependencyViolation: resource sg-069b01ce27e2bffd3 has a dependent object
	status code: 400, request id: 11278f08-4336-4f13-b335-bf0dc38e1ed8

During debugging, I noticed that kube couldn't be accessed with the kubeconfig generated by terraform output, which led to the discovery that this configmap had already been deleted. This cluster is now in a state where terraform can't destroy it, due to dependencies still existing in that security group, which @colstrom has speculated is due to cleanup of an eni expected to be managed by the worker node failing, because the worker node could no longer manage cluster resources after deletion of this configmap.

This configmap has also led to race conditions during deployment, where terraform apply has been failing with alarming frequency, with the following error:

module.services.kubernetes_config_map.aws_auth: Still creating... [30s elapsed]

Error: Post https://6A7BFC843CD4C7578DCB503446548A17.gr7.us-west-2.eks.amazonaws.com/api/v1/namespaces/kube-system/configmaps: dial tcp 34.218.122.126:443: i/o timeout

  on modules/services/aws_auth_cm.tf line 7, in resource "kubernetes_config_map" "aws_auth":
   7: resource "kubernetes_config_map" "aws_auth" {

Though I believe this is due to attempting to overwrite/patch it before the kube api is up.

deploy SCF on EKS automatically after spinning up an EKS cluster (which is working already) with DNS and other infra components required.

Instead of hardcoding provider credentials on a variables.tf, we could either pass them as env vars, or letting terraform take the credentials from the host. This allows for easier reuse of the terraform deployments on CI.

Having the credentials included in variables.tf simplifies the operator life when having several similar deployments from different accounts, but I would argue that this is somewhat rare. And anyways, the same could be achieved with a file containing the env vars to be loaded, or a different wrapper like https://github.com/SUSE/catapult.

See:
https://www.terraform.io/docs/providers/aws/index.html#environment-variables
https://www.terraform.io/docs/providers/google/provider_reference.html#full-reference

Stratos components must be added to the CAP deployment flow within Terraform script.

Hello.

Problem

For KubeCF (and CAP 2.0 then) all platforms should deploy with Helm 3. Right now, the terraform scripts fail because the helm provider that we are using is for Helm 2.

We need to keep both Helm 2 and Helm 3 terraform scripts around, to be able to deploy CAP 1.X and CAP 2.X though.

Workaround

For now, we are deploying as it is with Helm 2 with cap-terraform, and then reusing the kubeconfig on a folder with Helm 3. With catapult: BACKEND={gke,eks} make k8s which we have pinned to Helm 2, and then, HELM_VERSION=v3.3.1 BACKEND={gke,eks} make kubeconfig locally and on pipelines. This leaves Tiller pods on the deployment, for which we don't care for now.

At the moment we expect users to provide a ResourceGroup for spawning azure clusters. However, it would make things easier for users esp. via catapult if we handle its creation and deletion via terraform.

tiller service account should only have rights to SCF and UAA namespaces, not cluster-wide open access.

Currently kubeconfig produced by terraform has name aksk8scfg it appears that this name is not straightforward for person who is not familiar with project. So need to come up with more obvious name ( kubeconfig ? )

terraform destroy for the GKE cluster does not remove PVs. We need to delete the disks with a destroy-time provisioner:

https://www.terraform.io/docs/provisioners/index.html#destroy-time-provisioners