Deployments always fail as follows:

+ skaffold deploy -p ephemeral-deployment -f skaffold.yaml --build-artifacts=default-tags.json
Tags used in deployment:
 - soulshake/front-end -> soulshake/front-end:main
Starting deploy...
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /home/runner/.kube/config
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /home/runner/.kube/config
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /home/runner/.kube/config
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /home/runner/.kube/config
Release "microservice-demo-pr-front-end-2" has been upgraded. Happy Helming!
NAME: microservice-demo-pr-front-end-2
LAST DEPLOYED: Sat Oct 17 17:05:12 2020
NAMESPACE: pr-front-end-2
STATUS: deployed
REVISION: 2
TEST SUITE: None
Waiting for deployments to stabilize...
 - pr-front-end-2:deployment/carts-db is ready. [12/13 deployment(s) still pending]
 - pr-front-end-2:deployment/queue-master: running [kubectl --context ephemeral-demo-spot rollout status deployment queue-master --namespace pr-front-end-2 --watch=false]
 - stdout: ""
 - stderr: "error: deployment \"queue-master\" exceeded its progress deadline\n"
 - cause: exit status 1
 - pr-front-end-2:deployment/queue-master failed. Error: running [kubectl --context ephemeral-demo-spot rollout status deployment queue-master --namespace pr-front-end-2 --watch=false]
 - stdout: ""
 - stderr: "error: deployment \"queue-master\" exceeded its progress deadline\n"
 - cause: exit status 1.
12/13 deployment(s) failed
 - pr-front-end-2:deployment/carts: running [kubectl --context ephemeral-demo-spot rollout status deployment carts --namespace pr-front-end-2 --watch=false]
 - stdout: ""
 - stderr: "error: deployment \"carts\" exceeded its progress deadline\n"
 - cause: exit status 1
 - pr-front-end-2:deployment/carts failed. Error: running [kubectl --context ephemeral-demo-spot rollout status deployment carts --namespace pr-front-end-2 --watch=false]
 - stdout: ""
 - stderr: "error: deployment \"carts\" exceeded its progress deadline\n"
 - cause: exit status 1.
Error: Process completed with exit code 1.

This seems to be because t2.medium instances can run up to 17 pods. The demo project has 13; these plus the services of the ephemeral.run repo exceed the maximum, so new pods remain forever in Pending, causing the deployment to time out and fail.

It would be nice to check for this situation or document workarounds (add a node, increase the desired ASG size, change default instance type, etc).

I believe it's repo, workflow and that the bot user must be invited with write access to all repos involved.

The ap-south-1 region is hardcoded in a few places:

• platform-setup/terraform/vpc.tf
• ephemeral.run/.github/workflows/create_workflow.yaml
• ephemeral.run/.github/workflows/delete_workflow.yaml

Should these not be variables corresponding to the user-specified region instead?

It seems that the first environment that is started gets all the DNS entries routed to it. Once that environment is gone, any other running environments will get routed to.

Either that or it is random (but not on every load).

The Configure Access to EKS cluster workflow step always fails at kubectl get ns with error: You must be logged in to the server (Unauthorized).

Github actions apparently (even when it's not defined as a secret?) automatically replaces the account ID with *** in log output so this was finally tracked down by running kubectl --namespace=kube-system describe configmaps aws-auth and noticing an unfamiliar account ID, which I then found hardcoded in variables.tf.

Hi and thanks for your work on this promising project!

I'll use this issue to document a few of the stumbling blocks I encountered when trying this out for the first time.

Secrets

This document says to create two forks, front-end and ephemeral.run.

A following step says "Set the following secrets on your fork" but it's not clear which secrets need to go on which forked repo.

The intended values of the following secrets is unclear:

• DOCKERHUB_REPOSITORY -- it's not clear if this should be hub.docker.com/repository/registry-1.docker.io/soulshake/front-end/, soulshake/front-end, just soulshake, or what
• DOCKERHUB_USERNAME, DOCKERHUB_PASSWORD -- these are clear enough; however, it seems that the front-end repo is actually expecting to find DOCKER_USER and DOCKER_PASS
• GIT_HUB_ACCESS_TOKEN -- what permissions are required?
• AWS_CLUSTER_NAME -- corresponds to terraform output value cluster_name
• IAM_CLUSTER_USER -- it seems this is supposed to be the AWS_ACCESS_KEY_ID of the ephemeral-gha-user, corresponds to the terraform output value of ephemeral-gha-user_iam_creds_id
• IAM_CLUSTER_PASSWORD -- it seems this is supposed to be the AWS_SECRET_ACCESS_KEY of the ephemeral-gha-user, corresponds to the terraform output value of ephemeral-gha-user_iam_creds_secret

Non-secret secrets

It would be preferable to use something other than Github secrets to store the non-secret items, as all "secrets" are obfuscated in the action logs, making it harder to debug.

Other variables

It seems there are some additional variables that need to be updated in the workflow files (AWS_REGION, repository paths for create-or-update-comment steps, KUBE_CONTEXT?, etc).

KUBE_CONTEXT in particular was confusing -- to me, the name implies that it should be set to e.g. arn:aws:eks:ap-south-1:ACCOUNT_ID:cluster/ephemeral-demo-spot (as it appears in kubectl config get-contexts) but it seems it's actually supposed to be the short cluster name, e.g. ephemeral-demo-spot ?

Workflow files / front-end repo changes needed

GETTING-STARTED.md says to fork microservices-demo/front-end but that repo doesn't have the Github actions that actually trigger the build (see diff). The doc should either instead point to OpenGov/front-end or document the workflow yaml files that need to be added.

It's actually not clear (without some investigation) which workflow files need to go on which repo.

WIP

(Will continue to gather feedback for this issue, and will be happy to create a PR, however it might be preferable to rename some of these variables instead)