ms3inc / tavros Goto Github PK

Add the necessary annotations to use aws network load balancers instead of regular elastic load balancers

see:
https://github.com/Kong/kubernetes-ingress-controller/blob/f69ea6f047ce9c017b05c8250e70fee0e4f02104/deploy/manifests/base/service.yaml#L14

• Prefer Proven FOSS Components with Optional Support for Licensed Derivatives
• Apache Camel as the Default Integration Framework
• Spring Boot as the Base Application Framework
• DataSonnet as the Default Data Transformation Language
• OpenTracing for In-Process Tracing API
• Kubernetes as the Computing Platform
• Kops to Provision a Kubernetes Cluster
• Flux to Provide Platform GitOps
• Kubeseal to Securely Manage Secrets in GitOps
• Keycloak for Indetity and Access Management
• Kong as Kubernetes Ingress and API gateway
• PostgreSQL as the Platform's Default Database
• Gitea for a Lightweight Git Server
• Kuma for Service Mesh
• Jenkins for Continuous Delivery Jobs
• Sonarqube for Application Static Code Analysis
• Elastic Cloud for Observability Data Aggregation and Visualization
• Jaeger for Tracing with Elasticsearch Backend
• Nexus Repository Manager for Artifact Management
• Prefer Daemonsets Over Sidecars
• Spring Cloud Config for Application Configuration Management
• Prefer Kong Enterprise Edition
• Use Ansible as the Provisioning Engine
• Helm and Operators for Component Installation and Management
• Use Ansible Collection to Structure and Package Ansible Code
• Setup Sandbox and Production Kuma Meshes and Kong Ingress Controllers by Default
• Setup Sandbox and Production Keycloak Realms by Default
• cert-manager for Certificate Management
• Use Markdown Architectural Decision Records

The kubernetes ansible module provides some built-in options like waiting for resources that will make the code cleaner and easier to maintain. Also that's what Ansible Operators utilize, so it allows us to go in that direction if we decide to.

currently thinking about saving packaged lua plugins to nexus, and adding an init container that installs plugins described in a config map.

This means the pods will be restarted with a minor downtime.

Use --container-runtime containerd. Switching to containerd in order to address docker being deprecated as a runtime.

In order to enable TLS at the Gateway level we need to integrate cert-manager to provide server certificates on demand and handle rotation automatically.

In order to provide disaster recovery directions we need to document what needs to be backed up and how. Also how to restore from a failure. It's fine if this is manual for now, but would be good to think through automatic backups.

• kops cluster state in s3
• troubadour postgresql database

currently losing gitea state if shutdown

figure out kibana api to import logs and traces dashboards after provisioning

How / why?

Build an user interface that walks a customer through troubadour component configuration and generates the appropriate yaml configuration and triggers a Jenkins provision job

With the addition of #58, there are Jenkins environment variables for the SCM info that are being set with the configuration as code. The host is being dynamically set and the scm provider and scm creds are hardcoded. Support should be added to allow these values to be configured dynamically when gitea isn't enabled.

jenkins state is currently ephemeral, so far it seems it only loses job run history

In order to provide a seamless authentication experience to our clients we need to tie Gitea into Keycloak to provide single sign on.

If Gitea role is run without dry-run tag it will retrieve gitea_login information. It is possible for this key to lose the cookies_string value, and cause Add flux-cd as owner to tavros/platform task to fail later in Flux gitea-setup role.

In order for the customer to have access to all the installed components and the cluster itself, we need to provide those credentials in a secure manner. We need to determine what credentials are needed and how to transfer those over securely.

In order to provide further observability of application trace data we'll use Jaeger with a Elasticsearch backend so trace data can be correlated with log and metric data.

See ADR-0017 - Jaeger for Tracing with Elasticsearch Backend for more info

In order to be a repeatable process we need to use the same version in every run. We will upgrade manually when necessary.

In order to support an arbitrary setup of components, each component role should provision the user and database they need in the postgresql instance.

Test support on AWS EKS

• Add the two scripts for the pipeline jobs to config.j2
• Configure jenkins to have the build user vars plugin when it starts
• Configure jenkins to have the shared library settings when it starts

Additionally:

• A Jenkinsfile repo will need to be created publicly
• Repo will need to be cloned into gitea

A possible enhancement later could be to set the default gitea url in the jenkinsfile to what the url is going to be when it's cloned over, if that's possible

introduced in d4e2e94

the kustomize command needs to be changed to reorder=none to apply the source before the releases, needs to be tested since there's subfolders involved

In order to enable log aggregation, and general insight into application workloads through Kibana dashboards we need to add elastisearch and kibana

In order to be able to update Gitea from Jenkins, jenkins-ci needs to be an owner of the org. Otherwise a 403 is returned with the message: Given user is not allowed to create repository in organization.

Add retry strategy base on others at 30s delay between calls, and 30s retries for a 15 min effective wait.

Nexus has a setup wizard by default on admin first login. It changes the admin password and also disables anonymous logins.

related links
https://community.sonatype.com/t/is-that-possible-to-install-a-nexus-chart-with-a-given-initial-password/2962/2
https://issues.sonatype.org/browse/NEXUS-21909

https://blog.mimacom.com/automate-nexus/

specifically
kong ee user http requests
wait for keycloak instance

We have a test-playbook.yaml that is tested locally. In order to enable continuous integration we need to add some assertions and create a Jenkins job to be run on demand.

The test playbook should:

• provision kubernetes infrastructure from scratch
• setup our default components
• deploy a test application and run assertions against supported components, eg.: mesh isolation, https instead of http, etc
• destroy the kubernetes infrastructure
• be an on-demand ci job

support Azure cloud

until the jaeger operator officially supports adding a truststore from the elastic http certificate secret into the spark-dependencies cron job we'll need to generate and maintain that ourselves

we need all helm releases and custom resources to be named tavros instead of their generic names 'gitea', 'jaeger' so that clients can use those names in the event they want to have independent instances of those components

see https redirects in other ingresses for reference

Create a Jenkins job that takes in a configuration set of variables and runs the provision playbook

Needs to clean up vars.yaml in S3 durring this Task

Investigate what's are good request/limit resource allocations for postgresql as multiple components depend on that instance. Perhaps replicate and double limits.

gitea is currently disabled when running dry-run tests

In order to provide a single point of identity and access management, how do we delegate/map kong rbac capabilities to Keycloak?

Are there any internal points of extension? If not how can we manually keep them in sync?

In order to further provide gitops and continuous delivery, we'll setup a Kops git repository and Jenkins delivery pipelines as described here
https://github.com/kubernetes/kops/blob/master/docs/continuous_integration.md

Also must document an ADR

In order to provide static code qualitative analysis we'll use sonarqube.

See ADR-0015 - Sonarqube for Application Static Code Analysis for more information

In order to enable valuable ee features like admin and developer portals, and oidc or oauth2 plugins, we need to support EE installations of Kongs.

This includes a Keycloak Realm, likely aligned with the Kuma Mesh that the Kong instance is a gateway to. And OIDC plugin to provide SSO into the Admin and Dev Portals.