capactio / hub-manifests Goto Github PK

Description

We should use Capact CLI inside the capact.upgrade manifest to simplify the upgrade flow, and also maintain just a single upgrade process definition (inside CLI).

Reasons

Current problems:

• maintaining Capact installations, both in Capact CLI and Capact upgrade manifest
• The Capact upgrade manifest always takes CRD from main branch (See it in the actual manifest). In Capact CLI, we are support that (capactio/capact#576)

Description

• Use alpha manifest generation tooling
  • Single Helm and single Terraform manifest (e.g. Redis Helm chart and Redis managed solution - e.g. AWS ElastiCache for Redis?)
  • Test both the helm and terraform generators
• Add video recording for that
• document the process
  • publish the doc on website

Reason

For now the knowledge how to use it is internal, and we want to share it with other to get feedback if the generators as usable and if some parts need to be optimized.

Description

Add document about best practices for storing and creating Hub manifests.

Describe:

• Hierarchy, e.g. prefer cap.interface.database.postgresql instead of cap.interface.postgres
• How populator stores data e.g. generated path. Currently, they are calculated from dir structure (hierarchy).
• How to deal with multiple revisions for the same Action. Currently, we use file names, e.g. install-0.0.1, install-0.2.1

Reason

"Consistent code is easier to maintain, is easier to rationalize, requires less cognitive overhead, and is easier to migrate or update as new conventions emerge or classes of bugs are fixed."

• It's easier for us to review community PRs.
• External contributors don't need to figure out the correct approach each time they create manifests.

Use cases

• Content Developer - knows how to create quality, good content.

Description

This task is based on the investigation from capactio/capact#300. See the current doc here.

1. Modify Terraform scripts for Capact EKS provisioning
   1. Install iSCSI initiator on nodes
   2. Modify AutoScalingGroup and attach empty volume for every node
   3. This is an optional step during AWS EKS cluster provisioning
2. Create Hub manifests which installs & configures OpenEBS
   1. Install OpenEBS (via Helm - links in the OpenEBS investigation doc)
      1. Install OpenEBS API
      2. Install cStor
   2. Create CR CStorPoolCluster
      1. Link: https://github.com/openebs/cstor-operators/blob/master/docs/quick.md#provision-a-cstorpoolcluster
      2. Try to use volume label from input parameters and prepare mapping node: volume. If that’s not possible, require volume IDs from user as input params.
      3. Use the mapping in CR
   3. create output TypeInstances for storage pool and OpenEBS (to do easy uninstallation)
3. Make sure the docs are updated

NOTE: User is still required to provide storageClass during app installation.

Sample test scenario

1. Use the installation manifests
2. Install Mattermost with replicaCount: 1 and as shared storage set storageClass: cstor
3. Create a repository and push commit
4. Cordon the node
5. After some time, see if Mattermost still works and if there is still the repo with files you created. Make sure it is in a different zone.

Reason

OpenEBS could provide us a cross-cloud distributed storage without any vendor lock-in. It could also simplify implementation of the backup/restore operations for apps.

Description

Since Kubernetes 1.20, the evicted Pods stay in cluster with failed reason. For example:

cert-manager-webhook-c654fc8b-bqrnd                      1/1     Running     0          109m
cert-manager-webhook-c654fc8b-kqgwk                      0/1     Shutdown    0          20h
cert-manager-webhook-c654fc8b-st6kd                      0/1     Shutdown    0          16h
cert-manager-webhook-c654fc8b-x5kkq                      0/1     Shutdown    0          168m
ingress-nginx-controller-bf5464b58-g4ls8                 0/1     Shutdown    0          16h
ingress-nginx-controller-bf5464b58-rp2k2                 1/1     Running     0          109m

In our Capact upgrade Action we wait for cert-manager and ingress:

Unfortunately we use kubectl wait --for=condition=ready pod -n capact-system --selector=... which takes into account all Pods also those in failed state which were evicted.

A better approach is to use a dedicated command which understands a given workflow, e.g.: kubectl rollout status sts/neo4j-neo4j-core --watch

Expected behavior

Ignore failed Pods which don't impact a given workload

Description

1. Create MinIO installation manifests [open source] [2MD]
   - Helm-only
   - Bitnami version
   - Use our manifest generator
   - Use admin user
   - Outputs TypeInstance cap.type.database.object.aws.s3 with URL and related credentials for admin (cap.type.aws.credentials)
2. Create S3/Minio bucket [open source] [1MD]
   - Try MinIO CLI as it supports S3 as well
   - Interface: cap.interface.database.object.aws.s3.install, takes as an input:
   - TypeInstance cap.type.aws.credentials,
   - TypeInstance cap.type.database.object.aws.s3 - endpoint
   - Input parameters: bucket's name, policy
   - Double check it works for AWS S3 and MinIO as well

Reason

Add ability to install MinIO and create bucket to extend our Hub manifests base.

Use cases

• System User: Install MinIO on cluster
• System User: Create S3 bucket on S3 and MinIO

Description

Prepare simple manifests, much simpler than the Mattermost one. Think about something like whalesay example from Docker: https://docs.dockstore.org/en/feature-bcc-2020-notes/docker_instructions.html#part-a-running-containers
- Show the dependencies interchangeability - simple examples
- capactsay image? 🤔 With different text in a bubble for different Implementations (which output TypeInstance)

Reason

Currently, the Mattermost example is complicated to showcase it to the first time clients.

Description

It would be useful to have a dedicated section which discusses the process around contributing new abstractions/core types or updating previously existing abstractions/core types.

Reason

• Have an model in place for ensuring standardization and interoperability between shared abstractions.
• This can be used as a basis for an open governance model for OCF in the future.

Description

All Implementations for Interface capactio.capact.validation.hub.install are not runnable. After executing Capact Action for this Interface we will get error:

"Cannot render given action: while validating rendered Action: while linting workflow: templates.capact-root.steps[1].start-entrypoint template 'main' type is unknown (will retry - 1/15)"

Expected behavior

I can create and execute Action for capactio.capact.validation.hub.install

Steps to reproduce

capact act create cap.interface.capactio.capact.validation.hub.install

Ideas

This should be fixed, but it will be worth to add additional check for our manifest validation, to check whether Argo workflow is valid and can be rendered.

AC

• manifests are fixed

Description

Write proposal about our approach for automated Hub manifests testing.

• We want to test all Implementations inside a given GitHub repository
• In an automated way
• As GitHub Actions pipeline (but to verify whether we don't need to run such tests on external cluster, because of resource limits for GH Actions Runner)
• Should we support testing day-two operations?
• Cleanup: currently external cluster delete?
• Answer the question: Do we need to write some small testing framework? Or can we use something existing from OS community?
• Result: a document on capactio/capact repository summarizing the possibilities with a short comparison
• Nice to have: Prepare PoC to showcase the preferred idea

Reason

Currently, to test the manifests inside the hub-manifests repository, we need to manually run Actions. Our end goal is to have it automated, so that all Implementations are executed to check whether they work properly.

Use cases

• As a System Administrator, I want to make sure that the connected Public Hub contains valid, runnable manifests.
• As Capact Developer, I want to make sure any contribution to the hub-manifests repository is automatically validated on CI without a need to run Actions manually.

Additional links

• https://terratest.gruntwork.io/ - maybe it will be helpful for testing Terraform manifests

Description

When I tried to install some of the manifests using Dashboard UI, there is a problem with providing the additional-parameters for them.

Expected behavior

I want to be able to change all of the additional-parameters using Dashboard UI.

Actual behavior

I can't provide some of the additional input due to reasons:

Unsupported field schema for field root: Missing items definition.
A mapping of tags to assign to all resources.

Steps to reproduce

1. Login to Dashboard UI (dashboard.capact.local)
2. Go to Redis instalattion (cap.interface.database.redis)
3. Click create Action with random instance name
4. In the tab ,,Advanced Settings" choose Redis Install Action with the cap.implementation.aws.redis.install:0.1.0 and click ,,Provide input"

Description

It would be helpful to have the content-specific contribution guidelines, which could discuss the requirements, review process and strategy around content governance & lifecycle.

AC

• Document best practices for storing and creating Hub manifests:
  • Hierarchy, e.g. prefer cap.interface.database.postgresql instead of cap.interface.postgres
  • How populator stores data e.g. generated path. Currently, they are calculated from dir structure (hierarchy).
  • How to deal with multiple revisions for the same Action. Currently, we use file names, e.g. install-0.0.1, install-0.2.1
• Document review process and strategy around content governance & lifecycle.
• To discuss: Apart from CONTRIBUTING.md, publish it also on website in Community section

Reason

"Consistent code is easier to maintain, is easier to rationalize, requires less cognitive overhead, and is easier to migrate or update as new conventions emerge or classes of bugs are fixed."

• It's easier for us to review community PRs.
• External contributors don't need to figure out the correct approach each time they create manifests.
• There are no defined rules how the Hub content should be stored, which might be confusing for content developers.
• Transparency

Use cases

• Content Developer - knows how to create quality, good content.