arlonproj / arlon Goto Github PK

To make Arlon easier to use we need to create an automated build system that can publish the cli for Linux.

The end goal is to make using Arlon much easier, with a complied CLI users can download and go.
Users of any desktop OS should be able to download the CLI and use it.

• Allow users to install using APT

Other OS support will be delivered after Linux

• MACOS Allow users to install using brew #114
• Windows OS Investigate using https://chocolatey.org for Windows #115

Aha! Link: https://pf9.aha.io/features/ARLON-156

The command make docker-build , which internally invokes make test as a step fails on my Macbook having Apple m1 processor. While it works on another Mac having a processor with AMD architecture. The problem is with envtest [email protected] - the binary repo for this tool doesn't have a downloadable for Darwin (macOS) with ARM architecture (while both architectures are supported for Linux) since it's a new architecture on Mac.

Aha! Link: https://pf9.aha.io/features/ARLON-105

To simplify setting up Arlon we should have a bootstrap command that installs Arlon, ArgoCD and ClusterAPI into a users cluster.
Bootstrap should also handle clusters that already have ArgoCD, CAPI or both installed.

(We need to decide if installing CAPI Cloud Providers should be part of this)

Currently this task is subdivided into two seperate tasks for verify and install

• #143
• #144

Aha! Link: https://pf9.aha.io/features/ARLON-173

A bundle is currently fixed. If it accepts parameters, such as Helm values, those things cannot be overridden/customized by the end user. The only way to override parameters / values is to clone the bundle and modify the copy.

Aha! Link: https://pf9.aha.io/features/ARLON-107

Arlon must work with Cluster API clusterSpecs stored in Git for the following providers.

Arlon must be able to create the cluster as an ArgoCD application object based on the clusterSpec from a git repo, automatically add the new cluster to ArgoCD and then deploy the profile.

Background is here

Tier 2

• DigitalOcean

• IBM Cloud

• Equinix Metal (formerly Packet)

• Nested

• MAAS

• OpenStack

Aha! Link: https://pf9.aha.io/features/ARLON-153

Right now, the "clusterspec delete" command doesn't exist, but invoking it produces no errors, so it's confusing. Fix that too.

Aha! Link: https://pf9.aha.io/features/ARLON-108

As I have many existing clusters it would be great to be able to bulk apply Profiles to multiple clusters in a single command/action.

Aha! Link: https://pf9.aha.io/features/ARLON-160

Today, when a cluster C is deployed with profile P, and P is later modified (for e.g. by adding a new bundle B to it), C is not updated, meaning B is not added to C. Determine:

• What behaviors we want to support. For e.g. leave C unchanged, automatically update C, manually update C, or support all behaviors
• For manual and auto update, design and implement the mechanism
  Also need to update the documentation

Aha! Link: https://pf9.aha.io/features/ARLON-119

An ApplicationSet allows a single application to target multiple clusters. This may help with the update problem (#7)

Aha! Link: https://pf9.aha.io/features/ARLON-171

As user of Kubernetes in AWS it would great to see support for clusterSpecs that incorporated more of the complete clusterAPI specifications. The clusterSpec should be stored in Git and support all clusterAPI CRDs including elements such as.

• Region
• VPC
• Domain
• Instance Types
• Add-ons
• Autoscaling Config
• nodepools/ machine sets
• SSH keys
• Control plane types (EKS or Unmanaged)

Aha! Link: https://pf9.aha.io/features/ARLON-161

When a top-level arlon Cluster application is deleted in argocd, the child dynamic profile app (if the profile is dynamic) is deleted as expected, but the profile's child apps (one for each bundle) are not. This appears to be an ArgoCD bug, and it appears to affect application resources that appear under the templates/ subdirectory of a Helm chart.

Experiment:

• Keep an empty arlon cluster "cluster-a" by deleting (in git and manually as needed) all children apps.
• Manually create a parent app (using argocd app create) that has the same definition as the equivalent dynamic profile used in the bug scenario. The actual command I used is: argocd app create standalone-dynprofile --repo https://github.com/bcle/fleet-infra.git --path profiles/dyntest-1/mgmt --dest-namespace argocd --dest-server https://kubernetes.default.svc --auto-prune --sync-option Prune=true --sync-policy automated --helm-set clusterName=cluster-a
• Observe the 2 child apps being created and successfully synced
• Manually delete the parent app: argocd app delete -y standalone-dynprofile
• Observe that the 2 child apps are automatically deleted as expected
• Repeat the experiment, but this time by checking into git a profile.yaml file under the cluster's root app's templates/ directory
• Observe the creation of the child apps
• In git, remove profile.yaml and push the change
• Observe that the parent app is deleted, but not the child apps

Aha! Link: https://pf9.aha.io/features/ARLON-114

Unlike dynamic bundles, where a user-triggered change in git will result in live updates, dynamic profiles (DPs) and cluster specs (CSs) are defined and stored in arlon's "database", which is currently implemented as configmaps in the management cluster.

• A DP is defined in the database, and "compiled" into a git directory by the arlon library during a create or update operation. That directory is used as the source content for the argocd application that arlon creates for the DP. The directory is not meant to be inspected or manipulated by the end user because it'll get overwritten the next time the DP is updated through the arlon library.
• When a user wishes to change the composition of a DP, he/she needs to call the arlon library's profile.Update() function (typically via arlon profile update CLI command today).
• This means that DPs, despite using git internally (the compiled version), are not meant to be used in a gitops way by the user
• Same issue for CSs. There is a larger design issue for CS: how to update clusters.
• Possible solution: make DPs (and CSs) custom resources, and create a new arlon controller (or enhance existing one) to automatically compile DP custom resources.

Aha! Link: https://pf9.aha.io/features/ARLON-111

As an Arlon user leveraging the Create Cluster Command i would like Arlon to run pre-flight checks on the Profile.

So that i don't need to wait 15 minutes for Cluster API or ArgoCD to throw errors

Specfically the Pre Filight checks should look for

Incorrect YAML indentation

Broken links/references

non-existent Cluster API types (CRDs/APIs may not be installed on the cluster, OR the user may have simply miss-typed. )

Aha! Link: https://pf9.aha.io/features/ARLON-149

Arlon must work with Cluster API clusterSpecs stored in Git for the following providers.

Arlon must be able to create the cluster as an ArgoCD application object based on the clusterSpec from a git repo, automatically add the new cluster to ArgoCD and then deploy the profile.

Background is here

Tier 3

-[ ] BYOH

-[ ] Exoscale

-[ ] Sidero

-[ ] Hetzner

-[ ] Baidu Cloud

-[ ] Alibaba Cloud

-[ ] Tencent Cloud

Aha! Link: https://pf9.aha.io/features/ARLON-152

The controller is currently tested by running it externally and manually. It needs to go in-cluster.

Aha! Link: https://pf9.aha.io/features/ARLON-122

Active Diff enables users to compare any cluster running any profile to any other cluster or any bundle stored in git. Similar to the ArgoCD Diff but with a larger scope.

More background is here

With Active Diff users should be able to:

• Choose a bundle in Git and compare to any cluster running the bundle
• Choose a bundle in Git and compare it to any cluster whereby the diff searches for deployments/pods or namespaces of the same characteristics
• Compare a bundle on one cluster to another
• Compare a bundle to another bundle
• Compare a profile to a profile
• Compare a cluster to a profile

Aha! Link: https://pf9.aha.io/features/ARLON-157

This may involve adding a new ApiProvider property to the clusterspec object.

Aha! Link: https://pf9.aha.io/features/ARLON-170

As an Arlon user leveraging the Create Profile and Create ClusterSpec commands to wtite to git I expect Arlon to validate the YAML prior to writing it to Git.

So that i do not hit issues when attempting to use the newly created profile.

Specifically I would like it to

Look for malformated YAML

Incorrect ClusterAPI CRDs or APIs based on the version of ClusterAPI in my managment cluster and the installed providers

Aha! Link: https://pf9.aha.io/features/ARLON-150

Right now, a profile's bundle list is an arbitrary string that the user provides. It's meant to be a comma separated list. The following validations should take place:

• The bundle names must follow k8s resource naming requirements (they can't have special characters, etc...)
• The list must be comma separated if more than one bundle
• (Optional, needs some thought) The bundle names must refer to valid existing bundles. This is trickier than it seems, for e.g., what happens when a bundle is deleted? We don't have a full relational database backing the objects, so can't express constraints.

Aha! Link: https://pf9.aha.io/features/ARLON-172

As a user of Arlon I would like the CLI to allow cluster creation based on three inputs:

1. ClusterSpec,
2. Profile, and
3. Git repository

The create cluster command will read the ClusterSpec from the Git repo that Arlon is configured to work with, read the Profile from where Arlon is configured to work with and then write the new clusters configuration to the Git Repository specified.
This will mean that each cluster has a distinct structure in Git, that is separate to the Git Repository where each ClusterSpec and Profile is stored.

The create command will also interact with ArgoCD APIs and create the related application objects for the Cluster and Profile.

Aha! Link: https://pf9.aha.io/features/ARLON-102

It would be great to be able to clone a ClusterSpec, this would allow me to quickly create variations and test.

Aha! Link: https://pf9.aha.io/features/ARLON-159

.. so that these objects will be available in the container image built. When this container image is deployed in the management cluster, these objects can be applied using kubectl instead of manually running the steps in https://github.com/platform9/arlon#arlon-controller to deploy the controller and create the CRDs for cluster registration etc.

Aha! Link: https://pf9.aha.io/features/ARLON-104

To help users get started with Arlon we should create an example K8s policy object that demonstrates how Arlon can apply Policies to a cluster.
RBAC is a good example to go with as its used in all cluster types.
I suggest a Read-Only policy as this is non-destructive and materially useful for all customers.

Aha! Link: https://pf9.aha.io/features/ARLON-167

Currently arlon cli interacted with 3 entities:

1. Git repo
2. ArgoCD Server
3. K8s API Server of Management Cluster

It would be very beneficial to have a context management similar to how its done by argocd cli.
A context can be defined as combination of git repo (with credentials if there), ArgoCD Server connection information, k8s API Server connection information.

All the information about multiple contexts can be saved in ~/.arlon folder on user machine.

User should be able to perform following activities with the contexts:

1. Create new contexts
2. Delete contexts
3. Shift between contexts
4. Renew tokens in the context (good to have)

Once a context is set by the user, the individual commands having parameters about above 3 (git, k8s, argocd) can be treated as optional. If present, they override the set context.

Commands can be see as follows:

1. Create new context
   arlon context add <new_context_name> --argocd-server <argocd_addr> --gitrepo .....

2. Delete context
   arlon context delete <context_name>

3. Shift between context
   arlon context set <context_name>

4. Impact of other commands because of context. Below given is an example command:
   arlon create clusterspec [--context <context_to_use>] [--gitrepo <repo url>]..
   Notice in above command that git repo and context are optional parameters. Individual command to retrieve the git repo in following order:
   a. overridden param in command
   b. context present in the command
   c. command set globally
   Similarly do the same for argocd server & mgmt cluster also.

Aha! Link: https://pf9.aha.io/features/ARLON-164

From Chris Jones: (slack post: https://platform9.slack.com/archives/C02SEAQ0NB1/p1643832931650859)

The idea in my mind was to leverage Argo to manage CAPI objects as Argo Applications. This is essentially the demo that Spectro Cloud did during ArgoCon.
When i first thought about the end to end profile flow it was something like
Create CAPI YAML in Github -  Platform9 would enable this to be created automatically from an existing cluster.
Create Apps & K8s Resources in github - Platform9 would enable this to be created automatically from an existing cluster (we do this today for RBAC)
Create Profile from 1 &2
Arlon creates cluster from profile
Create Argo Apps from Profile (with any particular overrides, AWS region for example)
CAPI App for the cluster
Register the cluster into Argo (For the Apps and objects in the profile to deploy something needs to Automatically register the cluster into CAPI )
Once registered the Apps and K8s objects deploy,
App for all other objects
From your demo it would appear we have the following
Write cluster spec as plain text/YAML
Add cluster spec to Arlon so its stored as a CRD
Create Apps & K8s Resources in github OR local YAML/Helm files
Map Bundles to Apps and K8s Objects and store as CRD
Map Bundles to Profile as PRD
Create cluster by passing ClusterSpec and Profile
CAPI creates cluster
Registration controller registers the Cluster into Argo
Arlon then creates the argo Apps
You raised some valid points around Lifecycle that we need to think about.
One way around this is to mark the ClusterAPI Apps in Argo as Manual (disable sync) OR clone the YAML for each cluster in Github.
The issue with cloning is that there is no Change Once - Update Many Cluster ability.
Honestly we should look to support both methods, so that a user can choose,
"Deploy as baseline" where the CAPI YAML is cloned and sync is manual.
In this scenario we still need to enable a user to ask "which ClusterSpec and which version was cloned?" so that they can do a comparison between clusters
They should also be able to compare the clusters current state to that for which it is running, Assuming that they're using CAPI to augment the cluster.
"Deploy as cloned with sync" -  this could clone the YAML but keep sync on, such that changing the code in github would reflect a change in the Argo App which in turn is picked up by Cluster API and applied to the clusters
With this, items such as upgrades become a question, and i would need Anmol to help as i have not read up on how CAPI does upgrades.
"Deploy as Linked Cluster without Sync" - this would deploy directly from the CAPI Yaml in Github and work as a  "one-off" type cluster
"Deploy as Linked Cluster with Sync" - this would deploy directly from the CAPI Yaml in Github where any change to the ClusterSpec would reflect as a change in any cluster running off of the same ClusterSpec
The end goal is to have 100% of the cluster and apps defined in gitub,

Aha! Link: https://pf9.aha.io/features/ARLON-113

As a user of Arlon I would like the CLI to have a dedicated command for registering and unregistering the Git repo where my Profiles will be written.

So that I do not need to manually place the Profile and Git and manually generate the path that Arlon will use during the cluster deploy command.

It would also be good if I could specify which repository should be used for deploying clusters and their related ArgoCD elements (workspace repository) versus which repositories are used for storing the Bundle, Profile and ClusterSpec data.
This is especially important when we start supporting #54 Cloned Clusters as the entire contents stored in Git will be duplicated.

Aha! Link: https://pf9.aha.io/features/ARLON-146

Migrate our current docs to a more structured documentation site, most likely https://readthedocs.org/

This task is for doing the basic setup. Once done, all contributors can add to the documentation.

Aha! Link: https://pf9.aha.io/features/ARLON-103

We need a service that can automatically add ClusterAPI clusters to argoCD, so that the cluster create, and Profile Apply steps are 100% automated.
The automated registration should take seconds, such that once a cluster is running its attached to ArgoCD in under 30 seconds

Aha! Link: https://pf9.aha.io/features/ARLON-165

This kind of bundle contains a reference to a git repo and optional path.

Aha! Link: https://pf9.aha.io/features/ARLON-120

Now that arlon supports CAPI-AWS-kubeadm clusters, the issue of CNI came up. A CAPI kubeadm cluster starts with no CNI, so any bundles that install workloads (e.g. applications) will be stuck in the progressing state. This was a great opportunity for me to define a CNI setup as a bundle (I created a static bundle from a file downloaded from https://docs.projectcalico.org/v3.21/manifests/calico.yaml), and then add the bundle to the cluster's dynamic profile. As expected, the cluster was automatically updated with calico, and the workloads eventually deployed successfully. However, if CAPI EKS clusters were using the same dynamic profile, they would potentially break, since they are automatically configured with AWS VPC CNI out of the box, resulting in a conflict. So this raises important questions about how to treat CNIs. Possible approaches:

• Since CNIs are installable as bundles, keep them separate from cluster specs. Profiles should be defined with CNI bundles for clusters that have no OOB CNI, and separate profiles with no CNI bundle should be used for clusters that have CNI OOB. The problem with this is unnecessary duplication and proliferation of redundant profiles that only differ by one bundle.
• Make CNI a property of a cluster spec. This solves the profile duplication problem, but it means that CNIs (which again, are installable bundles) need special treatment by arlon, as opposed to managed by the user. For example, arlon would need to manage CNI releases, and test compatibility matrices.
• Allow clusters to have more than one profile. This would allow users to organize profiles into e.g. "infrastructure", "add-on", "config", and "apps", thereby eliminating duplication by allowing a cluster to use the exact set of profiles that it needs. The CNI bunlde could then be configured as part of a particular "infrastructure" profile, and that profile only needs to be included when absolutely needed by the cluster.

Aha! Link: https://pf9.aha.io/features/ARLON-169

Arlon requires testing and certification for each clusterAPI cloud provider.

Background is here

Tier 1

• 🚧 AWS #55
• Azure
• Azure Stack HCI
• Metal3
• GCP
• KubeVirt
• vSphere
• Openstack

Aha! Link: https://pf9.aha.io/features/ARLON-154

As an Arlon user with my own Profile (or Platform9 example) I would like to use Arlon to write the Profile directly to Git

So that i do not need to install and use seperate Git tools.
Having this in the Arlonctl will help streamline my work processes.

for example:
Arlon profile commit myFancyProfile -f http://github.com/pf9example/profiles/baseapps.yaml

Aha! Link: https://pf9.aha.io/features/ARLON-151

Arlon is built to simplify running clusters at scale using objects stored in Git and reconcile using ArgoCD. There are multiple ways that this can be achieved, for example, each cluster could have its own unique set of files in a defined Git hierarchy. Or, a set of clusters could all leverage a single set of fines in a defined hierarchy. Or a blend of Both.

Linked Mode is an approach that builds clusters from a central group of files that are shared.
Clusters are created from a ClusterSpec and Profile, users invoke Arlon cluster create linked <cluster_name> <cluster_spec> <profile> - This will deploy the cluster directly linking the clusters to the objects stored in Git.

This means any change to the ClusterSpec, profile or bundles in Git would reflect as a change in any/all cluster running off of the same ClusterSpec.

This is the default behavior of Arlon.

Read More Here

BACKGROUND

The end to end profile flow:

• Create CAPI YAML in Git - 
• Create Apps & K8s Resources in git-
• Create a ClusterSpec, Bundles and a Profile from each object

Arlon creates cluster from profile and the clusterSpec.

Register the cluster into Argo (For the Apps and objects in the profile to deploy something needs to Automatically register the cluster into CAPI )

Once registered the Apps and K8s objects deploy, App for all other objects

Aha! Link: https://pf9.aha.io/features/ARLON-163

It would at the minimum install the controller and CRD.

Aha! Link: https://pf9.aha.io/features/ARLON-121

As an Arlon user with my own YAML clusterSpec (or Platform9 example) I would like to use Arlon to write the bundle directly to Git

So that i do not need to install and use seperate Git tools.

Having this in the Arlonctl will help streamline my work processes.

Aha! Link: https://pf9.aha.io/features/ARLON-147

As cluster admin I would like to be able to apply profiles to my existing clusters. This makes Arlon significantly easier for me to adopt as I don't need to start from scratch. It will also help standardize my environment very quickly.

As I'm using EKS Arlon will need to work with my Private VPC and Public VPC clusters. Many clusters we run have private API endpoints for security reasons.

Aha! Link: https://pf9.aha.io/features/ARLON-166

Use case: the AWS cluster autoscaler YAML template used by Platform9 has some fields that are parameterized based on a unique "cluster UUID". Arlon does not use the concept of cluster UUIDs but it does assume (and verify) that cluster names are unique (within the arlon namespace in the mangement cluster). So it could automatically insert Helm values into the app representing the bundle. If the bundle happens to be a Helm chart, it could take advantage of that. If the bundle is (or references) a Kustomize template, then investigate generating the right Kustomization.

Aha! Link: https://pf9.aha.io/features/ARLON-106

As a user implementing GitOps I would like to see the potential impact of applying a bundle or profile prior to applying them to the target cluster.

So that i can be fully informed of the changes that will be made when ArgoCD applies the bundle.

More background is here

ArgoCD has a built in diff engine https://github.com/argoproj/gitops-engine/blob/master/pkg/diff/diff.go that can compare an ArgoCD App to its related source stored in GIt. This diff allows users to inspect the code when an app is out of sync. What it doesn’t do is expose the potential impact of delploying an ArgoCD App.

For example; If a user creates a bundle of RBAC policies and deploys them via ArgoCD they could potentially remove Role Bindings that ultimately break the cluster.

By building a pre-flight assesment diff users are better positioned to understand the impact of making a change.

The text based diff needs to be able to consume data from a running cluster and then compare it to objects stored in Git This feature needs to enable the following CLI based workflow.

New Pre-flight check

Run arlon pre-flight <<profile or bundle>> <<cluster>>

The CLI then collects the data form Git and the Cluster

The CLI generates a text based diff.

If multiple objects are present then the CLI should present each one by one.

The CLI should save the data for use later

Import Existing check

run arlon pre-flight -f <<existing data file>>

Diff is displayed

OSS Examples

Below are two open source diff engines that could be leveraged.

CLI based diff engine https://www.linuxlinks.com/diff-so-fancy-attractive-diffs-diff-highlight/
https://user-images.githubusercontent.com/51425288/171516023-ef7f6ca3-a55b-4662-a531-cb62b84b95a4.png

OR

https://www.linuxlinks.com/delta-viewer-git-diff-output/

https://user-images.githubusercontent.com/51425288/171516015-bbfa13c5-36a8-4cea-b08a-93124b802b93.png

Aha! Link: https://pf9.aha.io/features/ARLON-158

As an Arlon user with my own YAML based bundle (or Platform9 example) I would like to use Arlon to write the bundle directly to Git

So that i do not need to install and use seperate Git tools.

Having this in the Arlonctl will help streamline my work processes.

Aha! Link: https://pf9.aha.io/features/ARLON-148

• Inline bundle: should clusters using profile that includes the bundle be updated?
• Reference bundles: behavior should be clear: bundle itself doesn't change, but the git repo content does, so let ArgoCD do the update. But then should AutoSync flag be configurable?

Aha! Link: https://pf9.aha.io/features/ARLON-118

Similar to dynamic bundles and profiles.

Aha! Link: https://pf9.aha.io/features/ARLON-115

To help users get started we need a few example bundle apps, cert manager is a good option.
This will help users understand how Arlon works

Aha! Link: https://pf9.aha.io/features/ARLON-168

This approach uses the Profile and clusterSpec in git as a template where the original objects are duplicated such that any cluster using it runs from a an independent clone. This means changing the base profile has no impact on the cluster, however, changing the cloned object in Git will affect the single cluster that was built.

Read More Here

'Arlon cluster create cloned <cluster_name> <cluster_spec> ' -  this would clone the clusterSpec, the Profile and the Profiles Bundles.
Changing the objects in Git would reflect a change in the cluster

Aha! Link: https://pf9.aha.io/features/ARLON-162

Should arlon manage clusters that it didn't create? Possible workflows:

• "Import" or "register" an external cluster (how would it be represented?)
• Apply a profile to the cluster
• Un-apply the profile, causing all bundles from that profile to be removed from the cluster
• Unregister cluster

Other possible operations:

• Extract / capture a profile or set of bundles from an existing cluster (needs more thought / investigation)

Aha! Link: https://pf9.aha.io/features/ARLON-109

When deploying a new cluster, parameters (e.g. k8s version, node instance type, etc...) from the cluster spec are not actually used, so the cluster is deployed using default values from the Helm chart's values.yaml. This can be confirmed by looking at pkg/cluster/root_app.go's ConstructRootApp() function.

Aha! Link: https://pf9.aha.io/features/ARLON-112

The Arlon CLI needs to check for new versions and notify users to upgrade.
This could also be automatic, however this will require all changes to not be breaking changes.

At a minimum users should be able to run arlon version and see their current version

• Upgrade to be handled separately with arlon update to upgrade. #254
  The version command should return the current version and if a new version is available.

Aha! Link: https://pf9.aha.io/features/ARLON-155

Today, updating a cluster spec has no effects on existing clusters created from that spec.
Possible use cases:

• Updating kubernetes version (control plane, data plane, both)
• Changing the node count
• Perhaps changing the node type (for future cluster autoscaling scale-up operations, maybe?)

Aha! Link: https://pf9.aha.io/features/ARLON-110