microsoft / hydrate Goto Github PK

As a user I want my HLD to reflect the specific versions of deployments on my cluster.

As a developer I want to understand other use cases of cluster deployments. Identify if it is possible to differentiate between the services based on namespaces. Example: Kafka, Zookeeper

• Get sample output file from Nate
• Support disambiguating between stateful and stateless deployments

Differentiating stateless and stateful applications in the HLD

• Use comments in the component.yaml labeling stateless and stateful.

• Support using Helm to generate these and other applications that don't have a Fabrikate Definition.

As a user I want to be able to run hydrate in a docker container, so I can use the tool on any machine (with docker).

As a developer, I want users to be able to create Fabrikate components for future use. In the scenario where a deployment doesn't match an existing component, the user may want to create a Fabrikate component definition for the new deployment.

Potential Solutions

1. Guide the user using a "wizard" that helps them create a component.

2. Guide the user using instructions in documentation form.

Useful Links

Component Definition Spec

As a developer I want to collect telemetry about the usage of the application. I also want to store and view the telemetry data.

Resources:
Choose the right data store

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

As a developer I would like more feedback on the features from people with various levels of experience:

• People familiar with cloud computing, aks, kubernetes

• People not familiar with cloud computing, aks, kubernetes

Specific Actions

• Record a short 5 minute demo

• Share the demo with:

  • Team
  • CSE
  • Interns

• Include slide with context from bedrock

• Change the subtitle to something more broad that people that are not familiar with bedrock fabrikate can understand

• Explain what HLD is

• For the use case you can mention a fictitious company

• Create a short video 1-5 minutes

• Get Feedback during Team Sync

Example here

As a user, I want the components that are populated in my HLD to match the version I'm using on my cluster.

This information can be gained by introspecting into the resource, specifically looking at image tags, or potentially a piece of metadata labeled "version".

As a developer I'd like to associate an icon/logo with the project

Investigate adding a flag to allow the user to select what kubecontext they want to connect to.

As a user I want deployments that partially match Fabrikate components to be reflected in the component.yaml.

Partial Match

A partial match is a scenario where a deployment on the cluster exists in a component, but the component contains one or more deployments that aren't present on my cluster.

Example

Cluster: Prometheus
Component: Prometheus-Grafana

Potential Solutions

1. Allow the user to automatically use the more extensive component. The added functionality of the other deployments may prove to be useful for their cluster. A flag can be set that includes the partial match components in the component.yaml in a ready to use state with Fabrikate.

2. List the partial match components under a Partial Match comment header. Include with each component the deployments that partially matched the cluster, giving the user a better idea of what's on the cluster.

Chosen Solution

A combination of potential solutions 1 and 2.

• Identify Partial Match Deployments on the Cluster

• Create List of Fabrikate Components that Partially Match

• Add each item to a labelled "Partial Match" section with comments on next steps

• Update Tests to reflect changes

As a developer I want to see some output from my Azure Pipeline builds to gain more insight into failed builds, and to ensure that everything is running properly.

Figure out how to handle a kubeconfig file with mulitple clusters. Do we want to choose a single cluster, group/handle multiple clusters as a "multi-cluster/region" deployment?

On the demo, it was taking time to connect to the cluster, there was no indication of what the tool was doing.

Expected:
Before the cluster connection starts, display the "Connecting to cluster" message

Seen:
The "Connecting to cluster message" showed up after several seconds and right after the "Connected!" message showed up

• Include slide with context from bedrock
• Change the subtitle to something more broad that people that are not familiar with bedrock fabrikate can understand
• Explain what HLD is
• For the use case you can mention a fictitious company
• Create a short video 1-5 minutes

Need to check if the connection to the cluster succeeded before printing out a success message. Also need to add an error message if the connection failed.

Change the default output path of the component.yaml file to be in an /out folder. Currently, the file is written just outside of the hydrate module, but this causes problems when extracting the file from Docker. Creating a new /out folder would simplify the process of copying the file from the container to the host.

As a developer I want a place to put my namespace yams files. The Hydrate tool should create a manifests directory in the component.yaml output to support this. The tool should also tell me what I should guide me about what I should put inside of this directory.

Example:

name: "cloud-native"
generator: "static"
path: "./manifests"
subcomponents:
  - name: "elasticsearch-fluentd-kibana"
    source: "../fabrikate-elasticsearch-fluentd-kibana"
  - name: "prometheus-grafana"
    source: "../fabrikate-prometheus-grafana"
  - name: "istio"
    source: "../fabrikate-istio"
  - name: "kured"
    source: "../fabrikate-kured"

As a user, I want to authenticate my GitHub account with Hydrate so that GitHub requests are tied to my account. This ensures that when the IP address I'm using gets saturated with GitHub requests, I can be sure that my requests go through.

@andrebriggs @edaena : This is an issue I ran into while at OneWeek, there are so many devs using the Wifi that my GitHub requests fail when I run Hydrate. I think this is pretty low priority, but I wanted to create the issue for future reference.

As a user, I want the deployments living on my cluster to be reflected in the generated component.yaml, even if there isn't a Fabrikate component that exists for it.

Steps for handling these cases:

1. Use comments in the component.yaml to make a section for these "no match" deployments, since they will likely require user input before they are ready to be used by Fabrikate.

2. Guide the user through adding a source and generator to the component.yaml

   • Deployment lives on GitHub

   • Deployment uses a helm chart

As a user, I want Hydrate to generate HLDs that leverage umbrella components (components that contain multiple other components).

Example: fabrikate-cloud-native

Now that the repository is migrated, the pipeline needs to be setup again.

• Automatically run tests for PRs

• Run the linting script on the changed files

As a developer I would like have confidence about the build state of Hydrate. Having a build badge would help let me know tests have been run and that the code is functions according to business requirements.

As a developer I want to install the module with pip.
Test PyPI Link: https://test.pypi.org/project/hydrate/#description
An official PyPI release should be discussed in the future.

Context:

• Investigate how to package it
• Refactor the code
• Update Documentation with package usage and examples

As a user I want guidance on the parts that were not automatically generated or some insight into what I might be missing to port over to fabrikate.

• Determine what information to pre-populate in the yaml, for the additional services that were found in the cluster

• Determine what information can be added as a comment in the generated yaml.

• Add more guidance to the yaml by using comments.

• Add information in the comments about the versions of the components so the user is aware of what version were pre-populated

Information that can be pre-populated

• Deployments that match existing Fabrikate components
• Deployments that don't match existing Fabrikate components
  • Name of the deployment

Information that the user must provide

• Deployments that don't match existing Fabrikate components
  • Source repository link
  • Path (if necessary)
  • Generator (if using Helm)

User Steps

1. Use hydrate to generate a component.yaml
2. No Match deployments need information to be added OR they must be commented out.
3. Follow the Fabrikate Steps to generate your manifests

When running Hydrate on Docker with telemetry enabled, the program gets stuck; no telemetry data is printed, and the program never exits.

As a developer I want the docker image to be created after I push a change to master.

When Hydrate generates the namespaces.yaml file, it contains default namespaces such as "default", "kube-system", and "kube-public". This is redundant and may cause errors with Fabrikate.

@andrebriggs @edaena : Is this actually an issue? And can it be triaged? Thanks.