Feature Request: Support Istio `VirtualService` about forecastle HOT 15 CLOSED

Great idea @osterman!

@waseem-h what are your thoughts?

from forecastle.

@goruha I had a much simpler idea in mind.

@waseem-h yes, VirtualService is a CRD; I'm suggesting that there is a Forecastle CRD (name TBD). For example, there are a handful of service meshes today. Linkerd, HashiCorp Connect, Traefik Maesh, etc. Rather than try to add support for everything right now, maybe a generic resource that let's us get the same benefit.

My inspiration is how external-dns and cert-manager work. Both support annotations just like forecastle; however, they also define their own CRDs. I consider those CRDs an "escape hatch" which lets us leverage the power of the operator but without the limitations of the annotations that only work on certain resources.

For example,

• cert-manager provides the Issuer and Certificate CRDs.
• external-dns provides the DNSEndpoint

Here's the DNSEndpoint example for external-dns:

apiVersion: externaldns.k8s.io/v1alpha1
kind: DNSEndpoint
metadata:
  name: examplednsrecord
spec:
  endpoints:
  - dnsName: foo.bar.com
    recordTTL: 180
    recordType: A
    targets:
    - 192.168.99.216

Perhaps, forecastle could define ForecastleApp that would look like:

apiVersion: externaldns.k8s.io/v1alpha1
kind: ForecastleApp
metadata:
  name: my-istio-virtual-service
spec:
  endpoints:
  - appName: "My Virtual Service"
    expose: "true"
    url: 'https://myservice.example.org/auth/admin/'
    instance: "qa-forecastle"
    group: "example"

from forecastle.

@Nuru yeah and based on what you said and osterman suggested, I have opted to go for CRDs.
@osterman I have an implementation ready to support ForecastleApp CRD in #77. The PR is not yet finished but you can test it out by applying the chart at https://github.com/stakater/Forecastle/tree/forecastle-crd/deployments/kubernetes/chart/forecastle with this image: stakater/forecastle:SNAPSHOT-PR-77-5

The chart has some RBAC permissions added as well as the CRD itself. You can manually patch your RBAC if you want to install the CRD manually

kubectl apply -f https://raw.githubusercontent.com/stakater/Forecastle/forecastle-crd/deployments/kubernetes/chart/forecastle/templates/crd.yaml

Once the new version is running, you can create ForecastleApp resources like this:

apiVersion: forecastle.stakater.com/v1
kind: ForecastleApp
metadata:
  name: app-1
spec:
  name: My App
  url: https://some-url
  icon: https://icon-url
  group: monitoring

Let me know if this is what you wanted and if it works as expected. The PR needs some test cases and documentation and maybe some minor adjustments to the CRD as well so it is not final.

from forecastle.

Its a great idea. However, I don't have much experience using istio and how VirtualService works so it might take me some time to implement support for VirtualService, but it is definitely something that I would like to add to forecastle. I will comment when I start working on it here

from forecastle.

@osterman I wonder if you would like to make a PR? And then team can assist?

from forecastle.

@goruha or @aknysh might be interested in it.

from forecastle.

@rasheedamir @waseem-h what are your thoughts on supporting a CRD instead? That way there's an escape hatch for everyone.

from forecastle.

@osterman I'm not quite sure if I understood your question exactly. But since VirtualService is a CRD, and once implemented, we can make it optional by a flag or something similar

from forecastle.

@waseem-h the idea is to have a special section in config where you define resources that should be monitoring.

like

namespaceSelector:
  labelSelector:
    matchLabels:
      component: redis
    matchExpressions:
      - {key: tier, operator: In, values: [cache]}
  matchNames:
  - test
resources:
- apiVersion: extensions/v1beta1
  kind: Ingress
- apiVersion: networking.istio.io/v1alpha3
  kind: VirtualService
- apiVersion: networking.istio.io/v1alpha3
  kind: Gateway
title: Example
headerBackground:
headerForeground: "#ffffff"
instanceName: "Hello"

from forecastle.

@osterman thats a great idea. But don't you think its kind of the same as the custom apps feature? But it requires you to restart forecastle once you change the config. So, we can work on providing a CRD based on which forecastle populates the apps. The only issue I see with this approach is that you have to specify URLs again in the CRD as well, which might not be a good idea where people are using plain old manifests. Maybe we can provide a URL discovery mechanism with this CRD? But that would again have the dependency on VirtualService or any new resource that comes in.

Something like this might be more useful maybe:

apiVersion: forecastle.stakater.com/v1
kind: ForecastleApp
metadata:
  name: my-istio-virtual-service
spec:
  endpoints:
  - appName: "My Virtual Service"
    url: 'https://myservice.example.org/auth/admin/' # Either this
    urlFrom:                                    # Or this
      kind: VirtualService
      name: my-virtual-service
      namespace: my-ns
    instance: "qa-forecastle"
    group: "example"

Then we just have to add support for URL discovery for every new kind of service exposer resource which won't be required for new types. They can just hardcore the feature

What do you think about that?

from forecastle.

@waseem-h:

@osterman thats a great idea. But don't you think its kind of the same as the custom apps feature?

Yes/no. Custom apps are great, but as implemented not helpful for this use-case. We deploy an unlimited number of staging environments for every PR/branch. We want these to be easily discoverable in Forecastle. We cannot restart forecastle each time.

Then we just have to add support for URL discovery for every new kind of service exposer resource which won't be required for new types.

    urlFrom:                                    # Or this
      kind: VirtualService
      name: my-virtual-service
      namespace: my-ns

Yes, this sounds like a hybrid of what @goruha proposed and where I was coming from...

Looks reasonable to me.

from forecastle.

@waseem-h If you look at the experience and evolution of Prometheus/Grafana/Alert Manager you will see the same issues ultimately resolved by creating Custom Resource Definitions (CRDs). Yes, you can configure alert rules and scrape configs and Grafana dashboards in config files, but those require you to collect everything together, put it in a file or ConfigMap, then deploy the updated configuration and restart the services.

The fundamental design principle of Kubernetes is that applications should be able to declare everything they need to function and therefore be deployed as a standalone package anywhere in any cluster. There should be no need to change a configuration elsewhere in order to support a new app, because that leads to unmanageable dependencies when running a huge cluster of heterogeneous workloads. So everything an application needs is specified via resource definitions: CPU cycles, memory, persistent storage, service ports, exposure of endpoints (via Ingresses), and now ServiceMonitors to indicate how to get metrics into Prometheus, PrometheusRules to set up alerts based on those metrics, labeled ConfigMaps to add Grafana dashboards to visualize those metrics, and so on.

Perhaps the closest parallel to Forecastle is Cert Manager. Both can automatically perform their functions using Ingress annotations, simplifying deployment. However, in order for Cert Manager to work with other things that do not use Ingresses, it has the Certificate CRD that lets you specify the certificate you need and where to store it and it does the rest. This again follows the model that the app specifies what it needs and the rest of the system supplies it automatically. So we need something like that for Forecastle for the same reason Cert Manager does: not everything uses Ingresses.

But as with Prometheus etc., they still retain the ability to configure things via deploy-time config files, and there is no reason Forecastle cannot keep the exact same config files it is using now. New resources simply add to the configuration.

If you can parse the Virtual Service (with annotations) and not require a CRD, that would be more convenient, but as @osterman points out, that only covers Istio. You would still have a handful of other service meshes to support. Great if you want to do that, I'm all for it. It's just a matter of how much effort you want to put into it. The bottom-line requirement is only that an app be able to specify its link in Forecastle all by itself, and without necessarily using an Ingress.

from forecastle.

@osterman Support for CRD has been released in v1.0.30. Please follow the readme to create ForecastleApp CR's and they will be picked up by forecastle

from forecastle.

Rock on! @maximmi should be testing this out today.

from forecastle.

@waseem-h sorry for the late response. This worked out marvelously!

from forecastle.