kyma-project / helm-broker Goto Github PK

Description

We need to provide a strategy for updating the instances provisioned from given addons.

Scenario:

0. Register the addons repository in version 1.0
1. Provision addon from repo 1.0
2. Update repo addons to version 2.0
3. Update already created an instance from step 1 to version 2.0

We can consider re-using the Service Catalog functionality for updating already created ServiceInstance.

https://github.com/openservicebrokerapi/servicebroker/blob/v2.15/spec.md#updating-a-service-instance

AC

• proposal for update strategy for a given addon

Description

Array type in create-instance-scheme.json does not work.

Expected result

Expect this to work

{
    "$schema": "http://json-schema.org/draft-04/schema#",
    "properties": {
        "propertyName":{
            "type":"array",
            "items":{"type":"string"},
            "title":"title",
            "description":"description"
        }
    },
    "type": "object"
}

Actual result

Error provisioning ServiceInstance of ClusterServiceClass (K8S: "308dd6e2-ad87-4cf6-b97f-8948807e3a86" ExternalName: "service-name") at ClusterServiceBroker "helm-broker": Status: 400; ErrorMessage: while inserting instance operation to storage: while encoding entity: gob: type not registered for interface: []interface {}; Description: ; ResponseError: <nil>

Description
Error: failed to install CRD crds/addons-configuration.crd.yaml: unable to recognize "": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"

HEAD is now at 2b19eb0 Bump version to: v1.1.0
➜  helm-broker git:(2b19eb0) helm upgrade helm-broker charts/helm-broker --install --namespace helm-broker
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /Users/i039142/.kube/early-stage
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /Users/i039142/.kube/early-stage
Release "helm-broker" does not exist. Installing it now.
Error: failed to install CRD crds/addons-configuration.crd.yaml: unable to recognize "": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
➜  helm-broker git:(2b19eb0)

Expected result

➜  helm-broker git:(2b19eb0) kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.5", GitCommit:"c285e781331a3785a7f436042c65c5641ce8a9e9", GitTreeState:"clean", BuildDate:"2022-03-16T15:51:05Z", GoVersion:"go1.17.8", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.5", GitCommit:"5c99e2ac2ff9a3c549d9ca665e7bc05a3e18f07e", GitTreeState:"clean", BuildDate:"2021-12-16T08:32:32Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"}

Actual result

Steps to reproduce

HEAD is now at 2b19eb0 Bump version to: v1.1.0
➜  helm-broker git:(2b19eb0) helm upgrade helm-broker charts/helm-broker --install --namespace helm-broker
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /Users/i039142/.kube/early-stage
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /Users/i039142/.kube/early-stage
Release "helm-broker" does not exist. Installing it now.
Error: failed to install CRD crds/addons-configuration.crd.yaml: unable to recognize "": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"

**Troubleshooting**

<!-- Describe the steps you have already taken to solve the issue. -->

Description

Current documentation lacking a few things which are good to know when it comes to using/installing Helm Broker:

• In prerequisites section we do not specify the concrete version of required tools (you cannot always install the newest one because we do not support that)

• We do not specify for which k8s version Helm Broker works (e.g. we still do not support K8s 1.16 release)

• The installation process should be more precise on how to run minikube and how to init helm. Currently, user will get a problem with minikube installation because minikube start is not everything that needs to be set. Same thing for helm init . Suggestion: use kind instead of minikube.

  Add info that you need to clone our repo to be able to install Helm Broker chart,

  # install Service Catalog
  $helm install svc-cat/catalog --name catalog --namespace catalog
  
  # install Helm Broker
  helm install charts/helm-broker --name helm-broker --namespace helm-broker
  

  as you can see, commands are the same, but in the first case, it will work because the remote repo holds SeviceCatalog chart. In case of Helm Broker you need to have that chart locally.

  [stretch goal] Add option to install Helm Broker from remote repo

• Specify the concrete version of required tools in run-tests section

• create a devguide document with project structure and purpose of our directories (e.g. what should be put in /pkg, /hack etc. some things I've already commented in this PR description )

Description

From the Helm Broker Grafana charts we can see that the HelmBroker GetCatalog endpoint is called in each minute. Other brokers e.g. minibroker does not have such problems. We need to know if the metrics are wrong or the broker is really spammed.

Additionally, service catalog does not return the number of service classes and plans for the helm broker on /metrics endpoint and they are also not visible on chart. In comparison when we register the minibroker then all class&plans are logged.

Hint:

• check if metrics for namespaced and cluster-wide brokers are handled properly

Description

Helm Broker downloads the addon from the given Addon Repo URL which can be specified by our clients. Checkmarx report an issue that this approach can lead to resource exhaustion. We need to decide if that can be mitigated, or we should add logic for sanitizing the given URL.

See:

See:

Description

On our long-lasting clusters, the Helm-broker sometimes restart because of the memory/goroutines leak.

Check the graphs for the Helm Broker from the nightly cluster:

On this cluster, we are adding addons in the loop when executing the tests, the memory leak must be related to it.

Expected result

The Helm broker has a stable memory footprint.

Description

In the PR #52 we change our images to build them from scratch.
This change is necessary to get rid of the vulnerable libraries like curl.

The above PR meets the requirements of its story. But we still can do a small improvement in our helm-controller image which is still built from alpine.
It was very hard to use scratch in this case. helm-controller image contains multiple binaries and supports SSH auth.

My work on that scratch image was mainly done in Dockerfile:

FROM alpine:3.10 as certs
RUN apk update && apk --no-cache add ca-certificates

FROM eu.gcr.io/kyma-project/health-check-proxy:PR-6466 as health

FROM alpine:3.10 as builder
RUN apk add --no-cache git openssh mercurial

FROM scratch

ENV PATH=/usr/bin
ENV LD_LIBRARY_PATH=/usr/lib

COPY ./controller /root/controller
COPY --from=health /health-check /health-check
COPY --from=certs /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt

# Binaries
COPY --from=builder /usr/bin/ssh /usr/bin/ssh
COPY --from=builder /usr/bin/ssh-add /usr/bin/ssh-add
COPY --from=builder /usr/bin/ssh-agent /usr/bin/ssh-agent
COPY --from=builder /usr/bin/hg /usr/bin/hg
COPY --from=builder /usr/bin/git /usr/bin/git

# Libraries
COPY --from=builder /lib /lib
COPY --from=builder /usr/lib /usr/lib
COPY --from=builder /usr/share/git-core /usr/share/git-core
COPY --from=builder /usr/libexec/git-core /usr/libexec/git-core

LABEL [email protected]:kyma-project/helm-broker.git

CMD ["/root/controller"]

With that configured image, the helm-controller is working but it ends up with an error when we provide addons from the repository via SSH.

The error which I got:

while creating addons for repository from git::ssh://[email protected]/kyma-project/addons.git//addons/index.yaml?sshkey={key}: /usr/bin/git exited with 128: Cloning into '/tmp/cluster-addon-loader-dst/pnt4p5tvkg'...\nfatal: cannot run ssh -i /tmp/go-getter754833746: No such file or directory\nfatal: unable to fork\n"

I suppose that the container user or /tmp directory doesn't contain all the required permissions to do all the actions.

Description

It seems that during the bind template rendering, the helm built-in object
.Capabilities.KubeVersion.GitVersion is not properly populated. The observed error message on the binding is:

Events:
  Type     Reason          Age                From                                Message
  ----     ------          ----               ----                                -------
  Normal   Binding         70s                service-catalog-controller-manager  The binding is being created asynchronously
  Warning  BindCallFailed  69s (x2 over 69s)  service-catalog-controller-manager  Bind call failed: binding failed on error: while resolving bind data: while rendering bind yaml template: while rendering files: render error in "postgresql/templates/statefulset.yaml": template: postgresql/templates/_helpers.tpl:389:44: executing "postgresql.statefulset.apiVersion" at <.Capabilities.KubeVe...>: can't evaluate field GitVersion in type *version.Info

Steps to reproduce

Create an addon with a helm chart (e.g. https://github.com/bitnami/charts/tree/master/bitnami/postgresql) which uses the mentioned Capability (in _helpers.tpl?), like

{{- if semverCompare ">=1.4-0, <1.7-0" .Capabilities.KubeVersion.GitVersion -}}

create an instance, and try to bind to it (using some template in the bind.yaml).

Description

Update documentation about move to Helm3.

Related: #55

Description

Get below wrong docker images when doing deployment for release-1.1

eu.gcr.io/kyma-projecthelm-broker:v1.1.0 and eu.gcr.io/kyma-projecthelm-controller:v1.1.0

Expected result

eu.gcr.io/kyma-project/helm-broker:v1.1.0 and eu.gcr.io/kyma-project/helm-controller:v1.1.0

Actual result

eu.gcr.io/kyma-projecthelm-broker:v1.1.0 and eu.gcr.io/kyma-projecthelm-controller:v1.1.0

Steps to reproduce

➜ helm-broker git:(release-1.1) ✗ k create ns helm-broker
➜ helm-broker git:(release-1.1) ✗ helm upgrade helm-broker charts/helm-broker --install --namespace helm-broker

then get wrong docker images in pod

Troubleshooting

Add missing slash into
https://github.com/kyma-project/helm-broker/blob/release-1.1/charts/helm-broker/values.yaml#L40

global:
  containerRegistry:
    path: eu.gcr.io/kyma-project/

Description

The whole idea is already refined in the proposal

Consider going through the auth proxy instead of doing user validation in Helm Broker. In our flow, the user identity is always required, so we need to be sure that it is always present.

AC:

• implement proposal decision and fix the known issues
• update HB docs if needed

Description

Since implementing helm 3 memory usage has drastically increased.

The possible workaround is to use the SQL as a helm driver. Check for implementation here: https://github.com/helm/helm/blob/master/pkg/action/action.go#L412

Before 1.14 we must increase our memory settings in HB.

Example resources settings from kyma-installer:

 resources:
      limits:
        memory: 256Mi
      requests:
        memory: 32Mi

we also have migration which uses 2to3 migration supplied by helm and it uses the following settings:

resources:
      limits:
        memory: 1Gi
      requests:
        memory: 512Mi

Expected result

The Helm Broker memory usage is stable

Actual result

Memory grows in time while using the Helm Broker:

Steps to reproduce

Run the newest Helm Broker and provision instances to exceed the memory limit

Description

We should keep and version documentation (which is displayed in kyma-project.io) with helm-broker release process. Find a way how to then enable that in kyma-project.io website.

Reasons

No documentation in helm-broker repository itself.

AC

• documentation is part of the helm-broker repository
• adjust website to point to helm-broker docs for given internal version (we need to have a mappinbg between Kyma and HB version somehow)
• adjust Kyma HB docsTopics to point to HB repository (https://github.com/kyma-project/kyma/blob/master/resources/core/charts/docs/charts/content-ui/templates/docs-components-helm-broker-cdt.yaml)
• check content of existing HB Kyma documentation, we have some references to others components that we need to solve that somehow.
• POC about having our own helm-broker website. Check with community manager if that make sense in that point of time.

I'm running helm-broker in my development Kubernetes cluster for one month. Recently ETCD statefulset get OOM killed very easily.
I connected into the POD helm-broker-etcd-stateful-0, and found there are so many data in the etcd database.
i tried to use the following command to check the database entries.

etcdctl get h --prefix --limit=1 --write-out="json"

the response is

{"header":{"cluster_id":16971924815488459685,"member_id":9779250597200959907,"revision":383992,"raft_term":136},"kvs":[{"key":"aGVsbS1icm9rZXIvZW50aXR5L2FkZG9uaWQvY2x1c3Rlci8wYTZkNDAzNy0yZDdhLTRiYWUtOGU3Ny1jMmEyNmJmMDc1NTA=","create_revision":2,"mod_revision":2,"version":1,"value":"/7z/gQMBAQhhZGRvbkRTTwH/ggABDAECSUQBDAABBE5hbWUBDAABB1ZlcnNpb24BDAABC0Rlc2NyaXB0aW9uAQwAAQVQbGFucwH/mgABCE1ldGFkYXRhAf+cAAEEVGFncwH/oAABCFJlcXVpcmVzAf+KAAEIQmluZGFibGUBAgABE0JpbmRpbmdzUmV0cmlldmFibGUBAgABDVBsYW5VcGRhdGFibGUBAgABDVJlcG9zaXRvcnlVUkwBDAAAADv/mQQBASptYXBbaW50ZXJuYWwuQWRkb25QbGFuSURdZXRjZC5hZGRvblBsYW5EU08B/5oAAQwB/4QAAP+M/4MDAQL/hAABCgECSUQBDAABBE5hbWUBDAABC0Rlc2NyaXB0aW9uAQwAAQdTY2hlbWFzAf+UAAEIQ2hhcnRSZWYB/5YAAQtDaGFydFZhbHVlcwEKAAEITWV0YWRhdGEB/5gAAQxCaW5kVGVtcGxhdGUBCgABCEJpbmRhYmxlAQIAAQRGcmVlAQIAAABA/5MEAQEvbWFwW2ludGVybmFsLlBsYW5TY2hlbWFUeXBlXWludGVybmFsLlBsYW5TY2hlbWEB/5QAAQwB/4YAACf/hQMBAv+GAAECAQRUeXBlAf+IAAELRGVmaW5pdGlvbnMB/5IAAAD+Ahj/hwMBAQRUeXBlAf+IAAEkAQdWZXJzaW9uAQwAAQNSZWYBDAABCk11bHRpcGxlT2YBBAABB01heGltdW0BBAABEEV4Y2x1c2l2ZU1heGltdW0BAgABB01pbmltdW0BBAABEEV4Y2x1c2l2ZU1pbmltdW0BAgABCU1heExlbmd0aAEEAAEJTWluTGVuZ3RoAQQAAQdQYXR0ZXJuAQwAAQ9BZGRpdGlvbmFsSXRlbXMB/4gAAQVJdGVtcwH/iAABCE1heEl0ZW1zAQQAAQhNaW5JdGVtcwEEAAELVW5pcXVlSXRlbXMBAgABDU1heFByb3BlcnRpZXMBBAABDU1pblByb3BlcnRpZXMBBAABCFJlcXVpcmVkAf+KAAEKUHJvcGVydGllcwH/jAABEVBhdHRlcm5Qcm9wZXJ0aWVzAf+MAAEUQWRkaXRpb25hbFByb3BlcnRpZXMBCgABDERlcGVuZGVuY2llcwH/jAABBEVudW0B/44AAQRUeXBlAQwAAQVBbGxPZgH/kAABBUFueU9mAf+QAAEFT25lT2YB/5AAAQNOb3QB/4gAAQtEZWZpbml0aW9ucwH/kgABBVRpdGxlAQwAAQtEZXNjcmlwdGlvbgEMAAEHRGVmYXVsdAEQAAEGRm9ybWF0AQwAAQhFeGFtcGxlcwH/jgABBU1lZGlhAf+IAAEOQmluYXJ5RW5jb2RpbmcBDAAAABb/iQIBAQhbXXN0cmluZwH/igABDAAALP+LBAEBG21hcFtzdHJpbmddKmpzb25zY2hlbWEuVHlwZQH/jAABDAH/iAAAHP+NAgEBDltdaW50ZXJmYWNlIHt9Af+OAAEQAAAh/48CAQESW10qanNvbnNjaGVtYS5UeXBlAf+QAAH/iAAAHP+RBAEBC0RlZmluaXRpb25zAf+SAAEMAf+IAAAU/5UFAQEIQ2hhcnRSZWYB/5YAAAAT/6EDAQEHVmVyc2lvbgH/ogAAAC//lwMBARFBZGRvblBsYW5NZXRhZGF0YQH/mAABAQELRGlzcGxheU5hbWUBDAAAAP+q/5sDAQENQWRkb25NZXRhZGF0YQH/nAABCAELRGlzcGxheU5hbWUBDAABE1Byb3ZpZGVyRGlzcGxheU5hbWUBDAABD0xvbmdEZXNjcmlwdGlvbgEMAAEQRG9jdW1lbnRhdGlvblVSTAEMAAEKU3VwcG9ydFVSTAEMAAERUHJvdmlzaW9uT25seU9uY2UBAgABCEltYWdlVVJMAQwAAQZMYWJlbHMB/54AAAAW/50EAQEGTGFiZWxzAf+eAAEMAQwAACH/nwIBARNbXWludGVybmFsLkFkZG9uVGFnAf+gAAEMAAD+BKv/ggEkMGE2ZDQwMzctMmQ3YS00YmFlLThlNzctYzJhMjZiZjA3NTUwAQ9oZWxtLXBvc3RncmVzcWwBBTAuMC4xARpQb3N0Z3JlU1FMIGJ5IEhlbG0gQnJva2VyLgEBJDY3ZDA0M2VjLTkwNTgtNDllNS1iNmUxLTJiYzg2Njk1NjUwMQEkNjdkMDQzZWMtOTA1OC00OWU1LWI2ZTEtMmJjODY2OTU2NTAxAQZjdXN0b20BC0N1c3RvbSBwbGFuAQABOyH/owMBAv+kAAECAQROYW1lAQwAAQdWZXJzaW9uAQwAAAAY/6QBCnBvc3RncmVzcWwBBzguMTAuMTMAASx7InJlc291cmNlcyI6eyJyZXF1ZXN0cyI6eyJtZW1vcnkiOiI5Nk1pIn19fQEBBkN1c3RvbQAB/gKIY3JlZGVudGlhbDoKICAtIG5hbWU6IEhPU1QKICAgIHZhbHVlOiB7eyB0ZW1wbGF0ZSAicG9zdGdyZXNxbC5mdWxsbmFtZSIgLiB9fS57eyAuUmVsZWFzZS5OYW1lc3BhY2UgfX0uc3ZjLmNsdXN0ZXIubG9jYWwKICAtIG5hbWU6IFBPUlQKICAgIHZhbHVlRnJvbToKICAgICAgc2VydmljZVJlZjoKICAgICAgICBuYW1lOiB7eyB0ZW1wbGF0ZSAicG9zdGdyZXNxbC5mdWxsbmFtZSIgLiB9fQogICAgICAgIGpzb25wYXRoOiAneyAuc3BlYy5wb3J0c1s/KEAubmFtZT09InRjcC1wb3N0Z3Jlc3FsIildLnBvcnQgfScKICAtIG5hbWU6IFVTRVJOQU1FCiAgICB2YWx1ZToge3sgLlZhbHVlcy5wb3N0Z3Jlc3FsVXNlcm5hbWUgfX0KICAtIG5hbWU6IFBBU1NXT1JECiAgICB2YWx1ZUZyb206CiAgICAgIHNlY3JldEtleVJlZjoKICAgICAgICBuYW1lOiB7eyB0ZW1wbGF0ZSAicG9zdGdyZXNxbC5mdWxsbmFtZSIgLiB9fQogICAgICAgIGtleTogcG9zdGdyZXNxbC1wYXNzd29yZAogIHt7LSBpZiAuVmFsdWVzLnBvc3RncmVzcWxEYXRhYmFzZSB9fQogIC0gbmFtZTogREFUQUJBU0UKICAgIHZhbHVlOiB7eyAuVmFsdWVzLnBvc3RncmVzcWxEYXRhYmFzZSB9fQogIHt7LSBlbmQgfX0KICAtIG5hbWU6IFNTTF9NT0RFCiAgICB2YWx1ZTogZGlzYWJsZWQKAQEAAQEKUG9zdGdyZVNRTAEHYml0bmFtaQEaUG9zdGdyZVNRTCBieSBIZWxtIEJyb2tlci4BQGh0dHBzOi8vZ2l0aHViLmNvbS9iaXRuYW1pL2NoYXJ0cy90cmVlL21hc3Rlci9iaXRuYW1pL3Bvc3RncmVzcWwBG2h0dHBzOi8vYml0bmFtaS5jb20vc3VwcG9ydAI9aHR0cHM6Ly93d3cucG9zdGdyZXNxbC5vcmcvbWVkaWEvaW1nL2Fib3V0L3ByZXNzL2VsZXBoYW50LnBuZwEAAAECCGRhdGFiYXNlBWNhY2hlAgEA"}],"more":true,"count":191854}

How does helm broker generate so many entries into the etcd database?
Could it be the reason of OOM killed of etcd statefulset?

Description

POC with working solution where user with proper rights can provision redis addon.

Reasons

This will be Input for impl issue: #67

AC
Demo for:

• usecase 1: user with proper rights can provision redis addon
• usecase 2: user with lack of resource permission cannot successfuly provision redis addon
• usecase 3: check technical accounts to provision redis addon
  • service account (groups?)

Description

We need to find out how migration will work and create a proposal. Take into account that tiller has other helm releases, not from HB (should not be migrated). This task should have also a small POC with a working migration process.

AC:

• all old helm broker releases are migrated to helm3
• all old helm broker releases can be deleted without any permission issues
• create document under: docs/helm3

Description

Currently, in etcd storage, we are saving

• Charts as *chart.Chart (old chart struct from helm2)
• Instances with *chart.Config (old struct from helm2)

AC:

• document how this affect migrating to helm3 and how migration should look

Description

If not specified differently, the helm release name generated by the helm-broker will be

hb-<class>-<plan>-<instance’s external id>

(where the external id – if not explicitly given - will be defaulted by the catalog to a UUID as a string of length 36 (such as 9fc45a8c-15f9-4a27-9fae-dff8a7c2e6de).
On the other hand many helm charts generate resource names which are prefixed by the release name (which is not an immediate issue because resources names usually can be up to 253 characters long). But in certain situations these names (or even longer values) have to be set as label values (which are restricted to 63 characters). Prominent examples of such labels:

• statefulset.kubernetes.io/pod-name
• controller-revision-hash (on StatefulSet)

For example the value of the controller-revision-hash label of a StatefulSet could look like this

hb-<class>-<plan>-<instance's external id>-<statefulset name>-<10 digits>

Which would imply that the sum of the lengths of class, plan, and statefulset name cannot be more than 10, which is not realistic.

Clearly there are workarounds:

• Specify the externalID of the service instance explicitly at instance creation
• Override the name prefix of the artifacts generated by the chart (with fullnameOverride or similar).

However both workarounds are far from being optimal because they cause additional effort on the consumer side. It would be better if helm-broker itself could use shorter helm release names.

Steps to reproduce

Just create an addon called redis with a plan default around the bitnami redis chart (https://github.com/bitnami/charts/tree/master/bitnami/redis), and deploy ...

Description

$ kc logs -n kyma-system helm-broker-7fff98bd47-847z6 ctrl
Agent pid 8
{"level":"info","log":{"message":"Setting up client for manager","time":"2021-01-26T11:47:42.165Z"}}
{"level":"info","log":{"message":"Setting up manager","time":"2021-01-26T11:47:42.166Z"}}
I0126 11:47:43.378739       9 request.go:645] Throttling request took 1.013016466s, request: GET:https://api.mk4.dog.internal.canary.k8s.ondemand.com:443/apis/batch/v1?timeout=32s
{"level":"info","log":{"message":"Registering Components.","time":"2021-01-26T11:47:52.308Z"}}
{"level":"info","log":{"message":"Setting up schemes","time":"2021-01-26T11:47:52.308Z"}}
{"level":"info","log":{"message":"Disabling support for HTTP protocol because DevelopMode is set to false.","time":"2021-01-26T11:47:52.309Z"}}
{"level":"info","log":{"message":"Setting up controller","time":"2021-01-26T11:47:52.309Z"}}
time="2021-01-26T11:48:22Z" level=fatal msg="while waiting for etcd to be ready: while waiting for etcd: <nil>: status code 503: timed out waiting for the condition"

and then container has been restarted

Expected result

waiting loop handles that case and waits

Description

After installing Service Catalog and Helm Broker according to here i tried provsioning Redis from the sample addons located in the hack/examples directory. It failed.

Versions of the software used:

• Kubectl - 1.17.3
• Helm - 3.2.4
• Docker - 19.03
• Kind - 0.8.1

Expected result

Redis is provisioned

Actual result

Response from /last_operation

{
  "state": "failed",
  "description": "provisioning failed on error: while installing helm release:
    while installing release from chart with name [hb-redis-micro-kj] in namespace []:
    unable to build kubernetes objects from release manifest:
    unable to recognize \"\":
    no matches for kind \"Deployment\" in version \"extensions/v1beta1\""
}

Steps to reproduce

1. Follow this tutorial to install Helm Broker
2. Apply these addons
3. Provision the Redis addon on the micro plan
4. Check the status of the operation

Description

I wanted to install the SAP Samples XF Add-Ons from here: https://github.com/SAP-samples/xf-addons#installation

Installation fails when applying the ClusterAddonsConfiguration resource with the following logs from the Helm broker:

{"level":"info","log":{"message":"Start add ClusterAddonsConfiguration `xf-addons` process","time":"2020-07-09T22:12:32.314Z"}}
{"level":"info","log":{"message":"- add a finalizer","time":"2020-07-09T22:12:32.314Z"}}
{"level":"info","log":{"message":"- load addons and charts for each addon","time":"2020-07-09T22:12:32.352Z"}}
{"level":"info","log":{"message":"- create addons for \"github.com/sap/xf-addons//addons/index.yaml?ref=0.11\" repository","time":"2020-07-09T22:12:32.352Z"}}
{"level":"error","log":{"message":"while creating addons for repository from github.com/sap/xf-addons//addons/index.yaml?ref=0.11: while getting getter from factory: /usr/bin/git exited with 128: Cloning into
 '/tmp/cluster-addon-loader-dst/wdkdcvqdvp'...\nfatal: unable to access 'https://github.com/sap/xf-addons.git/': OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to github.com:443 \n","time":"2020-07-09T2
2:12:33.443Z"}}
{"level":"info","log":{"message":"- check duplicate ID addons alongside repositories","time":"2020-07-09T22:12:33.443Z"}}
{"level":"info","log":{"message":"- check duplicates ID addons in existing addons configurations","time":"2020-07-09T22:12:33.443Z"}}
{"level":"info","log":{"message":"- status: Failed","time":"2020-07-09T22:12:33.443Z"}}
{"level":"info","log":{"message":"Add ClusterAddonsConfiguration `xf-addons` process completed","time":"2020-07-09T22:12:33.448Z"}}

Expected result

I expected to have the ClusterAddonsConfiguration resource applied successfully and all Add-ons of that repository to be available in my Service Catalog.

Actual result

The ClusterAddonsConfiguration resource failed to apply; the Add-ons were not available in my Service Catalog.

Steps to reproduce

1. Provision minikube cluster:
   kyma provision minikube --cpus 6 --memory 12294 --timeout 15m0s
2. Create kyma-installer namespace for the next step:
   kubectl create namespace kyma-installer
3. Apply kyma-installer overrides:

cat <<EOF | kubectl apply -f -
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: disable-legacy-connectivity-override
     namespace: kyma-installer
     labels:
       installer: overrides
      kyma-project.io/installation: ""
  data:
    global.disableLegacyConnectivity: "true"
 EOF

4. Install Kyma with local branch set to release-1.13 and Compass & Compass Agent components enabled:
   kyma install --source=local
5. Apply ClusterAddonsConfiguration resource:

cat <<EOF | kubectl apply -f -
apiVersion: addons.kyma-project.io/v1alpha1
kind: ClusterAddonsConfiguration
metadata:
  name: xf-addons
spec:
  repositories:
    - url: "github.com/sap/xf-addons//addons/index.yaml?ref=0.11"
EOFF

Description

Looking at

https://github.com/kyma-project/helm-broker/blob/master/internal/broker/server.go#L145

it seems that the broker does not implement service instance updates (via PATCH) as described here:

https://github.com/openservicebrokerapi/servicebroker/blob/master/spec.md#updating-a-service-instance

(indeed the catalog controller receives a 405 error from the helm-broker, when updating e.g. the parameters of an existing instance).

On the other hand, the helm-broker documentation mentions a file ‘update-instance-schema.json‘ in the service plan metadata, which somehow indicates that instance updates are supported in principle.

If my understanding is correct, would it be possible to support instance updates (mostly parameter updates would be important for us, plan updates would be rather nice to have) ?

AC

• implement all investigations from #55
• use new HB in Kyma
• check current tests
• update documentation in Kyma

Description

Create release process for helm-broker repository where maintainers decides about new versions etc.

Reasons

Right now we don't have releases under helm-broker.

AC

• create official release by tagging repository by maintainer
• describe whole continuous integration process (for example, if PR merged what is a name of image)
• describe how CI flow looks like
• release process requirements from kyma included, like change log etc

Description
I can provisioning my addon via "Service Management UI" successfully, everything works well. But when de-provisioning the addon instance via clicking the "delete" button. It gives error message like:
Deprovision call failed; received error response from broker: Status: 400; ErrorMessage: while checking if instance is already deprovisioned: while getting instance operations from storage: while decoding returned entities: while decoding DSO: gob: name not registered for interface: "map[string]interface {}"; Description: ; ResponseError: <nil>

I also check the helm broker log via kubectl logs helm-broker-6c4b4cccb7-8l99f -c broker -n kyma-system. The log is shown as below, no much useful information.

{"level":"info","log":{"message":"started handling request","method":"DELETE","remote":"127.0.0.1:40424","request":"/ns/extension-installer/v2/service_instances/e9654c9e-dc81-4485-ba34-f734190671f0?accepts_incomplete=true\u0026plan_id=e868f0ba-1074-42de-8db4-e0c3ea0fdbdb\u0026service_id=ca0ebdb6-216b-465f-8fdf-499f4aecfcce","request_id":"b8255fcc-4e97-40f1-b5fe-31be22050e94","time":"2020-06-15T02:47:12.398Z"}}
{"level":"warning","log":{"message":"Server responds with error: [HTTP 400]: [while checking if instance is already deprovisioned: while getting instance operations from storage: while decoding returned entities: while decoding DSO: gob: name not registered for interface: \"map[string]interface {}\"] []","service":"server","time":"2020-06-15T02:47:12.399Z"}}
{"level":"info","log":{"message":"completed handling request","method":"DELETE","remote":"127.0.0.1:40424","request":"/ns/extension-installer/v2/service_instances/e9654c9e-dc81-4485-ba34-f734190671f0?accepts_incomplete=true\u0026plan_id=e868f0ba-1074-42de-8db4-e0c3ea0fdbdb\u0026service_id=ca0ebdb6-216b-465f-8fdf-499f4aecfcce","request_id":"b8255fcc-4e97-40f1-b5fe-31be22050e94","size":251,"status":400,"time":"2020-06-15T02:47:12.399Z","took":1685488}}
{"level":"info","log":{"message":"started handling request","method":"DELETE","remote":"127.0.0.1:40424","request":"/ns/extension-installer/v2/service_instances/98932185-e267-4a3f-a669-b0f651550bb6?accepts_incomplete=true\u0026plan_id=e868f0ba-1074-42de-8db4-e0c3ea0fdbdb\u0026service_id=ca0ebdb6-216b-465f-8fdf-499f4aecfcce","request_id":"ae8f25d5-8cb8-496a-8631-fc6a6af0d51a","time":"2020-06-15T02:47:12.596Z"}}
{"level":"warning","log":{"message":"Server responds with error: [HTTP 400]: [while checking if instance is already deprovisioned: while getting instance operations from storage: while decoding returned entities: while decoding DSO: gob: name not registered for interface: \"map[string]interface {}\"] []","service":"server","time":"2020-06-15T02:47:12.597Z"}}

So may I know how to go on debugging? By the way, I didn't have such issue in Kyma 1.11.0 before

Kyma version: 1.12.0
Kyma cluster URL: https://console.kongfu.kaka.shoot.canary.k8s-hana.ondemand.com/
Provider: GCP
Addon repo: github.com/SAP-samples/fsm-extension-installer-kyma/addons/index.yaml

Expected result
Addon service instance can be deleted successfully

Actual result
Can't de-provisioning addon service instance

Steps to reproduce

Troubleshooting

Description

Helm Broker does not retry failed creation of (Cluster)DocsTopic. Sometimes this action may fail because of connection issues, so it should be retried, see log:

while ensuring DocsTopic for addon 0a54e829-c352-4978-9291-c37f779e1900: while creating ClusterDocsTopic 0a54e829-c352-4978-9291-c37f779e1900: Post https://10.35.240.1:443/apis/cms.kyma-project.io/v1alpha1/clusterdocstopics: dial tcp 10.35.240.1:443: connect: connection refused

Also, DocsTopic is mixed with ClusterDocsTopic in the log.

Expected result

There are some retries during (Cluster)DocsTopic resources creation.

We should implement such retry i.e. here:

Right now we are failing with the first error

Actual result

Fails after the first time

Steps to reproduce

Random :(

Troubleshooting

Description

Provide a soft migration in the Helm broker code between hash parameters and flat parameters.

If the Instance from the database has a paramHash set, it cannot be considered as a conflicting one.
The corner case of that solution is that provisioned ServiceInstance can be parametrized with wrong values but we deal with the risk.

The risk of creating the ServiceInstance with the same ID, but with conflicting params are very low, but still, we should provide a test for it.

Description

We need to add more info to this readme https://github.com/kyma-project/helm-broker/blob/master/charts/helm-broker/README.md.

We need to describe the CI flow for testing chart:

• why and how we are using char-testing
• how to specify the custom parameters
• information about that the helm chart test is always building the helm-broker image from sources

Description

The controller does not process (Cluster)AddonsConfiguration with status Pending and ObservedGeneration equals to Meta.Generation. Such scenario can occur when the controller is killed just after the status Pending is set.

In our case, the helm-broker pod was restarted because of OOM while loading testing-addons.

Expected result

The controller must process an object with status Pending.

Actual result

After killing the controller, the new instance recevies the AddonsConfiguration but does not process it (all 'if' conditions are false):

        if addon.Meta.DeletionTimestamp != nil {
		c.log.Infof("Start delete %s", trace)

		if err := c.OnDelete(addon); err != nil {
			c.log.Errorf("while deleting %s process: %v", trace, err)
			return reconcile.Result{RequeueAfter: time.Second * 15}, errors.Wrapf(err, "while deleting %s", trace)
		}
		c.log.Infof("Delete %s process completed", trace)
		return reconcile.Result{}, nil
	}

	if addon.Status.ObservedGeneration == 0 {
		c.log.Infof("Start add %s process", trace)

		if err := c.PrepareForProcessing(addon); err != nil {
			c.log.Errorf("while preparing %s for processing: %v", trace, err)
			return reconcile.Result{}, errors.Wrapf(err, "while preparing %s for processing", trace)
		}
		if err := c.OnAdd(addon, addon.Status); err != nil {
			c.log.Errorf("while adding %s process: %v", trace, err)
			return reconcile.Result{}, errors.Wrapf(err, "while creating %s", trace)
		}
		c.log.Infof("Add %s process completed", trace)

	} else if addon.Meta.Generation > addon.Status.ObservedGeneration {
		c.log.Infof("Start update %s process", trace)

		lastStatus := addon.Status
		addon.Status = v1alpha1.CommonAddonsConfigurationStatus{}

		if err := c.OnAdd(addon, lastStatus); err != nil {
			c.log.Errorf("while updating %s process: %v", trace, err)
			return reconcile.Result{}, errors.Wrapf(err, "while updating %s", trace)
		}
		c.log.Infof("Update %s process completed", trace)
	}

Hint:
It seems the second condition should looks like:

if addon.Status.ObservedGeneration == 0 /*|| addon.Status.Phase == v1alpha1.AddonsConfigurationPending*/ {

Steps to reproduce

Hard to reproduce. To be sure the solution works fine, log the AddonConfioguration just after status update to pending. Then provide such AddonsConfiguration to the unit test.

Description

In Helm Broker, we are storing information about processed Addons and Charts.
During the provision request, we are rendering also the bind data, see:

results are stored in underlying storage (etcd) and used on bind request, see:

Problem

This generates a problem because we are becoming an owner of User sensitive data and such approach leads to security issues.

Solution

We should not render bind data on /provision request.
Instead, we should change the /bind action from synchronous to asynchronous and render bind data on each bind request. Thanks to that those secrets (user sensitive data) will be stored and maintained by k8s resources and HB will have those rendered data for a short period only during the bind action.

AC

• Helm Broker does not store user sensitive data
• Bind action still works and it's reliable
• New bind approach is well-documented
• Check it new bind approach does not break backward compatibility if yes then document that

Description
HelmBroker must not delete Cluster/ServiceBroker custom resource when user still have some service instances coming from HelmBroker (in both namespace and cluster wide scopes).

Reasons
With that new behaviour user will be able to delete service binding and service instance custom resources.

AC

• HB ServiceBroker/ClusterServiceBroker custom resources could be deleted in given namespace only when no HB ServiceInstances exist.
• describe new feature in documentation
• release HB and use in Kyma

Description
We need to check and compare using the pure helm3 client and flux (helm-operator). Which approach is more reasonable to use in Helm Broker.

AC:

• document with pros and cons of using: pure helm3 vs flux - helm-operator (PR #75)
• decision which approach should be used in Helm Broker

Description

Helm Broker etcd client does not specify proper timeouts. When the underlying etcd storage is down the Helm Broker operations hangs when trying to execute some operation on etcd e.g. save addon.

It's really hard to debug what's is going wrong because the whole HelmBroker freezes and stops processing other (Cluster)AddonsConfigurations. In container logs there is no information about the failure.

Expected result

When underlying etcd storage is not working then HB resets connection and prints that information in logs.

AC:

• proper timeouts are set on etcd client
• helm broker does not freeze if underlying etcd storage is not working
• helm broker prints error messages to logs in case of etcd connection failure
• helm broker check the connection to etcd on /ready endpoint

Description

Currently, there is no any build job which checks the chart. We need to have a chart validation in the build process. First, we need to define how should the verification looks like (only static chart validation or running in Minikube/Kind)

Reasons

It is easy to make a bug in the chart and the build is green

Attachments

Description

Reasons

Attachments

Description/Reasons

As Helm 3 has been released it's time to investigate and update Helm Broker and take advantage of removed tiller and role-based access control per client.

AC

• check and find out how to handle UI and kubectl access (idea: pass authenticated client/user from ServiceInstance custom resource to helm install) - extracted to: #102
• check, compare and document pros and cons of using:
  • pure helm3
  • flux - helm-operator
• find out how migration will work
  • documentation proposal
  • take into account that tiller has others helm releases not from HB (should not be migrated)
  • small poc with migration
• implement above investigation's results in the Helm Broker
• Update documentation - WIP (@piotrmiskiewicz)
• Decide if HB should be released or we should wait for impersonate user handling

Attachments

• check https://helm.sh/
• https://github.com/helm/helm-2to3
• check https://docs.fluxcd.io/projects/helm-operator/en/latest/

Description

The Helm broker builds on Prow are failing very often because of the random integration test fails.
That prevents our pipeline from creating a latest release.

Expected result

post-master-helm-broker job is always green

Actual result

post-master-helm-broker job fails often

Steps to reproduce

Merge a PR to helm-broker, check the result of the post-master-job

Description

Currently, we have 3 layers for versioning:

1. repository version which contains sets of addons
2. addons version <- problematic part
3. Helm Chart version which is wrapped a given addon

From the beginning, an addon was considered as an abstraction layer over a Helm chart which provides all information required to convert the chart into a Service Class. We added the bundle versions which are a new layer and directly not supported in Service Catalog.

The most problematic part is that the version of the addon is not used anywhere from the beginning and it's confusing for our users who need to create and maintain the addons repository.

We need to have a clear vision if we need the bundle version at all. If we cannot find any purpose for this versioning then we should deprecate that and remove it in the next version.

AC:

• a clear statement for the future of the addons versioning (if we can deprecate that and remove it in the next version)

Description

The helm-controller readiness probe is implemented in the dummy way to just return 200OK on every call.
We should return 200OK only when the controller is able to work.

We need to find the best way to implement the checking of the controller status.

Ideas:

• In SBU controller we do the test reconcile with the empty object and we expect it to process in order to indicate the readiness of the controller.
• We can wait until the controller-runtime probes implementation is ready and use it. kubernetes-sigs/controller-runtime#419

Description
Improve helm-broker readme.md and create sub-pages

AC

• in the main documentation leave just how to use helm-broker and project structure
• rest of the documentation move to the subpages of the /docs directory
• add a template file to each release with information how to run helm-broker with new released artifact

Description

Due to security, ETCD image should be bump to the latest version (ETCD chart also should be upgraded to the latest version). Additionally, secure communication (TLS) should be enabled for ETCD chart by default.