kyma-project / keda-manager Goto Github PK

Description
For the new modularization approach, we need documentation that is common for all the modules, e.g. general description (landing page), getting started, module configuration, etc.

AC

• Your audience is the end-user so the documentation must be written for them (not for the team that builds this module, or other SAP teams that wish to build a module, not for SRE, etc). Remember about the context. How much user knows by now (probably just high level knowledge about Kyma as a whole and that there are modules).
• The documentation must be consumable by the kyma-project.io home page (split into several information types/md files)
• Create a docs folder following the standard structure proposed in -> this issue
• In each module's documentation include:
  • General description of the functionality (if the module is mandatory, mention it!)
  • Getting Started - instructions on how to use the module. The module docs must mention the prerequisite that the module must be enabled. They must also include instructions on how to install/enable a module.
  • Module configuration
  • Instructional documents (currently called "dev tutorials" and "operational guides") - must be isolated, i.e. no dependencies on other tutorials or modules. An example.
  • If needed, troubleshooting
  • Instructions on how to give feedback

Reasons

Assignees

@kyma-project/technical-writers

Attachments
kyma-project/kyma#16421

Description
Implement automated release flow

AC

• only authorised user can trigger release flow
• github release flow should start upon new tag creation
• github release flow should create github release on the new tag
• github release flow should upload changelog as release content
• github release flow should upload artefects (keda-manager.yaml)

Reason
Automation of CD for keda-manager releases

Attachments
https://hub.tekton.dev/tekton/task/create-github-release
https://trstringer.com/github-actions-create-release-upload-artifacts/
btp-operator release flow

Description

Describe which tests are executed at which stage of development and refer that to the test strategy

Area

• Modularization
• Keda

Reasons
Keda is a reference module and it's readme file should guide other module developers how to fulfill test strategy

Description

Implement a minimal integration test that allows to verify if the keda module is working fine, i.e

• deploy a ScaledObject CR targetting a deployment
• verify the status of the ScaledObject CR
• verify if the underlying HPA is created and has a "ready" status
• verify deletion without orphan resources (ScaledObject & HPA)

Ideas

• define a dedicated make target that would allow us to make the extra test
• two options:
  • combine pull-keda-module-build with the test in one job
  • run the test in a dependant job (after pull-keda-module-build is finished) - preferred as it goes into worflows in the CI

Reasons
We as keda module providers we'd need to have an automated way to judge if a keda module version is ready for submission into a target channel.

Attachments

Description

We have to be able to clearly identify image version for each commit.
Drop the tag based versioning and use sha for versioning keda manager image

Reasons

Attachments

Description

The goal is to conform with k8s deletion mechanism and ensure it's implemented correctly with minimal impact on the module's functionality.

User will effectively remove keda module only if he first manually removes all ScaledObjects created by him.
If he doesn't do it, keda module deletion is initiated but remains blocked. The remaining user created Scaled objects should remain controlled by the keda workloads (they should not be orphaned).
Keda CR status reflects the blocked deletion (this status should also surface to Kyma CR status)
Once User removes all the CRs (Scaled objects), the deletion process will continue and there will be nothing left that belongs to keda module on the cluster.

AC:

• Implement the mechanism for the chosen strategy in the module.
• Update the module's documentation.
• Make sure to reflect the changes in tests

Reasons

To minimize any negative impact on the module's functionality and help the team to keep the module in a stable state.
To conform with k8s module lifecycle best practices.
To avoid orphaned objects that may become corrupted when the module is re-introduced

Description
Upgrade test on post-submit job fails, example

Happy Kyma-ing! :)
kubectl apply -f ../../template-k3d.yaml
error: the path "../../template-k3d.yaml" does not exist
make[1]: Leaving directory '/home/sa_117798148653314453801/keda-manager/hack/common'

Provide BASIC readme file for Keda Manager repository

AC
The following should be part of the readme

• What is in the repo? : manager and module (Keda in version xx) it installs
• What this code does : manages lifecycle of Keda installation on Kyma runtimes
• How to Create/Delete/Change Keda installation

Reason
Provide how Keda manager can be used to install Keda

Description

Bump Keda manifests to the latest version

Reasons

Currently used version uses deprecated HPA version:

autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscale

Attachments

https://github.com/kedacore/keda/releases/tag/v2.10.0

Description

For pre-submit phase we shout keep only:

• compiling,
• linting,
• unti testing
• building manager image
• integration tests of keda manager itself on k3d (Verify Keda CR status)
• integration tests of keda module installed by its manager (smoke of keda features)

For post-submit phase we should run additionally:

• building module + generate keda module template
• install module from template via lifecycle manager (Verify KYMA CR)
• upgrade test

Reason
Integration with rapidly changed alpha software should not block submitting PRs to keda manager

Description

Ensure keda-manager deployment schedules with resources specified in the spec.

Reasons

Without resource definition for keda manager it may not be able to schedule in congested cluster. Without any of that kubernetes assignes lowest QoS class for the workload and it would be served on best effort basis

Attachments
https://kubernetes.io/docs/tasks/configure-pod-container/quality-service-pod/

Description

Design and implement integration test that will check if upgrade between two keda-manager versions is possible.

The solution can be used to test:

• two versions of module coming from the same channel
• the promotion of module version from one channel to another

Reasons

• Compliance with Operational Test Management

Depends on:

• #74

Ideas

One step of upgradability test is to fetch and install "latest version from channel"
How to do it? What is the source of the "latest from channel"

it could be simply HEAD from main in the given channel folder, for example
https://github.com/kyma-project/kyma/blob/main/modules/alpha/moduletemplate-keda.yaml

We will implement three jobs

Description

The default target namespace for keda-manager and keda workloads should be kyme-system

Expected result

Keda manager and keda workloads should land in kyma-system.

Actual result

Keda manager is installed in keda-system and keda workloads land in keda namespace.

Steps to reproduce

1. Install keda on modular kyma.. i.e locally (following this guide)
2. Notice that keda manager is installed in keda-system and keda workloads land in keda system.

Troubleshooting

Description

Improve keda installation status verification to rely on k8s events instead of requeued verification of "3" of the keda deployments

Reasons

Right now we requeue verification of Keda installation
We should aim to have passive state resolution that bases on events comming from observed keda workloads.

In case of error we might end up in infinitely requeued operation.

Attachments

status verification

Description

A package with state functions has been added some time ago as an experiment. Right now we can pay a technical debt by adding/improving missing tests.

Attachments

The potentially similar implementation for the serverless-manager.

Description

Keda manager should deploy keda from images synced into europe-docker.pkg.dev/kyma-project registry

• configure image syncer
• edit the keda manifest to reference the europe-docker.pkg.dev/kyma-project registry
• remove explicit exceptions for keda images in warden's allowRegistries list

Description

Prerequisite: Fix dockerfile as it misses layer with module charts - #6

Define a pre- and post-submit jobs to keda-manager that would :

• unit tests the KEDA lifecycle management
• build the keda manager image

(STRETCH):

• generate module manifest ( follow up - #16 )

Reasons
Start with CI on an early stage developing keda operator
Kyma lifecycle manager needs an artefact with keda manager to pull and use ( apply on target runtimes )

Description

ATM the keda-manager owns kyma-system namespace, kyma-system is shared between multiple modules.
The deletion of the keda-managermodule may trigger the deletion of all keda-system content.

AC:

• keda-manager should not own kyma-system namespace

Reasons

The deletion of the keda-managermodule may trigger the deletion of all keda-system content

Attachments

• https://github.com/m00g3n/keda-operator/blob/add-finalizer/config/manager/manager.yaml#L1-L6

Description

Generated template.yaml does not include sample Keda CR in the spec.data: field.
Because of that the keda manager has no Keda to install after enabling keda module in the kyma runtime

Expected result

apiVersion: operator.kyma-project.io/v1alpha1
kind: ModuleTemplate
metadata:
  name: moduletemplate-keda
  namespace: kcp-system
  ...
spec:
  target: remote
  channel: stable
  data:
    apiVersion: operator.kyma-project.io/v1alpha1
    kind: Keda
    metadata:
      name: keda-sample
    spec:
      logging:
        operator:
          level: "debug"
  descriptor:
    component:
      componentReferences: []
     ```
     
**Actual result**

```yaml
apiVersion: operator.kyma-project.io/v1alpha1
kind: ModuleTemplate
metadata:
  name: moduletemplate-keda
  namespace: kcp-system
  ...
spec:
  target: remote
  channel: stable
  data:

  descriptor:
    component:
      componentReferences: []
     ```

<!-- Describe what happens instead. -->

**Steps to reproduce**

run `make module-build`

**Troubleshooting**

Could be that it is caused by our non-flat project structure and is dependant on #43 

Description

Describe how to run keda manager locally

Reasons

Developers need to integrate keda manager with lifecacle manager on their local k8s clusters

Assignees

@kyma-project/technical-writers

Attachments

Description

Provide automated tool (git action, tecton, etc) to release keda-manager artefacts as release artefacts:

• keda-manager.yaml (incl all manifests of keda-manager allowing to install it on any kubernetes)
• module-template (allowing to install it using lifecycle manager)

Reasons

Developers can release new keda manager version in an automated way to avoid human-mistakes

Attachments

Description

For better understanding and visibility, of what is going on during the de-installation process, we should share information about the process of the separated condition. We can call it Deleted. The condition should inform us when de-installation started and about an error if any would happen. Right now we emit events every time the process is starting and finishing - we can re-use communications from events in the condition.

(Optional) We can think about using the Warning state in our case when de-installation is failed because there are leftovers on the cluster. ( more info )

Expected behaviour
After uninstalation of keda manager no orphaned resources should be left in the runtime

Actual behaviour
After disabling keda module via Kyma CR there are two lease objects of keda manager and keda operatorleft in the runtime

{
    "namespace": "kyma-system",
    "kind": "Lease",
    "name": "4123c01c.operator.kyma-project.io",
    "apiVersion": "coordination.k8s.io/v1"
 },
  {
4296a4303,4308   },
   {
     "namespace": "kyma-system",
     "kind": "Lease",
     "name": "operator.keda.sh",
     "apiVersion": "coordination.k8s.io/v1"
}

https://sap-btp.slack.com/archives/C03BC9UVAR1/p1680248956529109

Description

Keda namespace is not deleted after keda uninstallation

apiVersion: v1
kind: Namespace
metadata:
  labels:
    control-plane: controller-manager
    kubernetes.io/metadata.name: keda
  name: keda
spec:
  finalizers:
  - kubernetes
status:
  conditions:
  - lastTransitionTime: "2022-10-31T13:26:04Z"
    message: 'Discovery failed for some groups, 1 failing: unable to retrieve the
      complete list of server APIs: external.metrics.k8s.io/v1beta1: the server is
      currently unable to handle the request'
    reason: DiscoveryFailed
    status: "True"
    type: NamespaceDeletionDiscoveryFailure
  - lastTransitionTime: "2022-10-31T13:25:57Z"
    message: All legacy kube types successfully parsed
    reason: ParsedGroupVersions
    status: "False"
    type: NamespaceDeletionGroupVersionParsingFailure
  - lastTransitionTime: "2022-10-31T13:26:40Z"
    message: All content successfully deleted, may be waiting on finalization
    reason: ContentDeleted
    status: "False"
    type: NamespaceDeletionContentFailure
  - lastTransitionTime: "2022-10-31T13:26:40Z"
    message: All content successfully removed
    reason: ContentRemoved
    status: "False"
    type: NamespaceContentRemaining
  - lastTransitionTime: "2022-10-31T13:25:57Z"
    message: All content-preserving finalizers finished
    reason: ContentHasNoFinalizers
    status: "False"
    type: NamespaceFinalizersRemaining
  phase: Terminating

Expected result

keda namespace should be deleted

Actual result

keda namespace remains in termination state

Steps to reproduce

Uninstall Keda form Kyma by editing Kyma CR

Troubleshooting

Description

When module deletion stuck due to existing scaled objects, it should be properly displayed in module status. Currently displayed message for Keda that stuck in deleting state is misleading (message is from previous state - ready):

status:
conditions:
- lastTransitionTime: '2023-05-16T12:28:07Z'
message: keda-operator and keda-operator-metrics-server ready
reason: Verified
status: 'True'
type: Installed
state: Deleting

It should be similar to the way BTP Operator does:

status:
conditions:
- lastTransitionTime: '2023-05-18T08:07:14Z'
message: >-
All service instances and bindings must be removed: 1 instance(s) and 0
binding(s)
reason: ServiceInstancesAndBindingsNotCleaned
status: 'False'
type: Ready
state: Deleting

Description

Introduce a makefile as a toolbox for developers that helps to install, test and deploy kyma with keda on k3d.

Reasons

Installing keda locally via keda manager requires a lot of commands and is cumbersome.

Attachments

Description
Run smoke test that installs keda module on k3d and verifies Keda CR and Kyma CR status

We have makefile target in hack folder that realises that

cd hack/local
make run

AC

• Test should be executed as a pre-submit job with every change in keda-manager
• Test should succeed only if Keda and Kyma CRs resolve to Ready state
• STRETCH: create the template for the job so that other modules could generate similar smoke test job from it

Reason
We need to have an automated verification if keda can be successfully installed on modular kyma.

Related
Depends on #56

Description

Provide development guide for module developer based on frontrunner module ( Keda) that should be a reference for other modules and should help getting started with new modules. Describe steps for developer that are relevant for him at different stages of module development.

AC:

Getting started:

• User needs to understand architecture.
  • links to high level goals of modularisation and architecture
  • pinpoint the integration points (interfaces, conventions that module must fulfil)
• User needs to create project scaffold
  • project structure
  • makefile (expected targets for conformance tests)
• User needs to write and run module in local mode (single k3d cluster)
• User needs to understand how to configure module (configurability via Keda Custom Resource)
• How to apply module (template) on any target kyma runtime

Productise module

• User needs to learn how to write CI/CD jobs:
  • building and publishing images (incl. how to use secrets ),
  • testing jobs
• How to submit module to a release channel
  • links do process
  • requirements
  • governance

Description

• enable keda module
• create a second keda resource => will be almost immediately ready
• delete duplicate of keda resource

Expected result
second keda resource should never be ready
resources created by the first keda resource should not be modified by any action taken towards the second keda resource

Actual result

• first keda resource will still be ready
• deletion of 2nd keda resource kills the actual keda installation. no keda operator nor keda metrics server is available after deletion of the 2nd keda-cr

Steps to reproduce

Troubleshooting

Description

Implement custom reconciliation loop as part of keda manager w/o dependency to module-manager`s declarative library and helm.

Reasons

Generic reconciliation loop from module-manager's declarative library is not allowing to hook in custom logic.
https://github.com/kyma-project/template-operator#default-declarative-reconciliation-and-status-handling

Description

Provide the DO NOT EDIT disclaimer to all objects that the manager is going to apply.

Example (from reconciler):

reconciler.kyma-project.io/managed-by-reconciler-disclaimer: |-
      DO NOT EDIT - This resource is managed by Kyma.
      Any modifications are discarded and the resource is reverted to the original state.

Description

Keda module manager should not only install keda workloads and CRDs but also add a configuration for kyma dashboard so that users can manage CRs that keda brings.

AC

• user should see a list of scaled objects under Configuration
• user can create a scaled objects using templates (cpu utilisation, prometheus)

Reasons

Leverage UI Extensibility for best usability of modular kyma

Attachments

https://github.com/kyma-project/busola/tree/main/docs/extensibility

Description
The README.md file for Keda Manager should be updated taking into account the recent changes.

AC:
README.md must contain the following information:

• General component description
• Prerequisites you need to have to work with the manager
• How to fully build own manager image
• How to start the manager on the cluster (for example k3s cluster)

Assignees

@grego952

Attachments

Description
Conditions of Keda CR reconciliation must be documented so that users can learn more about status of Keda installation

Attachments
Example:
https://github.com/kyma-project/btp-manager/blob/main/docs/operations.md#conditions

Description

Configure UI components for KEDA CR.

AC:

• user should be able to edit keda consumed resources and log level via edit foms, not yaml editor.
• user should be able to see detailed status (conditions) of the installed keda
• user should be able to read deteialed error messages in order to be able to troubleshoot (for example when deletion is stuck)

Reasons

Users will see all enabled modules on the dashboard's landing page. From there user could navigate to a CR view of the given module where he can configure it.

Description

Recently we refactored CI make targets but unfortunately introduced few bugs..
#139

Review/cleanup the makefile

• remove yq as its unused anymore (we no longer patch yaml resources using yq)
• use available kyma CLI commands whenever appliacable (i.e kyma alpha enable module instead of patching Kyma CR)
• get rid of dependencies to the hack/common package example

Address naming concerns from this PR
#1

• Naming of the CRD ( Keda vs KedaManagerConfig )
• naming of the keda manager's components.. i.e operator-manager-controller

Module is named: kyma-project.io/keda
Operator workload should be named : keda-operator
Original keda workload names should be kept as they are defined in the orginal charts in the keda upstream

Description

Changes in operator/config do not trigger job that builds module oci image

Expected result

Changes in operator/config should trigger module-build job

Actual result

Changes in operator/config should trigger module-build job

Attachment

see #37

Description

ATM we are pushing the module template file as an PCI layer into artefact registry.
Instead we should just print the module template while the module is build on CI

Description

Test basic lifecycle of Keda installation upon C(R)UD operations of Keda CR, I. e:

• add basic scaffold for operator tests - #3
• test if Keda's CRDs (i.e scaledObject) are applied on target runtime once Keda CR is applied
• test if updated properties of KEDA CR reflect the changes in the target deployment of keda controller (:warning:blocked by kyma-project/module-manager#94)
• test Keda installation cleanup once Keda CR is removed

Reasons

We need to have a base line for verification of the incoming changes

Related Issues
kyma-project/kyma#15290

Description

Automatically generate ModuleTemplate for Keda Manager

• ModuleTemplate yaml should be generated automatically upon merges to main branch
• the image of the keda manager resulting from the same commit should be referenced in the module template (synchronised dependant CI jobs)
• for now use github actions to wait for keda manager artefact and generate the template )
• module template should be attached to the keda-manager repo as artefact

Reason
Lifecycle manager needs Module Template that includes the information how to instal Keda Manager

Description

We need an efficient way to upgrade keda version that is installed by keda manager.
We should bind keda manager with keda version and make the version configurable.
Keda manifests should be automatically fetched from keda version.

Reasons

We should avoid using static keda manifests in our codebase but rather download them from keda upstream.

Attachments

Description

Create pipeline to test the latest Kubernetes API.

Exemplary scenario of the test:

• Create a cluster using the latest published stable Kubernetes cluster version
• Run module smoke tests

Reasons

• Compliance with Operational Test Management
  Operational Test Management

Description
Target namespace for keda module should be driven by the Keda CR namespace.
Currently keda module always installs in kyma-system - as its hardcoded in keda.yaml

Reason
Keda operator should allow for flexibility.

Description

Store module template for keda module (in alpha channel) so that infrastructure managers of modular kyma can use it.

Reasons
kyma-project/module-manager#157

Attachments
Source of module templates per channel

Description

In KEDA char environmental variables are passed as name-value pairs, the CRD spec defines it as map, this is is causing an error during the installation.

Description
Please move the operator content to the root folder

• api
• config
• ...
  Please merge the common files (gitignore, makefile, readme)

Reasons

Simplify project structure. The extra operator subfolder is artificial.
Additionally tooling for module creation may have assumptions on project structure

Attachments

See corresponding changes in template-operator

Description

After installing keda module the module manager is flooded with errors about unexpected empty response from external metrics API

Expected result

Module manager should log no errors

Actual result

E1019 13:33:16.429504 1 memcache.go:206] couldn't get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1
E1019 13:33:16.525036 1 memcache.go:206] couldn't get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1
E1019 13:33:16.625030 1 memcache.go:206] couldn't get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1
E1019 13:33:16.657362 1 memcache.go:206] couldn't get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1
E1019 13:33:16.719262 1 memcache.go:206] couldn't get resource list for external.metrics.k8s.io/v1beta1: Got empty response for: external.metrics.k8s.io/v1beta1

Steps to reproduce

Follow the steps to install keda
https://github.com/kyma-project/keda-manager#installation-in-modular-kyma-on-the-local-k3d-cluster

Observe logs of the module-manager in kcp-system namespace

Attachments

template.yaml.txt

Related Issues
kedacore/keda#1797

Description

• Enable protecode scan for keda and keda manager
• Enable whitesource scan for keda manager
• Enable checkmarx for keda manager

Additionally, please scan keda images manually using protecode to realize any potential release blockers

Reason
Rising awareness of potential CVEs being part of Keda manager.
Fulfil product standards to have all components scanned.

Description
Keda Manager should detect and fail in case custom keda installation (other than managed by kyma's keda-manager) already exists

Expected result

Installation should fail.. (Given CRD already exists...)
Keda CR status should reflect the keda installation collision

Actual result

Keda operator takes over the custom installation ( for example see annotations on keda namespace ) and uninstalls it upon Keda CR removal

Steps to reproduce

1. Deploy custom Keda installation manually i.e using kubectl apply -f https://github.com/kedacore/keda/releases/download/v2.8.0/keda-2.8.0.yaml
2. Install Keda via Keda Manager
3. Installation finishes with success
4. Keda operator takes over the custom one ( for example see annotations on keda namespace )
5. uninstall Keda CR
6. Custom Keda Installation disappears