kcp-dev / controller-runtime-example Goto Github PK

View Code? Open in Web Editor NEW

11.0 3.0 23.0 111 KB

License: Apache License 2.0

Go 73.81% Dockerfile 1.59% Makefile 24.60%

controller-runtime-example's Introduction

Deprecated

Example code was moved into the kcp-dev/controller-runtime repository itself!

controller-runtime-example

An example project that is multi-cluster aware and works with kcp

Description

This repository contains an example project that works with APIExports and multiple kcp workspaces. It demonstrates two reconcilers:

ConfigMap
1. Get a ConfigMap for the key from the queue, from the correct logical cluster
2. If the ConfigMap has labels["name"], set labels["response"] = "hello-$name" and save the changes
3. List all ConfigMaps in the logical cluster and log each one's namespace and name
4. If the ConfigMap from step 1 has data["namespace"] set, create a namespace whose name is the data value.
5. If the ConfigMap from step 1 has data["secretData"] set, create a secret in the same namespace as the ConfigMap, with an owner reference to the ConfigMap, and data["dataFromCM"] set to the data value.
Widget
1. Show how to list all Widget instances across all logical clusters
2. Get a Widget for the key from the queue, from the correct logical cluster
3. List all Widgets in the same logical cluster
4. Count the number of Widgets (list length)
5. Make sure .status.total matches the current count (via a patch)

Getting Started

Running on kcp

Build and push your image to the location specified by IMG:

make docker-build docker-push REGISTRY=<some-registry> IMG=controller-runtime-example:tag

Deploy the controller to kcp with the image specified by IMG:

make deploy REGISTRY=<some-registry> IMG=controller-runtime-example:tag

Uninstall resources

To delete the resources from kcp:

make uninstall

Undeploy controller

Undeploy the controller from kcp:

make undeploy

Contributing

See CONTRIBUTING.md

How it works

This project aims to follow the Kubernetes Operator pattern

It uses Controllers which provides a reconcile function responsible for synchronizing resources until the desired state is reached.

Test It Out

Install the required resources into kcp:

make install

Run your controller (this will run in the foreground, so switch to a new terminal if you want to leave it running):

make run

NOTE: You can also run this in one step by running: make install run

Modifying the API definitions

If you are editing the API definitions, regenerate the manifests using:

make manifests apiresourceschemas

NOTE: Run make --help for more information on all potential make targets

More information can be found via the Kubebuilder Documentation

License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

controller-runtime-example's People

Contributors

Stargazers

Watchers

controller-runtime-example's Issues

Controller not finding the widget resource when no apibinding created

It is a know issue and not directly related to the example that at least one APIBinding has first to be created for a controller to be able to watch a resource type through a virtual workspace.

As I have seen a couple of people loosing time in the troubleshooting of it in the context of simply running the controller runtime example I am proposing:

to have a word about it in the readme
to have an APIBinding manifest in the config/kcp directory that gets automatically created when the example is deployed (there is an open PR for that)

make deploy doesn’t work b/c of services

with a virgin kcp cluster

$ make deploy
/tmp/controller-runtime-example/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
cd config/manager && /tmp/controller-runtime-example/bin/kustomize edit set image controller=controller:latest
/tmp/controller-runtime-example/bin/kustomize build config/default | kubectl apply -f -
namespace/controller-runtime-example-system created
serviceaccount/controller-runtime-example-controller-manager created
role.rbac.authorization.k8s.io/controller-runtime-example-leader-election-role created
clusterrole.rbac.authorization.k8s.io/controller-runtime-example-kcp-manager-role created
clusterrole.rbac.authorization.k8s.io/controller-runtime-example-manager-role created
clusterrole.rbac.authorization.k8s.io/controller-runtime-example-metrics-reader created
clusterrole.rbac.authorization.k8s.io/controller-runtime-example-proxy-role created
rolebinding.rbac.authorization.k8s.io/controller-runtime-example-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/controller-runtime-example-kcp-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/controller-runtime-example-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/controller-runtime-example-proxy-rolebinding created
configmap/controller-runtime-example-manager-config created
apiexport.apis.kcp.dev/controller-runtime-example-data.my.domain created
apiresourceschema.apis.kcp.dev/controller-runtime-example-today.widgets.data.my.domain created
unable to recognize "STDIN": no matches for kind "Service" in version "v1"
unable to recognize "STDIN": no matches for kind "Deployment" in version "apps/v1"
make: *** [Makefile:112: deploy] Error 1

When compute is being added the Deployment API is added per default by the syncer. This is not the case of the Service API. This is not required by the core functionalities of the controller but referenced in the manifests for monitoring, auth-proxy and webhook. Of the three only auth-proxy is activated by default. As Robin mentioned this could get commented out.
Possibly the readme or the kustomization.yaml could be enhanced for making the user aware of it.

Add README

main controller-runtime-example is not working due to old API group

main of controller-runtime-example against main KCP, as the API group was renamed, we should probably pin to a commit that has this change + change manifests

/kind bug

controller runtime client list fails with error "Index with name cluster does not exist"

hi,
When I use latest https://github.com/kcp-dev/controller-runtime, the controller runtime client List is fails with error "Index with name cluster does not exist".

Steps to reproduce:
point to latest kcp-dev/controller-runtime, make change to compile the code and test the controller-runtime-example. you can use this branch to test https://github.com/prembhaskal/kcp-dev-controller-runtime-example/tree/client-list-issue


% export KUBECONFIG=/tmp/kcp.kubeconfig
% kubectl ws create prem-org --type organization
% kubectl ws prem-org

 % cat /tmp/data-api-binding.yaml 
apiVersion: apis.kcp.io/v1alpha1
kind: APIBinding
metadata:
  name: data.my.domain
spec:
  reference:
    export:
      name: data.my.domain
      path: root
  permissionClaims:
    - resource: "secrets"
      all: true
      state: Accepted
    - resource: "configmaps"
      all: true
      state: Accepted
    - resource: "namespaces"
      all: true
      state: Accepted
% kubectl apply -f /tmp/data-api-binding.yaml

% # create configmap with needed labels and data, check the output of the controller and see if anything wierd happens there.
% cat /tmp/test-cm.yaml 
apiVersion: v1
kind: ConfigMap
metadata:
  name: syncer-config
  labels:
    name: syncer-config
data:
  agent-target-cluster: "root:045aa3c9-017c-4824-a410-561b1e2ddef3:soumik-demo:cg-1"
  agent-target-name: "downstream"
  agent-target-namespace: "default"
  upstream-context: base
  secretData: syncer-config

% kubectl apply -f /tmp/test-cm.yaml

error seen

2024-01-29T23:23:54+05:30	INFO	Update: updated configMap	{"controller": "configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"syncer-config","namespace":"default"}, "namespace": "default", "name": "syncer-config", "reconcileID": "38ae15cb-9d1c-433d-a3d6-645970009745", "cluster": "29qso51vovx9kz1x"}
2024-01-29T23:23:54+05:30	INFO	Get: retrieved configMap	{"controller": "configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"syncer-config","namespace":"default"}, "namespace": "default", "name": "syncer-config", "reconcileID": "2b6493be-d719-4b86-9870-2ee1aa72fcea", "cluster": "29qso51vovx9kz1x"}
2024-01-29T23:23:54+05:30	ERROR	unable to list configmaps	{"controller": "configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"syncer-config","namespace":"default"}, "namespace": "default", "name": "syncer-config", "reconcileID": "2b6493be-d719-4b86-9870-2ee1aa72fcea", "cluster": "29qso51vovx9kz1x", "error": "Index with name cluster does not exist"}
github.com/kcp-dev/controller-runtime-example/controllers.(*ConfigMapReconciler).Reconcile
	/Users/bpremkumar/code/github.com/prembhaskal/kcp-dev-controller-runtime-example/controllers/configmap_controller.go:87
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/Users/bpremkumar/go/pkg/mod/github.com/kcp-dev/[email protected]/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/Users/bpremkumar/go/pkg/mod/github.com/kcp-dev/[email protected]/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/Users/bpremkumar/go/pkg/mod/github.com/kcp-dev/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/Users/bpremkumar/go/pkg/mod/github.com/kcp-dev/[email protected]/pkg/internal/controller/controller.go:227

Update README to note that both REGISTRY and IMG should be provided

The README indicates:

make docker-build docker-push IMG=<some-registry>/controller-runtime-example:tag

but it should be

make docker-build docker-push REGISTRY=<some-registry> IMG=controller-runtime-example:tag

because the makefile is using a default REGISTRY ?= localhost and is using ${REGISTRY}/${IMG}

Same for other commands using IMG

Server rejected event errors when leader election is enabled

When the leader election is enabled:

restConfig := ctrl.GetConfigOrDie()
...
mgr, err := kcp.NewClusterAwareManager(cfg, ctrl.Options{
	Scheme:                 scheme,
	MetricsBindAddress:     metricsAddr,
	Port:                   9443,
	HealthProbeBindAddress: probeAddr,
	LeaderElection:         true,
	LeaderElectionConfig:   restConfig,
	LeaderElectionID:       "68a0532d.my.domain",
})

(NOTE: I specify the original rest config so it doesn't use the VirtualWorkspace URL, otherwise, it would fail because the client used in leader election doesn't use the right round tripper.)

then the leader election code tries to create an event in the VirtualWorkspace URL, but it fails because it's using a wrong path (does not contain the expected /clusters/* section)

I0707 11:39:21.287347       1 round_trippers.go:553] POST https://192.168.1.133:6443/services/apiexport/root:plane:myapp/controller-runtime-example-data.my.domain/api/v1/namespaces/controller-runtime-example-system/events 403 Forbidden in 0 milliseconds
E0707 11:39:21.287479       1 event.go:267] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"68a0532d.my.domain.16ff8920aa62aeb2", GenerateName:"", Namespace:"controller-runtime-example-system", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"ConfigMap", Namespace:"controller-runtime-example-system", Name:"68a0532d.my.domain", UID:"bc0f0a11-21c8-4a6a-b632-6d5abe3a7532", APIVersion:"v1", ResourceVersion:"27429", FieldPath:""}, Reason:"LeaderElection", Message:"controller-runtime-example-controller-manager-65cd9fdd6c-5b58t_61618914-f476-4a88-86ce-4cfcbdf88ea4 became leader", Source:v1.EventSource{Component:"controller-runtime-example-controller-manager-65cd9fdd6c-5b58t_61618914-f476-4a88-86ce-4cfcbdf88ea4", Host:""}, FirstTimestamp:time.Date(2022, time.July, 7, 11, 39, 21, 286184626, time.Local), LastTimestamp:time.Date(2022, time.July, 7, 11, 39, 21, 286184626, time.Local), Count:1, Type:"Normal", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'forbidden: User "system:serviceaccount:controller-runtime-example-system:controller-runtime-example-controller-manager" cannot post path "/services/apiexport/root:plane:myapp/controller-runtime-example-data.my.domain/api/v1/namespaces/controller-runtime-example-system/events": Path not resolved to a valid virtual workspace' (will not retry!)
I0707 11:39:21.289195       1 round_trippers.go:553] POST https://192.168.1.133:6443/services/apiexport/root:plane:myapp/controller-runtime-example-data.my.domain/api/v1/namespaces/controller-runtime-example-system/events 403 Forbidden in 1 milliseconds
E0707 11:39:21.289449       1 event.go:267] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"68a0532d.my.domain.16ff8920aa62d112", GenerateName:"", Namespace:"controller-runtime-example-system", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Lease", Namespace:"controller-runtime-example-system", Name:"68a0532d.my.domain", UID:"b5d779ea-914b-41e5-aa3a-71ff06ee5349", APIVersion:"coordination.k8s.io/v1", ResourceVersion:"27430", FieldPath:""}, Reason:"LeaderElection", Message:"controller-runtime-example-controller-manager-65cd9fdd6c-5b58t_61618914-f476-4a88-86ce-4cfcbdf88ea4 became leader", Source:v1.EventSource{Component:"controller-runtime-example-controller-manager-65cd9fdd6c-5b58t_61618914-f476-4a88-86ce-4cfcbdf88ea4", Host:""}, FirstTimestamp:time.Date(2022, time.July, 7, 11, 39, 21, 286193426, time.Local), LastTimestamp:time.Date(2022, time.July, 7, 11, 39, 21, 286193426, time.Local), Count:1, Type:"Normal", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'forbidden: User "system:serviceaccount:controller-runtime-example-system:controller-runtime-example-controller-manager" cannot post path "/services/apiexport/root:plane:myapp/controller-runtime-example-data.my.domain/api/v1/namespaces/controller-runtime-example-system/events": Path not resolved to a valid virtual workspace' (will not retry!)

Instead of using the VirtualWorkspace URL, it should use the URL of the actual workspace that was provided as part of LeaderElectionConfig

The controller cannot work

Deployd controller follow the guide, seems it start up and start the watching:

I0926 08:31:12.492757       1 round_trippers.go:553] GET https://172.16.251.127:6443/services/apiexport/root:gold:dogs/controller-runtime-example-data.my.domain/clusters/%2A/api/v1/secrets?allowWatchBookmarks=true&resourceVersion=105000&timeoutSeconds=460&watch=true 200 OK in 3 milliseconds
I0926 08:31:12.492971       1 round_trippers.go:553] GET https://172.16.251.127:6443/services/apiexport/root:gold:dogs/controller-runtime-example-data.my.domain/clusters/%2A/apis/data.my.domain/v1alpha1/widgets?allowWatchBookmarks=true&resourceVersion=104997&timeoutSeconds=495&watch=true 200 OK in 3 milliseconds
I0926 08:31:12.494016       1 round_trippers.go:553] GET https://172.16.251.127:6443/services/apiexport/root:gold:dogs/controller-runtime-example-data.my.domain/clusters/%2A/api/v1/configmaps?allowWatchBookmarks=true&resourceVersion=104996&timeoutSeconds=381&watch=true 200 OK in 4 milliseconds

But the the reconcile do nothing event if I create cm or widget:

[root@experimental experimental]# kubectl get widget -A
NAMESPACE   NAME            AGE
default     widget-sample   11m
test        widget-sample   22m

Even more I query by the curl, still get nothing:

[root@experimental controller-runtime-example]# kubectl get --raw https://172.16.251.127:6443/services/apiexport/root:gold:dogs/controller-runtime-example-data.my.domain/clusters/\*/apis/data.my.domain/v1alpha1/namespaces/default/widgets
{"apiVersion":"data.my.domain/v1alpha1","items":[],"kind":"WidgetList","metadata":{"continue":"","resourceVersion":"105245"}}

am I missing something?

Raise exception: failed to wait for widget caches to sync: timed out waiting for cache to be synced

I deployed the controller to my virtual workspace: root:gold:dogs , it successed to sync to a KinD physics cluster.
but the pod keep CrashLoopBackOff, there are error raised in the logs:

290 1.6632054553562376e+09  ERROR   Could not wait for Cache to sync        {"controller": "configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "error": "failed to wait for configmap caches to sync:     timed out waiting for cache to be synced"}

300 1.6632054553565369e+09  ERROR   Could not wait for Cache to sync        {"controller": "widget", "controllerGroup": "data.my.domain", "controllerKind": "Widget", "error": "failed to wait for widget caches to     sync: timed out waiting for cache to be synced"}

Normally, it's caused by RBAC issue, I checked the clsuterrole, clusterrolebinding something, I think these are all as expectd, may I missing something?

APIExport names generated by make deploy and make install differ

When the user runs make deploythe name of the APIExport is controller-runtime-example-data.my.domain whereas the name generated by make install is data.my.domain.
This is confusing and can get messy when both methods are used in the same workspace.