kcp-dev / controller-runtime-example Goto Github PK
View Code? Open in Web Editor NEWLicense: Apache License 2.0
License: Apache License 2.0
with a virgin kcp cluster
$ make deploy
/tmp/controller-runtime-example/bin/controller-gen rbac:roleName=manager-role crd webhook paths="./..." output:crd:artifacts:config=config/crd/bases
cd config/manager && /tmp/controller-runtime-example/bin/kustomize edit set image controller=controller:latest
/tmp/controller-runtime-example/bin/kustomize build config/default | kubectl apply -f -
namespace/controller-runtime-example-system created
serviceaccount/controller-runtime-example-controller-manager created
role.rbac.authorization.k8s.io/controller-runtime-example-leader-election-role created
clusterrole.rbac.authorization.k8s.io/controller-runtime-example-kcp-manager-role created
clusterrole.rbac.authorization.k8s.io/controller-runtime-example-manager-role created
clusterrole.rbac.authorization.k8s.io/controller-runtime-example-metrics-reader created
clusterrole.rbac.authorization.k8s.io/controller-runtime-example-proxy-role created
rolebinding.rbac.authorization.k8s.io/controller-runtime-example-leader-election-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/controller-runtime-example-kcp-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/controller-runtime-example-manager-rolebinding created
clusterrolebinding.rbac.authorization.k8s.io/controller-runtime-example-proxy-rolebinding created
configmap/controller-runtime-example-manager-config created
apiexport.apis.kcp.dev/controller-runtime-example-data.my.domain created
apiresourceschema.apis.kcp.dev/controller-runtime-example-today.widgets.data.my.domain created
unable to recognize "STDIN": no matches for kind "Service" in version "v1"
unable to recognize "STDIN": no matches for kind "Deployment" in version "apps/v1"
make: *** [Makefile:112: deploy] Error 1
When compute is being added the Deployment API is added per default by the syncer. This is not the case of the Service API. This is not required by the core functionalities of the controller but referenced in the manifests for monitoring, auth-proxy and webhook. Of the three only auth-proxy is activated by default. As Robin mentioned this could get commented out.
Possibly the readme or the kustomization.yaml could be enhanced for making the user aware of it.
hi,
When I use latest https://github.com/kcp-dev/controller-runtime, the controller runtime client List is fails with error "Index with name cluster does not exist".
Steps to reproduce:
point to latest kcp-dev/controller-runtime, make change to compile the code and test the controller-runtime-example. you can use this branch to test https://github.com/prembhaskal/kcp-dev-controller-runtime-example/tree/client-list-issue
% export KUBECONFIG=/tmp/kcp.kubeconfig
% kubectl ws create prem-org --type organization
% kubectl ws prem-org
% cat /tmp/data-api-binding.yaml
apiVersion: apis.kcp.io/v1alpha1
kind: APIBinding
metadata:
name: data.my.domain
spec:
reference:
export:
name: data.my.domain
path: root
permissionClaims:
- resource: "secrets"
all: true
state: Accepted
- resource: "configmaps"
all: true
state: Accepted
- resource: "namespaces"
all: true
state: Accepted
% kubectl apply -f /tmp/data-api-binding.yaml
% # create configmap with needed labels and data, check the output of the controller and see if anything wierd happens there.
% cat /tmp/test-cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: syncer-config
labels:
name: syncer-config
data:
agent-target-cluster: "root:045aa3c9-017c-4824-a410-561b1e2ddef3:soumik-demo:cg-1"
agent-target-name: "downstream"
agent-target-namespace: "default"
upstream-context: base
secretData: syncer-config
% kubectl apply -f /tmp/test-cm.yaml
error seen
2024-01-29T23:23:54+05:30 INFO Update: updated configMap {"controller": "configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"syncer-config","namespace":"default"}, "namespace": "default", "name": "syncer-config", "reconcileID": "38ae15cb-9d1c-433d-a3d6-645970009745", "cluster": "29qso51vovx9kz1x"}
2024-01-29T23:23:54+05:30 INFO Get: retrieved configMap {"controller": "configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"syncer-config","namespace":"default"}, "namespace": "default", "name": "syncer-config", "reconcileID": "2b6493be-d719-4b86-9870-2ee1aa72fcea", "cluster": "29qso51vovx9kz1x"}
2024-01-29T23:23:54+05:30 ERROR unable to list configmaps {"controller": "configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "ConfigMap": {"name":"syncer-config","namespace":"default"}, "namespace": "default", "name": "syncer-config", "reconcileID": "2b6493be-d719-4b86-9870-2ee1aa72fcea", "cluster": "29qso51vovx9kz1x", "error": "Index with name cluster does not exist"}
github.com/kcp-dev/controller-runtime-example/controllers.(*ConfigMapReconciler).Reconcile
/Users/bpremkumar/code/github.com/prembhaskal/kcp-dev-controller-runtime-example/controllers/configmap_controller.go:87
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/Users/bpremkumar/go/pkg/mod/github.com/kcp-dev/[email protected]/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/Users/bpremkumar/go/pkg/mod/github.com/kcp-dev/[email protected]/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/Users/bpremkumar/go/pkg/mod/github.com/kcp-dev/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/Users/bpremkumar/go/pkg/mod/github.com/kcp-dev/[email protected]/pkg/internal/controller/controller.go:227
When the leader election is enabled:
restConfig := ctrl.GetConfigOrDie()
...
mgr, err := kcp.NewClusterAwareManager(cfg, ctrl.Options{
Scheme: scheme,
MetricsBindAddress: metricsAddr,
Port: 9443,
HealthProbeBindAddress: probeAddr,
LeaderElection: true,
LeaderElectionConfig: restConfig,
LeaderElectionID: "68a0532d.my.domain",
})
(NOTE: I specify the original rest config so it doesn't use the VirtualWorkspace URL, otherwise, it would fail because the client used in leader election doesn't use the right round tripper.)
then the leader election code tries to create an event in the VirtualWorkspace URL, but it fails because it's using a wrong path (does not contain the expected /clusters/* section)
I0707 11:39:21.287347 1 round_trippers.go:553] POST https://192.168.1.133:6443/services/apiexport/root:plane:myapp/controller-runtime-example-data.my.domain/api/v1/namespaces/controller-runtime-example-system/events 403 Forbidden in 0 milliseconds
E0707 11:39:21.287479 1 event.go:267] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"68a0532d.my.domain.16ff8920aa62aeb2", GenerateName:"", Namespace:"controller-runtime-example-system", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"ConfigMap", Namespace:"controller-runtime-example-system", Name:"68a0532d.my.domain", UID:"bc0f0a11-21c8-4a6a-b632-6d5abe3a7532", APIVersion:"v1", ResourceVersion:"27429", FieldPath:""}, Reason:"LeaderElection", Message:"controller-runtime-example-controller-manager-65cd9fdd6c-5b58t_61618914-f476-4a88-86ce-4cfcbdf88ea4 became leader", Source:v1.EventSource{Component:"controller-runtime-example-controller-manager-65cd9fdd6c-5b58t_61618914-f476-4a88-86ce-4cfcbdf88ea4", Host:""}, FirstTimestamp:time.Date(2022, time.July, 7, 11, 39, 21, 286184626, time.Local), LastTimestamp:time.Date(2022, time.July, 7, 11, 39, 21, 286184626, time.Local), Count:1, Type:"Normal", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'forbidden: User "system:serviceaccount:controller-runtime-example-system:controller-runtime-example-controller-manager" cannot post path "/services/apiexport/root:plane:myapp/controller-runtime-example-data.my.domain/api/v1/namespaces/controller-runtime-example-system/events": Path not resolved to a valid virtual workspace' (will not retry!)
I0707 11:39:21.289195 1 round_trippers.go:553] POST https://192.168.1.133:6443/services/apiexport/root:plane:myapp/controller-runtime-example-data.my.domain/api/v1/namespaces/controller-runtime-example-system/events 403 Forbidden in 1 milliseconds
E0707 11:39:21.289449 1 event.go:267] Server rejected event '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"68a0532d.my.domain.16ff8920aa62d112", GenerateName:"", Namespace:"controller-runtime-example-system", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Lease", Namespace:"controller-runtime-example-system", Name:"68a0532d.my.domain", UID:"b5d779ea-914b-41e5-aa3a-71ff06ee5349", APIVersion:"coordination.k8s.io/v1", ResourceVersion:"27430", FieldPath:""}, Reason:"LeaderElection", Message:"controller-runtime-example-controller-manager-65cd9fdd6c-5b58t_61618914-f476-4a88-86ce-4cfcbdf88ea4 became leader", Source:v1.EventSource{Component:"controller-runtime-example-controller-manager-65cd9fdd6c-5b58t_61618914-f476-4a88-86ce-4cfcbdf88ea4", Host:""}, FirstTimestamp:time.Date(2022, time.July, 7, 11, 39, 21, 286193426, time.Local), LastTimestamp:time.Date(2022, time.July, 7, 11, 39, 21, 286193426, time.Local), Count:1, Type:"Normal", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'forbidden: User "system:serviceaccount:controller-runtime-example-system:controller-runtime-example-controller-manager" cannot post path "/services/apiexport/root:plane:myapp/controller-runtime-example-data.my.domain/api/v1/namespaces/controller-runtime-example-system/events": Path not resolved to a valid virtual workspace' (will not retry!)
Instead of using the VirtualWorkspace URL, it should use the URL of the actual workspace that was provided as part of LeaderElectionConfig
The README indicates:
make docker-build docker-push IMG=<some-registry>/controller-runtime-example:tag
but it should be
make docker-build docker-push REGISTRY=<some-registry> IMG=controller-runtime-example:tag
because the makefile is using a default REGISTRY ?= localhost
and is using ${REGISTRY}/${IMG}
Same for other commands using IMG
I deployed the controller
to my virtual workspace: root:gold:dogs
, it successed to sync to a KinD physics cluster.
but the pod keep CrashLoopBackOff
, there are error raised in the logs:
290 1.6632054553562376e+09 ERROR Could not wait for Cache to sync {"controller": "configmap", "controllerGroup": "", "controllerKind": "ConfigMap", "error": "failed to wait for configmap caches to sync: timed out waiting for cache to be synced"}
300 1.6632054553565369e+09 ERROR Could not wait for Cache to sync {"controller": "widget", "controllerGroup": "data.my.domain", "controllerKind": "Widget", "error": "failed to wait for widget caches to sync: timed out waiting for cache to be synced"}
Normally, it's caused by RBAC issue, I checked the clsuterrole
, clusterrolebinding
something, I think these are all as expectd, may I missing something?
When the user runs make deploy
the name of the APIExport is controller-runtime-example-data.my.domain
whereas the name generated by make install
is data.my.domain
.
This is confusing and can get messy when both methods are used in the same workspace.
Deployd controller follow the guide, seems it start up and start the watching:
I0926 08:31:12.492757 1 round_trippers.go:553] GET https://172.16.251.127:6443/services/apiexport/root:gold:dogs/controller-runtime-example-data.my.domain/clusters/%2A/api/v1/secrets?allowWatchBookmarks=true&resourceVersion=105000&timeoutSeconds=460&watch=true 200 OK in 3 milliseconds
I0926 08:31:12.492971 1 round_trippers.go:553] GET https://172.16.251.127:6443/services/apiexport/root:gold:dogs/controller-runtime-example-data.my.domain/clusters/%2A/apis/data.my.domain/v1alpha1/widgets?allowWatchBookmarks=true&resourceVersion=104997&timeoutSeconds=495&watch=true 200 OK in 3 milliseconds
I0926 08:31:12.494016 1 round_trippers.go:553] GET https://172.16.251.127:6443/services/apiexport/root:gold:dogs/controller-runtime-example-data.my.domain/clusters/%2A/api/v1/configmaps?allowWatchBookmarks=true&resourceVersion=104996&timeoutSeconds=381&watch=true 200 OK in 4 milliseconds
But the the reconcile do nothing event if I create cm
or widget
:
[root@experimental experimental]# kubectl get widget -A
NAMESPACE NAME AGE
default widget-sample 11m
test widget-sample 22m
Even more I query by the curl, still get nothing:
[root@experimental controller-runtime-example]# kubectl get --raw https://172.16.251.127:6443/services/apiexport/root:gold:dogs/controller-runtime-example-data.my.domain/clusters/\*/apis/data.my.domain/v1alpha1/namespaces/default/widgets
{"apiVersion":"data.my.domain/v1alpha1","items":[],"kind":"WidgetList","metadata":{"continue":"","resourceVersion":"105245"}}
am I missing something?
It is a know issue and not directly related to the example that at least one APIBinding has first to be created for a controller to be able to watch a resource type through a virtual workspace.
As I have seen a couple of people loosing time in the troubleshooting of it in the context of simply running the controller runtime example I am proposing:
main of controller-runtime-example
against main KCP, as the API group was renamed, we should probably pin to a commit that has this change + change manifests
/kind bug
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.