Comments (6)
teoincontatto
commented on June 24, 2024
After some retries the operator Pod is able to start since the Secret for webhook are created (somehow) by the olm operator. Now I am facing another issue that is an extra service account (stackgres-restapi) that OLM created for the previous version of the operator get removed.
Can someone guide me to find out what is the root cause of that? I think the olm operator error about "no owner roles found" have something to do. From the code there seems to be multiple reasons why that error is shown and debugging just make me think this method contains some hints about the actual check that is failing: https://github.com/operator-framework/operator-lifecycle-manager/blob/master/pkg/controller/operators/olm/requirements.go#L260
from operator-lifecycle-manager.
teoincontatto
commented on June 24, 2024
After rebuilding olm with trace level I was able to found that the extra serviceaccount was indeed the cause of the error "no owner roles found". I am not sure why it get removed but seems that previous version of the operator was creating it since not aware that it was the task of olm operator to create it. Shouldn't olm re-create an extra service account if it does not exists? Is this a bug??
from operator-lifecycle-manager.
teoincontatto
commented on June 24, 2024
Here is the log with trace level I was talking about above:
time="2024-01-19T11:19:41Z" level=trace msg="popped queue" item="{update stackgres-65aa3592/stackgres.v1.8.0-snapshot}" queue-length=0
time="2024-01-19T11:19:41Z" level=debug msg="syncing CSV" csv=stackgres.v1.8.0-snapshot id=lvUDy namespace=stackgres-65aa3592 phase=Pending
time="2024-01-19T11:19:41Z" level=debug msg="annotations correct" annotationTargets= opgroupTargets=
time="2024-01-19T11:19:41Z" level=debug msg="csv in operatorgroup" csv=stackgres.v1.8.0-snapshot id=IaRPT namespace=stackgres-65aa3592 opgroup=stackgres phase=Pending
time="2024-01-19T11:19:41Z" level=debug msg="no intersecting operatorgroups provide the same apis" apis="SGBackup.v1.stackgres.io,SGCluster.v1.stackgres.io,SGConfig.v1.stackgres.io,SGDbOps.v1.stackgres.io,SGDistributedLogs.v1.stackgres.io,SGInstanceProfile.v1.stackgres.io,SGObjectStorage.v1beta1.stackgres.io,SGPoolingConfig.v1.stackgres.io,SGPostgresConfig.v1.stackgres.io,SGScript.v1.stackgres.io,SGShardedBackup.v1.stackgres.io,SGShardedCluster.v1alpha1.stackgres.io,SGShardedDbOps.v1.stackgres.io" csv=stackgres.v1.8.0-snapshot id=IaRPT namespace=stackgres-65aa3592 phase=Pending
time="2024-01-19T11:19:41Z" level=debug msg="checking if csv is replacing an older version"
time="2024-01-19T11:19:42Z" level=trace msg="enqueuing resource event" event="{update stackgres-65aa3592/stackgres}"
time="2024-01-19T11:19:42Z" level=trace msg="popped queue" item="{update stackgres-65aa3592/stackgres}" queue-length=0
2024-01-19T11:19:42Z DEBUG controllers.operator reconciling operator {"request": {"name":"stackgres.stackgres-65aa3592"}}
time="2024-01-19T11:19:42Z" level=debug msg="subscription has changed, requeuing installed csv" csv=stackgres.v1.8.0-snapshot
2024-01-19T11:19:42Z DEBUG controllers.adoption reconciling subscription {"request": {"name":"stackgres","namespace":"stackgres-65aa3592"}}
2024-01-19T11:19:42Z DEBUG controllers.operator reconciling operator {"request": {"name":"stackgres.stackgres-65aa3592"}}
time="2024-01-19T11:19:42Z" level=debug msg="perm.ServiceAccountName: stackgres-operator"
time="2024-01-19T11:19:42Z" level=debug msg="perm.ServiceAccountName: stackgres-restapi"
time="2024-01-19T11:19:42Z" level=trace msg="appending permission status" key=stackgres-operator status="{ v1 ServiceAccount stackgres-operator Present [{rbac.authorization.k8s.io v1 PolicyRule Satisfied cluster rule:{\"verbs\":[\"get\",\"list\",\"watch\",\"update\",\"create\",\"delete\",\"patch\"],\"apiGroups\":[\"\",\"apps\",\"extensions\",\"rbac.authorization.k8s.io\",\"batch\"],\"resources\":[\"pods\",\"pods/exec\",\"pods/log\",\"services\",\"endpoints\",\"endpoints/restricted\",\"persistentvolumeclaims\",\"configmaps\",\"secrets\",\"deployments\",\"statefulsets\",\"serviceaccounts\",\"namespaces\",\"roles\",\"rolebindings\",\"events\",\"cronjobs\",\"jobs\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied cluster rule:{\"verbs\":[\"get\",\"list\"],\"apiGroups\":[\"storage.k8s.io\"],\"resources\":[\"storageclasses\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied cluster rule:{\"verbs\":[\"create\",\"watch\",\"list\",\"get\",\"update\",\"patch\",\"delete\"],\"apiGroups\":[\"stackgres.io\"],\"resources\":[\"sgclusters\",\"sgpgconfigs\",\"sginstanceprofiles\",\"sgpoolconfigs\",\"sgbackupconfigs\",\"sgbackups\",\"sgdistributedlogs\",\"sgdbops\",\"sgobjectstorages\",\"sgscripts\",\"sgshardedclusters\",\"sgshardedbackups\",\"sgshardeddbops\",\"sgconfigs\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied cluster rule:{\"verbs\":[\"update\"],\"apiGroups\":[\"stackgres.io\"],\"resources\":[\"sgconfigs/status\",\"sgclusters/status\",\"sgdistributedlogs/status\",\"sgclusters/finalizers\",\"sgpgconfigs/finalizers\",\"sginstanceprofiles/finalizers\",\"sgpoolconfigs/finalizers\",\"sgbackupconfigs/finalizers\",\"sgbackups/finalizers\",\"sgdistributedlogs/finalizers\",\"sgdbops/finalizers\",\"sgobjectstorages/finalizers\",\"sgscripts/finalizers\",\"sgshardedclusters/finalizers\",\"sgshardedbackups/finalizers\",\"sgshardeddbops/finalizers\",\"sgconfigs/finalizers\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied cluster rule:{\"verbs\":[\"update\"],\"apiGroups\":[\"\",\"apps\",\"batch\"],\"resources\":[\"statefulsets/finalizers\",\"persistentvolumeclaims/finalizers\",\"deployments/finalizers\",\"services/finalizers\",\"endpoints/finalizers\",\"cronjobs/finalizers\",\"jobs/finalizers\",\"pods/finalizers\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied cluster rule:{\"verbs\":[\"get\"],\"apiGroups\":[\"apiextensions.k8s.io\"],\"resources\":[\"customresourcedefinitions\"],\"resourceNames\":[\"sgconfigs.stackgres.io\",\"sgclusters.stackgres.io\",\"sginstanceprofiles.stackgres.io\",\"sgpgconfigs.stackgres.io\",\"sgpoolconfigs.stackgres.io\",\"sgbackups.stackgres.io\",\"sgbackupconfigs.stackgres.io\",\"sgobjectstorages.stackgres.io\",\"sgdbops.stackgres.io\",\"sgdistributedlogs.stackgres.io\",\"sgshardedclusters.stackgres.io\",\"sgshardedbackups.stackgres.io\",\"sgshardeddbops.stackgres.io\",\"sgscripts.stackgres.io\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied cluster rule:{\"verbs\":[\"list\"],\"apiGroups\":[\"apiextensions.k8s.io\"],\"resources\":[\"customresourcedefinitions\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied cluster rule:{\"verbs\":[\"list\",\"get\",\"watch\",\"create\"],\"apiGroups\":[\"snapshot.storage.k8s.io\"],\"resources\":[\"volumesnapshots\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied cluster rule:{\"verbs\":[\"get\"],\"apiGroups\":[\"apiextensions.k8s.io\"],\"resources\":[\"customresourcedefinitions\"],\"resourceNames\":[\"prometheuses.monitoring.coreos.com\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied cluster rule:{\"verbs\":[\"list\"],\"apiGroups\":[\"apiextensions.k8s.io\"],\"resources\":[\"customresourcedefinitions\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied cluster rule:{\"verbs\":[\"list\",\"get\",\"create\",\"delete\",\"update\",\"patch\"],\"apiGroups\":[\"monitoring.coreos.com\"],\"resources\":[\"servicemonitors\",\"podmonitors\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied cluster rule:{\"verbs\":[\"list\",\"get\"],\"apiGroups\":[\"monitoring.coreos.com\"],\"resources\":[\"prometheus\",\"prometheuses\",\"podmonitors\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied cluster rule:{\"verbs\":[\"list\",\"get\"],\"apiGroups\":[\"operators.coreos.com\"],\"resources\":[\"operators\"]}}]}"
time="2024-01-19T11:19:42Z" level=trace msg="appending permission status" key=stackgres-restapi status="{ v1 ServiceAccount stackgres-restapi NotPresent Service account does not exist []}"
time="2024-01-19T11:19:42Z" level=debug msg="permissions/requirements not met" minKubeMet=true permMet=false reqMet=true
time="2024-01-19T11:19:42Z" level=debug msg="checking if csv is replacing an older version"
time="2024-01-19T11:19:42Z" level=info msg="requirements were not met" csv=stackgres.v1.8.0-snapshot id=IaRPT namespace=stackgres-65aa3592 phase=Pending
time="2024-01-19T11:19:42Z" level=debug msg="opgroup is global" csv=stackgres.v1.8.0-snapshot opgroup=stackgres
time="2024-01-19T11:19:42Z" level=debug msg="perm.ServiceAccountName: stackgres-operator"
time="2024-01-19T11:19:42Z" level=debug msg="perm.ServiceAccountName: stackgres-restapi"
time="2024-01-19T11:19:42Z" level=trace msg="appending permission status" key=stackgres-operator status="{ v1 ServiceAccount stackgres-operator Present [{rbac.authorization.k8s.io v1 PolicyRule Satisfied cluster rule:{\"verbs\":[\"get\",\"list\",\"watch\",\"update\",\"create\",\"delete\",\"patch\"],\"apiGroups\":[\"\",\"apps\",\"extensions\",\"rbac.authorization.k8s.io\",\"batch\"],\"resources\":[\"pods\",\"pods/exec\",\"pods/log\",\"services\",\"endpoints\",\"endpoints/restricted\",\"persistentvolumeclaims\",\"configmaps\",\"secrets\",\"deployments\",\"statefulsets\",\"serviceaccounts\",\"namespaces\",\"roles\",\"rolebindings\",\"events\",\"cronjobs\",\"jobs\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied cluster rule:{\"verbs\":[\"get\",\"list\"],\"apiGroups\":[\"storage.k8s.io\"],\"resources\":[\"storageclasses\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied cluster rule:{\"verbs\":[\"create\",\"watch\",\"list\",\"get\",\"update\",\"patch\",\"delete\"],\"apiGroups\":[\"stackgres.io\"],\"resources\":[\"sgclusters\",\"sgpgconfigs\",\"sginstanceprofiles\",\"sgpoolconfigs\",\"sgbackupconfigs\",\"sgbackups\",\"sgdistributedlogs\",\"sgdbops\",\"sgobjectstorages\",\"sgscripts\",\"sgshardedclusters\",\"sgshardedbackups\",\"sgshardeddbops\",\"sgconfigs\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied cluster rule:{\"verbs\":[\"update\"],\"apiGroups\":[\"stackgres.io\"],\"resources\":[\"sgconfigs/status\",\"sgclusters/status\",\"sgdistributedlogs/status\",\"sgclusters/finalizers\",\"sgpgconfigs/finalizers\",\"sginstanceprofiles/finalizers\",\"sgpoolconfigs/finalizers\",\"sgbackupconfigs/finalizers\",\"sgbackups/finalizers\",\"sgdistributedlogs/finalizers\",\"sgdbops/finalizers\",\"sgobjectstorages/finalizers\",\"sgscripts/finalizers\",\"sgshardedclusters/finalizers\",\"sgshardedbackups/finalizers\",\"sgshardeddbops/finalizers\",\"sgconfigs/finalizers\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied cluster rule:{\"verbs\":[\"update\"],\"apiGroups\":[\"\",\"apps\",\"batch\"],\"resources\":[\"statefulsets/finalizers\",\"persistentvolumeclaims/finalizers\",\"deployments/finalizers\",\"services/finalizers\",\"endpoints/finalizers\",\"cronjobs/finalizers\",\"jobs/finalizers\",\"pods/finalizers\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied cluster rule:{\"verbs\":[\"get\"],\"apiGroups\":[\"apiextensions.k8s.io\"],\"resources\":[\"customresourcedefinitions\"],\"resourceNames\":[\"sgconfigs.stackgres.io\",\"sgclusters.stackgres.io\",\"sginstanceprofiles.stackgres.io\",\"sgpgconfigs.stackgres.io\",\"sgpoolconfigs.stackgres.io\",\"sgbackups.stackgres.io\",\"sgbackupconfigs.stackgres.io\",\"sgobjectstorages.stackgres.io\",\"sgdbops.stackgres.io\",\"sgdistributedlogs.stackgres.io\",\"sgshardedclusters.stackgres.io\",\"sgshardedbackups.stackgres.io\",\"sgshardeddbops.stackgres.io\",\"sgscripts.stackgres.io\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied cluster rule:{\"verbs\":[\"list\"],\"apiGroups\":[\"apiextensions.k8s.io\"],\"resources\":[\"customresourcedefinitions\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied cluster rule:{\"verbs\":[\"list\",\"get\",\"watch\",\"create\"],\"apiGroups\":[\"snapshot.storage.k8s.io\"],\"resources\":[\"volumesnapshots\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied cluster rule:{\"verbs\":[\"get\"],\"apiGroups\":[\"apiextensions.k8s.io\"],\"resources\":[\"customresourcedefinitions\"],\"resourceNames\":[\"prometheuses.monitoring.coreos.com\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied cluster rule:{\"verbs\":[\"list\"],\"apiGroups\":[\"apiextensions.k8s.io\"],\"resources\":[\"customresourcedefinitions\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied cluster rule:{\"verbs\":[\"list\",\"get\",\"create\",\"delete\",\"update\",\"patch\"],\"apiGroups\":[\"monitoring.coreos.com\"],\"resources\":[\"servicemonitors\",\"podmonitors\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied cluster rule:{\"verbs\":[\"list\",\"get\"],\"apiGroups\":[\"monitoring.coreos.com\"],\"resources\":[\"prometheus\",\"prometheuses\",\"podmonitors\"]}} {rbac.authorization.k8s.io v1 PolicyRule Satisfied cluster rule:{\"verbs\":[\"list\",\"get\"],\"apiGroups\":[\"operators.coreos.com\"],\"resources\":[\"operators\"]}}]}"
time="2024-01-19T11:19:42Z" level=trace msg="appending permission status" key=stackgres-restapi status="{ v1 ServiceAccount stackgres-restapi NotPresent Service account does not exist []}"
time="2024-01-19T11:19:42Z" level=debug msg="lift roles/rolebindings to clusterroles/rolebindings" csv=stackgres.v1.8.0-snapshot opgroup=stackgres
time="2024-01-19T11:19:42Z" level=info msg="couldn't ensure RBAC in target namespaces" csv=stackgres.v1.8.0-snapshot error="no owned roles found" id=lvUDy namespace=stackgres-65aa3592 phase=Pending
time="2024-01-19T11:19:42Z" level=debug msg="done syncing CSV" csv=stackgres.v1.8.0-snapshot id=lvUDy namespace=stackgres-65aa3592 phase=Pending
time="2024-01-19T11:19:42Z" level=trace msg="requeuing with rate limiting" cache-key=stackgres-65aa3592/stackgres.v1.8.0-snapshot item="{update stackgres-65aa3592/stackgres.v1.8.0-snapshot}" requeues=2
E0119 11:19:42.543718 1 queueinformer_operator.go:319] sync {"update" "stackgres-65aa3592/stackgres.v1.8.0-snapshot"} failed: no owned roles found
Is there a way to set trace level with some olm command parameter or environment variable?
from operator-lifecycle-manager.
ddelnano
commented on June 24, 2024
@teoincontatto have you had any updates since your last message? I'm trying to upgrade the pixie operator to bundle OLM v0.26.0 and I'm seeing this same problem consistently with our next release candidate build.
from operator-lifecycle-manager.
teoincontatto
commented on June 24, 2024
After going back to this I was able to reproduce the issue locally with the same pipeline. I had to set up a Vagrantfile and a script in order to make it work (see attachments Vagrantfile.zip).
It turned out the problem was related to the caBundle field in webhooks that was set to a dummy value. I can not really recall why it was like that but something related to an old Kubernetes version restriction (maybe field was required before?!). The OLM operator was randomly setting the dummy value after changing it (maybe a bug?). Removing the value for caBundle field solved the issue but without reproducing it was impossible to tell what was happening (sometimes the problem didn't appear and the test passed).
from operator-lifecycle-manager.
ddelnano
commented on June 24, 2024
Thanks for providing that context. I was able to debug my problem and while the error message matched what was reported here, my issue was self inflicted and is resolved.
from operator-lifecycle-manager.
Related Issues (20)
- [Flake] End-to-end.[It] Subscription can reconcile InstallPlan status
- as long as there is a manual Subscription, other automatic operators cannot be automatically installed or upgraded HOT 1
- Add InstallPlan Finalizers to OLM
- OLM happily creates duplicate `PackageManifest` resources in the same namespace HOT 3
- Resolution error `constraints not satisfiable: no operator found from catalog` is not cleared after CatalogSource is made available again HOT 1
- OperatorGroup documentation with label selectors provides incorrect example format
- nil pointer panic when deleting a CSV HOT 4
- Subscription Conditions should be set to false, instead of being removed.
- Cannot render multiple bundles with same name in semver template
- Allow force recreate of controller-manager pods through CSV upgrade options and be smarter about upgrade monitoring
- CSV stuck in install loop with auth-delegator already exists error
- OLM generate bundle silently drops k8s resources existing in manifests HOT 5
- InstallPlan is missing expected OwnerReference to Subscription HOT 1
- I want to use my privet registry for all images
- The catalog source pod can not migrate when the node becomes NotReady HOT 2
- Subscription is stuck when installing same operator multiple times into different namespaces at different dates
- Improve Makefile to avoid unnecessarily shelling out multiple times for the same variables
- support cacheless catalogsources
- Deployment "olm/packageserver" never appears on install HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from operator-lifecycle-manager.