Comments (9)
phoracek
commented on July 23, 2024
Thanks for reporting this. Could you please attach HCO logs here?
from hyperconverged-cluster-operator.
DirectedSoul1
commented on July 23, 2024
shared it, Thanks
from hyperconverged-cluster-operator.
phoracek
commented on July 23, 2024
Log is full of:
{
"level": "error",
"ts": 1554220550.0038,
"logger": "kubebuilder.controller",
"msg": "Reconciler error",
"controller": "hyperconverged-controller",
"request": "kubevirt\/hyperconverged-cluster",
"error": "cdis.cdi.kubevirt.io \"cdi-hyperconverged-cluster\" already exists",
"stacktrace": "github.com/kubevirt/hyperconverged-cluster-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error
/home/rhallisey/src/github.com/kubevirt/hyperconverged-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128github.com/kubevirt/hyperconverged-cluster-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/home/rhallisey/src/github.com/kubevirt/hyperconverged-cluster-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217
github.com/kubevirt/hyperconverged-cluster-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
/home/rhallisey/src/github.com/kubevirt/hyperconverged-cluster-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158
github.com/kubevirt/hyperconverged-cluster-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
/home/rhallisey/src/github.com/kubevirt/hyperconverged-cluster-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
github.com/kubevirt/hyperconverged-cluster-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil
/home/rhallisey/src/github.com/kubevirt/hyperconverged-cluster-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
github.com/kubevirt/hyperconverged-cluster-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until
/home/rhallisey/src/github.com/kubevirt/hyperconverged-cluster-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"
}There is no mention of "Network" in the log. My guess is, that every Reconcile run, we fail and exit on CDI, so handling never gets to the network part.
@DirectedSoul1 was your environment clean or have you deployed HCO there before?
It fails with "CDI already exists", but we are checking for existence of the object on https://github.com/kubevirt/hyperconverged-cluster-operator/blob/master/pkg/controller/hyperconverged/hyperconverged_controller.go#L196.
It also seems like KubeVirt updates are followed with "Skip reconcile: %s CR already exists" (%s is due to #17). For CDI however, it is followed with "Creating a new %s CR".
@rthallisey could anyone from CDI take a look at that please? I don't want to point fingers at anyone, but it seems like the problem is there.
from hyperconverged-cluster-operator.
phoracek
commented on July 23, 2024
Beggining of the log, the rest is just repeating the error:
{"level":"info","ts":1554241717.8002312,"logger":"cmd","msg":"Go Version: go1.10.8"}
{"level":"info","ts":1554241717.8002756,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1554241717.800285,"logger":"cmd","msg":"Version of operator-sdk: v0.5.0+git"}
{"level":"info","ts":1554241717.8042269,"logger":"leader","msg":"Trying to become the leader."}
{"level":"info","ts":1554241717.9016047,"logger":"leader","msg":"Found existing lock with my name. I was likely restarted."}
{"level":"info","ts":1554241717.9016297,"logger":"leader","msg":"Continuing as the leader."}
{"level":"info","ts":1554241717.9788105,"logger":"cmd","msg":"Registering Components."}
{"level":"info","ts":1554241717.9790463,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"hyperconverged-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1554241717.9791713,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"hyperconverged-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1554241717.9792376,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"hyperconverged-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1554241717.9792972,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"hyperconverged-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1554241718.1422763,"logger":"metrics","msg":"Metrics Service object created","Service.Name":"hyperconverged-cluster-operator","Service.Namespace":"kubevirt"}
{"level":"info","ts":1554241718.142457,"logger":"cmd","msg":"Starting the Cmd."}
{"level":"info","ts":1554241718.2556615,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"hyperconverged-controller"}
{"level":"info","ts":1554241718.4515696,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"hyperconverged-controller","worker count":1}
{"level":"info","ts":1554241718.4517422,"logger":"controller_hyperconverged","msg":"Reconciling HyperConverged operator","Request.Namespace":"kubevirt","Request.Name":"hyperconverged-cluster"}
{"level":"info","ts":1554241718.4517922,"logger":"controller_hyperconverged","msg":"HCO CR contains KubeVirt Image Pull Policy","imagePullPolicy":"IfNotPresent"}
{"level":"info","ts":1554241718.4518185,"logger":"controller_hyperconverged","msg":"HCO CR contains CDI Image Pull Policy","imagePullPolicy":"IfNotPresent"}
{"level":"info","ts":1554241718.4518635,"logger":"controller_hyperconverged","msg":"Creating a new CR","Kind":"KubeVirt"}
{"level":"info","ts":1554241718.458969,"logger":"controller_hyperconverged","msg":"Creating a new CR","Kind":"CDI"}
{"level":"info","ts":1554241718.4656014,"logger":"controller_hyperconverged","msg":"Creating a new CR","Kind":"NetworkAddonsConfig"}
{"level":"info","ts":1554241718.4782228,"logger":"controller_hyperconverged","msg":"Reconciling HyperConverged operator","Request.Namespace":"","Request.Name":"hyperconverged-cluster"}
{"level":"info","ts":1554241718.4783592,"logger":"controller_hyperconverged","msg":"Reconciling HyperConverged operator","Request.Namespace":"kubevirt","Request.Name":"hyperconverged-cluster"}
{"level":"info","ts":1554241718.4784544,"logger":"controller_hyperconverged","msg":"HCO CR contains KubeVirt Image Pull Policy","imagePullPolicy":"IfNotPresent"}
{"level":"info","ts":1554241718.4785082,"logger":"controller_hyperconverged","msg":"HCO CR contains CDI Image Pull Policy","imagePullPolicy":"IfNotPresent"}
{"level":"info","ts":1554241718.478559,"logger":"controller_hyperconverged","msg":"Skip reconcile: CR already exists","Kind":"KubeVirt"}
{"level":"info","ts":1554241718.4786012,"logger":"controller_hyperconverged","msg":"Creating a new CR","Kind":"CDI"}
{"level":"error","ts":1554241718.4927685,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"hyperconverged-controller","request":"kubevirt/hyperconverged-cluster","error":"cdis.cdi.kubevirt.io \"cdi-hyperconverged-cluster\" already exists","stacktrace":"github.com/kubevirt/hyperconverged-cluster-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/phoracek/Code/go/src/github.com/kubevirt/hyperconverged-cluster-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/kubevirt/hyperconverged-cluster-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/phoracek/Code/go/src/github.com/kubevirt/hyperconverged-cluster-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/kubevirt/hyperconverged-cluster-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/home/phoracek/Code/go/src/github.com/kubevirt/hyperconverged-cluster-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/kubevirt/hyperconverged-cluster-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/home/phoracek/Code/go/src/github.com/kubevirt/hyperconverged-cluster-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/kubevirt/hyperconverged-cluster-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/phoracek/Code/go/src/github.com/kubevirt/hyperconverged-cluster-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/kubevirt/hyperconverged-cluster-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/home/phoracek/Code/go/src/github.com/kubevirt/hyperconverged-cluster-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
It is also worth noting, that HCO failed during its first start (other operators were not deployed yet). Vanilla operators cannot dynamically register new Kinds after they are created. You might need to fix it with this: https://github.com/kubevirt/cluster-network-addons-operator/blob/master/pkg/util/k8s/dynamicrestmapper.go (edit: Tried to fix it, dynamic REST mapper didn't help)
from hyperconverged-cluster-operator.
rthallisey
commented on July 23, 2024
I think this issue highlights well two stability issues:
- HCO fails on the first missing CRD
- We don't account for CRs already created
Issue 1), I think we can be more forgiving of missing CRDs and raise it in the logs, on the HCO cr, and in OLM. Issue 2), should be a basic function of the object reconciliation.
from hyperconverged-cluster-operator.
rthallisey
commented on July 23, 2024
What version of k8s with minikube?
from hyperconverged-cluster-operator.
phoracek
commented on July 23, 2024
I tested in on kubevirtci cluster. Once it passed, but then it failed with the same error as for @DirectedSoul1
oc v3.9.0
kubernetes v1.9.1+cbc5b49
features: Basic-Auth GSSAPI Kerberos SPNEGO
Server http://127.0.0.1:32773
openshift v3.11.0+8de5c34-71
kubernetes v1.11.0+d4cacc0
from hyperconverged-cluster-operator.
DirectedSoul1
commented on July 23, 2024
What version of k8s with minikube?
root@shegde🎩minikube version
minikube version: v0.35.0
root@shegde🎩kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:08:12Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-02-28T13:30:26Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
from hyperconverged-cluster-operator.
rthallisey
commented on July 23, 2024
Replacing this issue with two separate issues. #21 and #20
from hyperconverged-cluster-operator.
Related Issues (20)
- Helm chart for deployment HOT 7
- Components deployed by HCO 1.4.1 on OKD 4.8 are still reported as using k8s 1.22 deprecated APIs HOT 2
- Consider switching to kubevirt.io/api from kubevirt.io/client-go HOT 2
- Update configure-custom-golden-images doc section
- ssp-operator Pod fails to start: No matches for kind infrastructure HOT 2
- Future Release Branches Frozen For Merging | branch:release-4.13 HOT 4
- Make Sidecar feature gate configurable HOT 3
- Add API to set CDI's DataVolumeTTLSeconds field HOT 4
- Do not set the LiveMigration feature gate in KV
- HCO deployment via OLM on plain k8s is failing due to SSP operator HOT 1
- Bridge networking failed HOT 1
- Expose new APIs to adjust KubeVirt rate limiting HOT 6
- Unable to build the hyperconverged-cluster-operator HOT 2
- Resources Limits HOT 1
- SCC diff problem when running cluster-sync on top of HCO HOT 4
- Operator does not install because a file limit is reached HOT 3
- Auto-generated files creation date changes which leads to frequent rebases HOT 5
- Create KubeVirt rate limiting static profiles HOT 2
- [docs] tests.md is outdated HOT 4
- Unable to deploy HCO on CRC HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hyperconverged-cluster-operator.