Hello, bentoML team.
I'm recently trying to use bentoML and Yatai on our on-premise K8S cluster, but somehow it failed because we don't have LB service on our cluster. Is there any guide or workarounds to deploy Yatai on non-cloud K8S?
Thank you.
Followings are a few error messages.
2022-06-07T06:36:31.318Z INFO controller-runtime.manager.controller.deployment getting Deployment ... {"reconciler group": "component.yatai.ai", "reconciler kind": "Deployment", "name": "deployment", "namespace": ""}
2022-06-07T06:36:31.318Z INFO controller-runtime.manager.controller.deployment Deployment getting successfully {"reconciler group": "component.yatai.ai", "reconciler kind": "Deployment", "name": "deployment", "namespace": ""}
2022-06-07T06:36:31.318Z INFO controller-runtime.manager.controller.deployment creating namespace yatai-components ... {"reconciler group": "component.yatai.ai", "reconciler kind": "Deployment", "name": "deployment", "namespace": ""}
2022-06-07T06:36:31.318Z INFO controller-runtime.manager.controller.deployment namespace yatai-components creation successfully {"reconciler group": "component.yatai.ai", "reconciler kind": "Deployment", "name": "deployment", "namespace": ""}
2022-06-07T06:36:31.322Z INFO controller-runtime.manager.controller.deployment Installing CertManagerComponent ... {"reconciler group": "component.yatai.ai", "reconciler kind": "Deployment", "name": "deployment", "namespace": ""}
2022-06-07T06:36:31.322Z INFO controller-runtime.manager.controller.deployment crd certificates.cert-manager.io already exists, so skipping install cert-manager {"reconciler group": "component.yatai.ai", "reconciler kind": "Deployment", "name": "deployment", "namespace": ""}
2022-06-07T06:36:31.322Z INFO controller-runtime.manager.controller.deployment Installed CertManagerComponent successfully {"reconciler group": "component.yatai.ai", "reconciler kind": "Deployment", "name": "deployment", "namespace": ""}
2022-06-07T06:36:31.325Z INFO controller-runtime.manager.controller.deployment Installing YataiDeploymentOperatorComponent ... {"reconciler group": "component.yatai.ai", "reconciler kind": "Deployment", "name": "deployment", "namespace": ""}
2022-06-07T06:36:31.326Z INFO controller-runtime.manager.controller.deployment installing crd from file helm-charts/yatai-deployment-operator/crds/deployments.yaml ... {"reconciler group": "component.yatai.ai", "reconciler kind": "Deployment", "name": "deployment", "namespace": ""}
2022-06-07T06:36:31.361Z INFO controller-runtime.manager.controller.deployment crd bentodeployments.serving.yatai.ai updated successfully {"reconciler group": "component.yatai.ai", "reconciler kind": "Deployment", "name": "deployment", "namespace": ""}
2022-06-07T06:36:31.361Z INFO controller-runtime.manager.controller.deployment getting helm release yatai ... {"reconciler group": "component.yatai.ai", "reconciler kind": "Deployment", "name": "deployment", "namespace": ""}
2022-06-07T06:36:31.368Z INFO controller-runtime.manager.controller.deployment found helm release yatai, status: deployed {"reconciler group": "component.yatai.ai", "reconciler kind": "Deployment", "name": "deployment", "namespace": ""}
2022-06-07T06:36:31.369Z INFO controller-runtime.manager.controller.deployment Installed YataiDeploymentOperatorComponent successfully {"reconciler group": "component.yatai.ai", "reconciler kind": "Deployment", "name": "deployment", "namespace": ""}
2022-06-07T06:36:31.373Z INFO controller-runtime.manager.controller.deployment Installing CSIDriverImagePopulatorComponent ... {"reconciler group": "component.yatai.ai", "reconciler kind": "Deployment", "name": "deployment", "namespace": ""}
2022-06-07T06:36:31.373Z INFO controller-runtime.manager.controller.deployment getting helm release yatai-csi-driver-image-populator ... {"reconciler group": "component.yatai.ai", "reconciler kind": "Deployment", "name": "deployment", "namespace": ""}
2022-06-07T06:36:31.376Z INFO controller-runtime.manager.controller.deployment found helm release yatai-csi-driver-image-populator, status: deployed {"reconciler group": "component.yatai.ai", "reconciler kind": "Deployment", "name": "deployment", "namespace": ""}
2022-06-07T06:36:31.377Z INFO controller-runtime.manager.controller.deployment Installed CSIDriverImagePopulatorComponent successfully {"reconciler group": "component.yatai.ai", "reconciler kind": "Deployment", "name": "deployment", "namespace": ""}
2022-06-07T06:36:31.380Z INFO controller-runtime.manager.controller.deployment Installing IngressControllerComponent ... {"reconciler group": "component.yatai.ai", "reconciler kind": "Deployment", "name": "deployment", "namespace": ""}
2022-06-07T06:36:31.382Z INFO controller-runtime.manager.controller.deployment getting helm release yatai-ingress-controller ... {"reconciler group": "component.yatai.ai", "reconciler kind": "Deployment", "name": "deployment", "namespace": ""}
2022-06-07T06:36:31.390Z INFO controller-runtime.manager.controller.deployment found helm release yatai-ingress-controller, status: failed {"reconciler group": "component.yatai.ai", "reconciler kind": "Deployment", "name": "deployment", "namespace": ""}
2022-06-07T06:36:31.393Z INFO controller-runtime.manager.controller.deployment Installed IngressControllerComponent successfully {"reconciler group": "component.yatai.ai", "reconciler kind": "Deployment", "name": "deployment", "namespace": ""}
2022-06-07T06:36:31.396Z INFO controller-runtime.manager.controller.deployment Installing MinioComponent ... {"reconciler group": "component.yatai.ai", "reconciler kind": "Deployment", "name": "deployment", "namespace": ""}
2022-06-07T06:36:31.396Z INFO controller-runtime.manager.controller.deployment installing crd from file helm-charts/minio-operator/crds/minio.min.io_tenants.yaml ... {"reconciler group": "component.yatai.ai", "reconciler kind": "Deployment", "name": "deployment", "namespace": ""}
2022-06-07T06:36:31.627Z INFO controller-runtime.manager.controller.deployment crd tenants.minio.min.io updated successfully {"reconciler group": "component.yatai.ai", "reconciler kind": "Deployment", "name": "deployment", "namespace": ""}
2022-06-07T06:36:31.627Z INFO controller-runtime.manager.controller.deployment getting helm release yatai-minio ... {"reconciler group": "component.yatai.ai", "reconciler kind": "Deployment", "name": "deployment", "namespace": ""}
2022-06-07T06:36:31.639Z INFO controller-runtime.manager.controller.deployment found helm release yatai-minio, status: failed {"reconciler group": "component.yatai.ai", "reconciler kind": "Deployment", "name": "deployment", "namespace": ""}
2022-06-07T06:36:31.640Z INFO controller-runtime.manager.controller.deployment getting ingress-controller service external ip... {"reconciler group": "component.yatai.ai", "reconciler kind": "Deployment", "name": "deployment", "namespace": ""}
2022-06-07T06:36:31.640Z ERROR controller-runtime.manager.controller.deployment getting ingress-controller service external ip failed {"reconciler group": "component.yatai.ai", "reconciler kind": "Deployment", "name": "deployment", "namespace": "", "error": "the external ip of service yatai-ingress-controller-ingress-nginx-controller on namespace yatai-components is empty!", "errorVerbose": "the external ip of service yatai-ingress-controller-ingress-nginx-controller on namespace yatai-components is empty!\ngithub.com/bentoml/yatai-deployment-comp-operator/controllers.(*IngressControllerComponent).getIngressControllerServiceIps\n\t/workspace/controllers/deployment_controller.go:294\ngithub.com/bentoml/yatai-deployment-comp-operator/controllers.(*MinioComponent).Install\n\t/workspace/controllers/deployment_controller.go:510\ngithub.com/bentoml/yatai-deployment-comp-operator/controllers.(*DeploymentReconciler).doReconcile\n\t/workspace/controllers/deployment_controller.go:211\ngithub.com/bentoml/yatai-deployment-comp-operator/controllers.(*DeploymentReconciler).Reconcile\n\t/workspace/controllers/deployment_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:214\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371"}
github.com/bentoml/yatai-deployment-comp-operator/controllers.(*DeploymentReconciler).doReconcile
/workspace/controllers/deployment_controller.go:211
github.com/bentoml/yatai-deployment-comp-operator/controllers.(*DeploymentReconciler).Reconcile
/workspace/controllers/deployment_controller.go:126
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:298
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:214
2022-06-07T06:36:31.641Z ERROR controller-runtime.manager.controller.deployment Failed to install MinioComponent {"reconciler group": "component.yatai.ai", "reconciler kind": "Deployment", "name": "deployment", "namespace": "", "error": "the external ip of service yatai-ingress-controller-ingress-nginx-controller on namespace yatai-components is empty!", "errorVerbose": "the external ip of service yatai-ingress-controller-ingress-nginx-controller on namespace yatai-components is empty!\ngithub.com/bentoml/yatai-deployment-comp-operator/controllers.(*IngressControllerComponent).getIngressControllerServiceIps\n\t/workspace/controllers/deployment_controller.go:294\ngithub.com/bentoml/yatai-deployment-comp-operator/controllers.(*MinioComponent).Install\n\t/workspace/controllers/deployment_controller.go:510\ngithub.com/bentoml/yatai-deployment-comp-operator/controllers.(*DeploymentReconciler).doReconcile\n\t/workspace/controllers/deployment_controller.go:211\ngithub.com/bentoml/yatai-deployment-comp-operator/controllers.(*DeploymentReconciler).Reconcile\n\t/workspace/controllers/deployment_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:214\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371"}
github.com/bentoml/yatai-deployment-comp-operator/controllers.(*DeploymentReconciler).Reconcile
/workspace/controllers/deployment_controller.go:126
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:298
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:214
2022-06-07T06:36:31.649Z ERROR controller-runtime.manager.controller.deployment Reconciler error {"reconciler group": "component.yatai.ai", "reconciler kind": "Deployment", "name": "deployment", "namespace": "", "error": "the external ip of service yatai-ingress-controller-ingress-nginx-controller on namespace yatai-components is empty!", "errorVerbose": "the external ip of service yatai-ingress-controller-ingress-nginx-controller on namespace yatai-components is empty!\ngithub.com/bentoml/yatai-deployment-comp-operator/controllers.(*IngressControllerComponent).getIngressControllerServiceIps\n\t/workspace/controllers/deployment_controller.go:294\ngithub.com/bentoml/yatai-deployment-comp-operator/controllers.(*MinioComponent).Install\n\t/workspace/controllers/deployment_controller.go:510\ngithub.com/bentoml/yatai-deployment-comp-operator/controllers.(*DeploymentReconciler).doReconcile\n\t/workspace/controllers/deployment_controller.go:211\ngithub.com/bentoml/yatai-deployment-comp-operator/controllers.(*DeploymentReconciler).Reconcile\n\t/workspace/controllers/deployment_controller.go:126\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:214\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:214