At present, modifying starrocksBeSpec.config and starrocksFESpec.config merely updates the configmap without restarting the pod, thereby config changes fails to take effect.
The correct approach shall updating the configmap and restarting the pod to effectuate the updated configuration.

Version: 1.3.5

Problems
It seems that it tries to create a sa instead of accepting an existing sa.

Reproduce Steps:

1. create a sa to a namespace e.g. celostar
2. set these two fields with the same sa name in values.yaml : starrocksFESpec.serviceAccount starrocksFESpec.serviceAccount
3. helm install with the values.yaml will get this error:

Error: INSTALLATION FAILED: rendered manifests contain a resource that already exists. Unable to continue with install: ServiceAccount "celostar" in namespace "celostar" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "starrocks"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "celostar"

k8s version: v1.24.8
An error occurred while adding custom resources
kubectl apply -f https://raw.githubusercontent.com/StarRocks/starrocks-kubernetes-operator/main/deploy/starrocks.com_starrocksclusters.yaml
error msg is
The CustomResourceDefinition "starrocksclusters.starrocks.com" is invalid: metadata.annotations: Too long: must have at most 262144 bytes

We want move our starrocks cluster into k8s, and need to implement fe,be,broker controller. Please let me know if you are already working on it.

need to differentiate builtin labels and user defined labels. Allow users add user-defined labels whatever they want, but only update services pod selector with builtin (operator determined fixed labels) so service selector will be always stable, won't be affected by user-defined labels. won't cause rolling upgrade issue.

StarRocks/starrocks#22768

Running the delete command in OpenShift gets me stuck and won't return cursor to me.

atwong@Alberts-MBP ~ % kubectl delete -f https://raw.githubusercontent.com/StarRocks/starrocks-kubernetes-operator/main/examples/starrocks/starrocks-fe-and-be.yaml
starrockscluster.starrocks.com "starrockscluster-sample" deleted

使用operator安装的时候，是否有方法配置starrocks的时区

We need these two fields be configurable via helm chart values.yaml

• imagePullSecrets: to pull image from private registry
• annotations: to assume a role from an AWS account

apiVersion: v1
kind: ServiceAccount
metadata:
  name: kube-starrocks
  namespace: celostar
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::xxx:role/yyy <=========
imagePullSecrets:
  - name: docker-ecr-pull <==============
secrets:
  - name: kube-starrocks-dockercfg-h5wzz

@shileifu @kevincai
Can you take a look of this issue?
Thank you!

When setting starrocksFESpec.feEnvVars in the values.yaml got this error.
e.g.

helm install starrocks . -n celostar --create-namespace \
  -f values-eu-dev-1-cs.yaml --render-subchart-notes
Error: INSTALLATION FAILED: 
unable to build kubernetes objects from release manifest: error validating "": 
error validating data: 
ValidationError(StarRocksCluster.spec.starRocksFeSpec): 
unknown field "feEnvVars" in com.starrocks.v1alpha1.StarRocksCluster.spec.starRocksFeSpec

Hi,
if we can add StarRocksBeSpec and StarRocksFeSpec for pod template support will more flexibility for the Operator deployment such as we can use some sidecar to collect StarRocks logs.

When setting storageSpec.name in the value.yaml file, it actually initialized it as the volume name instead of storageclassName in the eventual manifest.

values.yaml

    # fe storageSpec for persistent meta data.
    storageSpec:
      # the storageClass name for fe meta persistent volume. if name is empty used emptyDir.
      name: "gp2" <====================

generate statefulset manifest

  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      creationTimestamp: null
      name: gp2 <===================== mistakenly set it as the volume name
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 100Gi
      volumeMode: Filesystem
      storageClassName: gp2 <=================== should set spec.storageClassName

How did we get here. StarRocks/starrocks#22767

executed

oc create sa starrocks-sa
oc adm policy add-scc-to-user privileged -z starrocks-sa
oc set sa deploy starrocks-controller starrocks-sa

now I get this error in the starrocks-controller pod

E0428 23:37:25.428177       1 leaderelection.go:330] error retrieving resource lock starrocks/c6c79638.starrocks.com: leases.coordination.k8s.io "c6c79638.starrocks.com" is forbidden: User "system:serviceaccount:starrocks:starrocks-sa" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "starrocks"

The failure and recovery mechanism needs to be re-designed for the k8s environment for the following reasons:

1. StarRocks includes a clone process to address missing replicas in situations where the number of alive replicas is insufficient, such as when a BackEnd (BE) node fails. However, in a Kubernetes (k8s) environment, the failure mode of a BE node is different. Specifically, if a BE fails a k8s liveness probe check, which may be due to reasons like VM node failure, the BE will not be permanently dead. Instead, the k8s control plane will schedule the BE on a different VM node in the form of a stateful set pod.

2. Data tablets are not lost when a BE is detected as dead. This is because the data tablets associated with the BE are typically stored in a Persistent Volume that is backed by a separate block storage service, such as EBS or Azure Disk Storage. These block storage services have much higher durability than a commodity disk, and they do not necessarily fail when the BE fails. Consequently, when a BE stateful set pod is rescheduled to a different node, the data tablet will still be available via the Persistent Volume.

3. it is important to note that the end-to-end time that k8s takes to reschedule a new BE might be longer than the 5 minutes threshold that StarRocks uses to consider a BE as dead. Please check the timeline graph of how a pod is reschedule to a new node on node failure.
   

Ref:

• https://www.percona.com/blog/2021/01/20/drain-kubernetes-nodes-wisely/
• AWS EBS Availability & Durability

The setting of starrocksFESpec.beEnvVars doesn't taking effect

#Context
We need to use the load selector and toleration to deploy SRs to dedicated node group.

nodeSelector: {}

tolerations: []

affinity: {}

What isn't said in the README is what is the target Operator Capability Level for this operator. https://sdk.operatorframework.io/docs/overview/operator-capabilities/

env： Kubernetes v1.26.1
operator: starrocks/operator:latest
deployment: starrocks/fe-ubuntu:2.5.4 starrocks/be-ubuntu:2.5.4

fe work well , be can't be started

[Mon Apr 17 08:00:50 UTC 2023] Empty $CONFIGMAP_MOUNT_PATH env var, skip it!
[Mon Apr 17 08:00:50 UTC 2023] Add myself (starrockscluster-be-0.starrockscluster-be-search.open.svc.k8s.com:9050) into FE ...
ERROR 1064 (HY000) at line 1: Unexpected exception: Same backend already exists[starrockscluster-be-0.starrockscluster-be-search.open.svc.k8s.com:9050]
[Mon Apr 17 08:00:50 UTC 2023] run start_be.sh
/opt/starrocks/be/bin/start_backend.sh: line 185: 807 Illegal instruction (core dumped) ${START_BE_CMD} "$@" >> ${LOG_FILE} 2>&1 < /dev/null

@shileifu @kevincai

If I make a change to the resource specifications in the values.yml file, such as increasing the requested CPU from 2 to 3, without changing the number of replicas, upgrading the Helm release will not trigger an upgrade of the pod. However, if I increase the number of replicas from 3 to 4, all pods will receive the updated resource specifications.

Or else when you do the exercise, testing or restart pods, you'll lose the data.

Can we add the support of imagePullSecrets for pulling image from private repos.

We need this support for all services: fe/be/cn/operator.
On the Helm Chart value file, it could be a filed under image .e.g image.imagePullSecrets
https://github.com/StarRocks/helm-charts/blob/main/charts/kube-starrocks/values.yaml#LL16-L19C14

The current implementation is using tcpSocket for livenessProbe/readinessProbe/startupProbe
This is a less preferred way of implementing Readiness/Liveness probe.
Detailed discussion can be found in this tech blog:
Stop Using TCP Health Checks for Kubernetes Application

Suggest using the http endpoint for k8s probes
Starrocks Official Doc: Check the health status of a cluster

      livenessProbe:
        tcpSocket:
          port: 9050
        timeoutSeconds: 1
        periodSeconds: 5
        successThreshold: 1
        failureThreshold: 3
      readinessProbe:
        tcpSocket:
          port: 9060
        timeoutSeconds: 1
        periodSeconds: 5
        successThreshold: 1
        failureThreshold: 3
      startupProbe:
        tcpSocket:
          port: 8060
        timeoutSeconds: 1
        periodSeconds: 5
        successThreshold: 1
        failureThreshold: 60

Way to reproduce the error:
run go test ./... from the repo root dir to reproduce it.

$ go test ./... 
?       github.com/StarRocks/starrocks-kubernetes-operator/cmd  [no test files]
?       github.com/StarRocks/starrocks-kubernetes-operator/common       [no test files]
?       github.com/StarRocks/starrocks-kubernetes-operator/components/offline_job       [no test files]
?       github.com/StarRocks/starrocks-kubernetes-operator/components/register_container        [no test files]
ok      github.com/StarRocks/starrocks-kubernetes-operator/internal/fe  (cached)
I0131 08:29:18.092066 3563771 starrockscluster_controller.go:80] StarRocksClusterReconciler reconciler the update crd name starrockscluster-sample namespace default
I0131 08:29:18.092842 3563771 k8sutils.go:56] Creating resource service namespace default name fe-domain-search kind 
I0131 08:29:18.092858 3563771 k8sutils.go:56] Creating resource service namespace default name starrockscluster-sample-fe-service kind 
I0131 08:29:18.092912 3563771 fe_statefulset.go:68] FeController buildStatefulSetParams [{"metadata":{"name":"fe-storage","creationTimestamp":null},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"10Gi"}},"storageClassName":"shard-data"},"status":{}}]
I0131 08:29:18.093138 3563771 fe_pod.go:75] FeController buildPodTemplate vols [{"name":"fe-storage","persistentVolumeClaim":{"claimName":"fe-storage"}}]
I0131 08:29:18.093281 3563771 k8sutils.go:56] Creating resource service namespace default name starrockscluster-sample-fe kind 
I0131 08:29:18.093305 3563771 statefulset.go:128] the statefulset new hash value 1286400549 old have value 4246122645 oldGeneration 0 new Generation 0
I0131 08:29:18.093309 3563771 fe_controller.go:93] FeController Sync exist statefulset not equals to new statefuslet
I0131 08:29:18.093313 3563771 k8sutils.go:64] Updating resource service namespace default name starrockscluster-sample-fe kind l&TypeMeta{Kind:,APIVersion:,}
I0131 08:29:18.094360 3563771 cn_controller.go:51] CnController Sync the cn component is not needed namespace default starrocks cluster name starrockscluster-sample
I0131 08:29:18.094366 3563771 starrockscluster_controller.go:141] StarRocksClusterReconciler reconcile status.
I0131 08:29:18.094368 3563771 starrockscluster_controller.go:178] StarRocksClusterReconciler reconciler namespace default name starrockscluster-sample
I0131 08:29:18.094440 3563771 starrockscluster_controller.go:80] StarRocksClusterReconciler reconciler the update crd name starrockscluster-sample namespace default
I0131 08:29:18.094542 3563771 k8sutils.go:56] Creating resource service namespace default name fe-domain-search kind 
I0131 08:29:18.094555 3563771 k8sutils.go:56] Creating resource service namespace default name starrockscluster-sample-fe-service kind 
--- FAIL: TestStarRocksClusterReconciler_FeReconcileSuccess (0.00s)
panic: cannot parse '': quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$' [recovered]
        panic: cannot parse '': quantities must match the regular expression '^([+-]?[0-9.]+)([eEinumkKMGTP]*[-+]?[0-9]*)$'

goroutine 14 [running]:
testing.tRunner.func1.2({0x16736e0, 0xc0003ffae0})
        /snap/go/10030/src/testing/testing.go:1396 +0x24e
testing.tRunner.func1()
        /snap/go/10030/src/testing/testing.go:1399 +0x39f
panic({0x16736e0, 0xc0003ffae0})
        /snap/go/10030/src/runtime/panic.go:884 +0x212
k8s.io/apimachinery/pkg/api/resource.MustParse({0x0, 0x0})
        /home/d.liu/go/pkg/mod/k8s.io/[email protected]/pkg/api/resource/quantity.go:139 +0x1ce
github.com/StarRocks/starrocks-kubernetes-operator/pkg/fe_controller.(*FeController).buildStatefulSetParams(_, _, _)
        /home/d.liu/git/dengliu/starrocks-kubernetes-operator/pkg/fe_controller/fe_statefulset.go:60 +0x8fc
github.com/StarRocks/starrocks-kubernetes-operator/pkg/fe_controller.(*FeController).Sync(0xc000a73920, {0x1b025e8, 0xc000044158}, 0xc000338160)
        /home/d.liu/git/dengliu/starrocks-kubernetes-operator/pkg/fe_controller/fe_controller.go:75 +0x7aa
github.com/StarRocks/starrocks-kubernetes-operator/pkg.(*StarRocksClusterReconciler).Reconcile(0xc000a73590, {0x1b025e8, 0xc000044158}, {{{0x188475d, 0x7}, {0x189baa7, 0x17}}})
        /home/d.liu/git/dengliu/starrocks-kubernetes-operator/pkg/starrockscluster_controller.go:119 +0x522
github.com/StarRocks/starrocks-kubernetes-operator/pkg.TestStarRocksClusterReconciler_FeReconcileSuccess(0x0?)
        /home/d.liu/git/dengliu/starrocks-kubernetes-operator/pkg/starrockscluster_controller_test.go:171 +0x9bc
testing.tRunner(0xc000583380, 0x196d180)
        /snap/go/10030/src/testing/testing.go:1446 +0x10b
created by testing.(*T).Run
        /snap/go/10030/src/testing/testing.go:1493 +0x35f
FAIL    github.com/StarRocks/starrocks-kubernetes-operator/pkg  0.019s
?       github.com/StarRocks/starrocks-kubernetes-operator/pkg/apis     [no test files]
?       github.com/StarRocks/starrocks-kubernetes-operator/pkg/apis/starrocks/v1alpha1  [no test files]
?       github.com/StarRocks/starrocks-kubernetes-operator/pkg/be_controller    [no test files]
?       github.com/StarRocks/starrocks-kubernetes-operator/pkg/cn_controller    [no test files]
?       github.com/StarRocks/starrocks-kubernetes-operator/pkg/common   [no test files]
?       github.com/StarRocks/starrocks-kubernetes-operator/pkg/common/hash      [no test files]
ok      github.com/StarRocks/starrocks-kubernetes-operator/pkg/common/resource_utils    (cached)
ok      github.com/StarRocks/starrocks-kubernetes-operator/pkg/fe_controller    (cached)
?       github.com/StarRocks/starrocks-kubernetes-operator/pkg/k8sutils [no test files]
?       github.com/StarRocks/starrocks-kubernetes-operator/scripts/docs/template        [no test files]
FAIL

The issue is around container support. Support isn't provided on the container level, it's done on the host level. So Suse will not support (provide big fixes, etc etc) for a Red Hat container (only best effort). So when there are issues between the container (different linux distro) and host (another different linux distro), you have finger pointing on who should fix what. The biggest on prem market is Red Hat.

https://www.redhat.com/en/blog/limits-compatibility-and-supportability-containers

版本：1.6.1
service配置NodePort type，nodePort 配置不生效

action

• k delete pod starrockscluster-fe-0 starrockscluster-fe-1 starrockscluster-fe-2

• fe version 2.4.3

• fe.log
  2023-03-22 03:26:52,488 INFO (main|1) [StarRocksFE.start():123] StarRocks FE starting... 2023-03-22 03:26:52,644 INFO (main|1) [FrontendOptions.initAddrUseFqdn():211] Use FQDN init local addr, FQDN: starrockscluster-fe-0.fe-domain-search.tmp-mgr.svc.cluster.local, IP: 10.116.28.181 2023-03-22 03:26:52,876 INFO (main|1) [Auth.grantRoleInternal():822] grant operator to 'root'@'%', isReplay = true 2023-03-22 03:26:52,897 INFO (main|1) [PrivilegeManager.initBuiltinRoleUnlocked():285] create built-in role root[-1] 2023-03-22 03:26:52,902 INFO (main|1) [PrivilegeManager.initBuiltinRoleUnlocked():285] create built-in role db_admin[-2] 2023-03-22 03:26:52,903 INFO (main|1) [PrivilegeManager.initBuiltinRoleUnlocked():285] create built-in role cluster_admin[-3] 2023-03-22 03:26:52,903 INFO (main|1) [PrivilegeManager.initBuiltinRoleUnlocked():285] create built-in role user_admin[-4] 2023-03-22 03:26:52,904 INFO (main|1) [PrivilegeManager.initBuiltinRoleUnlocked():285] create built-in role public[-5] 2023-03-22 03:26:52,904 INFO (main|1) [GlobalStateMgr.initAuth():976] using new privilege framework.. 2023-03-22 03:26:53,094 INFO (main|1) [NodeMgr.getHelperNodes():589] get helper nodes: [starrockscluster-fe-0.fe-domain-search.tmp-mgr.svc.cluster.local:9010] 2023-03-22 03:26:53,144 INFO (main|1) [NodeMgr.getClusterIdAndRoleOnStartup():389] finished to get cluster id: 1639540152, role: FOLLOWER and node name: starrockscluster-fe-0.fe-domain-search.tmp-mgr.svc.cluster.local_9010_1679455382038 2023-03-22 03:26:53,145 INFO (main|1) [BDBEnvironment.ensureHelperInLocal():357] skip check local environment because helper node and local node are identical. 2023-03-22 03:26:53,169 INFO (main|1) [BDBEnvironment.setupEnvironment():267] start to setup bdb environment for 1 times 2023-03-22 03:26:53,553 INFO (UNKNOWN starrockscluster-fe-0.fe-domain-search.tmp-mgr.svc.cluster.local_9010_1679455382038(-1)|1) [BDBEnvironment.setupEnvironment():277] add helper[starrockscluster-fe-0.fe-domain-search.tmp-mgr.svc.cluster.local:9010] as ReplicationGroupAdmin 2023-03-22 03:26:53,560 WARN (UNKNOWN starrockscluster-fe-0.fe-domain-search.tmp-mgr.svc.cluster.local_9010_1679455382038(-1)|1) [StateChangeExecutor.notifyNewFETypeTransfer():62] notify new FE type transfer: UNKNOWN 2023-03-22 03:26:53,578 INFO (UNKNOWN starrockscluster-fe-0.fe-domain-search.tmp-mgr.svc.cluster.local_9010_1679455382038(-1)|1) [BDBEnvironment.setupEnvironment():297] replicated environment is all set, wait for state change... 2023-03-22 03:27:03,580 INFO (UNKNOWN starrockscluster-fe-0.fe-domain-search.tmp-mgr.svc.cluster.local_9010_1679455382038(-1)|1) [BDBEnvironment.setupEnvironment():305] state change done, current role UNKNOWN 2023-03-22 03:27:03,585 INFO (UNKNOWN starrockscluster-fe-0.fe-domain-search.tmp-mgr.svc.cluster.local_9010_1679455382038(-1)|1) [BDBEnvironment.setupEnvironment():309] end setup bdb environment after 1 times 2023-03-22 03:27:03,588 INFO (UNKNOWN starrockscluster-fe-0.fe-domain-search.tmp-mgr.svc.cluster.local_9010_1679455382038(-1)|1) [GlobalStateMgr.loadImage():1293] image does not exist: /opt/starrocks/fe/meta/image/image.0 2023-03-22 03:27:03,589 INFO (stateChangeExecutor|77) [StateChangeExecutor.runOneCycle():85] begin to transfer FE type from INIT to UNKNOWN 2023-03-22 03:27:03,590 INFO (stateChangeExecutor|77) [StateChangeExecutor.runOneCycle():179] finished to transfer FE type to INIT 2023-03-22 03:27:05,589 INFO (UNKNOWN starrockscluster-fe-0.fe-domain-search.tmp-mgr.svc.cluster.local_9010_1679455382038(-1)|1) [GlobalStateMgr.waitForReady():1019] wait globalStateMgr to be ready. FE type: INIT. is ready: false 2023-03-22 03:27:07,589 INFO (UNKNOWN starrockscluster-fe-0.fe-domain-search.tmp-mgr.svc.cluster.local_9010_1679455382038(-1)|1) [GlobalStateMgr.waitForReady():1019] wait globalStateMgr to be ready. FE type: INIT. is ready: false 2023-03-22 03:27:09,590 INFO (UNKNOWN starrockscluster-fe-0.fe-domain-search.tmp-mgr.svc.cluster.local_9010_1679455382038(-1)|1) [GlobalStateMgr.waitForReady():1019] wait globalStateMgr to be ready. FE type: INIT. is ready: false 2023-03-22 03:27:11,590 INFO (UNKNOWN starrockscluster-fe-0.fe-domain-search.tmp-mgr.svc.cluster.local_9010_1679455382038(-1)|1) [GlobalStateMgr.waitForReady():1019] wait globalStateMgr to be ready. FE type: INIT. is ready: false 2023-03-22 03:27:13,591 INFO (UNKNOWN starrockscluster-fe-0.fe-domain-search.tmp-mgr.svc.cluster.local_9010_1679455382038(-1)|1) [GlobalStateMgr.waitForReady():1019] wait globalStateMgr to be ready. FE type: INIT. is ready: false 2023-03-22 03:27:15,592 INFO (UNKNOWN starrockscluster-fe-0.fe-domain-search.tmp-mgr.svc.cluster.local_9010_1679455382038(-1)|1) [GlobalStateMgr.waitForReady():1019] wait globalStateMgr to be ready. FE type: INIT. is ready: false 2023-03-22 03:27:17,592 INFO (UNKNOWN starrockscluster-fe-0.fe-domain-search.tmp-mgr.svc.cluster.local_9010_1679455382038(-1)|1) [GlobalStateMgr.waitForReady():1019] wait globalStateMgr to be ready. FE type: INIT. is ready: false

• fe.warn.log
  2023-03-22 03:26:53,560 WARN (UNKNOWN starrockscluster-fe-0.fe-domain-search.tmp-mgr.svc.cluster.local_9010_1679455382038(-1)|1) [StateChangeExecutor.notifyNewFETypeTransfer():62] notify new FE type transfer: UNKNOWN

how to add 10.10.15.170 master to /etc/hosts

thanks!

Hi team,

Can I set service annotations in config, like:

apiVersion: starrocks.com/v1alpha1
kind: StarRocksCluster
metadata:
  name: example
  namespace: starrocks
spec:
  starRocksFeSpec:
    service:
      type: LoadBalance
      annotations:
        service.beta.kubernetes.io/aws-load-balancer-internal: "true"
        service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
        service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: 'true'
        service.beta.kubernetes.io/aws-load-balancer-type: nlb

Thanks.

missing log err in operator autoscaler

Starrocks FE/BE services has multiple sources of logs. It is not appropriate to redirect all logs to stdout and stderr.
A more proper way is to setup log collection from each log file. This will require adding annotation to the pod spec, and mount the log files as hostPath volumes.
It would be great to add a config to the helm template with values.yml values.log.enableDatadog to transparent set up these annotations and hostPath volumes

Ref: https://docs.datadoghq.com/containers/kubernetes/log/?tab=helm#examples---log-collection-from-file-configured-in-an-annotation

e.g.

ad.datadoghq.com/<CONTAINER_IDENTIFIER>.logs: |
  [
    {"type":"file","path":"/opt/starrocks/be/be.INFO","source":"file","service":"starrocks-be"},
    {"type":"file","path":"/opt/starrocks/be/be.WARNING","source":"file","service":"starrocks-be"},
  ]

ad.datadoghq.com/<CONTAINER_IDENTIFIER>.logs: |
  [
    {"type":"file","path":"/opt/starrocks/fe/fe.log","source":"file","service":"starrocks-fe"},
    {"type":"file","path":"/opt/starrocks/fe/fe.out","source":"file","service":"starrocks-fe"},
    {"type":"file","path":"/opt/starrocks/fe/fe.warn.log","source":"file","service":"starrocks-fe"},
    {"type":"file","path":"/opt/starrocks/fe/fe.audit.log","source":"file","service":"starrocks-fe"},
    {"type":"file","path":"/opt/starrocks/fe/fe.dump.log","source":"file","service":"starrocks-fe"},
  ]    

In the operator source file pkg/fe_controller/fe_pod.go, the variable fe_config_path is defined as "/etc/starrocks/fe/conf". However, when I downloaded the image starrocks/fe-ubuntu:2.5.2, its configuration path was located at /opt/starrocks/fe/conf, which does not match the value of fe_config_path.
I am curious to know why fe_config_path is not defined as "/opt/starrocks/fe/conf".

目前部署环境如下：

• K8s：1.18.10
• starrocks/operator:v1.1、starrocks/fe:2.4.1、starrocks/be:2.4.1

看 API 文档 StarRocksCluster 中不支持加 nodeSelector

已有了一套物理机集群(FE+BE)，想借助k8s部署CN实现计算弹性伸缩，如何将k8s集群中的FE+CN add进物理机集群中呢

stream load data to starrocks that deployed on k8s only allow clients in same k8s cluster. can we should provide a mode for client to load data from out k8s.

Have you tested it with kube-monkey or something similar to ensure failure-resilient ?

Currently, operator just simply shutdown BE node when updating the replica number of BE.
This will cause data loss.
It should be changed to use DECOMMISSION to scale-in the cluse.
As DECOMMISSION is an async operation, operator must monitor its status until it is completed before proceeding with the lower-level Kubernetes statefulset scale-in.

cc @kevincai @shileifu @imay

@kevincai @shileifu @imay

This is found on Chart V1.3.5

Currently both be/storage and be/log are mounted to the same PV. This will have negative impact to the performance and using unnecessarily be/storage space for logging.

Suggesting mounting be/log to an emptyDir as we have already done this in the fe config

be storage manifest sample:

spec:
  volumes:
    - name: be-storage
      persistentVolumeClaim:
        claimName: be-storage-kube-starrocks-be-0

      volumeMounts:
        - name: be-storage
          mountPath: /opt/starrocks/be/storage
        - name: be-storage
          mountPath: /opt/starrocks/be/log

fe storage manifest sample:

spec:
  volumes:
    - name: fe-meta
      persistentVolumeClaim:
        claimName: fe-meta-kube-starrocks-fe-0
    - name: fe-log
      emptyDir: {}
      
      volumeMounts:
        - name: fe-meta
          mountPath: /opt/starrocks/fe/meta
        - name: fe-log
          mountPath: /opt/starrocks/fe/log

It would be great for SR team to clean up the unused values in values.yaml file.
This effort can help reduce unnecessary questions/distraction to both the chart author and helm users.

E.g.

starrocksCluster:
  # Provide a name for starprocks cluster
  name: ""
  namespace: ""
  # annotations for starrocks cluster
  annotations: {} <======

@kevincai
Deploying FE/BE stateful set fails due to some Security constraint in Openshift.
Workaround has been provided below but it would be nice to solve this issue in a better way.

mv: cannot move '/opt/starrocks/fe/conf/fe.conf' to '/opt/starrocks/fe/conf/fe.conf.bak': Permission denied
ln: failed to create symbolic link '/opt/starrocks/fe/conf/fe.conf': Permission denied
ERROR 2003 (HY000): Can't connect to MySQL server on 'xxxx-fe-service.xxxx:9030' (111)

Workaround

1. add fsGroup: 1000 to both starrocksBeSpec and starrocksFESpec.
   E.g. in the helm values.yaml

  starrocksFESpec:
    # Number of replicas to deploy for a fe statefulset.
    replicas: 1
...
    fsGroup: 1000 <======= add this line

2. role bind the sa to system:openshift:scc:anyuid

deploy the following role binding

> kubectl apply -f rolebinding.yml

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: system:openshift:scc:anyuid
  namespace: you-name-space
  managedFields:
    - manager: oc
      operation: Update
      apiVersion: rbac.authorization.k8s.io/v1

subjects:
  - kind: ServiceAccount
    name: your-sa  <============ replace it with the service account you use for starrocks cluster
    namespace: you-name-space <============ replace it with the name space you use for starrocks cluster
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:openshift:scc:anyuid 

cc @imay @hagonzal

For better security,
we need to add support to configure the Starrocks cluster with custom user name and passoword.
The password could be configure from a k8s secret.

Proposed place in the helm chart values.yaml:

starrocksOperator:
  # provide a name for operator deployment.
  userName:
  password: 

StarRocks/starrocks#21974

Starrocks has custom metrics available for pulling.
However, it requires manual steps to configure Prometheus server to pull the metrics.
We need to make the operator automatically setup the metrics pulling and connect it to datadog.

Ref:

https://docs.starrocks.io/en-us/latest/administration/Monitor_and_Alert
https://docs.datadoghq.com/containers/kubernetes/prometheus/?tab=kubernetesadv2

Is there a way to pass the default username and password for FE and BE? Inside the fe_entrypoint.sh and be_entrypoint.sh it is using username: root and password: as a default. When I change the root password and I increase the number of replica of BE, the BE will fail when it tries to join FE.

add_self()
{
    local svc=$1
    start=`date +%s`
    local timeout=$PROBE_TIMEOUT

    while true
    do
        log_stderr "Add myself ($MY_SELF:$HEARTBEAT_PORT) into FE ..."
        timeout 15 mysql --connect-timeout 2 -h $svc -P $FE_QUERY_PORT -u root --skip-column-names --batch -e "ALTER SYSTEM ADD BACKEND \"$MY_SELF:$HEARTBEAT_PORT\";"
        memlist=`show_backends $svc`
        if echo "$memlist" | grep -q -w "$MY_SELF" &>/dev/null ; then
            break;
        fi

        let "expire=start+timeout"
        now=`date +%s`
        if [[ $expire -le $now ]] ; then
            log_stderr "Time out, abort!"
            exit 1
        fi

        sleep $PROBE_INTERVAL

    done
}

This line
timeout 15 mysql --connect-timeout 2 -h $svc -P $FE_QUERY_PORT -u root --skip-column-names --batch -e "ALTER SYSTEM ADD BACKEND \"$MY_SELF:$HEARTBEAT_PORT\";"

support mapping volumes in the oeprator

mapping of TLS trust store / keystore in case of MTLS to the BE nodes

This is needed for collecting logs and metrics from datadog agent.

  template:
    metadata:
      annotations:
        kubectl.kubernetes.io/default-container: manager
      labels:
        app: {{ template "kube-starrocks.name" . }}-operator
        version: {{ $.Chart.Version }}

I started 3 FE pods with nfs storage. It works normally if I does not restart any pod.
When I restart one of the FE pods, it will recover to the ready state.
But when I execute the following command for restarting all FE pods simultaneously:
[kubectl delete -f starrocks-fe.yaml]. At this time, the data in the NFS storage has been retained.
Then execute [kubectl apply -f starrocks-fe.yaml] , the state of fe-0 is in an unready state always.
I found that The FE pods must be started simultaneously, and the number of FE pods started must be the same as the number of FE pods started in the previous startup for the FE pods to start up normally. Otherwise, if only start up a single fe pod，it will remain in an unready state.
So，I tried to fix the source code of operator. In [func NewStatefulset], under [Spec: appv1.StatefulSetSpec], I added PodManagementPolicy: appv1.ParallelPodManagement,
And also I modified the startup command in operator as following:
Before:
Command: []string{"/opt/starrocks/fe_entrypoint.sh"},
After:
var command = []string{
"/bin/bash",
"-c",
"if [ -f "/opt/starrocks/fe/meta/image/ROLE" ]; then /opt/starrocks/fe/bin/start_fe.sh; else /opt/starrocks/fe_entrypoint.sh starrockscluster-sample-fe-service.starrocks; fi",
}
Command: command,

After the above changes, FE pods startup simultaneously, and the problem disappeared.
Can the author of Starrocks Operator tell me if my modification is correct?