Kvass is a Prometheus horizontal auto-scaling solution , which uses Sidecar to generate special config file only containes part of targets assigned from Coordinator for every Prometheus shard.
License: Apache License 2.0
有prometheus监控kubernetes的完整配置文件参考吗，试过直接拷贝prometheus-operator的配置文件应用，但是不行。如果可以请指点下实现思路，谢谢！
prometheus-operator和kvass有些功能有冲突。比如部署prometheus方式上。
保留prometheus-operator的原因是保留其中很多功能，比如serviceMonitor服务发现方式等。
GOROOT=D:\Go #gosetup
GOPATH=D:\goProjects #gosetup
D:\Go\bin\go.exe build -o C:\Users\sWX992771\AppData\Local\Temp___go_build_coordinator_go_main_go_sidecar_go.exe -gcflags "all=-N -l" D:\goProjects\src\kvass-0.1.2\cmd\kvass\coordinator.go D:\goProjects\src\kvass-0.1.2\cmd\kvass\main.go D:\goProjects\src\kvass-0.1.2\cmd\kvass\sidecar.go #gosetup
"D:\developTools\GoLand 2020.1.3\plugins\go\lib\dlv\windows\dlv.exe" --listen=localhost:57806 --headless=true --api-version=2 --check-go-version=false --only-same-user=false exec C:\Users\sWX992771\AppData\Local\Temp___go_build_coordinator_go_main_go_sidecar_go.exe -- #gosetup
API server listening at: 127.0.0.1:57806
Kvass is a noninvasive prometheusURL scaling solution,
it allows prometheusURL shard with "target" granules.
Usage:
Kvass [command]
Available Commands:
coordinator coordinator manager all prometheus shard
help Help about any command
sidecar sidecar manager one prometheus shard
Flags:
-h, --help help for Kvass
Use "Kvass [command] --help" for more information about a command.
Vmagent accepts Prometheus scrape configs. It would be great if kvass could balance load among a group of vmagents too. Vmagent exposes various metrics at /metrics page, wchich could be used during rebalancing decisions. See https://victoriametrics.github.io/vmagent.html#monitoring
场景：10个job，每个job100w+的series，设置的--shard.max-series数量为90w。
现象：kvass会一直调度新增shard来实现分配，但实际并未分配。
影响：扩容到一定程度会导致k8s集群瘫痪。(kvass部署在k8s集群中)。
建议：是否可以增加逻辑判断，对于超过设置阈值的以日志的形式通知提示，并停止这种无效调度。
对比采用kvass分片和原生promethues不分片的采集数，不管用scrape_samples_scraped还是rate(prometheus_remote_storage_samples_in_total[5m])，发现kvass的结果值总是比原生不分片的prometheus要小，让人感觉是不是少了。。请问这个该怎么确认
我没有修改prometheus配置，集群也没有增删节点，因此总的series个数没有变化
现在想通过修改max-series来测试prometheus水平扩展能力
发现max-series的配置由大到小时，prometheus副本数会增加，每一个prometheus的series个数满足要求
但是max-series的配置由小到大时，prometheus副本数没有减少（等待了idle时间）
coordinator的配置：--shard.max-series=40000 --shard.delete-pvc=false --shard.max-idle-time=1h --shard.min-shard=1
测试发现，如果现有prometheus挂掉，coordinator发现了但是没有新建prometheus转移target。coordiantor日志如下：
time="2021-06-25T00:20:40Z" level=warning msg="Statefulset prometheus UpdatedReplicas != Replicas, skipped" component="shard manager"
time="2021-06-25T00:21:40Z" level=warning msg="Statefulset prometheus is not ready, try wait 2m" component="shard manager"
time="2021-06-25T00:21:40Z" level=warning msg="Statefulset prometheus is not ready, still waiting" component="shard manager"
time="2021-06-25T00:22:40Z" level=warning msg="Statefulset prometheus is not ready, still waiting" component="shard manager"
time="2021-06-25T00:23:40Z" level=info msg="prometheus-1 is not ready" component=coordinator
time="2021-06-25T00:23:40Z" level=info msg="prometheus-0 is not ready" component=coordinator
time="2021-06-25T00:23:40Z" level=info msg="need space 58203" component=coordinator
time="2021-06-25T00:23:40Z" level=warning msg="shard group prometheus-0 is unHealth, skip apply change" component=coordinator
time="2021-06-25T00:23:40Z" level=warning msg="shard group prometheus-1 is unHealth, skip apply change" component=coordinator
kvass sidecar 报如下错误，请教一下是什么问题？
copy data to prometheus failed write tcp 127.0.0.1:8008->127.0.0.1:50370: write: broken pipe" component="target manager"
除了第一个容器prometheus-rep-0-0，后面启动的容器中kvass sidecar 都报这个错误
数据目录/prometheus 挂载了pvc，看跟这个issuse很像#7 ，目前使用kvass版本0.0.3
I'm testing kvass-prometheus.
1st attempt: I create 1 metrics pod and 1 prometheus pod
root@master:~/kvass# kubectl get pod
NAME READY STATUS RESTARTS AGE
kvass-coordinator-f89599b75-5q5kz 2/2 Running 0 101s
metrics-64c679f867-bmxzc 1/1 Running 0 102s
prometheus-rep-0-0 3/3 Running 1 101s
thanos-query-849d699f8b-xp9zq 1/1 Running 0 100s
2nd attempt: I scale up metrics pod to 20 replicas. Prometheus pod scale up very well
root@master:~/kvass# kubectl get pod
NAME READY STATUS RESTARTS AGE
kvass-coordinator-f89599b75-5q5kz 2/2 Running 0 9m16s
metrics-64c679f867-5fs75 1/1 Running 0 3m11s
metrics-64c679f867-6z5ff 1/1 Running 0 3m11s
metrics-64c679f867-8r7l8 1/1 Running 0 3m11s
metrics-64c679f867-bgwsp 1/1 Running 0 3m11s
metrics-64c679f867-bmxzc 1/1 Running 0 9m17s
metrics-64c679f867-bt584 1/1 Running 0 3m11s
metrics-64c679f867-c5cf5 1/1 Running 0 3m11s
metrics-64c679f867-fxns2 1/1 Running 0 3m11s
metrics-64c679f867-hzl8g 1/1 Running 0 3m11s
metrics-64c679f867-j5bzb 1/1 Running 0 3m11s
metrics-64c679f867-k7wdc 1/1 Running 0 3m11s
metrics-64c679f867-kcvc6 1/1 Running 0 3m11s
metrics-64c679f867-r68kg 1/1 Running 0 3m11s
metrics-64c679f867-rptqp 1/1 Running 0 3m11s
metrics-64c679f867-sc47p 1/1 Running 0 3m11s
metrics-64c679f867-sgfv4 1/1 Running 0 3m11s
metrics-64c679f867-snrbb 1/1 Running 0 3m11s
metrics-64c679f867-tkdf8 1/1 Running 0 3m11s
metrics-64c679f867-ts244 1/1 Running 0 3m11s
metrics-64c679f867-x8tnj 1/1 Running 0 3m11s
prometheus-rep-0-0 3/3 Running 1 9m16s
prometheus-rep-0-1 3/3 Running 1 2m39s
prometheus-rep-0-2 3/3 Running 1 2m39s
prometheus-rep-0-3 3/3 Running 1 2m28s
prometheus-rep-0-4 3/3 Running 1 2m18s
prometheus-rep-0-5 3/3 Running 1 2m8s
prometheus-rep-0-6 3/3 Running 1 2m8s
prometheus-rep-0-7 3/3 Running 1 117s
prometheus-rep-0-8 3/3 Running 1 107s
prometheus-rep-0-9 3/3 Running 1 106s
thanos-query-849d699f8b-xp9zq 1/1 Running 0 9m15s
root@master:~/kvass#
3rd attempt: I reduce the metrics pod to 5 replicas, but prometheus pod didn't scale down, still 9 repicas
root@master:~/kvass# kubectl get pod
NAME READY STATUS RESTARTS AGE
kvass-coordinator-f89599b75-5q5kz 2/2 Running 0 13m
metrics-64c679f867-6z5ff 1/1 Running 0 7m23s
metrics-64c679f867-bmxzc 1/1 Running 0 13m
metrics-64c679f867-c5cf5 1/1 Running 0 7m23s
metrics-64c679f867-sgfv4 1/1 Running 0 7m23s
metrics-64c679f867-ts244 1/1 Running 0 7m23s
prometheus-rep-0-0 3/3 Running 1 13m
prometheus-rep-0-1 3/3 Running 1 6m51s
prometheus-rep-0-2 3/3 Running 1 6m51s
prometheus-rep-0-3 3/3 Running 1 6m40s
prometheus-rep-0-4 3/3 Running 1 6m30s
prometheus-rep-0-5 3/3 Running 1 6m20s
prometheus-rep-0-6 3/3 Running 1 6m20s
prometheus-rep-0-7 3/3 Running 1 6m9s
prometheus-rep-0-8 3/3 Running 1 5m59s
prometheus-rep-0-9 3/3 Running 1 5m58s
thanos-query-849d699f8b-xp9zq 1/1 Running 0 13m
Can you add support for scale-down as well?
#12 (comment)
似乎并没有在0.0.4版本完成
prometheus release：https://github.com/prometheus/prometheus/releases/tag/v2.28.0
issue：#8839
调研试用，希望支持此功能，升级一下prometheus版本，kvass回归验证，重新release。
我用0.0.4版本测试自己环境的prometheus，拉取target都正常
但是换成0.0.5版本后，原有的target都没了，同时增加prometheus_shards这个job，但是我并不需要
查看sidecar和coordinator都没有错误日志，查看prometheus配置确实没有target，应该是0.0.5版本代码逻辑修改导致
我希望能让每2个普罗米修斯采集同样的数据，通过thanos来做去重，保证高可用性。但使用了kvass之后，无法控制让他们去采集重复的数据。请问是你们是如何保证高可用的？
./kvass coordinator --config.file /opt/prometheus/prometheus_cd.yml
Error: unable to load in-cluster configuration, KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT must be defined
强制依赖吗？
我发现有很多prom实例上的target是0。但是实际确启动了一堆的prom实例
只更新coordinator的配置更新pod后。发现prometheus也全部重建
When I deploy a new prometheus, I had met the error below. Is this a known bug?
k8s version:
1.16.6
kvass version:
tag 0.5
Log:
2021/03/04 07:30:41 http: panic serving 127.0.0.1:43278: runtime error: invalid memory address or nil pointer dereference
goroutine 680 [running]:
net/http.(*conn).serve.func1(0xc00099a000)
/usr/local/go/src/net/http/server.go:1772 +0x139
panic(0x22a1580, 0x3e505f0)
/usr/local/go/src/runtime/panic.go:973 +0x3e3
tkestack.io/kvass/pkg/sidecar.(*Proxy).ServeHTTP.func1(0xc000871480, 0xc0084105d6f1662b, 0x226bdee68e, 0x3e7a0a0, 0xc0008cfa58, 0xc0005dff20, 0xc0008cfa68, 0x2bbc780, 0xc00079a000)
/Users/Gone/code/kvass/pkg/sidecar/proxy.go:90 +0x50
tkestack.io/kvass/pkg/sidecar.(*Proxy).ServeHTTP(0xc0005dff20, 0x2bbc780, 0xc00079a000, 0xc000802900)
/Users/Gone/code/kvass/pkg/sidecar/proxy.go:113 +0x8b2
net/http.serverHandler.ServeHTTP(0xc0006de460, 0x2bbc780, 0xc00079a000, 0xc000802900)
/usr/local/go/src/net/http/server.go:2807 +0xa3
net/http.(*conn).serve(0xc00099a000, 0x2bc5b00, 0xc00099c000)
/usr/local/go/src/net/http/server.go:1895 +0x86c
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2933 +0x35c
prometheus采用了类似如下的配置：
- job_name: "node"
file_sd_configs:
- refresh_interval: 1m
files:
- "/etc/prometheus/config_pvc/node/*.yml"
/etc/prometheus/config_pvc/node/下会动态更新需要监控的target文件，但发现coordinator日志有如下报错，实际上文件确实存在的：
level=error ts=2021-06-04T03:18:06.090Z caller=file.go:344 component="discovery manager scrape" discovery=file msg="Error reading file" path=/etc/prometheus/config_pvc/node/xxxxxxxxxxxxxxx.yml err="open /etc/prometheus/config_pvc/node/xxxxxxxxxxxxxxxyml: no such file or directory"
When I testing kvass, sometimes kvass sidecar will be killed because of OOM, and then the targets scraped by that shard will loss.
According to the source code, coordinator will cache the targets info in 'scraping', the cache seems like never update anywhere. While sidecar encounter some error, the injected target will lost but coordinator will not know.
I think kvass should keep the cache in coordinator and sidecar the same.
coordinator日志一直输出：
time="2021-03-04T09:42:03Z" level=warning msg="config of shard-0 is not up to date" component=coordinator
time="2021-03-04T09:42:03Z" level=info msg="need space 48158" component=coordinator
time="2021-03-04T09:42:03Z" level=warning msg="shard group shard-0 is unHealth, skip apply change" component=coordinator
time="2021-03-04T09:42:03Z" level=warning msg="shard group shard-1 is unHealth, skip apply change" component=coordinator
prometheusUI target为空，且curl 获取不到任何数据
curl http://10.222.30.58:9090/api/v1/query?query=alertmanager_build_info
{"status":"success","data":{"resultType":"vector","result":[]}}
1、完善kvass非 ON k8s部署文档,包括生产配置最佳实践及建议指引文档
查看prometheus的静态配置，除了给指标使用的label外，还有一些默认label和__parm开头的label，保留目的是什么
coordinator运行时创建2个prometheus，后来采集的target和series也没有变化，但是发现coordinator创建了第3个prometheus
查看第3个prometheus没有发现coordinator分配的target，此prometheus为空配置运行，kvass版本为0.1.0
所有的自动发现抓取target都把scheme改成了http协议，如果有需要https,比如kubelet 这种就会报400错误
I found that after scale down, the pod may be created again because the pod still terminating.
Here is the code:
func (s *shardManager) Shards() ([]*shard.Shard, error) { pods, err := s.getPods(s.sts.Spec.Selector.MatchLabels) // **Here will also get the pods that is terminating** if err != nil { return nil, errors.Wrap(err, "list pod") } ... return ret, nil }
As the title show, can I set a min replicas for a prometheus sts, and then kvass inject the target to the prometheus pod evenly?
https://github.com/tkestack/kvass/blob/master/examples/coordinator.yaml
rancher/coreos-prometheus-config-reloader:v0.32.0似乎是帮助prometheus重载配置的，dockerhub中找到这个image但没发现用法或者代码仓库地址。
可以描述一下它的作用以及在kvass中的使用场景？
在两个exporter 的指标有关联的情况下
比如 我同一个节点的 node_exporter 和. cadvisor 的采集尽量分配在同一个 prom 上， 这样prom 本地做 record rule 要省不少事情
Hello,
this project sounds really interesting and I would like to use it.
Unfortunately you haven't wrote any documentation, which makes it hard for me to understand the full capability of your software.
Therefore I want to ask you some questions if it is possible or not.
I have two types of targets I want to scrape. One type needs a replica set with a and b and the other one only needs to be scraped by a singe Prometheus. Is kvass capable of creating a replica set of Proms with external_labels set in the Prometheus config, so that Thanos can us the label for de-duplication? I also want to add an external_label to each Prometheus which automatically increases. I should also mention that my targets are not inside any k8s cluster and are written in a static config.
For example:
data:
prometheus.yml: |-
global:
scrape_interval: 15s
evaluation_interval: 15s
external_labels:
cluster: some-cluster
prom_replica: a
prom: prom-node-n
data:
prometheus.yml: |-
global:
scrape_interval: 15s
evaluation_interval: 15s
external_labels:
cluster: some-cluster
prom_replica: b
prom: prom-node-n
prom-node-n where n stands for a number which would count up for each replica set.
Is this possible with kvass?
增删Job后，coordinator发现了series变化，但是连接prometheus发现异常，导致prometheus没有加载最新的配置
以增加job为例，coordinator一直打印如下日志
time="2021-03-26T01:53:22Z" level=info msg="need space 1567" component=coordinator
time="2021-03-26T01:53:22Z" level=warning msg="shard group prometheus-0 is unHealth, skip apply change" component=coordinator
time="2021-03-26T01:53:22Z" level=warning msg="shard group prometheus-1 is unHealth, skip apply change" component=coordinator
time="2021-03-26T01:53:32Z" level=warning msg="config of prometheus-1 is not up to date, expect md5 = 15756264197780680581, shard md5 = 9559724437703473867" component=coordinator
time="2021-03-26T01:53:32Z" level=warning msg="config of prometheus-0 is not up to date, expect md5 = 15756264197780680581, shard md5 = 9559724437703473867" component=coordinator
prometheus运行正常，sidecar也没有报错，不知道为什么cooridnator判别prometheus为unHealth
发现kvass sidecar通过配置--web.proxy-addr暴露端口8008，同时注入到prometheus的配置增加proxy_url: http://127.0.0.1:8008
请问这两块配置存在的作用是什么，是把prometheus的拉取请求代理到sidecar吗，那prometheus拉取相关的指标（scrape开头的）还准确吗
【问题描述】: 利用promethues/jmx_exporter 以http服务方式 http://ip:port/metrics 暴露服务的jmx信息，然后kvass + thanos + promethues 架构抓取jmx 指标，但是在promethues query 侧却查不到相应的监控指标，已排除可能存在的网络连问题，且自己通过 curl http://ip:port/metrics 是有指标数据返回的，通过单点promethues 直接配置scrape_config 抓取jmx ，在query 端是可以查到相应的指标数据的；
exporter 吐出的指标类似如下格式：
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageAfterGc_init{type="GarbageCollector",key="CodeHeap 'profiled nmethods'",} 2555904.0
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageAfterGc_init{type="GarbageCollector",key="G1 Old Gen",} 3.154116608E10
java_lang_G1_Young_Generation_LastGcInfo_memoryUsageAfterGc_init{type="GarbageCollector",key="G1 Survivor Space",} 0.0
PS： 很多指标 {} 内部后面都多了一个逗号；
所以，请问下是否是数据格式的原因导致了指标抓取失败？
time="2021-03-11T03:19:22Z" level=info msg="config inject completed" component=injector
2021/03/11 03:20:38 http: proxy error: context canceled
time="2021-03-11T03:21:17Z" level=info msg="config inject completed" component=injector
2021/03/11 03:22:08 http: superfluous response.WriteHeader call from tkestack.io/kvass/pkg/sidecar.(*Proxy).ServeHTTP.func1 (proxy.go:90)
2021/03/11 03:22:38 http: proxy error: context canceled
2021/03/11 03:23:08 http: proxy error: context canceled
2021/03/11 03:23:38 http: proxy error: context canceled
2021/03/11 03:24:08 http: proxy error: context canceled
2021/03/11 03:24:38 http: proxy error: context canceled
2021/03/11 03:25:08 http: proxy error: context canceled
2021/03/11 03:25:38 http: proxy error: context canceled
2021/03/11 03:26:38 http: proxy error: context canceled
2021/03/11 03:26:47 http: superfluous response.WriteHeader call from tkestack.io/kvass/pkg/sidecar.(*Proxy).ServeHTTP.func1 (proxy.go:90)
2021/03/11 03:27:38 http: proxy error: context canceled
2021/03/11 03:28:08 http: proxy error: context canceled
2021/03/11 03:28:27 http: superfluous response.WriteHeader call from tkestack.io/kvass/pkg/sidecar.(*Proxy).ServeHTTP.func1 (proxy.go:90)
2021/03/11 03:28:38 http: proxy error: context canceled
2021/03/11 03:30:44 http: superfluous response.WriteHeader call from tkestack.io/kvass/pkg/sidecar.(*Proxy).ServeHTTP.func1 (proxy.go:90)
2021/03/11 03:30:44 http: superfluous response.WriteHeader call from tkestack.io/kvass/pkg/sidecar.(*Proxy).ServeHTTP.func1 (proxy.go:90)
现在如果想要达到多副本的目的，是不是只能创建多个Deployment了，然后再对--shard.selector=app.kubernetes.io/name=prometheus 传入不同的参数？，但是这样还得部署多个Deployment， 感觉不是很优雅，能否提供一个参数，可以选择副本的数量。
想咨询下，Targets 分片后，告警规则配置在什么地方比较好？
用 Thanos，且 Targets 分片之后，意味着只有通过 Thanos 聚合层查询，才能拿到完整全部的数据。在 TSDB 数据层，通过分片方案降低单 Prometheus 数据压力的方案是很友好的。
但是，Thanos Ruler 官方不是很推荐建议使用 Ruler 来做告警。比如性能、比如查询耗时等，官方建议告警规则，最好是配置在 Prometheus 本身。请问，如果用 Kvass，最佳的告警规则配置方式是什么呢？还是 Thanos Ruler 来配置吗？
hello ，想了解是否支持consul 动态发现，有没有支持的计划？之前看文章有介绍”Kvass coordinaor引用了原生Prometheus的服务发现代码，用于实现与Prometheus 100%兼容的服务发现能力“ ，使用中发现consul_sd_configs 并不能很好的支持，报如题所示的错误。是哪里没配置对吗？
分片缩容讲到当分片被清空后，分片会变为闲置状态，经过一段时间后（等待分片数据被删除或者被上传至对象存储），分片被删除
请问如果设置了-shard.max-idle-time后，是不是也要等待prometheus分片数据被删除的时间，即选择二者的最大值
另外，prometheus分片被删除是指的内存中还是硬盘上删除
time="2021-09-27T12:49:50Z" level=error msg="reload failed: marshal config: parsing YAML file /etc/prometheus/config_out/prometheus.env.yaml: yaml: unmarshal errors:\n line 1519: field http_sd_configs not found in type config.ScrapeConfig" component=web
比如对以下事件暴露出metrics有助于后期运维监控
- 当前need space数量
- 发生change scale事件
- config inject
- target transfer
...
环境信息：
k8s集群共118个节点
每个prom的max-series 为80w，共6个prom
资源为8G、2Cpu
我使用一个集群外部的prom通过federate接口，抓取prometheus-rep-0-(0-5)pod中prom的数据时，发生OOMKilled
xxx1@localhost:~$ kubectl get po
NAME READY STATUS RESTARTS AGE
kvass-coordinator-58f4b8df79-47svs 2/2 Running 0 2h
prom-lua-proxy-deployment-758d58f6c-rql9l 1/1 Running 0 7m
prometheus-node1-7474c8758c-wkqg7 2/2 Running 1 23d
prometheus-rep-0-0 2/3 OOMKilled 5 2h
prometheus-rep-0-1 3/3 Running 3 2h
prometheus-rep-0-2 3/3 Running 4 2h
prometheus-rep-0-3 3/3 Running 3 2h
prometheus-rep-0-4 3/3 Running 2 2h
prometheus-rep-0-5 3/3 Running 1 2h
tensorflow-serving-58997667b8-6kkrv 1/1 Running 0 146d
thanos-query-6b69d8c69d-6hln6 1/1 Running 0 1d
config.yaml配置
kind: ConfigMap
apiVersion: v1
metadata:
name: prometheus-config
namespace: xxx-dev
data:
prometheus.yml: |-
global:
scrape_interval: 45s
scrape_timeout: 45s
evaluation_interval: 45s
external_labels:
server: 'k8s-monitor'
scrape_configs:
- job_name: 'kubernetes-u-nodes'
kubernetes_sd_configs:
- role: node
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: true
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
- target_label: __address__
replacement: kubernetes.default.svc:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
coordinator.yaml
apiVersion: v1
kind: Service
metadata:
name: kvass-coordinator
labels:
app.kubernetes.io/name: kvass-coordinator
spec:
ports:
- name: http
port: 9090
targetPort: http
selector:
app.kubernetes.io/name: kvass-coordinator
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/name: kvass-coordinator
name: kvass-coordinator
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: kvass-coordinator
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
type: RollingUpdate
template:
metadata:
labels:
app.kubernetes.io/name: kvass-coordinator
spec:
#serviceAccountName: prometheus
serviceAccountName: xxx-dev-user
containers:
- name: config-reload
args:
- --reload-url=http://localhost:9090/-/reload
- --config-file=/etc/prometheus/config/prometheus.yml
- --config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
image: rancher/coreos-prometheus-config-reloader:v0.32.0
imagePullPolicy: Always
resources:
limits:
memory: 240Mi
requests:
memory: 120Mi
volumeMounts:
- mountPath: /etc/prometheus/config_out
name: config-out
- mountPath: /etc/prometheus/config
name: config
#- image: tkestack/kvass:v0.0.3
- image: kvass/urmsone-kvass
imagePullPolicy: Always
#imagePullPolicy: Never
args:
- coordinator
- --shard.max-series=1000000
- --shard.selector=app.kubernetes.io/name=prometheus
- --shard.namespace=$(NAMESPACE)
- --config.file=/etc/prometheus/config_out/prometheus.env.yaml
env:
- name: NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
ports:
- containerPort: 9090
name: http
protocol: TCP
volumeMounts:
- mountPath: /etc/prometheus/config
name: config
- mountPath: /etc/prometheus/config_out
name: config-out
name: kvass
resources:
limits:
cpu: 4
memory: 4Gi
requests:
cpu: 4
memory: 2Gi
volumes:
- name: config
configMap:
name: prometheus-config
defaultMode: 420
- emptyDir: {}
name: config-out
- emptyDir: {}
name: tls-assets
prometheus-rep-0.yaml
kind: Service
apiVersion: v1
metadata:
name: prometheus
namespace: xxx-dev
labels:
app.kubernetes.io/name: prometheus
spec:
type: ClusterIP
clusterIP: None
selector:
app.kubernetes.io/name: prometheus
ports:
- name: web
protocol: TCP
port: 8080
targetPort: web
- name: grpc
port: 10901
targetPort: grpc
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
labels:
app.kubernetes.io/name: prometheus
name: prometheus-rep-0
namespace: xxx-dev
spec:
# must set as Parallel
podManagementPolicy: Parallel
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app.kubernetes.io/name: prometheus
kvass/rep: "0"
serviceName: prometheus
template:
metadata:
labels:
app.kubernetes.io/name: prometheus
kvass/rep: "0"
spec:
containers:
- name: thanos
image: thanosio/thanos:v0.16.0-rc.1
args:
- sidecar
- --tsdb.path=/prometheus
- --prometheus.url=http://localhost:8080
- --reloader.config-file=/etc/prometheus/config/prometheus.yml
- --reloader.config-envsubst-file=/etc/prometheus/config_out/prometheus.env.yaml
ports:
- name: http-sidecar
containerPort: 10902
- name: grpc
containerPort: 10901
livenessProbe:
httpGet:
port: 10902
path: /-/healthy
readinessProbe:
httpGet:
port: 10902
path: /-/ready
volumeMounts:
- mountPath: /etc/prometheus/config_out
name: config-out
- mountPath: /etc/prometheus/config
name: config
- name: kvass
args:
- sidecar
image: kvass/urmsone-kvass
imagePullPolicy: Always
resources:
limits:
memory: 4Gi
cpu: 4
requests:
memory: 2Gi
cpu: 2
volumeMounts:
- mountPath: /etc/prometheus/config_out
name: config-out
ports:
- containerPort: 8080
name: web
protocol: TCP
- name: prometheus
args:
- --storage.tsdb.path=/prometheus
- --storage.tsdb.retention.time=3h
- --web.enable-lifecycle
- --storage.tsdb.no-lockfile
- --storage.tsdb.max-block-duration=2h
- --storage.tsdb.min-block-duration=2h
- --config.file=/etc/prometheus/config_out/prometheus_injected.yaml
image: prom/prometheus:v2.19.2
resources:
limits:
memory: 8Gi
cpu: 2
requests:
memory: 8Gi
cpu: 2
ports:
- containerPort: 9090
name: web
protocol: TCP
volumeMounts:
- mountPath: /etc/prometheus/config
name: config
- mountPath: /etc/prometheus/config_out
name: config-out
- mountPath: /prometheus
name: data
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
serviceAccountName: xxx-dev-user
securityContext:
runAsUser: 0
volumes:
- name: data
emptyDir: {}
- name: config
configMap:
name: prometheus-config
defaultMode: 420
- emptyDir: {}
name: config-out
- emptyDir: {}
name: tls-assets
updateStrategy:
rollingUpdate:
partition: 0
type: RollingUpdate
coordinator管理prometheus开始运行正常，后来自己挂掉重启，挂掉日志如下
time="2021-03-31T10:25:47Z" level=info msg="prometheus-1 need update targets" component="shard manager" shard=prometheus-1 sts=prometheus
fatal error: concurrent map iteration and map write
goroutine 509 [running]:
runtime.throw(0x27f20af, 0x26)
/home/wushuai/go/src/runtime/panic.go:1116 +0x72 fp=0xc0028c9ba8 sp=0xc0028c9b78 pc=0x4376b2
runtime.mapiternext(0xc0028c9c80)
/home/wushuai/go/src/runtime/map.go:853 +0x554 fp=0xc0028c9c28 sp=0xc0028c9ba8 pc=0x410cf4
tkestack.io/kvass/pkg/discovery.(*TargetsDiscovery).ActiveTargetsByHash(0xc0000bbf80, 0x0)
/home/wushuai/gospace/src/github.com/tkestack/kvass/pkg/discovery/discovery.go:113 +0x17c fp=0xc0028c9d00 sp=0xc0028c9c28 pc=0x15642dc
tkestack.io/kvass/pkg/discovery.(*TargetsDiscovery).ActiveTargetsByHash-fm(0xc001b27c40)
/home/wushuai/gospace/src/github.com/tkestack/kvass/pkg/discovery/discovery.go:109 +0x2a fp=0xc0028c9d20 sp=0xc0028c9d00 pc=0x202c54a
tkestack.io/kvass/pkg/coordinator.(*Coordinator).runOnce(0xc00014db00, 0xc0009f4900, 0x0)
/home/wushuai/gospace/src/github.com/tkestack/kvass/pkg/coordinator/coordinator.go:100 +0x25d fp=0xc0028c9e50 sp=0xc0028c9d20 pc=0x1596ebd
tkestack.io/kvass/pkg/coordinator.(*Coordinator).runOnce-fm(0x0, 0x0)
/home/wushuai/gospace/src/github.com/tkestack/kvass/pkg/coordinator/coordinator.go:85 +0x2a fp=0xc0028c9e78 sp=0xc0028c9e50 pc=0x159c1ca
tkestack.io/kvass/pkg/utils/wait.RunUntil(0x2b588e0, 0xc000128000, 0x2b8d9a0, 0xc000688930, 0x2540be400, 0xc0028c9ef8, 0x0, 0x0)
/home/wushuai/gospace/src/github.com/tkestack/kvass/pkg/utils/wait/wait.go:35 +0x63 fp=0xc0028c9eb8 sp=0xc0028c9e78 pc=0x1568283
tkestack.io/kvass/pkg/coordinator.(*Coordinator).Run(0xc00014db00, 0x2b588e0, 0xc000128000, 0x11, 0x0)
/home/wushuai/gospace/src/github.com/tkestack/kvass/pkg/coordinator/coordinator.go:75 +0x7b fp=0xc0028c9f18 sp=0xc0028c9eb8 pc=0x1596bfb
main.glob..func1.8(0x0, 0x441256)
/home/wushuai/gospace/src/github.com/tkestack/kvass/cmd/kvass/coordinator.go:207 +0x9a fp=0xc0028c9f78 sp=0xc0028c9f18 pc=0x202be7a
golang.org/x/sync/errgroup.(*Group).Go.func1(0xc000aadec0, 0xc001878060)
/home/wushuai/gospace/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:57 +0x59 fp=0xc0028c9fd0 sp=0xc0028c9f78 pc=0x15001d9
runtime.goexit()
/home/wushuai/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc0028c9fd8 sp=0xc0028c9fd0 pc=0x46f8e1
created by golang.org/x/sync/errgroup.(*Group).Go
/home/wushuai/gospace/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:54 +0x66
goroutine 1 [semacquire, 7908 minutes]:
sync.runtime_Semacquire(0xc000aaded0)
/home/wushuai/go/src/runtime/sema.go:56 +0x45
sync.(*WaitGroup).Wait(0xc000aadec8)
/home/wushuai/go/src/sync/waitgroup.go:130 +0x65
golang.org/x/sync/errgroup.(*Group).Wait(0xc000aadec0, 0xc004ab1040, 0xc000abf860)
/home/wushuai/gospace/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:40 +0x31
main.glob..func1(0x3b35960, 0xc000598d20, 0x0, 0x5, 0x0, 0x0)
/home/wushuai/gospace/src/github.com/tkestack/kvass/cmd/kvass/coordinator.go:215 +0x1085
github.com/spf13/cobra.(*Command).execute(0x3b35960, 0xc000598cd0, 0x5, 0x5, 0x3b35960, 0xc000598cd0)
/home/wushuai/gospace/pkg/mod/github.com/spf13/[email protected]/command.go:842 +0x47c
github.com/spf13/cobra.(*Command).ExecuteC(0x3b35c00, 0x166fc4bbac19f505, 0x3b4d020, 0x0)
/home/wushuai/gospace/pkg/mod/github.com/spf13/[email protected]/command.go:950 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
/home/wushuai/gospace/pkg/mod/github.com/spf13/[email protected]/command.go:887
main.main()
/home/wushuai/gospace/src/github.com/tkestack/kvass/cmd/kvass/main.go:42 +0x11c
日志很多，这里只截取一部分，同时发现goroutine个数特别大，是不是存在协程泄露的问题
我修改了prometheus配置中某个job的metric_relabel_configs，指标过滤在原有process和go基础上添加http开头的，coordinator感知到了，sidecar感知不到，sidecar的启动参数配置如下：
/kvass sidecar --config.file=/etc/prometheus/config/prometheus.yml --config.output-file=/etc/prometheus/config_out/prometheus.yml
查看sidecar的--config.file和--config.output-file，发现sidecar没有感知配置file的修改从而去重载output-file
/etc/prometheus/config/prometheus.yml
- job_name: 'kubernetes-nodes'
kubernetes_sd_configs:
- role: node
scrape_interval: 1m
scrape_timeout: 1m
relabel_configs:
- source_labels: [__address__]
regex: '(.*):10250'
replacement: '${1}:10255'
target_label: __address__
- source_labels: [__meta_kubernetes_node_name]
action: replace
target_label: host
- source_labels: [__address__]
action: replace
target_label: instance
metric_relabel_configs:
- source_labels: [__name__]
regex: '(process|go|http)_.+'
action: drop
/etc/prometheus/config_out/prometheus.yml
- job_name: kubernetes-nodes
honor_timestamps: true
scrape_interval: 1m
scrape_timeout: 1m
metrics_path: /metrics
scheme: http
proxy_url: http://127.0.0.1:8008
relabel_configs:
- separator: ;
regex: __invalid_label_(.+)
replacement: $1
action: labelmap
metric_relabel_configs:
- source_labels: [__name__]
separator: ;
regex: (process|go)_.+ ——缺少修改的http
replacement: $1
action: drop
Please help to make a release version include fix: register all SD type and push to docker hub.
Thanks a lot
我部署配置了kvass sidecar的prometheus，发现Pod总会重启一次，查看日志发现kvaas sidecar启动后连接prometheus的url，因为连接不上导致程序挂掉，这里能否在连接时增加从重试逻辑，避免第一次启动时总是显示重启次数为1
root@mgt01:~# kubectl get pod|grep prometheus
prometheus-0 3/3 Running 1 37m
prometheus-1 3/3 Running 1 30m
prometheus-2 3/3 Running 1 30m
root@mgt01:~# kubectl logs prometheus-0 kvass -p
time="2021-02-18T07:45:21Z" level=info msg="config inject completed" component=injector
panic: apply config: Post "http://127.0.0.1:9090/-/reload": dial tcp 127.0.0.1:9090: connect: connection refused
goroutine 1 [running]:
main.glob..func2(0x3b39100, 0x3b82c70, 0x0, 0x0, 0x0, 0x0)
/home/runner/work/kvass/kvass/cmd/kvass/sidecar.go:109 +0xbae
github.com/spf13/cobra.(*Command).execute(0x3b39100, 0x3b82c70, 0x0, 0x0, 0x3b39100, 0x3b82c70)
/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:842 +0x47c
github.com/spf13/cobra.(*Command).ExecuteC(0x3b38e60, 0x1664c7ddbad24102, 0x3b50180, 0x0)
/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:950 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
/home/runner/go/pkg/mod/github.com/spf13/[email protected]/command.go:887
main.main()
/home/runner/work/kvass/kvass/cmd/kvass/main.go:42 +0x11c
我想测试修改prometheus配置后，coordinator能否感知到并同步配置到prometheus；修改配置后发现prometheus完全重建了（先删除再新建），cooridnator日志如下：
time="2021-06-16T00:37:20Z" level=warning msg="Statefulset prometheus UpdatedReplicas != Replicas, skipped" component="shard manager"
level=info ts=2021-06-16T00:37:27.302Z caller=kubernetes.go:263 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
W0616 00:37:27.302653 1 reflector.go:424] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:156: watch of *v1.Service ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
level=info ts=2021-06-16T00:37:27.304Z caller=kubernetes.go:263 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
level=info ts=2021-06-16T00:37:27.306Z caller=kubernetes.go:263 component="discovery manager scrape" discovery=kubernetes msg="Using pod service account via in-cluster config"
time="2021-06-16T00:37:30Z" level=info msg="need space 57074" component=coordinator
time="2021-06-16T00:37:30Z" level=info msg="change scale to 1" component="shard manager" sts=prometheus
time="2021-06-16T00:37:40Z" level=error msg="get targets status info from prometheus-0 failed, url = http://100.101.245.226:8080: http get: Get "http://100.101.245.226:8080/api/v1/shard/targets/\": dial tcp 100.101.245.226:8080: connect: connection refused" component=coordinator
time="2021-06-16T00:37:40Z" level=error msg="get runtime info from prometheus-0 failed : http get: Get "http://100.101.245.226:8080/api/v1/shard/runtimeinfo/\": dial tcp 100.101.245.226:8080: connect: connection refused" component=coordinator
time="2021-06-16T00:37:40Z" level=info msg="need space 57074" component=coordinator
time="2021-06-16T00:37:40Z" level=warning msg="shard group prometheus-0 is unHealth, skip apply change" component=coordinator
shard-38 28分执行迁移，shard-40 series数量的确增加，但3个周期后，shard-38的series没有执行剔除，导致series数量不断增加，增加无故内存消耗。
我的理解是：3个周期指的是prometheus拉取三次数据之后，剔除shard-38的数量。
I like the concept of this project.
I'm curious about the latency in service discovery when a new target is created.
And in an environment of new deployments are created and destroyed frequently, I think there should be some latency or malfunction in service discovery. And there is the delay in prometheus when reloading config I guess?
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.