taosdata / taoskeeper Goto Github PK

License: MIT License

Dockerfile 0.32% Go 99.68%

taoskeeper's Introduction

TaosKeeper

TDengine Metrics Exporter, you can obtain the running status of TDengine version 3.0 by performing several simple configurations.

This tool uses TDengine RESTful API, so you could just build it without TDengine client.

Build

Get the source codes

git clone https://github.com/taosdata/taoskeeper
cd taoskeeper

compile

go mod tidy
go build

Install

If you build the tool by your self, just copy the taoskeeper binary to your PATH.

sudo install taoskeeper /usr/bin/

Start

Before start, you should configure some options like database IP, port or the prefix and others for exported metrics.

in /etc/taos/taoskeeper.toml.

# Start with debug middleware for gin
debug = false

# Listen port, default is 6043
port = 6043

# log level
loglevel = "info"

# go pool size
gopoolsize = 50000

# interval for TDengine metrics
RotationInterval = "15s"

[tdengine]
host = "127.0.0.1"
port = 6041
username = "root"
password = "taosdata"

# list of taosAdapter that need to be monitored
[taosAdapter]
address = ["127.0.0.1:6041"]

[metrics]
# metrics prefix in metrics names.
prefix = "taos"

# database for storing metrics data
database = "log"

# export some tables that are not super table
tables = []

[environment]
# Whether running in cgroup.
incgroup = false

Now you could run the tool:

taoskeeper

If you use systemd, copy the taoskeeper.service to /lib/systemd/system/ and start the service.

sudo cp taoskeeper.service /lib/systemd/system/
sudo systemctl daemon-reload
sudo systemctl start taoskeeper

To start taoskeeper whenever OS rebooted, you should enable the systemd service:

sudo systemctl enable taoskeeper

So if use systemd, you'd better install it with these lines all-in-one:

go mod tidy
go build
sudo install taoskeeper /usr/bin/
sudo cp taoskeeper.service /lib/systemd/system/
sudo systemctl daemon-reload
sudo systemctl start taoskeeper
sudo systemctl enable taoskeeper

Docker

Here is an example to show how to build this tool in docker:

Before building, you should configure ./config/taoskeeper.toml with proper parameters and edit Dockerfile. Take following as example.

FROM golang:1.17.2 as builder

WORKDIR /usr/src/taoskeeper
COPY ./ /usr/src/taoskeeper/
ENV GO111MODULE=on \
    GOPROXY=https://goproxy.cn,direct
RUN go mod tidy && go build

FROM alpine:3
RUN mkdir -p /etc/taos
COPY --from=builder /usr/src/taoskeeper/taoskeeper /usr/bin/
COPY ./config/taoskeeper.toml /etc/taos/taoskeeper.toml
EXPOSE 6043
CMD ["taoskeeper"]

If you already have taosKeeper binary file, you can build this tool like:

FROM ubuntu:18.04
RUN mkdir -p /etc/taos
COPY ./taoskeeper /usr/bin/
COPY ./taoskeeper.toml /etc/taos/taoskeeper.toml
EXPOSE 6043
CMD ["taoskeeper"]

FAQ

Error occurred: Connection refused, while taosKeeper was starting

Answer: taoskeeper relies on restful interfaces to query data. Check whether the taosAdapter is running or whether the taosAdapter address in taoskeeper.toml is correct.
Why detection metrics displayed by different TDengine's inconsistent with taoskeeper monitoring?

Answer: If a metric is not created in TDengine, taoskeeper cannot get the corresponding test results.

Cannot receive log from TDengine server.

Answer: Modify /etc/taos/taos.cfg file and add parameters like:

monitor                  1  // start monitor
monitorInterval          30  // send log interval (s)
monitorFqdn              localhost 
monitorPort              6043 // taosKeeper port
monitorMaxLogs           100

taoskeeper's People

Contributors

Stargazers

Watchers

Forkers

jandison zhengqin agile6v petaverse-cloud laiyongtao buyfakett

taoskeeper's Issues

服务启动时显示创建数据库错误

#taoskeeper
WARN[0000] create database sql: create database if not exists log model=report module=taosKeeper
ERRO[0000] create database log error ReadString: expects " or n, but found 4, error found in #10 byte of ...|ed_rows",4,4]],"rows|..., bigger context ...|"affected_rows"],"column_meta":[["affected_rows",4,4]],"rows":1,"data":[[0]]}|... error="ReadString: expects " or n, but found 4, error found in #10 byte of ...|ed_rows",4,4]],"rows|..., bigger context ...|"affected_rows"],"column_meta":[["affected_rows",4,4]],"rows":1,"data":[[0]]}|..." model=report module=taosKeeper
panic: ReadString: expects " or n, but found 4, error found in #10 byte of ...|ed_rows",4,4]],"rows|..., bigger context ...|"affected_rows"],"column_meta":[["affected_rows",4,4]],"rows":1,"data":[[0]]}|...

goroutine 1 [running]:
github.com/taosdata/taoskeeper/api.(*Reporter).createDatabase(0x4000291440)
/opt/taoskeeper/api/report.go:227 +0x180
github.com/taosdata/taoskeeper/api.(*Reporter).Init(0x4000291440, {0x18a83d0, 0x4000329a00})
/opt/taoskeeper/api/report.go:60 +0xc4
main.main()
/opt/taoskeeper/main.go:26 +0x60

运行环境：
aarch64/ubuntu 16/go 1.21.4
下载了 aarch64-server安装，TDengine-server-2.6.0.34-Linux-aarch64.tar.gz，启动taosd/taosadapter
下载 taoskeeper源代码，按文档步骤编译、安装、运行后，出现以上错误

go build时报错请问该怎么解决

go mod tidy 就过不去怎么办？

check sum 各种报错，最后只好手动调成一致
要求influxdata/[email protected] 但是它又要求harlow/kinesis-consumer v0.3.6 ，但是这个版本被作者删了，根本都不存在了，也不知的怎么改，我就把telegraf版本改成1.24.0 不知道最后行不行
influxdata/telegraf改成1.24.0之后，requires google.golang.org/grpc/[email protected] 一个什么很奇怪的版本，也是根本找不到的，现在就卡在这里，根本没法继续安装了.....

求教程

主节点安装taoskeeper后，另外两节点报错http-report failed to conn to server, reason:connection refused, dst:localhost:6043

我们是在linux centos 7.9系统上搭建了3.0.1.3版本的3节点集群，在主节点安装了taoskeeper，主节点配置文件内容如下：

Start with debug middleware for gin

debug = false

Listen port, default is 6043

port = 6043

log level

loglevel = "error"

go pool size

gopoolsize = 50000

interval for TDengine metrics

RotationInterval = "15s"

[tdengine]
host = "192.168.13.170"
port = 6041
username = ""
password = ""

list of taosAdapter that need to be monitored

[taosAdapter]
address = ["127.0.0.1:6041","192.168.13.170:6041"]

[metrics]

metrics prefix in metrics names.

prefix = "taos"

database for storing metrics data

database = "log"

另外两节点都报错

12/13 21:04:19.387492 00004591 UTL ERROR http-report failed to conn to server, reason:connection refused, dst:localhost:6043
12/13 21:04:19.387509 00004591 UTL ERROR failed to send monitor msg

请问是什么问题呢

DNode 资源使用情况

集群有三个节点，第一个节点安装了taoskeeper，但是grafana上的报表“DNode 资源使用情况”只有第一个节点data1有监控信息

这个是需要每个节点都安装taoskeeper吗？

如何进行云原生支持呢

K8s Pormetheus 如何进行集成呢

无法创建正确的表名称

taoskeeper

INFO[0000] execute sql: create table if not exists cluster_info (ts timestamp, first_ep binary(134), first_ep_dnode_id int, version binary(12), master_uptime float, monitor_interval int,dbs_total int, tbs_total int, stbs_total int, dnodes_total int, dnodes_alive int, mnodes_total int, mnodes_alive int, vgroups_total int, vgroups_alive int, vnodes_total int, vnodes_aliveint, connections_total int, protocol int ) tags (cluster_id nchar(32)) model=report module=taosKeeper
INFO[0000] execute sql: create table if not exists d_info (ts timestamp, status binary(10)) tags (dnode_id int, dnode_ep nchar(255), cluster_id nchar(32)) model=report module=taosKeeper
INFO[0000] execute sql: create table if not exists m_info (ts timestamp, role binary(10)) tags (mnode_id int, mnode_ep nchar(255), cluster_id nchar(32)) model=report module=taosKeeper
INFO[0000] execute sql: create table if not exists dnodes_info (ts timestamp, uptime float, cpu_engine float, cpu_system float, cpu_cores float, mem_engine int, mem_system int, mem_totalint, disk_engine int, disk_used bigint, disk_total bigint, net_in float, net_out float, io_read float, io_write float, io_read_disk float, io_write_disk float, req_select int, req_select_rate float, req_insert int, req_insert_success int, req_insert_rate float, req_insert_batch int, req_insert_batch_success int, req_insert_batch_rate float, errors int, vnodes_num int, masters int, has_mnode int, has_qnode int, has_snode int, has_bnode int ) tags (dnode_id int, dnode_ep nchar(255), cluster_id nchar(32)) model=report module=taosKeeper
INFO[0000] execute sql: create table if not exists data_dir (ts timestamp, name nchar(200), level int, avail bigint, used bigint, total bigint) tags (dnode_id int, dnode_ep nchar(255), cluster_id nchar(32)) model=report module=taosKeeper
INFO[0000] execute sql: create table if not exists log_dir (ts timestamp, name nchar(200), avail bigint, used bigint, total bigint) tags (dnode_id int, dnode_ep nchar(255), cluster_id nchar(32)) model=report module=taosKeeper
INFO[0000] execute sql: create table if not exists temp_dir(ts timestamp, name nchar(200), avail bigint, used bigint, total bigint ) tags (dnode_id int, dnode_ep nchar(255), cluster_id nchar(32)) model=report module=taosKeeper
INFO[0000] execute sql: create table if not exists vgroups_info (ts timestamp, vgroup_id int, database_name binary(33), tables_num int, status binary(512) ) tags (dnode_id int, dnode_ep nchar(255), cluster_id nchar(32)) model=report module=taosKeeper
INFO[0000] execute sql: create table if not exists vnodes_role (ts timestamp, vnode_role binary(10) ) tags (dnode_id int, dnode_ep nchar(255), cluster_id nchar(32)) model=report module=taosKeeper
INFO[0000] execute sql: create table if not exists logs (ts timestamp, level binary(10), content nchar(1024)) tags (dnode_id int, dnode_ep nchar(255), cluster_id nchar(32)) model=reportmodule=taosKeeper
INFO[0000] execute sql: create table if not exists log_summary(ts timestamp, error int, info int, debug int, trace int ) tags (dnode_id int, dnode_ep nchar(255), cluster_id nchar(32)) model=report module=taosKeeper
INFO[0000] execute sql: create table if not exists grants_info(ts timestamp, expire_time int, timeseries_used int, timeseries_total int ) tags (dnode_id int, dnode_ep nchar(255), cluster_id nchar(32)) model=report module=taosKeeper
INFO[0000] execute sql: create table if not exists keeper_monitor (ts timestamp, cpu float, mem float, total_reports int ) tags (identify nchar(50)) model=report module=taosKeeper