Code Monkey home page Code Monkey logo

bailongma's Introduction

通过Bailongma API服务程序接入Prometheus和Telegraf

注意:Bailongma项目功能已经被 taosAdapter 项目功能完全实现并取代。

TDengine在原生连接器通过TAOS SQL写入数据之外,还支持通过API服务程序来接入Prometheus和Telegraf的数据,仅需在Prometheus和Telegraf添加相关配置,即可将数据直接写入TDengine中,并按规则自动创建库和相关表项,无需任何代码,以及提前在TDengine做任何配置。 这篇博文用Docker容器快速搭建一个Devops监控Demo即是采用bailongma将Prometheus和Telegraf的数据写入TDengine中的示例,可以参考。

Bailongma

Bailongma是TDengine团队开发的开源数据接入API服务,目前支持Prometheus和Telegraf通过增加配置后直接将数据写入TDengine中。

从源代码编译Bailongma

目前bailongma需要从源码进行编译后运行,因此需要从github上下载Bailongma的源码后,使用Golang语言编译器编译生成可执行文件。 在开始编译前,需要准备好以下条件:

  • Linux操作系统的服务器
  • 安装好Golang,1.10版本以上
  • 对应的TDengine版本。因为用到了TDengine的客户端动态链接库,因此需要安装好和服务端相同版本的TDengine程序;比如服务端版本是TDengine 2.0.0,则在bailongma所在的linux服务器(可以与TDengine在同一台服务器,或者不同服务器) Bailongma项目中有两个文件夹blm_prometheus和blm_telegraf,分别存放了prometheus和Telegraf的写入API程序,编译方法都相同。以prometheus写入程序为例,编译过程如下

cd blm_prometheus
go build

一切正常的情况下,就会在对应的目录下生成一个blm_prometheus的可执行程序。同样的操作步骤编译成blm_telegraf的可执行文件。

Prometheus

Prometheus作为Cloud Native Computing Fundation毕业的项目,在性能监控以及K8S性能监控领域有着非常广泛的应用。TDengine可以通过bailongma API服务程序实现无代码的快速接入,高效的将数据写入TDengine中。并通过Grafana来查询TDengine中的数据

安装Prometheus

通过Prometheus的官网下载安装。下载地址

配置Prometheus

参考Prometheus的配置文档,在Prometheus的配置文件中的<remote_write>和<remote_read>部分,增加以下配置

  • url: bailongma API服务提供的URL, 参考下面的blm_prometheus启动示例章节 启动Prometheus后,可以通过taos客户端查询确认数据是否成功写入。

启动blm_prometheus程序

blm_prometheus程序有以下选项,在启动blm_prometheus程序时可以通过设定这些选项来设定blm_prometheus的配置。

--tdengine-ip 
TDengine服务端的IP地址,缺省值为127.0.0.1

--tdengine-name
如果TDengine安装在一台具备域名的服务器上,也可以通过配置TDengine的域名来访问TDengine。在K8S环境下,可以配置成TDengine所运行的service name

--batch-size 
blm_prometheus会将收到的prometheus的数据拼装成TDengine的写入请求,这个参数控制一次发给TDengine的写入请求中携带的数据条数。

--dbname
设置在TDengine中创建的数据库名称,blm_prometheus会自动在TDengine中创建一个以dbname为名称的数据库,缺省值是prometheus。

--dbuser
设置访问TDengine的用户名,缺省值是'root'

--dbpassword
设置访问TDengine的密码,缺省值是'taosdata'

--port
blm_prometheus对prometheus提供服务的端口号。

启动示例

通过以下命令启动一个blm_prometheus的API服务

./blm_prometheus -port 8088

则在prometheus的配置文件中,假设blm_prometheus所在服务器的IP地址为"10.1.2.3",<remote_write> 和 <remote_read> 部分增加url为

remote_write:
  - url: "http://10.1.2.3:8088/receive"

remote_read:
  - url: "http://10.1.2.3:8088/pull"

查询prometheus写入数据

prometheus产生的数据格式如下:

{
Timestamp: 1576466279341,
Value: 37.000000, 
apiserver_request_latencies_bucket {
component="apiserver", 
instance="192.168.99.116:8443", 
job="kubernetes-apiservers", 
le="125000", 
resource="persistentvolumes", s
cope="cluster",
verb="LIST", 
version=“v1" 
}

其中,apiserver_request_latencies_bucket为prometheus采集的时序数据的名称,后面{}中的为该时序数据的标签。blm_prometheus会以时序数据的名称在TDengine中自动创建一个超级表,并将{}中的标签转换成TDengine的tag值,Timestamp作为时间戳,value作为该时序数据的值。 因此在TDengine的客户端中,可以通过以下指令查到这个数据是否成功写入。

use prometheus;
select * from apiserver_request_latencies_bucket;

Telegraf

TDengine能够与开源数据采集系统Telegraf快速集成,整个过程无需任何代码开发。

安装Telegraf

目前TDengine支持Telegraf 1.7.4以上的版本。用户可以根据当前的操作系统,到Telegraf官网下载安装包,并执行安装。下载地址如下:https://portal.influxdata.com/downloads

配置Telegraf

修改Telegraf配置文件/etc/telegraf/telegraf.conf中与TDengine有关的配置项。

在output plugins部分,增加[[outputs.http]]配置项:

  • url: bailongma API服务提供的URL, 参考下面的启动示例章节
  • data_format: "json"
  • json_timestamp_units: "1ms"

在agent部分:

  • hostname: 区分不同采集设备的机器名称,需确保其唯一性
  • metric_batch_size: 100,允许Telegraf每批次写入记录最大数量,增大其数量可以降低Telegraf的请求发送频率。

关于如何使用Telegraf采集数据以及更多有关使用Telegraf的信息,请参考Telegraf官方的文档

启动blm_telegraf程序

blm_telegraf程序有以下选项,在启动blm_telegraf程序时可以通过设定这些选项来设定blm_telegraf的配置。

--host 
TDengine服务端的IP地址,缺省值为空

--batch-size 
blm_telegraf会将收到的telegraf的数据拼装成TDengine的写入请求,这个参数控制一次发给TDengine的写入请求中携带的数据条数。

--dbname
设置在TDengine中创建的数据库名称,blm_telegraf会自动在TDengine中创建一个以dbname为名称的数据库,缺省值是prometheus。

--dbuser
设置访问TDengine的用户名,缺省值是'root'

--dbpassword
设置访问TDengine的密码,缺省值是'taosdata'

--port
blm_telegraf对telegraf提供服务的端口号。

启动示例

通过以下命令启动一个blm_telegraf的API服务

./blm_telegraf -host 127.0.0.1 -port 8089

则在telegraf的配置文件中,假设blm_telegraf所在服务器的IP地址为"10.1.2.3",在output plugins部分,增加[[outputs.http]]配置项:

url = "http://10.1.2.3:8089/telegraf"

查询telegraf写入数据

telegraf产生的数据格式如下:

{
  "fields": {
    "usage_guest": 0, 
    "usage_guest_nice": 0,
    "usage_idle": 89.7897897897898, 
    "usage_iowait": 0,
    "usage_irq": 0,
    "usage_nice": 0,
    "usage_softirq": 0,
    "usage_steal": 0,
    "usage_system": 5.405405405405405, 
    "usage_user": 4.804804804804805
  },
  "name": "cpu", 
  "tags": {
    "cpu": "cpu2",
    "host": "bogon" 
    },
  "timestamp": 1576464360 
}

其中,name字段为telegraf采集的时序数据的名称,tags字段为该时序数据的标签。blm_telegraf会以时序数据的名称在TDengine中自动创建一个超级表,并将tags字段中的标签转换成TDengine的tag值,Timestamp作为时间戳,fields字段中的值作为该时序数据的值。 因此在TDengine的客户端中,可以通过以下指令查到这个数据是否成功写入。

use telegraf;
select * from cpu;

bailongma's People

Contributors

anjia0532 avatar chenrulongmaster avatar fangpanpan avatar huskar-t avatar kim-up avatar liuyq-617 avatar sangshuduo avatar xiaopingcs avatar zitsen avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

bailongma's Issues

工程编译问题

HI:
如果可以的话,帮忙在一个干净环境里重新编译测试看看。
1、go.mod是不是可以直接写在工程里?而不是用go mod init
2、默认情况下promethues的版本不存在prompb
3、需要依赖TDengine,但是TDengine, submodule clone问题,这里存在git clone权限不足
4、解决了prompb module问题,但是传入promb变量有问题,这里估计是prompb版本问题

谢谢。

编译 blm_prometheus 报错

root@dcim:/opt/Bailongma# go mod tidy
go: finding module for package github.com/emirpasic/gods/sets/hashset
go: finding module for package github.com/prometheus/common/model
go: finding module for package github.com/prometheus/prometheus/prompb
go: finding module for package github.com/taosdata/driver-go/taosSql
go: finding module for package github.com/golang/snappy
go: finding module for package github.com/gogo/protobuf/proto
go: found github.com/gogo/protobuf/proto in github.com/gogo/protobuf v1.3.2
go: found github.com/golang/snappy in github.com/golang/snappy v0.0.4
go: found github.com/prometheus/common/model in github.com/prometheus/common v0.31.1
go: found github.com/prometheus/prometheus/prompb in github.com/prometheus/prometheus v2.5.0+incompatible
go: found github.com/emirpasic/gods/sets/hashset in github.com/emirpasic/gods v1.12.0
go: finding module for package github.com/grpc-ecosystem/grpc-gateway/utilities
go: finding module for package github.com/taosdata/driver-go/taosSql
go: finding module for package github.com/grpc-ecosystem/grpc-gateway/runtime
go: found github.com/grpc-ecosystem/grpc-gateway/runtime in github.com/grpc-ecosystem/grpc-gateway v1.16.0
go: found github.com/grpc-ecosystem/grpc-gateway/utilities in github.com/grpc-ecosystem/grpc-gateway v1.16.0
go: finding module for package github.com/taosdata/driver-go/taosSql
bailongma/v2/blm_prometheus imports
        github.com/taosdata/driver-go/taosSql: no matching versions for query "latest"
root@dcim:/opt/Bailongma/blm_prometheus# go get github.com/taosdata/driver-go/taosSql
go get github.com/taosdata/driver-go/taosSql: no matching versions for query "upgrade"

项目 github.com/taosdata/driver-go 已发布 v2 版本,详见:taosdata/driver-go

启动后运行一会儿报错

panic: Checktable error value type

goroutine 22 [running]:
main.SerilizeTDengine(0xc00059c5d0, 0xc0005956d0, 0xc, 0xc00059c600, 0x177289bb8b0, 0x736fdd, 0x8, 0xc000210000, 0xe, 0xc0005eb050, ...)
/root/blm_telegraf/server.go:346 +0xc6a
main.ProcessData(0xc0000f27e0, 0x1, 0x1, 0x736fdd, 0x8, 0xc000210000, 0xe, 0x0, 0x0)
/root/blm_telegraf/server.go:392 +0x1030
main.ProcessReq(0xc000580000, 0x4d2, 0x63f, 0xc000210000, 0xe, 0x0, 0x0)
/root/blm_telegraf/server.go:284 +0x3d5
main.NodeProcess(0x4, 0x8, 0x9)
/root/blm_telegraf/server.go:210 +0xce
created by main.main
/root/blm_telegraf/server.go:139 +0x164
host: tdengine:0 port: 10202 database: telegraf%

The file "server.go" can't build in TDengine 2.0.3.1 (blm_promepheus)

I want to rebuild the project as directed by the link:https://www.taosdata.com/cn/documentation/insert/#Prometheus直接写入
But I cannot go build, and get ERR:

execSql Error: syntax error near ")
" sqlcmd: create table if not exists MD5_648e4536f5d4184314287d187677867c using node_network_address_assign_type tags("623716d73194f57a558f0d1cb4964378",)

I modify the file "server.go" and add a line after line 529:

line 529: sqlcmd = sqlcmdhead + md5V2(tagHash) + ""," + sqlcmd
line 530: sqlcmd = strings.Replace(sqlcmd, ",)\n", ")\n", -1)

But it still cannot run normally after modification. Attached is the generated error log, please help update the code or indicate whether there is a problem in my configuration.
This is the table of project configuration:

ServerName IP Port APP
taos1 10.197.190.17 6030~6042 Tdengine 2.0.3.1
taos2 10.197.190.18 6030~6042 Tdengine 2.0.3.1
taos3 10.197.190.19 6030~6042 Tdengine 2.0.3.1
app 10.197.190.20 3000 Grafana 7.1.5
Telegraf
6051 blm_telegraf
Prometheus 10.197.190.21 9090 Prometheus
6052 blm_prometheus

blm_prometheus.log
prometheus.yml.log
server.go.log
taosdlog.0.log

离线环境无法编译

/root/Bailongma-feature-openfalcon/blm_telegraf
[root@master2 blm_telegraf]# go build
go: github.com/emirpasic/[email protected]: Get "https://proxy.golang.org/github.com/emirpasic/gods/@v/v1.12.0.mod": dial tcp: lookup proxy.golang.org on 114.114.114.114:53: read udp 192.168.0.212:36019->114.114.114.114:53: i/o timeout
go: downloading github.com/taosdata/driver-go v0.0.0-20210722071233-99176d298e90
go: downloading github.com/emirpasic/gods v1.12.0
go: github.com/emirpasic/[email protected]: Get "https://proxy.golang.org/github.com/emirpasic/gods/@v/v1.12.0.mod": dial tcp: lookup proxy.golang.org on 114.114.114.114:53: read udp 192.168.0.212:36019->114.114.114.114:53: i/o timeout
[root@master2 blm_telegraf]# ls
Dockerfile  go.mod  go.sum  README.md  server.go  Telegraf写入TDengine的API.md
[root@master2 blm_telegraf]# go version
go version go1.17.1 linux/amd64

prometheus 配置了BLM 没有数据写到TDengine,也没有报错

问题:prometheus 配置了BLM 没有数据写到TDengine,也没有报错
prometheus 版本:2.24.0
TDengine 版本:2.0.15

1、prometheus的启动日志:
image
2、prometheus的配置:
image
3、BIM服务也是正常的,在安装了prometheus的机器上,直接http请求:
image
4、BLM的启动参数,并且进程也在的:
image
5、然后TDengine的机器上查不到数据,连数据库都没有创建:
image
6、实际在grafana里面是有数据写入prometheus的:
image

启动报错 connection refused

./blm_prometheus -port 8088 启动,终端直接报错
Get "http://172.23.11.4:6020/rest/sql": dial tcp 172.23.11.4:6020: connect: connection refused Get "http://172.23.11.4:6020/rest/sql": dial tcp 172.23.11.4:6020: connect: connection refused Get "http://172.23.11.4:6020/rest/sql": dial tcp 172.23.11.4:6020: connect: connection refused Get "http://172.23.11.4:6020/rest/sql": dial tcp 172.23.11.4:6020: connect: connection refused Get "http://172.23.11.4:6020/rest/sql": dial tcp 172.23.11.4:6020: connect: connection refused Get "http://172.23.11.4:6020/rest/sql": dial tcp 172.23.11.4:6020: connect: connection refused Get "http://172.23.11.4:6020/rest/sql": dial tcp 172.23.11.4:6020: connect: connection refused Get "http://172.23.11.4:6020/rest/sql": dial tcp 172.23.11.4:6020: connect: connection refused

查看taosd log 如下:
06/08 20:35:32.069964 00023174 MND ERROR msg:0x7f6e44000d60, app:0x66a table:0.prometheus.mysql_info_schema_table_size, failed to get table meta, table not exist 06/08 20:35:32.070581 00023175 MND ERROR msg:0x7f6e44000d60, app:0x66c table:0.prometheus.mysql_info_schema_table_size, failed to get table meta, table not exist 06/08 20:35:32.071263 00023174 MND ERROR msg:0x7f6e44000d60, app:0x66e table:0.prometheus.mysql_info_schema_table_size, failed to get table meta, table not exist 06/08 20:35:32.071779 00023175 MND ERROR msg:0x7f6e44000d60, app:0x670 table:0.prometheus.mysql_info_schema_table_size, failed to get table meta, table not exist 06/08 20:35:32.072324 00023174 MND ERROR msg:0x7f6e44000d60, app:0x672 table:0.prometheus.mysql_info_schema_table_size, failed to get table meta, table not exist 06/08 20:35:32.072893 00023175 MND ERROR msg:0x7f6e44000d60, app:0x674 table:0.prometheus.mysql_info_schema_table_size, failed to get table meta, table not exist 06/08 20:35:32.073430 00023174 MND ERROR msg:0x7f6e44000d60, app:0x676 table:0.prometheus.mysql_info_schema_table_size, failed to get table meta, table not exist 06/08 20:35:33.020489 00023174 MND ERROR msg:0x7f6e44000d60, app:0x678 table:0.prometheus.md5_ed032e52a5b5bb7a4b190fad66074a46, failed to get table meta, table not exist

文档说库和表会自动创建,但实际log显示都没创建

blm_prometheus 可能存在内存泄漏,使用内存一直在增长

使用 bailongma master 以及 prometheus v2.23.0 编译,修改部分语法错误(prometheus v2.23.0同v2.5.0部分struct不兼容) 运行后,内存持续增长( v2.5.0 也有同样的问题,所以才升级到 v2.23.0 测试)

运行: go tool pprof 'http://localhost:8080/debug/pprof/heap'

File: blm_prometheus
Build ID: 1d27bd0ba96f37057ab7eb9aa9f5e516fd4b7cf2
Type: inuse_space
Time: Dec 24, 2020 at 5:31pm (CST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 1370.07MB, 97.38% of 1406.88MB total
Dropped 32 nodes (cum <= 7.03MB)
Showing top 10 nodes out of 31
flat flat% sum% cum cum%
690.07MB 49.05% 49.05% 1001.09MB 71.16% github.com/prometheus/prometheus/prompb.(*TimeSeries).Unmarshal
311.02MB 22.11% 71.16% 311.02MB 22.11% github.com/prometheus/prometheus/prompb.(*Label).Unmarshal
166.04MB 11.80% 82.96% 166.04MB 11.80% bufio.NewWriterSize (inline)
114.43MB 8.13% 91.09% 114.43MB 8.13% bufio.NewReaderSize
52.51MB 3.73% 94.82% 1053.59MB 74.89% github.com/prometheus/prometheus/prompb.(*WriteRequest).Unmarshal
9.50MB 0.68% 95.50% 9.50MB 0.68% runtime.malg
9.50MB 0.68% 96.18% 12.50MB 0.89% net/textproto.(*Reader).ReadMIMEHeader
7.50MB 0.53% 96.71% 92.10MB 6.55% net/http.(*conn).readRequest
7MB 0.5% 97.21% 27.50MB 1.95% net/http.readRequest
2.50MB 0.18% 97.38% 1056.09MB 75.07% main.main.func1

过一段时间后,运行

File: blm_prometheus
Build ID: 1d27bd0ba96f37057ab7eb9aa9f5e516fd4b7cf2
Type: inuse_space
Time: Dec 24, 2020 at 5:38pm (CST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 1743.78MB, 97.58% of 1787.07MB total
Dropped 34 nodes (cum <= 8.94MB)
Showing top 10 nodes out of 31
flat flat% sum% cum cum%
872.86MB 48.84% 48.84% 1262.88MB 70.67% github.com/prometheus/prometheus/prompb.(*TimeSeries).Unmarshal
390.02MB 21.82% 70.67% 390.02MB 21.82% github.com/prometheus/prometheus/prompb.(*Label).Unmarshal
212.18MB 11.87% 82.54% 212.18MB 11.87% bufio.NewWriterSize (inline)
147.56MB 8.26% 90.80% 147.56MB 8.26% bufio.NewReaderSize
66.65MB 3.73% 94.53% 1329.53MB 74.40% github.com/prometheus/prometheus/prompb.(*WriteRequest).Unmarshal
15MB 0.84% 95.37% 18MB 1.01% net/textproto.(*Reader).ReadMIMEHeader
14.51MB 0.81% 96.18% 14.51MB 0.81% runtime.malg
12MB 0.67% 96.85% 126.14MB 7.06% net/http.(*conn).readRequest
9MB 0.5% 97.35% 37.51MB 2.10% net/http.readRequest
4MB 0.22% 97.58% 1333.53MB 74.62% main.main.func1

系统运行环境:

Linux xxx 3.10.0-1127.19.1.el7.x86_64 #1 SMP Tue Aug 25 17:23:54 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
CentOS Linux release 7.9.2009 (Core)
go version go1.15.2 linux/amd64

Bailongma can not connect to taos db cluster in kubernetes

Hi, @sangshuduo
I deployed an tdengine cluster in k8s followed TDengine-Operator, following is my tdengine pods and service, and it worked.
image

then i deployed bailongma
image

bailongma can not connect to taos. following is logs
image

and i can connect to tdengine by restful api in blm_prometheus container.

image

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.