taosdata / bailongma Goto Github PK
View Code? Open in Web Editor NEWAn data transporter for tdengine
License: GNU Affero General Public License v3.0
An data transporter for tdengine
License: GNU Affero General Public License v3.0
/root/Bailongma-feature-openfalcon/blm_telegraf
[root@master2 blm_telegraf]# go build
go: github.com/emirpasic/[email protected]: Get "https://proxy.golang.org/github.com/emirpasic/gods/@v/v1.12.0.mod": dial tcp: lookup proxy.golang.org on 114.114.114.114:53: read udp 192.168.0.212:36019->114.114.114.114:53: i/o timeout
go: downloading github.com/taosdata/driver-go v0.0.0-20210722071233-99176d298e90
go: downloading github.com/emirpasic/gods v1.12.0
go: github.com/emirpasic/[email protected]: Get "https://proxy.golang.org/github.com/emirpasic/gods/@v/v1.12.0.mod": dial tcp: lookup proxy.golang.org on 114.114.114.114:53: read udp 192.168.0.212:36019->114.114.114.114:53: i/o timeout
[root@master2 blm_telegraf]# ls
Dockerfile go.mod go.sum README.md server.go Telegraf写入TDengine的API.md
[root@master2 blm_telegraf]# go version
go version go1.17.1 linux/amd64
HI:
如果可以的话,帮忙在一个干净环境里重新编译测试看看。
1、go.mod是不是可以直接写在工程里?而不是用go mod init
2、默认情况下promethues的版本不存在prompb
3、需要依赖TDengine,但是TDengine, submodule clone问题,这里存在git clone权限不足
4、解决了prompb module问题,但是传入promb变量有问题,这里估计是prompb版本问题
谢谢。
是否能将blm作为Opentsdb和tdegine的转换器,将opentsdb数据格式转换后存入tdengine
Hi, @sangshuduo
I deployed an tdengine cluster in k8s followed TDengine-Operator, following is my tdengine pods and service, and it worked.
bailongma can not connect to taos. following is logs
and i can connect to tdengine by restful api in blm_prometheus container.
root@dcim:/opt/Bailongma# go mod tidy
go: finding module for package github.com/emirpasic/gods/sets/hashset
go: finding module for package github.com/prometheus/common/model
go: finding module for package github.com/prometheus/prometheus/prompb
go: finding module for package github.com/taosdata/driver-go/taosSql
go: finding module for package github.com/golang/snappy
go: finding module for package github.com/gogo/protobuf/proto
go: found github.com/gogo/protobuf/proto in github.com/gogo/protobuf v1.3.2
go: found github.com/golang/snappy in github.com/golang/snappy v0.0.4
go: found github.com/prometheus/common/model in github.com/prometheus/common v0.31.1
go: found github.com/prometheus/prometheus/prompb in github.com/prometheus/prometheus v2.5.0+incompatible
go: found github.com/emirpasic/gods/sets/hashset in github.com/emirpasic/gods v1.12.0
go: finding module for package github.com/grpc-ecosystem/grpc-gateway/utilities
go: finding module for package github.com/taosdata/driver-go/taosSql
go: finding module for package github.com/grpc-ecosystem/grpc-gateway/runtime
go: found github.com/grpc-ecosystem/grpc-gateway/runtime in github.com/grpc-ecosystem/grpc-gateway v1.16.0
go: found github.com/grpc-ecosystem/grpc-gateway/utilities in github.com/grpc-ecosystem/grpc-gateway v1.16.0
go: finding module for package github.com/taosdata/driver-go/taosSql
bailongma/v2/blm_prometheus imports
github.com/taosdata/driver-go/taosSql: no matching versions for query "latest"
root@dcim:/opt/Bailongma/blm_prometheus# go get github.com/taosdata/driver-go/taosSql
go get github.com/taosdata/driver-go/taosSql: no matching versions for query "upgrade"
项目 github.com/taosdata/driver-go 已发布 v2 版本,详见:taosdata/driver-go
./blm_prometheus -port 8088 启动,终端直接报错
Get "http://172.23.11.4:6020/rest/sql": dial tcp 172.23.11.4:6020: connect: connection refused Get "http://172.23.11.4:6020/rest/sql": dial tcp 172.23.11.4:6020: connect: connection refused Get "http://172.23.11.4:6020/rest/sql": dial tcp 172.23.11.4:6020: connect: connection refused Get "http://172.23.11.4:6020/rest/sql": dial tcp 172.23.11.4:6020: connect: connection refused Get "http://172.23.11.4:6020/rest/sql": dial tcp 172.23.11.4:6020: connect: connection refused Get "http://172.23.11.4:6020/rest/sql": dial tcp 172.23.11.4:6020: connect: connection refused Get "http://172.23.11.4:6020/rest/sql": dial tcp 172.23.11.4:6020: connect: connection refused Get "http://172.23.11.4:6020/rest/sql": dial tcp 172.23.11.4:6020: connect: connection refused
查看taosd log 如下:
06/08 20:35:32.069964 00023174 MND ERROR msg:0x7f6e44000d60, app:0x66a table:0.prometheus.mysql_info_schema_table_size, failed to get table meta, table not exist 06/08 20:35:32.070581 00023175 MND ERROR msg:0x7f6e44000d60, app:0x66c table:0.prometheus.mysql_info_schema_table_size, failed to get table meta, table not exist 06/08 20:35:32.071263 00023174 MND ERROR msg:0x7f6e44000d60, app:0x66e table:0.prometheus.mysql_info_schema_table_size, failed to get table meta, table not exist 06/08 20:35:32.071779 00023175 MND ERROR msg:0x7f6e44000d60, app:0x670 table:0.prometheus.mysql_info_schema_table_size, failed to get table meta, table not exist 06/08 20:35:32.072324 00023174 MND ERROR msg:0x7f6e44000d60, app:0x672 table:0.prometheus.mysql_info_schema_table_size, failed to get table meta, table not exist 06/08 20:35:32.072893 00023175 MND ERROR msg:0x7f6e44000d60, app:0x674 table:0.prometheus.mysql_info_schema_table_size, failed to get table meta, table not exist 06/08 20:35:32.073430 00023174 MND ERROR msg:0x7f6e44000d60, app:0x676 table:0.prometheus.mysql_info_schema_table_size, failed to get table meta, table not exist 06/08 20:35:33.020489 00023174 MND ERROR msg:0x7f6e44000d60, app:0x678 table:0.prometheus.md5_ed032e52a5b5bb7a4b190fad66074a46, failed to get table meta, table not exist
文档说库和表会自动创建,但实际log显示都没创建
I want to rebuild the project as directed by the link:https://www.taosdata.com/cn/documentation/insert/#Prometheus直接写入
But I cannot go build, and get ERR:
execSql Error: syntax error near ")
" sqlcmd: create table if not exists MD5_648e4536f5d4184314287d187677867c using node_network_address_assign_type tags("623716d73194f57a558f0d1cb4964378",)
I modify the file "server.go" and add a line after line 529:
line 529: sqlcmd = sqlcmdhead + md5V2(tagHash) + ""," + sqlcmd
line 530: sqlcmd = strings.Replace(sqlcmd, ",)\n", ")\n", -1)
But it still cannot run normally after modification. Attached is the generated error log, please help update the code or indicate whether there is a problem in my configuration.
This is the table of project configuration:
ServerName | IP | Port | APP |
---|---|---|---|
taos1 | 10.197.190.17 | 6030~6042 | Tdengine 2.0.3.1 |
taos2 | 10.197.190.18 | 6030~6042 | Tdengine 2.0.3.1 |
taos3 | 10.197.190.19 | 6030~6042 | Tdengine 2.0.3.1 |
app | 10.197.190.20 | 3000 | Grafana 7.1.5 |
Telegraf | |||
6051 | blm_telegraf | ||
Prometheus | 10.197.190.21 | 9090 | Prometheus |
6052 | blm_prometheus |
blm_prometheus.log
prometheus.yml.log
server.go.log
taosdlog.0.log
由于网络等客观原因,部分用户自行编译存在难以克服的困难,如果能够提供支持主流操作系统平台的预编译二进制文件,会更好的推广taosd。期待答复
../../../../gowork/pkg/mod/github.com/taosdata/[email protected]/taosSql/result.go:21:10: fatal error: 'taos.h' file not found
#include <taos.h>
^~~~~~~~
1 error generated.
使用 bailongma master 以及 prometheus v2.23.0 编译,修改部分语法错误(prometheus v2.23.0同v2.5.0部分struct不兼容) 运行后,内存持续增长( v2.5.0 也有同样的问题,所以才升级到 v2.23.0 测试)
运行: go tool pprof 'http://localhost:8080/debug/pprof/heap'
File: blm_prometheus
Build ID: 1d27bd0ba96f37057ab7eb9aa9f5e516fd4b7cf2
Type: inuse_space
Time: Dec 24, 2020 at 5:31pm (CST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 1370.07MB, 97.38% of 1406.88MB total
Dropped 32 nodes (cum <= 7.03MB)
Showing top 10 nodes out of 31
flat flat% sum% cum cum%
690.07MB 49.05% 49.05% 1001.09MB 71.16% github.com/prometheus/prometheus/prompb.(*TimeSeries).Unmarshal
311.02MB 22.11% 71.16% 311.02MB 22.11% github.com/prometheus/prometheus/prompb.(*Label).Unmarshal
166.04MB 11.80% 82.96% 166.04MB 11.80% bufio.NewWriterSize (inline)
114.43MB 8.13% 91.09% 114.43MB 8.13% bufio.NewReaderSize
52.51MB 3.73% 94.82% 1053.59MB 74.89% github.com/prometheus/prometheus/prompb.(*WriteRequest).Unmarshal
9.50MB 0.68% 95.50% 9.50MB 0.68% runtime.malg
9.50MB 0.68% 96.18% 12.50MB 0.89% net/textproto.(*Reader).ReadMIMEHeader
7.50MB 0.53% 96.71% 92.10MB 6.55% net/http.(*conn).readRequest
7MB 0.5% 97.21% 27.50MB 1.95% net/http.readRequest
2.50MB 0.18% 97.38% 1056.09MB 75.07% main.main.func1
过一段时间后,运行
File: blm_prometheus
Build ID: 1d27bd0ba96f37057ab7eb9aa9f5e516fd4b7cf2
Type: inuse_space
Time: Dec 24, 2020 at 5:38pm (CST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 1743.78MB, 97.58% of 1787.07MB total
Dropped 34 nodes (cum <= 8.94MB)
Showing top 10 nodes out of 31
flat flat% sum% cum cum%
872.86MB 48.84% 48.84% 1262.88MB 70.67% github.com/prometheus/prometheus/prompb.(*TimeSeries).Unmarshal
390.02MB 21.82% 70.67% 390.02MB 21.82% github.com/prometheus/prometheus/prompb.(*Label).Unmarshal
212.18MB 11.87% 82.54% 212.18MB 11.87% bufio.NewWriterSize (inline)
147.56MB 8.26% 90.80% 147.56MB 8.26% bufio.NewReaderSize
66.65MB 3.73% 94.53% 1329.53MB 74.40% github.com/prometheus/prometheus/prompb.(*WriteRequest).Unmarshal
15MB 0.84% 95.37% 18MB 1.01% net/textproto.(*Reader).ReadMIMEHeader
14.51MB 0.81% 96.18% 14.51MB 0.81% runtime.malg
12MB 0.67% 96.85% 126.14MB 7.06% net/http.(*conn).readRequest
9MB 0.5% 97.35% 37.51MB 2.10% net/http.readRequest
4MB 0.22% 97.58% 1333.53MB 74.62% main.main.func1
系统运行环境:
Linux xxx 3.10.0-1127.19.1.el7.x86_64 #1 SMP Tue Aug 25 17:23:54 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
CentOS Linux release 7.9.2009 (Core)
go version go1.15.2 linux/amd64
现有的 tdengine/blm_prometheus镜像版本太老了,重新发布一个吧
prometheus 写入 TDEngine 后,原有几百个grafana 表达式都需要改成 类sql 形式,这个有工具可以处理吗,数量太多了
panic: Checktable error value type
goroutine 22 [running]:
main.SerilizeTDengine(0xc00059c5d0, 0xc0005956d0, 0xc, 0xc00059c600, 0x177289bb8b0, 0x736fdd, 0x8, 0xc000210000, 0xe, 0xc0005eb050, ...)
/root/blm_telegraf/server.go:346 +0xc6a
main.ProcessData(0xc0000f27e0, 0x1, 0x1, 0x736fdd, 0x8, 0xc000210000, 0xe, 0x0, 0x0)
/root/blm_telegraf/server.go:392 +0x1030
main.ProcessReq(0xc000580000, 0x4d2, 0x63f, 0xc000210000, 0xe, 0x0, 0x0)
/root/blm_telegraf/server.go:284 +0x3d5
main.NodeProcess(0x4, 0x8, 0x9)
/root/blm_telegraf/server.go:210 +0xce
created by main.main
/root/blm_telegraf/server.go:139 +0x164
host: tdengine:0 port: 10202 database: telegraf%
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.