bbdshow / qelog Goto Github PK

Lightweight, high performance, low cost logging service. Support alarm, metrics statistics, quick query, access and so on

License: MIT License

Go 95.69% Dockerfile 0.58% Makefile 0.46% HTML 2.98% Shell 0.30%

uber-zap lightweight high-performance deploy-simple alarm

qelog's Introduction

Qelog

Qelog is a small, low cost and light operation and maintenance of the log service. The purpose of its birth is to solve the problem of small and medium-sized team multi-service group log and alarm.

It has been running stably for 2+ years and has been rolling over 100+TB of data.

中文文档

admin manager example address username：admin passwd：123456

Log System Features:

Client

qezap is the Go version Client of the project. Wrap Uber-zap ,Extension for local and remote implementations of WriteSyncer.
Local extension: log cutting, compression, retention period, dynamic switching level, etc.
Remote expansion: The buffer is packed and compressed for transmission log data, and the error backup is retried to ensure that the data is not lost. The GRPC(default) HTTP protocol is supported, and the I/O control ensures the bandwidth usage is controllable and the memory usage is low.

Log receiver server

The Receiver process can be expanded horizontally to ensure high availability and high performance.
Configure multiple storage instances on the Receiver to improve the storage capacity and write performance of the cluster.
Alarm module detects alarm rules for each log. The hit can be delivered according to the rules and different alarm methods. Currently supported DingTalk | Telegram
Implement data fragmentation storage rules, support automatic capacity management, monitoring and early warning. Store separate instances of extensions without bottlenecks due to middleware.
Log statistics, level distribution, and trend report.

Log manager server

Based on the vue-element-admin modification,support rich query dimensions, such as level, keyword, TraceId, ClientIP, multi-level conditions, etc. Due to cost and performance issues ** Full-text indexing ** is currently not supported
Friendly operation interaction, simple configuration, efficient content display, almost can be done out of the box quick used.
Cluster capacity statistics query, manual intervention, and other functions.

Performance test tool

/tools Includes benchmark and memory footprint tests.
Different operating environments and configurations have different performance. Therefore, you can conduct a comprehensive analysis to preliminarily check whether the service meets the requirements of scenarios.

Design drawing

Usage

Qezap Client import your project

go get -u github.com/bbdshow/qelog/qezap

Client use example

Quick Deploy

default use single node deploy

Docker Deploy

git clone https://github.com/bbdshow/qelog.git
cd qelog
# build docker image
make image
# docker-compose start container
docker-compose up -d

Binary Deploy

git clone https://github.com/bbdshow/qelog.git
cd qelog
make
cd bin
# ./qelog or nohup ./qelog &

Service cluster deployment

Cluster deployment time, pay attention to mongo configuration file and qelog service running mode "cluster_admin" | "cluster_receiver", use docker-compose.yaml for cluster layout...

Thank you for your support. If it is useful to you, I hope the Star can support you. If you have any questions, please Issues, keep updating and solve problems.

qelog's People

Contributors

Stargazers

Watchers

Forkers

isgasho todo233 zz quenlang 13meimei showsmall ninefive ajunlonglive geqiaoyu xpp777 wooveep

qelog's Issues

无法访问示例网站

按文档部署服务端失败

我尝试了docker版和二进制版都失败了。是否是部署文档没有更新，或者我是步骤错了。

docker版

git clone https://github.com/bbdshow/qelog.git
cd qelog
# build docker image
make image
# docker-compose start container
docker-compose up -d


docker-compose up -d

[+] Running 1/2
 ✘ qelog Error                                                                              7.0s
 ⠙ mongodb Pulling                                                                          7.0s
Error response from daemon: pull access denied for qelog, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

二进制版

git clone https://github.com/bbdshow/qelog.git
cd qelog
make
cd bin

fatal: destination path 'qelog' already exists and is not an empty directory.
export GOPROXY="https://goproxy.io,direct"
mkdir -p ./bin && rm -r ./bin
mkdir -p ./bin/configs && cp -r configs ./bin
mkdir -p ./bin/web && cp -r web ./bin/web
go mod tidy
go: downloading github.com/bbdshow/bkit v0.3.8
go: downloading github.com/bbdshow/qelog/api v1.1.0
....

./qelog
current running server mode: single
2023/09/12 23:01:21 init GRPC pusher success
2023/09/12 23:01:23 Pusher:grpc rpc error: code = Unavailable desc = last connection error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:31082: connect: connection refused"
2023/09/12 23:01:23 Writer:remote push packet Push Unavailable
2023/09/12 23:01:23 Pusher:grpc rpc error: code = Unavailable desc = last connection error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:31082: connect: connection refused"
2023/09/12 23:01:23 Writer:packet retry PushPacket Push Unavailable
2023/09/12 23:01:24 Pusher:grpc rpc error: code = Unavailable desc = last connection error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:31082: connect: connection refused"
2023/09/12 23:01:24 Writer:packet retry PushPacket Push Unavailable
2023/09/12 23:01:25 Pusher:grpc rpc error: code = Unavailable desc = last connection error: connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:31082: connect: connection refused"

容器化部署Make报错

make image

+ mkdir -p ./bin
+ rm -r ./bin
+ go build -ldflags -s -o bin/receiver/qelog_receiver cmd/receiver/main.go
# golang.org/x/net/http2
/go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:417:45: undefined: os.ErrDeadlineExceeded
note: module requires Go 1.17
The command '/bin/sh -c set -eux;         mkdir -p ./bin && rm -r ./bin;         go build -ldflags "-s" -o bin/receiver/qelog_receiver cmd/receiver/main.go;         go build -ldflags "-s" -o bin/admin/qelog_admin cmd/admin/main.go;' returned a non-zero code: 2
make: *** [Makefile:32: image] Error 2