jilieryuyi / wing-binlog-go Goto Github PK

wing-binlog-go is a mysql(mariadb) master-slave replication client, that uses replication protocol to read events from Mysql or MariaDB's binlog build on top of go-mysql, and send json data via Http, TCP、 Redis queue、kafka or subscribe. It allows you to receive events like insert, update and delete with the original data.

• wing-binlog-go is a high performance Mysql/MariaDB middleware
• wing-binlog-go is a light weight Mysql/MariaDB data watch system

wing-binlog-go can run on OSX, Linux and Windows system.

Installation:

• On Xnix, run ./bin/build.sh
• On Windows, run bin/build.bat

Project status The project is test with:

• Mysql 5.5, 5.6 and 5.7
• MariaDB 5.5, 10.0, 10.1 and 10.2
• golang version 1.8+
• consul v1.0.2

Important Mysql server settings

[mysqld]
expire_logs_days         = 10
max_binlog_size          = 100M
server-id		 = 1
log_bin			 = /var/log/mysql/mysql-bin.log
binlog-format            = row

The system variables server_id, log_bin and binlog-format are madantory, and row based replication is required for the project.

Features:

• TCP protocol support
• Http protocal support
• Support client group, broadcast and filters

Use cases:

• MySQL to NoSQL replication
• MySQL to search engine replication, such as elasticsearch
• Realtime analytics
• Audit

Usage:

• Help: ./wing-binlog-go -help
• Start service: ./wing-binlog-go
• Show version information: ./wing-binlog-go -version
• Stop service: ./wing-binlog-go -stop
• Reload service: ./wing-binlog-go -reload all|http|tcp

Known issues:

1. The underlying package go-mysql does not support compressed binlog
2. Schema changes after initial binlog position may cause incorrect json package

Special thanks:

• https://github.com/go-sql-driver/mysql
• https://github.com/siddontang/go-mysql
• https://github.com/BurntSushi/toml
• https://github.com/takama/daemon
• https://github.com/hashicorp/consul
• https://github.com/sirupsen/logrus

Contributors

• jilieryuyi
• mia0x75

Help:

QQ group: 535218312 document