Code Monkey home page Code Monkey logo

flume-ng-sql-source-json's Introduction

flume-ng-sql-source-json

这个项目是改造于https://github.com/keedio/flume-ng-sql-source.git 因keedio的插件flume-ng-sql-source只支持csv的格式,如果开始同步之后,数据库表需要增减字段,则会给开发者造成很大的困扰。所以我添加了一个分支版本,用来将数据以JSON的格式,同步到kafka,字段语义更加清晰。

部署方式和flume-ng-sql-source的一致,参照:http://www.cnblogs.com/yucy/p/7845105.html

flume-ng-sql-source-json's People

Contributors

yucy avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

flume-ng-sql-source-json's Issues

json格式转换报错

您好,请教一下,我把插件放入flume/lib目录下,运行的时候报下面的错误,是什么原因呢
图片

查询offset问题

哈喽,在HibernaterHelper.java这个类的184-193这块offset的代码是有问题吧?
if (!rowsList.isEmpty()){
if (sqlSourceHelper.isCustomQuerySet()){ sqlSourceHelper.setCurrentIndex(rowsList.get(rowsList.size()-1).get(sqlSourceHelper.getOrderBy()).toString());
}
else{ sqlSourceHelper.setCurrentIndex(Integer.toString((Integer.parseInt(sqlSourceHelper.getCurrentIndex())+ rowsList.size())));
}
}
因为这边一次查询10000条,而if (sqlSourceHelper.isCustomQuerySet())
这里不为空,那么这里将会一直在查询10000-20000这个范围的数据

无法准实时从mysql导入hdfs

你好,请教一下
我用的是
flume-release-1.7.0 + flume-ng-sql-source-1.5.0 + Hadoop 2.7.6 + MySQL 5.7.17
都是分别从源码编译打包部署的,
flume.conf如下:


agent1.channels = ch1
agent1.sinks = h1
agent1.sources = ss1

agent1.channels.ch1.type = memory

agent1.sources.ss1.channels = ch1
agent1.sources.ss1.type = org.keedio.flume.source.SQLSource

agent1.sources.ss1.hibernate.connection.url = jdbc:mysql://129.1.99.1:3306/test
agent1.sources.ss1.hibernate.connection.user = root
agent1.sources.ss1.hibernate.connection.password = root
agent1.sources.ss1.table = user
agent1.sources.ss1.start.from = 0
agent1.sources.ss1.custom.query = select * from user where id > $@$ order by id asc
agent1.sources.ss1.batch.size = 1000
agent1.sources.ss1.max.rows = 1000
agent1.sources.ss1.run.query.delay=15000
agent1.sources.ss1.hibernate.connection.driver_class = com.mysql.jdbc.Driver
agent1.sources.ss1.hibernate.dialect = org.hibernate.dialect.MySQL5Dialect
agent1.sources.ss1.connection.autocommit = true

agent1.sources.ss1.status.file.path = /opt/flume/status
agent1.sources.ss1.status.file.name = ss1.status

agent1.sinks.h1.channel = ch1
agent1.sinks.h1.type = hdfs
agent1.sinks.h1.hdfs.path = hdfs://129.1.99.5:9000/flume/mysql/%y-%m-%d/%H%M/%S
agent1.sinks.h1.hdfs.writeFormat = Text
agent1.sinks.h1.hdfs.rollSize = 268435456
agent1.sinks.h1.hdfs.rollInterval = 0
agent1.sinks.h1.hdfs.rollCount = 0
agent1.sinks.h1.hdfs.fileType = DataStream
agent1.sinks.h1.hdfs.threadsPoolSize = 10

agent1.sinks.h1.hdfs.filePrefix = evt-
agent1.sinks.h1.hdfs.useLocalTimeStamp = true
agent1.sinks.h1.hdfs.round = true
agent1.sinks.h1.hdfs.roundValue = 10
agent1.sinks.h1.hdfs.roundUnit = minute

agent1.sinks.h1.hdfs.retryInterval = 60


启动后最终会定时的打印
2018-05-16 17:37:17,588 (PollableSourceRunner-SQLSource-ss1) [DEBUG - org.hibernate.engine.jdbc.spi.SqlStatementLogger.logStatement(SqlStatementLogger.java:109)] select * from user where id > 89 order by id asc limit ?

sql中会多一个 ‘limit ?’
数据可以正常读取写入到hdfs,但是flume运行中在mysql添加新数据是无法准实时新增到hdfs中的

想请问下可能是哪里出了问题?
谢谢!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.