Comments (10)
配置serverTimeZone的作用有两个:
- 对debezium而言,它会先从mysql端获取默认的时区,如果没有则使用 配置的database.serverTimezone.
- 对flink-cdc而言,会用指定的serverTimeZone来转换timestamp的时间(但只对 sql 有效)
所以目前暂时DataStreaApi只满足了第一个,还需要自己参照 RowDataDebeziumDeserializeSchema的实现,把UTC时间转换成指定的serverTimeZone的时间
from flink-cdc.
你的 mysql的time_zone是什么?在mysql客户端上看下: show variable like '%zone%', 我的是 system_time_zone是CST, time_zone是SYSTEM。 然后mysql的字段是timestamp类型,启动cdc时,开始的snapshot没问题,更新后就会有时差,但是加上serverTimeZone为 “Asia/Shanghai”就没有问题。
from flink-cdc.
dbz默认都会把timestamp类型字段转换成UTC,如果需要转到指定时区,需要参考实现CustomConverter接口,对timestamp类型字段做处理,参考 #28
from flink-cdc.
你的 mysql的time_zone是什么?在mysql客户端上看下: show variable like '%zone%', 我的是 system_time_zone是CST, time_zone是SYSTEM。 然后mysql的字段是timestamp类型,启动cdc时,开始的snapshot没问题,更新后就会有时差,但是加上serverTimeZone为 “Asia/Shanghai”就没有问题。
我的mysql中的 system_time_zone也是CST, time_zone也是SYSTEM。加上.serverTimeZone("Asia/Shanghai")后,cdc启动后凡是字段是timestamp类型的都有时差
from flink-cdc.
dbz默认都会把timestamp类型字段转换成UTC,如果需要转到指定时区,需要参考实现CustomConverter接口,对timestamp类型字段做处理,参考 #28
需要自己写debezium的Converter是么?我直接使用debezium的时候是通过写converter实现时间转换的。
from flink-cdc.
dbz默认都会把timestamp类型字段转换成UTC,如果需要转到指定时区,需要参考实现CustomConverter接口,对timestamp类型字段做处理,参考 #28
的确是这样没错,但是我看了代码,servertimezone的配置就是启到了这个作用,把UTC的时间转换成指定的timezone的时间,所以,实现converter的方法和加server-time-zone的方法其实是一样的。
from flink-cdc.
你的 mysql的time_zone是什么?在mysql客户端上看下: show variable like '%zone%', 我的是 system_time_zone是CST, time_zone是SYSTEM。 然后mysql的字段是timestamp类型,启动cdc时,开始的snapshot没问题,更新后就会有时差,但是加上serverTimeZone为 “Asia/Shanghai”就没有问题。
我的mysql中的 system_time_zone也是CST, time_zone也是SYSTEM。加上.serverTimeZone("Asia/Shanghai")后,cdc启动后凡是字段是timestamp类型的都有时差
你用的是stream api, 你看下你是不是用了 RowDataDebeziumDeserializeSchema这个deserialize,这个sql是默认的,如果是stream api的话,要自己实现。类似下面:
SourceFunction<String> sourceFunction = MySQLSource.<String>builder() .hostname("localhost") .port(3306) .databaseList("inventory") // monitor all tables under inventory database .username("flinkuser") .password("flinkpw") .deserializer(new CoustomDebeziumDeserializationSchema()) // converts SourceRecord to String .build();
from flink-cdc.
配置serverTimeZone的作用有两个:
- 对debezium而言,它会先从mysql端获取默认的时区,如果没有则使用 配置的database.serverTimezone.
- 对flink-cdc而言,会用指定的serverTimeZone来转换timestamp的时间(但只对 sql 有效)
所以目前暂时DataStreaApi只满足了第一个,还需要自己参照 RowDataDebeziumDeserializeSchema的实现,把UTC时间转换成指定的serverTimeZone的时间
原来需要自己去自定义序列化后的结果去解决时区问题是么?
from flink-cdc.
你的 mysql的time_zone是什么?在mysql客户端上看下: show variable like '%zone%', 我的是 system_time_zone是CST, time_zone是SYSTEM。 然后mysql的字段是timestamp类型,启动cdc时,开始的snapshot没问题,更新后就会有时差,但是加上serverTimeZone为 “Asia/Shanghai”就没有问题。
我的mysql中的 system_time_zone也是CST, time_zone也是SYSTEM。加上.serverTimeZone("Asia/Shanghai")后,cdc启动后凡是字段是timestamp类型的都有时差
你用的是stream api, 你看下你是不是用了 RowDataDebeziumDeserializeSchema这个deserialize,这个sql是默认的,如果是stream api的话,要自己实现。类似下面:
SourceFunction<String> sourceFunction = MySQLSource.<String>builder() .hostname("localhost") .port(3306) .databaseList("inventory") // monitor all tables under inventory database .username("flinkuser") .password("flinkpw") .deserializer(new CoustomDebeziumDeserializationSchema()) // converts SourceRecord to String .build();
我用的是StringDebeziumDeserializationSchema,没找到使用RowDataDebeziumDeserializeSchema的demo
from flink-cdc.
配置serverTimeZone的作用有两个:
- 对debezium而言,它会先从mysql端获取默认的时区,如果没有则使用 配置的database.serverTimezone.
- 对flink-cdc而言,会用指定的serverTimeZone来转换timestamp的时间(但只对 sql 有效)
所以目前暂时DataStreaApi只满足了第一个,还需要自己参照 RowDataDebeziumDeserializeSchema的实现,把UTC时间转换成指定的serverTimeZone的时间
原来需要自己去自定义序列化后的结果去解决时区问题是么?
如果用DataStream API,的确是要自己去实现deserialize
from flink-cdc.
Related Issues (20)
- SQL Server does not support scan. startup. mode: initial only HOT 2
- flink cdc 2.4.2版本关于报错io.debezium.DebeziumException: Failed to deserialize data of EventHeaderV4 的问题 HOT 1
- Exception in thread "main" java.lang.NoSuchMethodError: 'void org.apache.flink.streaming.runtime.operators.sink.CommitterOperatorFactory.<init>(org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink, boolean, boolean)' HOT 3
- Information that has disappeared from github appears on the web page. HOT 1
- DataStreamSource oracle ID HOT 1
- Information that has disappeared from github appears on the web page. HOT 2
- supported mysql8.4? HOT 1
- Flink 1.18.1 and 1.19.0 having build jdk 1.8 and causing incompatibilities with Java 17 HOT 3
- update operation does not take effect HOT 5
- mysql pipeline connector lost table which database and table with the same name
- Caused by: java.lang.ClassCastException: cannot assign instance of org.apache.doris.flink.cfg.DorisOptions to field org.apache.flink.cdc.connectors.doris.sink.DorisMetadataApplier.dorisOptions of type org.apache.doris.flink.cfg.DorisOptions in instance of org.apache.flink.cdc.connectors.doris.sink.DorisMetadataApplier HOT 1
- flink-cdc sql sqlserver HOT 4
- [ERROR] Could not execute SQL statement. Reason: java.io.StreamCorruptedException: unexpected block data HOT 1
- lost data when gtid with Multiple transaction id on mgr cluster HOT 3
- flinkcdc3 official website operation error HOT 5
- When restoring a task from a savepoint, tables that were previously synchronized but later deleted still exist in a state HOT 1
- java.lang.IllegalStateException: Failed to send request to coordinator: com.ververica.cdc.runtime.operators.schema.event.ReleaseUpstreamRequest@5b433541 HOT 3
- addSplits only Snapshot,not foud binlog,so, incrementing data don't sync to sink,need help HOT 2
- 驱动程序无法通过使用安全套接字层(SSL)加密与 SQL Server 建立安全连接。错误:“Unexpected rethrowing”。 HOT 1
- flink1.18.1 + flink-cdc3.0.1 无法通过配置scanNewlyAddedTableEnabled=true实现无需重启作业能够识别符合tableList(正则表达式)的表采集数据 HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from flink-cdc.