Comments (4)
shizhengchao
commented on August 25, 2024
这的确是一个问题，在这篇文章中提到了 https://www.jianshu.com/p/66adedf25ee9?utm_campaign=haruki ，目前这个项目使用的debezium不是最新的。
这个应该是debezium再加的功能，不确定最新版本有没有实现：
修改同步任务的table.whitelist后，debezium可以自动完成新增表的全量和增量同步，并且这个过程不会干扰原有的同步任务；当新老两批同步任务进度相近时，合二为一，只使用同一个BinlogReader完成后续的stream同步。
这个可能是一个优化点。 @wuchong
from flink-cdc.
zanmeipaul
commented on August 25, 2024
@shizhengchao 谢谢你的回答，我今天试了debezium 1.3.0 版本，确实有这个功能。 目前文档上没有开放出来，默认是关闭的，我设置了 snapshot.new.tables 为 parallel 后生效。 目前不知这个功能是否稳定
from flink-cdc.
wuchong
commented on August 25, 2024
我们会在近期升级 debezium 依赖。
from flink-cdc.
PatrickRen
commented on August 25, 2024
Closing this issue because it was created before version 2.3.0 (2022-11-10). Please try the latest version of Flink CDC to see if the issue has been resolved. If the issue is still valid, kindly report it on Apache Jira under project Flink with component tag Flink CDC. Thank you!
from flink-cdc.
Related Issues (20)
- SQL Server does not support scan. startup. mode: initial only HOT 2
- flink cdc 2.4.2版本关于报错io.debezium.DebeziumException: Failed to deserialize data of EventHeaderV4 的问题 HOT 1
- Exception in thread "main" java.lang.NoSuchMethodError: 'void org.apache.flink.streaming.runtime.operators.sink.CommitterOperatorFactory.<init>(org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink, boolean, boolean)' HOT 3
- Information that has disappeared from github appears on the web page. HOT 1
- DataStreamSource oracle ID HOT 1
- Information that has disappeared from github appears on the web page. HOT 2
- supported mysql8.4? HOT 1
- Flink 1.18.1 and 1.19.0 having build jdk 1.8 and causing incompatibilities with Java 17 HOT 3
- update operation does not take effect HOT 5
- mysql pipeline connector lost table which database and table with the same name
- Caused by: java.lang.ClassCastException: cannot assign instance of org.apache.doris.flink.cfg.DorisOptions to field org.apache.flink.cdc.connectors.doris.sink.DorisMetadataApplier.dorisOptions of type org.apache.doris.flink.cfg.DorisOptions in instance of org.apache.flink.cdc.connectors.doris.sink.DorisMetadataApplier HOT 1
- flink-cdc sql sqlserver HOT 4
- [ERROR] Could not execute SQL statement. Reason: java.io.StreamCorruptedException: unexpected block data HOT 1
- lost data when gtid with Multiple transaction id on mgr cluster HOT 3
- flinkcdc3 official website operation error HOT 5
- When restoring a task from a savepoint, tables that were previously synchronized but later deleted still exist in a state HOT 1
- java.lang.IllegalStateException: Failed to send request to coordinator: com.ververica.cdc.runtime.operators.schema.event.ReleaseUpstreamRequest@5b433541 HOT 3
- addSplits only Snapshot，not foud binlog，so, incrementing data don't sync to sink,need help HOT 2
- 驱动程序无法通过使用安全套接字层(SSL)加密与 SQL Server 建立安全连接。错误:“Unexpected rethrowing”。 HOT 1
- flink1.18.1 + flink-cdc3.0.1 无法通过配置scanNewlyAddedTableEnabled=true实现无需重启作业能够识别符合tableList(正则表达式)的表采集数据 HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from flink-cdc.