Comments (4)
shizhengchao
commented on August 25, 2024
这个错不是驱动问题，是flink的connector没找到，在pom文件中引入相关依赖即可:
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-jdbc_${scala.binary.version}</artifactId>
<version>1.11.1</version>
什么是flink connector，可以看官方文档: https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/connectors.
from flink-cdc.
zzy-data-coder
commented on August 25, 2024
这个错不是驱动问题，是flink的connector没找到，在pom文件中引入相关依赖即可:
<groupId>org.apache.flink</groupId> <artifactId>flink-connector-jdbc_${scala.binary.version}</artifactId> <version>1.11.1</version>什么是flink connector，可以看官方文档: https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/connectors.
谢谢您，我在flink_home/lib下放了flink-connector-jdbc_2.11-1.11.2的connector解决了问题。但是有了一个新问题，还望指教。
问题描述：
在MySQL同步数据到postgresql时，执行flink job时报的错，
我的flink sql 如下
insert into dot(id,dotno,orders,channelname,etl_time) select id,dotno,orders,channelname,etl_time from dot_mysql;
报错如下：
2020-11-11 10:33:46
org.apache.flink.runtime.JobException: Recovery is suppressed by NoRestartBackoffTimeStrategy
at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.handleFailure(ExecutionFailureHandler.java:116)
at org.apache.flink.runtime.executiongraph.failover.flip1.ExecutionFailureHandler.getFailureHandlingResult(ExecutionFailureHandler.java:78)
at org.apache.flink.runtime.scheduler.DefaultScheduler.handleTaskFailure(DefaultScheduler.java:192)
at org.apache.flink.runtime.scheduler.DefaultScheduler.maybeHandleTaskFailure(DefaultScheduler.java:185)
at org.apache.flink.runtime.scheduler.DefaultScheduler.updateTaskExecutionStateInternal(DefaultScheduler.java:179)
at org.apache.flink.runtime.scheduler.SchedulerBase.updateTaskExecutionState(SchedulerBase.java:503)
at org.apache.flink.runtime.jobmaster.JobMaster.updateTaskExecutionState(JobMaster.java:386)
at sun.reflect.GeneratedMethodAccessor69.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:284)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:199)
at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152)
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
at akka.actor.ActorCell.invoke(ActorCell.scala:561)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
at akka.dispatch.Mailbox.run(Mailbox.scala:225)
at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.RuntimeException: Writing records to JDBC failed.
at org.apache.flink.connector.jdbc.internal.JdbcBatchingOutputFormat.checkFlushException(JdbcBatchingOutputFormat.java:142)
at org.apache.flink.connector.jdbc.internal.JdbcBatchingOutputFormat.writeRecord(JdbcBatchingOutputFormat.java:148)
at org.apache.flink.streaming.api.functions.sink.OutputFormatSinkFunction.invoke(OutputFormatSinkFunction.java:87)
at org.apache.flink.streaming.api.functions.sink.SinkFunction.invoke(SinkFunction.java:52)
at org.apache.flink.table.runtime.operators.sink.SinkOperator.processElement(SinkOperator.java:86)
at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:717)
at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:692)
at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:672)
at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:52)
at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:30)
at StreamExecCalc$249.processElement(Unknown Source)
at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:717)
at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:692)
at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:672)
at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:52)
at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:30)
at org.apache.flink.streaming.api.operators.StreamSourceContexts$ManualWatermarkContext.processAndCollect(StreamSourceContexts.java:305)
at org.apache.flink.streaming.api.operators.StreamSourceContexts$WatermarkContext.collect(StreamSourceContexts.java:394)
at com.alibaba.ververica.cdc.debezium.internal.DebeziumChangeConsumer.emitRecords(DebeziumChangeConsumer.java:159)
at com.alibaba.ververica.cdc.debezium.internal.DebeziumChangeConsumer.emitRecordsUnderCheckpointLock(DebeziumChangeConsumer.java:145)
at com.alibaba.ververica.cdc.debezium.internal.DebeziumChangeConsumer.handleBatch(DebeziumChangeConsumer.java:113)
at io.debezium.embedded.ConvertingEngineBuilder.lambda$notifying$2(ConvertingEngineBuilder.java:81)
at io.debezium.embedded.EmbeddedEngine.run(EmbeddedEngine.java:812)
at io.debezium.embedded.ConvertingEngineBuilder$2.run(ConvertingEngineBuilder.java:170)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Writing records to JDBC failed.
at org.apache.flink.connector.jdbc.internal.JdbcBatchingOutputFormat.checkFlushException(JdbcBatchingOutputFormat.java:142)
at org.apache.flink.connector.jdbc.internal.JdbcBatchingOutputFormat.flush(JdbcBatchingOutputFormat.java:167)
at org.apache.flink.connector.jdbc.internal.JdbcBatchingOutputFormat.lambda$open$0(JdbcBatchingOutputFormat.java:120)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
... 3 more
Caused by: java.lang.RuntimeException: Writing records to JDBC failed.
... 10 more
Caused by: java.lang.RuntimeException: Writing records to JDBC failed.
... 10 more
Caused by: java.lang.RuntimeException: Writing records to JDBC failed.
... 10 more
Caused by: java.lang.RuntimeException: Writing records to JDBC failed.
... 10 more
from flink-cdc.
shizhengchao
commented on August 25, 2024
你可以调试下 JdbcBatchingOutputFormat的open方法，在open的时候就有异常了，然后writeRecord中，checkFlushException发现 flushException 不为null， 启动时把断点打到flush方法，看下是什么异常。
from flink-cdc.
leonardBang
commented on August 25, 2024
This should be related to sink
from flink-cdc.
Related Issues (20)
- SQL Server does not support scan. startup. mode: initial only HOT 2
- flink cdc 2.4.2版本关于报错io.debezium.DebeziumException: Failed to deserialize data of EventHeaderV4 的问题 HOT 1
- Exception in thread "main" java.lang.NoSuchMethodError: 'void org.apache.flink.streaming.runtime.operators.sink.CommitterOperatorFactory.<init>(org.apache.flink.api.connector.sink2.TwoPhaseCommittingSink, boolean, boolean)' HOT 3
- Information that has disappeared from github appears on the web page. HOT 1
- DataStreamSource oracle ID HOT 1
- Information that has disappeared from github appears on the web page. HOT 2
- supported mysql8.4? HOT 1
- Flink 1.18.1 and 1.19.0 having build jdk 1.8 and causing incompatibilities with Java 17 HOT 3
- update operation does not take effect HOT 5
- mysql pipeline connector lost table which database and table with the same name
- Caused by: java.lang.ClassCastException: cannot assign instance of org.apache.doris.flink.cfg.DorisOptions to field org.apache.flink.cdc.connectors.doris.sink.DorisMetadataApplier.dorisOptions of type org.apache.doris.flink.cfg.DorisOptions in instance of org.apache.flink.cdc.connectors.doris.sink.DorisMetadataApplier HOT 1
- flink-cdc sql sqlserver HOT 4
- [ERROR] Could not execute SQL statement. Reason: java.io.StreamCorruptedException: unexpected block data HOT 1
- lost data when gtid with Multiple transaction id on mgr cluster HOT 3
- flinkcdc3 official website operation error HOT 5
- When restoring a task from a savepoint, tables that were previously synchronized but later deleted still exist in a state HOT 1
- java.lang.IllegalStateException: Failed to send request to coordinator: com.ververica.cdc.runtime.operators.schema.event.ReleaseUpstreamRequest@5b433541 HOT 3
- addSplits only Snapshot，not foud binlog，so, incrementing data don't sync to sink,need help HOT 2
- 驱动程序无法通过使用安全套接字层(SSL)加密与 SQL Server 建立安全连接。错误:“Unexpected rethrowing”。 HOT 1
- flink1.18.1 + flink-cdc3.0.1 无法通过配置scanNewlyAddedTableEnabled=true实现无需重启作业能够识别符合tableList(正则表达式)的表采集数据 HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from flink-cdc.