starcwang / canal_mysql_elasticsearch_sync Goto Github PK

数据库中的这个数据，同步时会报错。

你好，数据库数据一直处于变动中，第一次的时候设置结束主键，比如数据库主键目前最大是1000，但是数据库一直在运行中，我怕出现问题，把最大的主键设置1200，实际数据并没有到1200，那下次同步的时候主键是从多少开始呢，还有第一次手动运行全量之后，后面会自动运行同步数据吗？

maxPK问题: 很多时候maxPK ≠ totalCount. 这里用maxPK会导致进度计算不准确.

com.star.sync.elasticsearch.scheduling.CanalScheduling:
if (batchId != -1 && entries.size() > 0) {
entries.forEach(entry -> {
*****这里只能看到有update
logger.info("entrytype="+entry.getHeader().getEventType());
// entry.
if (entry.getEntryType() == EntryType.ROWDATA) {
publishCanalEvent(entry);
}
});
com.star.sync.elasticsearch.listener.UpdateCanalListener:
doSync这个操作都没执行过

什么情况呢？？

这里的条件应该用do while语句来防止第一次的时候数据为一条时不触发同步操作，因为查询出来的MinId跟MaxId是同一个值。

求助，RT

增量同步不断进行中，当手动触发了全量同步时，增量同步会暂停吗？如果会，当这次全量同步结束后，增量同步再次启动，会收到旧的数据变更事件吗？

运行./startup.sh ，提示: No such file or directory

目前可以通过springboot启动访问开放的api进行全量导入到es，但是我在更改数据库数据后，发现没有同步到es，控制台也没反应

Error starting ApplicationContext. To display the auto-configuration report re-run your application with 'debug' enabled.
[2024-02-22 21:10:37.076] [ERROR] [main] [o.s.boot.SpringApplication] --- Application startup failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'canalScheduling': Injection of resource dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'getCanalConnector' defined in class path resource [com/star/sync/elasticsearch/client/CanalClient.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.alibaba.otter.canal.client.CanalConnector]: Factory method 'getCanalConnector' threw exception; nested exception is com.alibaba.otter.canal.protocol.exception.CanalClientException: com.alibaba.otter.canal.protocol.exception.CanalClientException: java.net.ConnectException: Connection refused: connect
at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.postProcessPropertyValues(CommonAnnotationBeanPostProcessor.java:321)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1264)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543)
at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:693)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:360)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:303)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1118)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1107)
at com.star.sync.elasticsearch.CanalMysqlElasticsearchSyncApplication.main(CanalMysqlElasticsearchSyncApplication.java:20)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'getCanalConnector' defined in class path resource [com/star/sync/elasticsearch/client/CanalClient.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.alibaba.otter.canal.client.CanalConnector]: Factory method 'getCanalConnector' threw exception; nested exception is com.alibaba.otter.canal.protocol.exception.CanalClientException: com.alibaba.otter.canal.protocol.exception.CanalClientException: java.net.ConnectException: Connection refused: connect
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:599)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1173)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1067)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:513)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:208)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1138)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1066)
at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.autowireResource(CommonAnnotationBeanPostProcessor.java:518)
at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.getResource(CommonAnnotationBeanPostProcessor.java:496)
at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor$ResourceElement.getResourceToInject(CommonAnnotationBeanPostProcessor.java:627)
at org.springframework.beans.factory.annotation.InjectionMetadata$InjectedElement.inject(InjectionMetadata.java:169)
at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88)
at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.postProcessPropertyValues(CommonAnnotationBeanPostProcessor.java:318)
... 17 common frames omitted
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [com.alibaba.otter.canal.client.CanalConnector]: Factory method 'getCanalConnector' threw exception; nested exception is com.alibaba.otter.canal.protocol.exception.CanalClientException: com.alibaba.otter.canal.protocol.exception.CanalClientException: java.net.ConnectException: Connection refused: connect
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:189)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
... 34 common frames omitted
Caused by: com.alibaba.otter.canal.protocol.exception.CanalClientException: com.alibaba.otter.canal.protocol.exception.CanalClientException: java.net.ConnectException: Connection refused: connect
at com.alibaba.otter.canal.client.impl.ClusterCanalConnector.connect(ClusterCanalConnector.java:73)
at com.star.sync.elasticsearch.client.CanalClient.getCanalConnector(CanalClient.java:39)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162)
... 35 common frames omitted
Caused by: com.alibaba.otter.canal.protocol.exception.CanalClientException: java.net.ConnectException: Connection refused: connect
at com.alibaba.otter.canal.client.impl.SimpleCanalConnector.doConnect(SimpleCanalConnector.java:171)
at com.alibaba.otter.canal.client.impl.SimpleCanalConnector.connect(SimpleCanalConnector.java:97)
at com.alibaba.otter.canal.client.impl.ClusterCanalConnector.connect(ClusterCanalConnector.java:63)
... 41 common frames omitted
Caused by: java.net.ConnectException: Connection refused: connect
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Net.java:482)
at sun.nio.ch.Net.connect(Net.java:474)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:647)
at com.alibaba.otter.canal.client.impl.SimpleCanalConnector.doConnect(SimpleCanalConnector.java:132)
... 43 common frames omitted
Disconnected from the target VM, address: '127.0.0.1:64466', transport: 'socket'

Process finished with exit code 1

正在找这方面的解决方案，感觉这个很靠谱，感谢分享

我看批量导入从i每次加stepSize，但是如果最后一次加stepSize超过了maxPK最后一次就不会执行了。