Code Monkey home page Code Monkey logo

nacos-sync's Introduction

Nacos Sync

Function

  • Console: provide API and console for management
  • Worker: provide the service registration synchronization.

Architecture

Architecture Topology

            +-------------+
     +----> |NacosClusterA|
     |      +-------------+               +-------------+
     |                                    |NacosClusterB|
Pull |                                    +--+----------+
Info |      +------------+                   ^
     |      |ZooKeeper   |                   |
     |      +--+---------+                   | Push Info
     |         ^ Pull Info                   |
     |         |                             |
     |        ++-----------------------------+--+
     <--------+  NacosSync1, NacosSync2,....    |
              +---+-------------------------+---+
                  |                         |
                  |                         |
                  |                         |
                  |       +---------+       |
                  +-----> |NacosSync| <-----+
                          |Database |
                          +---------+

Architecture HighLights

  • All registration information will be stored in NacosSync DB.
  • Multiple NacosSync instances will perform the same job.
    • Multiple NacosSync instances ensure high availability.
    • Multiple NacosSync instances performing the same job ensure the simplicity.
    • NacosCluster target will dedup the synchronization information from Nacos.

Quick Start:

NacosSync Migration User Guide

Support migration type

Source Target Support Note
Nacos Nacos Yes Only supports the same version of Nacos migration,especially the version of 0.8 migrates to 1.0 or above.
Nacos Zookeeper Yes Only support registery center of Dubbo
Nacos Consul Yes Only support registery center of Spring Cloud
Nacos Eureka Yes Only support registery center of Spring Cloud
Zookeeper Nacos Yes Only support registery center of Dubbo
Consul Nacos Yes Only support registery center of Spring Cloud
Eureka Nacos Yes Only support registery center of Spring Cloud

Manual Goal

  • Start the NacosSync service
  • Use a simple example to demonstrate how to migrate a Dubbo client registered in the Zookeeper Registry to the Nacos Registry

Prerequisites

Before you begin, install the following:

  • 64bit OS: Linux/Unix/Mac/Windows supported, Linux/Unix/Mac recommended.
  • 64bit JDK 1.8+: downloads, JAVA_HOME settings.
  • Maven 3.5.2+: downloads, settings.
  • MySql 5.6.+

Download & Build From Release

There are two ways to get NacosSync.

  • Download run package
  • Download source code from Github
cd nacos-sync/
mvn clean package -U

The path to the target file:

nacos-sync/nacossync-distribution/target/nacos-sync-0.5.0.tar.gz

After extracting the installation package, the directory structure:

nacos-sync
├── LICENSE
├── NOTICE
├── bin
│   ├── nacosSync.sql
│   ├── shutdown.sh
│   └── startup.sh
├── conf
│   ├── application.properties
│   └── logback-spring.xml
├── logs
└── nacos-sync-server.jar

Initialize The DB

The default is Mysql database, which can support other relational databases

  • Build db schema, the default schema name nacos_sync.
  • Tables do not need to be created separately, which is conducive to hibernate's automatic table creation function.
  • If the automatic table creation fails, you can build the table nacosSync.sql, the table statement is in the bin folder.

DB Configuration

In the bin folder, application.properties:

spring.datasource.url=jdbc:mysql://127.0.0.1:3306/nacos_sync?characterEncoding=utf8
spring.datasource.username=root
spring.datasource.password=root

Start Server

$ nacosSync/bin:
sh startup.sh  start

Admin Console

http://127.0.0.1:8083/#/serviceSync

Advanced Configuration

Full Synchronization from Zookeeper to Nacos (Dubbo)

When “*” is entered in the “Service Name” field of this form, it will fully synchronize all services from Zookeeper to Nacos, but only when using Dubbo. img_1.png

Full Synchronization from Nacos to Nacos

When “All” is entered in the “Service Name” field of this form, it will automatically synchronize all registered services within the default group of the current cluster.

This description explains the functionality clearly for English-speaking users. img_2.png

nacos-sync's People

Contributors

alios-studio avatar chenhao26-nineteen avatar cherishcai avatar davidlei08 avatar dependabot[bot] avatar dragontalon avatar fanyanming2016 avatar ijustyce avatar jefferson-chern avatar jerrygutripleup avatar loadchange avatar majorhe1 avatar mi-cool avatar mingyixu avatar nanamikon avatar oliverwqcwrw avatar paderlol avatar paderzhang avatar seancool avatar shalk avatar wyp12 avatar xyohn avatar yangyshdan avatar yanlinly avatar yongchao9 avatar zrlw avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

nacos-sync's Issues

Implement SPI on NacosSync for SyncManager

Implement SPI on NacosSync for SyncService.

Currently the logic of SyncManager is not elegant enough. We need to implement SPI for SyncService.

The ultimate goal is that,

  1. User implement XXXSyncService based on Sync Service.
  2. NacosSync Register XXX SyncService automatically through SPI mechanism.

maven仓库在pom里没有配置,是配置在本地的setting文件里了么

maven仓库在pom里没有配置,是配置在本地的setting文件里了么,有些插件下载不了

比如pom.xml中的以下插件:

<plugin>
                <groupId>com.github.vongosling</groupId>
                <artifactId>dependency-mediator-maven-plugin</artifactId>
                <version>1.0.2</version>
            </plugin>
            <plugin>
                <groupId>org.codehaus.mojo</groupId>
                <artifactId>clirr-maven-plugin</artifactId>
                <version>2.7</version>
            </plugin>

Compile error

Project compilation error is caused by internal dependence.

运行中遇到一些问题的汇总

版本:
最后一个commit 是 6bc3057

运行期间问题
1、程序中context已经改为nacossync, 但是前端还是用skywalker, 这样页面会出现404
2、SyncService这个接口已经定义了, 但是缺少一个地方在SyncManagerService中通过register方法来注册
3、目前最新版的nacos sdk, 是无法同步集群或者服务具体信息的, 这些是不是应该在nacos-sync做, 还是通过后台界面手动去做

NacosSync User Guide

We Need a NacosSync User Guide:
1、How to Install NacosSync Server(Cluster,Stand-alone)
2、How to Sync Data(Others to Nacos and Nacos to Others)
3、How to Add Sync Cluster

新功能:同步ZK数据同步到Nacos

背景

Dubbo在2.6.5版本支持了Nacos作为其服务发现框架,也就是,以后Dubbo用户,可以使用Nacos了,问题来了,Dubbo支持ZK作为其服务发现框架,并且占很大一部分比例,(Dubbo+zk)的存量用户想迁移到Dubbo+Nacos的模式,如何支持呢?

方案1:客户端全量升级

暴力模式,全量停机,把Provider和Consumer全部重新升级至Dubbo2.6.5版本,此方式适用于可接受停机的产品业务

优点

是不需要引入其他中心系统,
简单

缺点

有业务方场景限制

方案2:客户端全量升级+切流

统一接入层引流,这个要求业务系统的基础设置要比较完善,在统一接入层能够将流量切走,同时成本也比较大,需要一样规模的同一个集群进行切换升级,这个类似蓝绿发布,相比方案1,增加了切流及的操作,来达到业务不停机的目的
image

缺点

有业务方场景限制

方案3:客户端支持(多发布+多订阅)

Dubbo客户端支持多发布,多订阅模式,即Provider可以将自己的地址发布到Zk集群的同时,又发布到Nacos集群,Consumer订阅时,可以订阅Zk集群的服务地址同时也能订阅到Nacos集群的服务地址
image

优点

不需要引入其他中心系统

缺点

Dubbo客户端改动量大
客户端的可控性不在手上,难以推动快速落地
客户端有升级顺序限制(先升Provider,会导致老的Consumer调用不到,产生容量问题)

迁移步骤:

1、新搭建一个Nacos集群
2、必须先升级Dubbo Consumer客户端,可以订阅到2端集群的数据,并且聚合使用(这里不会出现负载不均的情况)
2、再Dubbo Provider升级客户端,此时Nacos集群中,Zk集群中,都有该Provider地址
4、关闭原ZK集群,升级完毕

方案4:NacosSync双向同步

注:“新”的意思是升级了Dubbo2.6.5连接到Naocs注册中心的客户端

新Dubbo Provider 老Dubbo Provider
新Dubbo Consumer 支持 支持
老Dubbo Consumer 支持 支持

优点

相比方案3,改动点都在NacosSync,可控

缺点

Nacos目前还不支持Nacos同步到zk,目前已经在做的开发计划支持zk同步到Nacos
双向同步功能支持时,需要对服务进行打标,避免循环同步,但是需要ZK能支持服务打标功能?目前没有
NacosSync的高可用模式下,不支持双向同步,高可用模式是指NaocsSync集群部署,原因是目前NacosSync同步到目标集群,是多台NacosSync重复同步,需要目标集群支持去重功能

迁移步骤

新搭建一个Nacos集群
新搭建一个NacosSync集群
升级老客户端到新客户端(Provider和Consumer无顺序升级要求)
升级完毕,关闭zk集群,关闭NacosSync集群

方案5:NacosSync单向同步

新Dubbo Provider 老Dubbo Provider
新Dubbo Consumer 支持 支持
老Dubbo Consumer 不支持 支持

最大问题是:没办法支持老的Dubbo Consumer订阅新的Dubbo Provider!

若无法解决这个问题,可操作性也没有!除非增加机器成本,即新Provider用新机器,这样老的DubboConsumer就不会因为容量问题出现风险

迁移步骤

1、 搭建Nacos集群
2、 搭建NacosSync集群
3、新申请Provider机器,连接到Nacos集群
4、老的Consumer升级到新版本Consumer
5、老的Provider升级到新版本Provider
6、升级完毕,关闭zk集群,关闭NacosSync集群

image

新Dubbo Provider 老Dubbo Provider
新Dubbo Consumer 支持 支持
老Dubbo Consumer 支持 支持

缺点

需要推动Dubbo Provider支持多发布功能

迁移步骤

1、搭建Nacos集群
2、搭建NacosSync集群
3、升级老的Provider到多发布版本
4、老的Consumer升级到新版本Consumer
5、升级完毕,关闭zk集群,关闭NacosSync集群

方案选型

注:高=3,中=2,低=1

开发成本 机器成本 风险成本 用户成本 维护成本 推动成本 场景限制 成本分
方案1 1 1 3 1 1 1 8
方案2 1 1 1 1 1 1 6
方案3 2 1 1 1 1 2 8
方案4 3 1 1 1 2 1 9
方案5 1 2 1 1 2 1 9
方案6 2 1 1 1 2 2 9

when deleting zk sync to Nacos, throw the NPE

Issue

when deleting zk sync to Nacos, throw the NPE

  • exception log:
019-01-28 17:35:37.652 [SkyWalker-Timer-schedule-pool-5] INFO  com.alibaba.nacossync.timer.QuerySyncTaskTimer - 从数据库中查询到一个删除任务,发出一个同步事件:TaskDO(id=2, taskId=3be2941c2587f760efa5e726771cdefd, sourceClusterId=a1959b18bc9857c4fc2377bfd32207fb, destClusterId=889f7787de0127418ac68c02056a4689, serviceName=com.paderlol.nacos.api.DemoService, version=1.0.0, groupName=zk, nameSpace=null, taskStatus=DELETE, workerIp=192.168.10.197, operationId=b3b047ed-27e2-4f46-a29e-bfed47fc01a4)
2019-01-28 17:35:49.700 [SkyWalker-Timer-schedule-pool-2] INFO  c.a.n.extension.holder.AbstractServerHolder - starting create cluster server,clusterId=889f7787de0127418ac68c02056a4689
2019-01-28 17:36:25.788 [SkyWalker-Timer-schedule-pool-2] ERROR c.a.n.e.impl.ZookeeperSyncToNacosServiceImpl - delete task from zookeeper to nacos was failed, taskId:37f67153bd3ade067690b76728c53331
java.lang.NullPointerException: null
	at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
	at com.alibaba.nacos.client.naming.core.HostReactor.getSerivceInfo0(HostReactor.java:199)
	at com.alibaba.nacos.client.naming.core.HostReactor.getServiceInfo(HostReactor.java:219)
	at com.alibaba.nacos.client.naming.NacosNamingService.getAllInstances(NacosNamingService.java:191)
	at com.alibaba.nacos.client.naming.NacosNamingService.getAllInstances(NacosNamingService.java:185)
	at com.alibaba.nacossync.extension.impl.ZookeeperSyncToNacosServiceImpl.delete(ZookeeperSyncToNacosServiceImpl.java:156)
	at com.alibaba.nacossync.extension.SyncManagerService.delete(SyncManagerService.java:49)
	at com.alibaba.nacossync.event.listener.EventListener.listenerDeleteTaskEvent(EventListener.java:74)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.google.common.eventbus.EventSubscriber.handleEvent(EventSubscriber.java:74)
	at com.google.common.eventbus.SynchronizedEventSubscriber.handleEvent(SynchronizedEventSubscriber.java:47)
	at com.google.common.eventbus.EventBus.dispatch(EventBus.java:322)
	at com.google.common.eventbus.EventBus.dispatchQueuedEvents(EventBus.java:304)
	at com.google.common.eventbus.EventBus.post(EventBus.java:275)
	at com.alibaba.nacossync.timer.QuerySyncTaskTimer$CheckRunningStatusThread.lambda$run$0(QuerySyncTaskTimer.java:86)
	at java.util.ArrayList.forEach(ArrayList.java:1257)
	at com.alibaba.nacossync.timer.QuerySyncTaskTimer$CheckRunningStatusThread.run(QuerySyncTaskTimer.java:71)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset$$$capture(FutureTask.java:308)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Optimize getting cluster connections

Issue

  1. AbstractServerHolder 's get method makes double lock operation
  2. AbstractServerHolder 's createServer method maybe invalid call in once request,eg: ZookeeperServerHolder

Summary

  1. Remove ReentrantLock from AbstractServerHolder class
  2. Using the Supplier class to lazily load the serverAddress

希望能提供批量同步功能

现状:
现在nacosSync只能手动一个个添加SyncTaskEvent,通过service地址去zk里寻址并匹配version和group。

问题:
原遗留项目provider配置不规范,有的没有group,有的没有version。通过人工比对来追加效率很低下,给zk->nacos推进带来困难。

对策:
能否与dubbo ops 联动(或引用部分实现), 达到同纬度检索功能(service,ip, application),通过选择service的方式来实现批量增加SyncTaskEvent功能。

添加全量同步功能

目前想将erueka切换成nacos,eureka上的服务比较多,按照目前的做法,得一个服务一个服务配置,配得要吐啊。建议添加全量同步功能。

NacosSync执行环境不同,导致同步异常

环境1
nacosSync : window
nacos: linux
zk:linux

结果:zk异常,且同步不成功
一个同步zk会多出一个与同步对应的奇怪的文件

环境2
nacosSync : linux
nacos: linux
zk:linux

结果:zk正常,且同步成功

Console Task Manager Optimize

目前用户使用同步功能时,对前端的交互会有gap,目前的“删除”,应该时“暂停”同步任务的意思,
同时,需要增加一个真实“删除”的语义

image

1、“删除”的文案改成“暂停”
2、操作里面增加一个“删除”的操作

java.lang.NullPointerException: task from zk--> nacos

2019-01-24 17:51:34.649 ERROR com.alibaba.nacossync.extension.impl.ZookeeperSyncToNacosServiceImpl Line:149 - sync task from zookeeper to nacos was failed, taskId:3475b1b5c930dfe1f4f19001155c7e87
java.lang.NullPointerException: null
at com.alibaba.nacossync.extension.impl.ZookeeperSyncToNacosServiceImpl.sync(ZookeeperSyncToNacosServiceImpl.java:93)
at com.alibaba.nacossync.extension.SyncManagerService.sync(SyncManagerService.java:55)
at com.alibaba.nacossync.event.listener.EventListener.listenerSyncTaskEvent(EventListener.java:60)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.google.common.eventbus.EventSubscriber.handleEvent(EventSubscriber.java:74)
at com.google.common.eventbus.SynchronizedEventSubscriber.handleEvent(SynchronizedEventSubscriber.java:47)
at com.google.common.eventbus.EventBus.dispatch(EventBus.java:322)
at com.google.common.eventbus.EventBus.dispatchQueuedEvents(EventBus.java:304)
at com.google.common.eventbus.EventBus.post(EventBus.java:275)
at com.alibaba.nacossync.timer.QuerySyncTaskTimer$CheckRunningStatusThread.lambda$run$0(QuerySyncTaskTimer.java:80)
at java.util.ArrayList.forEach(ArrayList.java:1249)
at com.alibaba.nacossync.timer.QuerySyncTaskTimer$CheckRunningStatusThread.run(QuerySyncTaskTimer.java:71)

Console FE project optimization && upgrading

  1. Remove cookie tool and use FE friendly localStorage to record i18n information;
  2. Upgrade projects all depend on the latest (Exception:react-redux,v5.1.1);
  3. Code style, follow the guidance of Alibaba f2e committee
  4. Introduce eslint strict specification project code style, install eslint-config-ali

Nacos集群多园区异地多活

tim 20181222231644

   目前生产上是两个城市的三个园区分别布署了三个注册中心集群,服务注册在某一园区的集群后,其他园区能够同步数据,实现异地多活的跨园区高可用方案。
   集群进行扩容时,新加的机器同步集群数据,F5添加新地址 实现优雅横向扩容。

nacos-sync log

when a sync task run,I get this log:[从数据库中查询到一个删除任务,发出一个同步事件:...],and noting else.The task is done,or failed,or running.I do not know anything.Should we get more infomation when the sync task started.

如何制定namespace??

debug display nameSpace is null

com.alibaba.nacossync.timer.QuerySyncTaskTimer Line:81 - 从数据库中查询到一个同步任务,发出一个同步事件:TaskDO(id=10, taskId=3475b1b5c930dfe1f4f19001155c7e87, sourceClusterId=8afc6b7d4758bc2bcdaa93ad2017a304, destClusterId=d0cb5c71a5576c5839ba6f341c90b84c, serviceName=ndev-sync, version=0, groupName=ndev, nameSpace=null, taskStatus=SYNC, workerIp=192.168.100.50, operationId=1a869552-842d-42da-9f03-c2bb39cf8a36)

两个关于启动的bug

  1. 本仓库README中,写启动命令是
$ nacosSync/bin:
sh startup.sh  restart

应该是

$ nacosSync/bin:
sh startup.sh  start
  1. 0.3.0版本, startup.sh 中:
JAVA_OPT="${JAVA_OPT} -jar ${BASE_DIR}/nacosSync-server.0.1.0.jar"

应该改为

JAVA_OPT="${JAVA_OPT} -jar ${BASE_DIR}/nacosSync-server.0.3.0.jar"

Eureka and Consul can only be valid once when syncing to Nacos

Issue

Eureka and Consul can only be valid once when syncing to Nacos
Because once after synchronization task execution, scheduled tasks will not perform the task synchronization off again

Solution

The new scheduled task to perform Eureka and Consul synchronization method, Eureka and Consul synchronization subscription class event to notify polling task

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.