api7 / apisix-seed Goto Github PK

When the password of nacos is ‘xQ+/OfYv+q’, the apisix-seed will not start.

discovery:
  nacos:
    host:
      - "http://reader:xQ+/[email protected]:80"
    prefix: /nacos
    weight: 100
    timeout:
      connect: 2000
      send: 2000
      read: 5000

I hope that the account and password will try a new field for configuration.

检测到 api7/apisix-seed 一共引入了112个开源组件，存在2个漏洞

漏洞标题：Buger Jsonparser 安全漏洞
缺陷组件：github.com/buger/[email protected]
漏洞编号：CVE-2020-35381
漏洞描述：Buger Jsonparser是Buger个人开发者的一个基于Go语言的用于与json格式数据进行交互的代码库。
jsonparser 1.0.0 存在安全漏洞，该漏洞允许攻击者可利用该漏洞通过GET调用导致拒绝服务。
影响范围：(∞, 1.1.1)
最小修复版本：1.1.1
缺陷组件引入路径：github.com/api7/apisix-seed@->github.com/buger/[email protected]

另外还有2个漏洞，详细报告：https://mofeisec.com/jr?p=a4d102

Issue Description

I have been using Apisix-seed with ZooKeeper for my project, and I've encountered an issue with registering multiple instances in different JSON objects within a single node.

Expected Behavior

I expected Apisix-seed to handle multiple instances registering themselves in different JSON objects within a node.

Actual Behavior

Instead, Apisix-seed only accepts a single JSON object with a list of service instances info(ip,port,weight) for registration.

Possible Solution

One possible solution to address this issue could be to modify the Apisix-seed code to support multiple JSON objects within a node for registration.

apisix-seed在自己的容器里部署起来了，但是没有日志输出，请问是什么原因？

Since APISiX configuration information is stored in etcd, we also need to have the ability to interact with etcd. We need to get the service discovery-related configuration items, which determine what the program should do. Since APISIX can dynamically change its configuration at runtime, we need to be able to read the data in etcd and watch it for updates. Finally, we also need to be able to modify the configuration entries in etcd so that APISIX can obtain the results of service discovery.

We can refer to the relevant implementation in APISIX-Dashboard:
apisix-dashboard/storage

https://github.com/api7/apisix-seed/blob/main/internal/discoverer/zookeeper.go#L110-L116

When the apisix creates an upstream with service discovery , this upstream couldn't be recognized by the apisix-seed.

I checked the source code and I found that when initializing A6Conf, it is not possible to unmarshal the upstream.

Hope this bug can be solved quickly.

//   internal/core/message/a6conf.go
func NewA6Conf(value []byte) (*A6Conf, error) {
	a6 := &A6Conf{
		All: make(map[string]interface{}),
	}
	err := unmarshal(value, a6)
	if err != nil {
		return nil, err
	}
	return a6, nil
}

func unmarshal(data []byte, v *A6Conf) error {
	err := json.Unmarshal(data, v)
	if err != nil {
		return err
	}

	err = json.Unmarshal(data, &v.All)
	if err != nil {
		return err
	}

	return nil
}

The Watcher needs to listen for changes in the configuration of the relevant entity (upstream/route/service) in etcd and filter out the content of the configuration that is relevant for service discovery. Watcher needs to construct service discovery query messages based on the change events and pass them to the corresponding Discoverer for processing.

We should support dns for endpoint of etcd and any other Service Discovery Center. This is more friendly to containerized deployments

When we logout all instances in nacos via api like:

curl -X DELETE 'http://127.0.0.1:8848/nacos/v1/ns/instance?serviceName=APISIX-NACOS&ip=127.0.0.1&port=10000'

We expect the node information in etcd to be updated as well, but it not.

ETCDConfig.Timeout is not assigned a value from conf, resulting in the timeout in the configuration file never taking effect.

source code: https://github.com/api7/apisix-seed/blob/bd8fb778b32a3c18a01685fb88729935307ef54f/internal/conf/conf.go#L102C1-L109C2

From a practical point of view, the service discovery takes place in two places:

1. apisix discovery module;
2. apisix-seed;

Is it expected by design?
Why not completely take over service discovery on the CP side

Apisix-seed zookeeper:

path:
/zookeeper/serviceInstanceName

data:

[{
  "host": "127.0.0.1",
  "port": 8191
}]

Curator Service Discovery

https://curator.apache.org/docs/service-discovery#curator-service-discovery

path:

base path
       |_______ service A name
                    |__________ instance 1 id --> (serialized ServiceInstance)
                    |__________ instance 2 id --> (serialized ServiceInstance)
                    |__________ ...
       |_______ service B name
                    |__________ instance 1 id --> (serialized ServiceInstance)
                    |__________ instance 2 id --> (serialized ServiceInstance)
                    |__________ ...
       |_______ ...

ServiceInstance schema
https://curator.apache.org/apidocs/org/apache/curator/x/discovery/ServiceInstance.html

data:

{
  "name": "Service A name",
  "id": "127.0.0.1:20991",
  "address": "127.0.0.1",
  "port": 20991,
  "payload": {},
  "serviceType": "DYNAMIC"
}

We need to package the nacos-sdk-go to implement the nacos discoverer. We need to support service discovery, service subscription and unsubscription. Discoverer needs to pass messages to other components through the defined message format in apisix-seed/comm.

Hi, does this line of code need to determine whether the length of the msg is 0?

just like this:

if len(msgs) > 0 {
    ch <- msgs
}

Rewriter will act as a single point component for writing to etcd for future operations such as privilege control on Discoverer. When the Discoverer gets the latest service discovery information, it will pass the data to the Rewriter, which will do the final write to etcd.

The first one:

The second one:

Zookeeper autodiscovery with apisix-seed encounters the following errors

[error] 51#51: 4 [lua] config_etcd.lua:535: load_full_data(): failed to check item data of [/apisix/routes] err:property "upstream" validation failed: additional properties forbidden, found _discovery_type ,val: {"create_time":1722587116,"upstream":{"scheme":"http","_discovery_type":"zookeeper","hash_on":"vars","_service_name":"APISIX-ZK","pass_host":"pass","nodes":[{"priority":0,"weight":100,"host":"10.20.210.37","port":30103}],"type":"roundrobin"},"hosts":["test.apisix.web"],"name":"test_zk","status":1,"uri":"/","id":"1","priority":0,"update_time":1722587116}, context: init_worker_by_lua*

[error] 50#50: *1945 [lua] init.lua:548: handle_upstream(): failed to set upstream: discovery zookeeper is uninitialized,

We can't assume the id is "/apisix/routes/1", as the prefix is configurable. We should use $prefix/routes/$name

What is the best practice to ensure high availability when deploying apisix-seed in a production environment？

zkclient : create /zookeeper/SoaInvokerService '[{"host":"127.0.0.1","port":7980,"weight":100}]'

admin api comand:

curl http://127.0.0.1:9180/apisix/admin/routes/1
-H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -i -d '
{
"uri": "/zk/test1",
"plugins": {
"grpc-transcode": {
"proto_id": "1",
"service": "SoaInvokerService",
"method": "Call"
}
},
"upstream": {
"scheme": "grpc",
"service_name": "SoaInvokerService",
"type": "roundrobin",
"discovery_type": "zookeeper"
}
}'

1. add a new upstream in apisix dashboard as follow before start apisix-seed, config preview like this:
   { "timeout": { "connect": 6, "send": 6, "read": 6 }, "type": "roundrobin", "scheme": "grpc", "discovery_type": "nacos", "pass_host": "pass", "name": "apisix-seed", "service_name": "share.ShareService", "keepalive_pool": { "idle_timeout": 60, "requests": 1000, "size": 320 } }
2. start apisix-seed with nacos as discovery， apisix-seed will rewrite nodes to etcd, and i get upstream config like this:
   { "_discovery_type": "nacos", "_service_name": "share.ShareService", "create_time": 1699364548, "id": "486262354254758591", "keepalive_pool": { "idle_timeout": 60, "requests": 1000, "size": 320 }, "name": "apisix-seed", "nodes": [ { "host": "10.66.36.202", "port": 5018, "weight": 10 } ], "pass_host": "pass", "scheme": "grpc", "timeout": { "connect": 6, "read": 6, "send": 6 }, "type": "roundrobin", "update_time": 1699364548 }

it added field “_discovery_type” and "_service_name" which caused apisix err:
[error] 51#51: *251621 [lua] config_etcd.lua:843: failed to fetch data from etcd: failed to check item data of [/apisix/upstreams] err:additional properties forbidden, found _discovery_type, etcd key: /apisix/upstreams, context: ngx.timer

apisix version: 3.6.0

We are using apisix seed, and after deploying the application for a period of time, the application will experience a CPU usage rate of 100%. It was found that this is because the watcher.handleWatch method will receive endless messages from empty etcd. Here is the log:

"level":"debug","time":"2023-04-17T15:31:15.808098376+08:00","caller":"storer/etcd.go:209","message":"etcd watch prefix[/SERVICES/apisix/routes] event: {{16855301504475757525 1118403134429716127 0 134530 {} [] 0} [] 459241403 true false <nil> }"}
{"level":"info","time":"2023-04-17T15:31:15.80901158+08:00","caller":"components/watcher.go:109","message":"Watcher handleWatch receive msgs from etcd: routes, 0, []"}
{"level":"info","time":"2023-04-17T15:31:15.80903039+08:00","caller":"components/watcher.go:109","message":"Watcher handleWatch receive msgs from etcd: routes, 0, []"}
// repeat output Watcher handleWatch receive msgs from etcd: routes, 0

Then we added the judgment that the message is empty in etcd.Watch:

// Watch for changes on a key
func (s *EtcdV3) Watch(ctx context.Context, prefix string) <-chan []*message.Message {
	eventChan := s.client.Watch(ctx, prefix, clientv3.WithPrefix())
	ch := make(chan []*message.Message, 1)

	go func() {
		defer close(ch)

		for event := range eventChan {
+			logger.Debugf("etcd watch prefix[%s] event: %v", prefix, event)
			msgs := make([]*message.Message, 0, 16)

			for _, ev := range event.Events {
+				logger.Infof("watch changed, key: %s, version: %d", key, ev.Kv.Version)
				// ...
			}

+			if len(msgs) > 0 {
+				logger.Infof("Watcher handleWatch receive msgs from etcd: %+v, %+v", len(msgs), msgs)
+				ch <- msgs
+			}
		}
	}()

	return ch
}

From the log, if len(msgs) > 0 is working, because it doesn't output "Watcher handleWatch receive msgs xxx" with caller storer/etcd.go

And my watcher.handleWatch looks like this:

func (w *Watcher) handleWatch(s *storer.GenericStore) {
	logger.Debugf("Watcher handleWatch: %+v", s)
	ch := s.Watch()

	for {
		select {
		case <-w.ctx.Done():
			logger.Infof("Watcher handleWatch exit")
			return
		case msgs := <-ch: // receive msgs from etcd
			wg := sync.WaitGroup{}
			wg.Add(len(msgs))
+			logger.Infof("Watcher handleWatch receive msgs from etcd: %+v, %+v", len(msgs), msgs)
			for _, msg := range msgs {
				w.sem <- struct{}{}
				tc := tracer.NewTracer()
				ctx, span := tc.NewSpanConsumerKind(tc.RootCtx)
				go w.handleValue(ctx, msg, &wg, s)
				span.End()
			}
			wg.Wait()
		}
	}
}

I don't know what happened, and looking forward to your reply, thanks.

now ,apisix-seed work : APISIX-Seed subscribes the specified service name to the service registry to obtain changes to the corresponding service. by discovery of zk

if zk data changes, how guarantee timeliness ?