api7 / amesh Goto Github PK

Create a service without endpoint can reproduce it.

When accessing non-service grid services in the sidecar proxy container of the apisix application, a 503 gateway is prompted, such as Baidu

Encountered a problem probably because the delete doesn't take effect inside apisix.

AmeshPluginConfig Resource

The resource AmeshPluginConfig is a custom resource that defines all the plugin configurations that will be used by the APISIX.
A minimum sample configuration is as follows:

apiVersion: apisix.apache.org/v1alpha1
kind: AmeshPluginConfig
metadata:
  name: go-plugins
spec:
  type: pre-req
  plugins:
  - name: say
    config: "{\"body\":\"hello\"}"

This CRD indicates that the routes should be injected with the following plugin configuration:

{
  // ... Route configurations ...
  "plugins": {
    "ext-plugin-pre-req": {
       "conf": [
         {"name":"say", "value":"{\"body\":\"hello\"}"}
       ]
    } 
  },
}

If you only want the plugin to run on specific workloads, you can add the selector field for filtering.

apiVersion: apisix.apache.org/v1alpha1
kind: AmeshPluginConfig
metadata:
  name: go-plugins
spec:
  type: pre-req
  plugins: 
  - name: say
    config: "{\"body\":\"hello\"}"
  # selectors
  nodeSelector:
    kubernetes.io/hostname: node1
  selector:
    matchLabels:
      component: redis
    matchExpressions:
    - key: zone
      operator: In
      values:
      - foo
      - bar
    - key: environment
      operator: NotIn
      values:
      - dev

For more information about selectors, please refer to Resources that support set-based requirements.
Custom Plugins
Apart from the official plugins, users can also add custom plugins to the Amesh.
To make the image used by the sidecar include user custom plugins, the user needs to build a new image based on Amesh's APISIX image and the user's plugins.
Test Plan
We need a complete and solid test suite to validate this functionality, including the following cases:

• Test the plugin configuration is injected into the routes
• Test the plugin with a custom selector
• Test the plugin definition updates are reflected

1、chang config property worker_shutdown_timeout to 10s in confi.yaml
2、exec docker run command start amesh apisix
3、exec docker exec command enter the container
4、exec apisix reload command in amesh apisix container
5、wait 10 seconds or more
6、exec ps -ef command show processes，will find there has one work process status is shutting down, as follows:

[mwopr@1a01vlc0386zzzz apisix]$ ps -ef
UID PID PPID C STIME TTY TIME CMD
mwopr 1 0 0 18:03 ? 00:00:00 nginx: master process /usr/local/openresty/nginx/sbin/nginx -p /usr/local/apisix -c conf/nginx.conf
mwopr 39 1 0 18:03 ? 00:00:00 nginx: worker process is shutting down
mwopr 204 1 0 18:07 ? 00:00:00 nginx: worker process
mwopr 205 1 0 18:07 ? 00:00:00 nginx: worker process
mwopr 206 1 0 18:07 ? 00:00:00 nginx: worker process
mwopr 207 1 0 18:07 ? 00:00:00 nginx: worker process
mwopr 208 1 0 18:07 ? 00:00:00 nginx: cache manager process
mwopr 209 1 0 18:07 ? 00:00:00 nginx: privileged agent process
mwopr 297 54 0 18:10 pts/0 00:00:00 ps -ef

E2E test also.

Start amesh apisix when xds control plane is unavailable will stuck nginx work process, the result is that no work process, all are master process, and there are no log, as follows:

[mwopr@1a01vlc0386zzzz apisix]$ ps -ef
UID PID PPID C STIME TTY TIME CMD
mwopr 1 0 0 17:39 ? 00:00:00 nginx: master process /usr/local/openresty/nginx/sbin/nginx -p /usr/local/apisix -c conf/nginx.conf
mwopr 39 1 1 17:39 ? 00:00:00 nginx: master process /usr/local/openresty/nginx/sbin/nginx -p /usr/local/apisix -c conf/nginx.conf
mwopr 40 1 1 17:39 ? 00:00:00 nginx: master process /usr/local/openresty/nginx/sbin/nginx -p /usr/local/apisix -c conf/nginx.conf
mwopr 41 1 1 17:39 ? 00:00:00 nginx: master process /usr/local/openresty/nginx/sbin/nginx -p /usr/local/apisix -c conf/nginx.conf
mwopr 42 1 1 17:39 ? 00:00:00 nginx: master process /usr/local/openresty/nginx/sbin/nginx -p /usr/local/apisix -c conf/nginx.conf
mwopr 43 1 0 17:39 ? 00:00:00 nginx: cache manager process
mwopr 44 1 0 17:39 ? 00:00:00 nginx: cache loader process
mwopr 45 1 1 17:39 ? 00:00:00 nginx: master process /usr/local/openresty/nginx/sbin/nginx -p /usr/local/apisix -c conf/nginx.conf
mwopr 75 53 0 17:39 pts/0 00:00:00 ps -ef

related to #26

Test consumer:

kn run consumer --image curlimages/curl --image-pull-policy IfNotPresent --command sleep 1d

We have two apps, first one is httpbin2, deployment via kubectl command line

kn create deployment httpbin2 --image kennethreitz/httpbin --port 80 
kn expose deployment httpbin2 --port 80

The second is httpbin3, deployed via a yaml file

apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin3
  labels:
    app: httpbin3
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin3
  template:
    metadata:
      labels:
        app: httpbin3
    spec:
      containers:
      - name: httpbin
        image: kennethreitz/httpbin
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80
          protocol: TCP
          name: http
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin3
spec:
  selector:
    app: httpbin3
  ports:
  - name: http
    targetPort: 80
    port: 80
    protocol: TCP

The httpbin3 works perfectly, but httpbin2 doesn't.
If we deploy httpbin2 ONLY, the xds response vhost.domains is "*" so the route doesn't work.

If we deploy httpbin3 AFTER httpbin2, once httpbin3 is deployed, the httpbin2 vhost with correct domains is generated. But there is a large number of xds calls and logs are generated in the amesh log. Also, the empty domains route still exists.

If we deploy httpbin3 BEFORE httpbin2, the final result is the same as above. The only difference is that no exceed logs generated.

When xds control plane restart, the grpc connection between amesh and xds control plane will break, then amesh will try reconnect to xds control plane immediately, only one time. If try failed, amesh won't try any more, then amesh will exit, result apisix cannot receive any more configuration change.

We should keep one of them has a higher priority, even in a delete event.