How to connect etcd cluster with tls about apisix-docker HOT 16 CLOSED

@gxthrj
I am sure that I can communicate with 10.1.7.51 2379 in the same container

This is the startup parameter of my etcd cluster. I wonder if it is related to the "--enable-v2=true" parameter.

root     11701  2.4  3.1 10888476 322624 ?     Ssl  11月22 321:14 /usr/local/bin/etcd --listen-client-urls=https://0.0.0.0:2379 \
--initial-advertise-peer-urls=https://10.1.7.51:2380 --key-file=/etc/kubernetes/ssl/kube-etcd-10-1-7-51-key.pem \ 
--peer-key-file=/etc/kubernetes/ssl/kube-etcd-10-1-7-51-key.pem --peer-client-cert-auth=true \ 
--initial-cluster-token=etcd-cluster-1 --name=etcd-master1 --cert-file=/etc/kubernetes/ssl/kube-etcd-10-1-7-51.pem \
--advertise-client-urls=https://10.1.7.51:2379,https://10.1.7.51:4001 --enable-v2=true --data-dir=/var/lib/rancher/etcd/ \ 
--listen-peer-urls=https://0.0.0.0:2380 --initial-cluster-state=new --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem \ 
--peer-cert-file=/etc/kubernetes/ssl/kube-etcd-10-1-7-51.pem --initial-cluster=etcd-master1=https://10.1.7.51:2380,etcd-worker1=https://10.1.7.52:2380,etcd-worker2=https://10.1.7.53:2380 \ 
--trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --client-cert-auth=true --election-timeout=5000 --heartbeat-interval=500

The option --client-cert-auth=true is used in your etcd server, which forces the clients offer their own certificate to meet the mTLS, which is not supported by APISIX.

If the mLTS is not required necessarily, just remove it.

from apisix-docker.

I used this example: https://github.com/apache/apisix-docker/tree/master/example

from apisix-docker.

I set etcd host as:

etcd:
  host:                           # it's possible to define multiple etcd hosts addresses of the same etcd cluster.
    - "https://10.1.7.51:2379,https://10.1.7.52:2379,https://10.1.7.53:2379"

appear：

/usr/local/openresty/luajit/bin/luajit: /usr/bin/apisix:619: attempt to index local 'base' (a nil value)
stack traceback:
	/usr/bin/apisix:619: in function 'merge_conf'
	/usr/bin/apisix:616: in function 'merge_conf'
	/usr/bin/apisix:616: in function 'merge_conf'
	/usr/bin/apisix:660: in function 'read_yaml_conf'
	/usr/bin/apisix:826: in function </usr/bin/apisix:819>
	/usr/bin/apisix:1166: in main chunk
	[C]: at 0x00404c00
/usr/local/openresty/luajit/bin/luajit: /usr/bin/apisix:619: attempt to index local 'base' (a nil value)
stack traceback:
	/usr/bin/apisix:619: in function 'merge_conf'
	/usr/bin/apisix:616: in function 'merge_conf'
	/usr/bin/apisix:616: in function 'merge_conf'
	/usr/bin/apisix:660: in function 'read_yaml_conf'
	/usr/bin/apisix:826: in function </usr/bin/apisix:819>
	/usr/bin/apisix:1166: in main chunk
	[C]: at 0x00404c00
/usr/local/openresty/luajit/bin/luajit: /usr/bin/apisix:619: attempt to index local 'base' (a nil value)
stack traceback:
	/usr/bin/apisix:619: in function 'merge_conf'
	/usr/bin/apisix:616: in function 'merge_conf'
	/usr/bin/apisix:616: in function 'merge_conf'
	/usr/bin/apisix:660: in function 'read_yaml_conf'
	/usr/bin/apisix:826: in function </usr/bin/apisix:819>
	/usr/bin/apisix:1166: in main chunk
	[C]: at 0x00404c00

from apisix-docker.

Here is the latest conf/config.yaml file from APISIX project:

https://github.com/apache/apisix/blob/master/conf/config.yaml

Needs to update the current version

from apisix-docker.

@membphis Now this is my configuration file，
It still appears

got malformed version message: "" from etcd
got malformed version message: "" from etcd
got malformed version message: "" from etcd
got malformed version message: "" from etcd

cat apisix_conf/config.yaml

apisix:
  admin_key:
    -
      name: "admin"
      key: edd1c9f034335f136f87ad84b625c8f1 
      role: admin
etcd:
  host:
    - "https://10.1.7.51:2379"

from apisix-docker.

This is a bug, we don't respect the scheme when calling bin/apisix init_etcd.

from apisix-docker.

@Hyt05 I have two questions:

1. https://github.com/apache/apisix-docker/tree/master/example use docker-compose with network as below, make sure the etcd address can be reached.

networks:
  apisix:
    driver: bridge
    ipam:
      config:
      - subnet: 172.18.0.0/16

2. multi-etcd IPs can not be configured in a line , need to configure like this:

etcd:
  host:
    - "https://10.1.7.51:2379"
    - "https://10.1.7.52:2379"
    - "https://10.1.7.53:2379"

from apisix-docker.

@tokers Do you have a good solution

from apisix-docker.

This is a bug, we don't respect the scheme when calling bin/apisix init_etcd.

Sorry, i misunderstood this issue, just ignore it.

from apisix-docker.

@gxthrj In fact, I didn't use docker- compose.yml Etcd in, I just use the example docker- compose.yml To start apifix, the etcd connection address used in my configuration file is the etcd cluster deployed with RKE,

Just tested and used the configuration

etcd:
  host:
    - "https://10.1.7.51:2379"
    - "https://10.1.7.52:2379"
    - "https://10.1.7.53:2379"

Still appear：

got malformed version message: "" from etcd
got malformed version message: "" from etcd
got malformed version message: "" from etcd
got malformed version message: "" from etcd
got malformed version message: "" from etcd
got malformed version message: "" from etcd

from apisix-docker.

The error messsage got malformed version message: "" from etcd means APISIX can not connect to etcd. In most cases, the network cannot be connected between APISIX and etcd.

We need to check if etcd can be reached in APISIX container.

A feasible method is to use docker exec to enter a container which on the same network as the APISIX container, and then telnet 10.1.7.51 2379.

from apisix-docker.

@gxthrj
I am sure that I can communicate with 10.1.7.51 2379 in the same container

This is the startup parameter of my etcd cluster. I wonder if it is related to the "--enable-v2=true" parameter.

root     11701  2.4  3.1 10888476 322624 ?     Ssl  11月22 321:14 /usr/local/bin/etcd --listen-client-urls=https://0.0.0.0:2379 \
--initial-advertise-peer-urls=https://10.1.7.51:2380 --key-file=/etc/kubernetes/ssl/kube-etcd-10-1-7-51-key.pem \ 
--peer-key-file=/etc/kubernetes/ssl/kube-etcd-10-1-7-51-key.pem --peer-client-cert-auth=true \ 
--initial-cluster-token=etcd-cluster-1 --name=etcd-master1 --cert-file=/etc/kubernetes/ssl/kube-etcd-10-1-7-51.pem \
--advertise-client-urls=https://10.1.7.51:2379,https://10.1.7.51:4001 --enable-v2=true --data-dir=/var/lib/rancher/etcd/ \ 
--listen-peer-urls=https://0.0.0.0:2380 --initial-cluster-state=new --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem \ 
--peer-cert-file=/etc/kubernetes/ssl/kube-etcd-10-1-7-51.pem --initial-cluster=etcd-master1=https://10.1.7.51:2380,etcd-worker1=https://10.1.7.52:2380,etcd-worker2=https://10.1.7.53:2380 \ 
--trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --client-cert-auth=true --election-timeout=5000 --heartbeat-interval=500

from apisix-docker.

@Hyt05 what are the apisix and etcd versions?

from apisix-docker.

@membphis apisix version：2.1 、{"etcdserver":"3.4.3","etcdcluster":"3.4.0"}

from apisix-docker.

@tokers Thanks for your answer, I will try to remove this parameter

from apisix-docker.

@Hyt05 https://github.com/apache/apisix/pull/2940/files here is the related PR

feel free to reopen this if you still have an issue.

from apisix-docker.