NATS supercluster in kubernetes about k8s HOT 3 CLOSED

(fyi transferring this to the nats-io/k8s repo)

• Which loadbalancer type are you using for the setup? One option could be the NLB service from AWS like this: https://docs.nats.io/nats-on-kubernetes/nats-external-nlb that page only covers client connections but could change the port for gateway connections as well. Using the an NLB for AWS and NATS should be ok because that can be setup to ouse TLS. So you can create another service for the gateways as follows:

apiVersion: v1
kind: Service
metadata:
  name: nats-nlb-gw
  namespace: default
  labels:
    app: nats
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  ports:
  - name: nats
    port: 7522
    protocol: TCP
    targetPort: 7522
  selector:
    app: nats

• Another option without using the NLB, is to use something like external-dns to dynamically announce the routes and exposing each one of the NATS Servers public ip address and host:port (this is the prod setup for the connect.ngs.global for example).

# Create 3 nodes Kubernetes cluster
eksctl create cluster --name nats-k8s-cluster \
  --nodes 3 \
  --node-type=t3.large \
  --region=eu-west-1

# Get the credentials for your cluster
eksctl utils write-kubeconfig --name $YOUR_EKS_NAME --region eu-west-1

After that is done you get a set of 3 nodes with the example above:

 kubectl get nodes -o wide
NAME                                           STATUS   ROLES    AGE    VERSION   INTERNAL-IP      EXTERNAL-IP     OS-IMAGE         KERNEL-VERSION                  CONTAINER-RUNTIME
ip-192-168-10-213.us-east-2.compute.internal   Ready    <none>   124d   v1.12.7   192.168.10.213   3.17.184.16     Amazon Linux 2   4.14.123-111.109.amzn2.x86_64   docker://18.6.1
ip-192-168-45-209.us-east-2.compute.internal   Ready    <none>   124d   v1.12.7   192.168.45.209   18.218.52.122   Amazon Linux 2   4.14.123-111.109.amzn2.x86_64   docker://18.6.1
ip-192-168-65-15.us-east-2.compute.internal    Ready    <none>   124d   v1.12.7   192.168.65.15    3.15.38.138     Amazon Linux 2   4.14.123-111.109.amzn2.x86_64   docker://18.6.1

Then you can deploy NATS and create a headless service named nats which will represent the NATS Server nodes:

kubectl get svc nats -o wide
NAME   TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                                                 AGE   SELECTOR
nats   ClusterIP   None         <none>        4222/TCP,6222/TCP,8222/TCP,7777/TCP,7422/TCP,7522/TCP   36d   app=nats

Once deploying external-dns, you have to use a NodePort with something as follows to keep the nodes mapped by the external dns with the ones from the headless service:

apiVersion: v1
kind: Service
metadata:
  name: nats-nodeport
  labels:
    app: nats
  annotations:
    external-dns.alpha.kubernetes.io/hostname: nats.example.com
spec:
  type: NodePort
  selector:
    app: nats
  externalTrafficPolicy: Local
  ports:
  - name: client
    port: 4222
    nodePort: 30222 #  Arbitrary port to represent the external dns service, external-dns issue...
    targetPort: 4222  # NOTE: the NATS pods also use host ports

The external-dns process would be responsible of registering the public ips from the nodes to be serviced at nats.example.com.

from k8s.

thanks for the quick response. I have gone ahead with NLB and it works for me.

from k8s.

@vtomar01 I have the same situation, I have one cluster in eastus and another in westus. Both the region's vnets are connected with peering. I have a private load balancer added by k8s service for gateway ports and configured both to talk to each other. Experiencing 503 from Nats client when connected with an externally facing URL. Do you have any documentation on how you set up this?

from k8s.