groundwater / systems-talk Goto Github PK

The following is probably something I've ranted to you at one point or another, but here it is in word form.

I was thinking about my CoreOS cluster, and how I want to do service discovery, high-availability, circuit-breakers, etc. My plan is to run a local proxy on all hosts in the cluster, which I am going to give the incredibly overloaded name of the application router.

The app router will do a lot. I would love to separate concerns more, but I want to minimize the number of network hops involved. The routers job is to route network connections between apps in your cluster.

1. your app registers what dependencies it requires somewhere (tbd)
2. for each dependency, the app router uses etcd to resolve where that service resides
3. a unix domain socket is created named after the service at /services/$NAME.sock

Optionally the app router may setup a persistent TCP connection to the target host ahead of time. When a request comes in, the application should issue a single request to the local unix socket. The router then forwards the request to the remote hosts router, which forwards the request to another unix domain socket.

Only the router deals with network sockets. It might be better to have the router talk directly to an app listening on a public port, but for now I want the flexibility of intercepting requests on both sides.

There are a number of things you can introduce now

1. Complete network monitoring: the app router can report stats about all incoming/outgoing connections
2. Latency-monkey: the app routers can slow down your requests to test robustness of the system. You can literally dial in the chaos.
3. Circuit breakers: the app router can immediately error to applications when downstream latency starts to build up.

The most interesting piece however is that the data center network is now entirely a private interface. The only public interface that applications talk to are unix domain sockets.