Comments (5)
The idea is that gossip messages are not sent very often - Weave Mesh selects log(number of connections) peers to send to - so the scalability should be good.
However various other issues in the code mean that messages are sent way more often than this ideal. #101, #106 and #107 are attempts to improve matters, though work is ongoing to understand the full set of causes.
from mesh.
From my use-case of protokube (kubernetes/kops#7427) I'm seeing ~2 cores of CPU usage and ~3G of RAM usage with a fully connected mesh of ~300 nodes. This seems to highlight some serious scale limitations of weaveworks/mesh -- as that isn't even a very large cluster. More importantly the utilization ramp-up was more-or-less exponential as more nodes were added.
Even after I made a custom build with #107 fixed the CPU usage dropped to 1.6 cores -- which is still way too many (all the CPU time was being spent marshaling/unmarshaling the peer list being gossiped around).
There seem to be quite a few issues, a couple: (1) no concept of "suspect" state (2) peer messages include the list of all peers it has connected-- which scaled with cluster size. There are likely more but TBH I have decided to instead spend my time swapping protokube to a more robust/reliable gossip library.
from mesh.
We're seeing quite positive results from deferring gossip updates - #117 and #118.
Would you like to try those in your build?
from mesh.
peer messages include the list of all peers it has connected
It's worse than that - the topology message lists all the connections of all peers. In other words in a fully-connected cluster it's O(N^2).
However, for 300 nodes that might be 8MB per message, which needs something else to get to 1-2GB.
We found that the connections would each read a message then block on the Peers
lock to apply the update. So with 200 connections that's 1.6GB.
Changing the "everyone sends everything" behaviour is quite a big change, so ahead of that I felt that just slowing down the initial connections would help - #124 . After initial connection the updates only go to logN peers so we shouldn't get the massive spikes.
from mesh.
I'm going to close this issue now 0.4 is released - if you want to come back to the discussion please do.
from mesh.
Related Issues (20)
- make gossipInterval configurable HOT 1
- optimise Peer.Routes() for fully connected mesh of node topology
- redundant broadcast topology update in case of fully-connected mesh of node HOT 1
- rate-limit routes calculation done when a gossip topology update is received
- Topology update broadcasts are sent once per route calculation HOT 1
- cached broadcast routes are reset when performing routes.calculate() HOT 4
- high rate of peer.routes() calculation resuting from gossip broadcasts HOT 1
- high rate of Peers.garbageCollect() calls resulting from topology gossip
- in fullly connected mesh topolgy, topology update gossip's can get chatty
- randomNeighbours selection is biased
- TCP sends block forever
- encryptedTCPSender.Send() holds a lock over a TCP send HOT 1
- runtime error: invalid memory address or nil pointer dereference HOT 1
- Example for mesh to build a distributed app
- Provide a simple usage example
- Is this project still supported by weaveworks maintainers? HOT 1
- Whether the Router will ignore the localPeer in Router.Peers
- Project status? Is this dead? HOT 2
- There is something wrong
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mesh.