Document story regarding horizontally scalability about zulip HOT 3 CLOSED

Yeah, we haven't written up documentation on scalability stuff. The short story is:

(1) You can scale to a pretty large installation (O(~1000) concurrently active users using it to chat all day) with just a single reasonably large server sitting mostly idle for each of the frontend and database. I highly recommend SSD for the database (though I do that for any database), and having the frontend be a different server from the database (config for this is in puppet/zulip_internal and zproject/local_settings.py).

(2) If you're hosting multiple realms on the same server like we do with zulip.com, you can scale horizontally without any real code changes by adding additional frontends -- two frontends only need to talk to each other if they have users that exchange messages.

(3) Some code changes are required in order to make multiple frontend servers host the same Zulip realm. There are two issues: you need a shared memcached (should be easy to move that to a network service shared by multiple servers) and the Tornado event system for pushing to browsers currently has no mechanism for multiple frontend servers talking to each other. One can probably get a factor of 10 by just making it multiprocess on a single server; I have an old branch from working on this problem that I can make available if someone is interested in seriously working on this problem. Probably a week or two's work by someone with the right context to resolve these constraints.

(4) For reasonably high availability right now, it's easy to run a hot spare frontend server.

(5) If you have a very large realm (more than a few thousand users), some work will likely need to be done on the buddy list perf and UI to work nicely in that environment.

(6) The database load per user is pretty optimized and thus inexpensive, so I don't think you'll need multiple database masters unless below 10K users actually chatting all day (which can mean a lot more users in a realm!). There's config for streaming replication postgres slaves and pulling backups from those in puppet/zulip_internal.

I'm happy to expand more on this if you (or anyone else) still has questions! I think this thread can form the basis for an answer to live under docs/ in the codebase.

from zulip.

@timabbott I'd want we to discuss these things in live discussion. I invited you, Sir. https://gitter.im/turalsprofile/zulip?utm_source=share-link&utm_medium=link&utm_campaign=share-link

from zulip.

This is now covered in https://github.com/zulip/zulip/blob/master/README.prod.md#scalability-of-zulip

from zulip.