Improve memory footprint of queue workers about zulip HOT 6 CLOSED

Well there are two considerations:

• debuggability -- it will likely be easier to debug some things in the multi-process configuration than the multithreaded configuration.
• stability -- currently if a worker crashes, the rest of them keep running happily; with multithreading, it's possible for a worker crashing to bring them all down.

Both of these seem like things that shouldn't be a problem in the long term, but likely are important enough that I'd imagine we want to support both options for a transitional period...

from zulip.

The places in the code to start looking at this are zerver/worker/queue_processors.py and zerver/management/commands/process_queue.py.

To start with getting this working in the development environment, you can patch tools/run-dev.py to run things with your new tooling.

from zulip.

Would be very cool to make this multithreaded, the 4GB memory requirement for a production server is very high.

from zulip.

#367 will make the Zulip workers multithreaded in the development environment; there will be a bit more work to do this in production since we'll probably want to offer both configuration options...

from zulip.

@timabbott cool! I'm not sure why anyone would want non-multithreaded workers, or is it temporarily until multithreading is stable?

from zulip.

Totally agree about the transitional period.

from zulip.