jackdoe / back-to-back Goto Github PK

Hi! I read your blog on https://proxy.vulpes.one/gemini/berserk.red/~jack/we-got-it-all-wrong.txt and I think this is a really great concept. We've all drunk the cool-aid that makes it seem like microservices and queues fix everything but they don't.

https://ferd.ca/queues-don-t-fix-overload.html is an essay I like addressing this. There's another that more specifically calls out queues+microservices for being a sham but I can't find it anymore.

I think it's interesting that all that long time ago Unix got this right: a unix pipe has implicit backpressure at every connection. There is a small buffer between each process to allow each to do work without being directly slogged by the consumer, but when that buffer fills write(2) blocks until the consumer read(2)s some content. It would be interesting to experiment with changing that queue size (I think you can do this in Linux? I'm sure there's a sysctl somewhere).

I've been thinking of this principle of system design as the maxim:

Producers set the size, consumers set the speed.