Performance drop versus v1 about server HOT 7 CLOSED

@mochi-co , sharing some real-world performance data. This is one node in a much larger cluster. The node has 4 vCPU, 32 GB memory. Each client sends avg ~2KB (some can be much larger) data once every minute.
V1 with the setting BufferSize=64K, BufferBlockSize=8K:
V2 with FanPoolSize=128 * GOMAXPROCS, FanPoolQueueSize=1024

The CPU utilization is on par with v1. The memory improvement in V2 is very dramatic, well done!

from server.

./mqtt-stresser -broker tcp://192.168.11.52:1883 -num-clients=10 -num-messages=10000

(v1.3.2)
Median: 15990 msg/sec
Median: 16551 msg/sec
Median: 13630 msg/sec
Median: 20391 msg/sec

(v2.2.1)
Median: 16386 msg/sec
Median: 15545 msg/sec
Median: 14253 msg/sec
Median: 18597 msg/sec

Now I don't see performance loss versys v1.

I also see that it is resolved slow consumers problem. Good job.

from server.

@izarraga Wonderful, thank you!

from server.

I get the following results:

(v1.3.2)

• Receiving Median: 59284 msg/sec, Publishing Median: 29438 msg/sec
• Receiving Median: 67485 msg/sec, Publishing Median: 33231 msg/sec
• Receiving Median: 54457 msg/sec, Publishing Median: 30674 msg/sec
• Receiving Median: 75374 msg/sec, Publishing Median: 34337 msg/sec

(v2.0.4)

• Receiving Median: 41456 msg/sec, Publishing Median: 31356 msg/sec
• Receiving Median: 32336 msg/sec, Publishing Median: 29485 msg/sec
• Receiving Median: 33827 msg/sec, Publishing Median: 33531 msg/sec
• Receiving Median: 35704 msg/sec, Publishing Median: 27839 msg/sec

(v2.0.4) with FanPoolSize=8*1024, FanPoolQueueSize=128*128

• Receiving Median: 76887 msg/sec, Publishing Median: 21007 msg/sec
• Receiving Median: 46100 msg/sec, Publishing Median: 26947 msg/sec
• Receiving Median: 51609 msg/sec, Publishing Median: 26087 msg/sec
• Receiving Median: 50639 msg/sec, Publishing Median: 26639 msg/sec

I agree that there is a performance difference between 1.3.2 and v2.0.4, and it appears to be Receiving side. As I understand it, mqtt-stresser publishing and receiving values refers to the client side (adding time.Sleep(time.Millisecond * 50) to cl.WritePacket . reduces the throughout for receive).

This suggests the v2.0.4 is slower at issuing packets than v1.3.2, however the code controlling the receipt of packets from the client has remained largely the same, minus the circular buffer. As to why, I can't say yet. I will need to compare the code and see if there's any major differences.

One caveat is that although v1.3.2 was very fast, it was also very unstable and tended to freeze up under heavy or oversized traffic - this was one of the main driving decisions behind rewriting the packet processing in v2. Still, I think this is an unusual difference and deserves some investigation.

from server.

@thedevop Just seeing this now - this is wonderful to see! Hopefully the new changes we're discussing won't affect memory usage too severely 😅

from server.

Yeah, depending on the buffered chan size, at 1K, it may add ~2GB for 200K connections. There are some area we can potentially optimize to reduce the overall memory, for example use of sync.pool (can potentially reduce CPU as well). But that will add complexity, we can look into that in the future.

from server.

@izarraga Would you be able to recheck the benchmark on your machine against v2.2.0? I would be interested to see if there has been an improvement... Thank you!

from server.