performance: loxilb starts consuming 100% CPU only after a few seconds about loxilb HOT 4 OPEN

Loxilb has a garbage collector which monitors its connection-track entries. If a connection goes through its normal life cycle - e.g init, init-ack, est, fin etc, eBPF module itself cleans up the CT entries. But for half-cooked connections, the garbage collector comes into play. Currently, it is set to aggressive GC.One potential solution is to trigger GC only when there is a space pressure in its CT map.

from loxilb.

Thanks for bringing this to notice. It seems strange but we will have a look and update soon.

from loxilb.

Hi @luisgerhorst, we have tried to reproduce this with loxilb latest docker but in our test we couldn't find this issue. nginx and wrk seems to be taking only 10%. We used validation-wrk script to test this. If you are using some other config/steps then please share with us and we will try with them. You may join our slack channel, we will be able to assist you better.

from loxilb.

I'm sorry for the incomplete description. I have been running wrk2 at a much higher rate than the version merged (12.5k RPS, roughly 80% of the max. on my machine).

I run https://github.com/luisgerhorst/loxilb/blob/ccf029a1f6cf8b914b23909d4cc922a4c32662d0/cicd/tcpsctpperf/validation-wrk using OSE_PERF_STAT="perf stat" OSE_LOXILB_SERVERS=1 OSE_LATENCY_PAYLOAD_SIZE=1024 ./validation-wrk 2 60 $(pwd)/ 100 | tee v.log against loxilb v0.9. Using parca I was able to record a CPU trace of the behaviour I observed:

The screenshot has the x axis separated into 6 segments. The dark purple line is loxilb which is mostly idle in segments 3 and 4 and ramps up in segments 2, 5, and 6. The red/green line in segments 3-6 are wrk2 and nginx (red/orange in 1-2). The light purple line is parca which periodically processes the cpu samples collected.

Here's the CPU profile while loxilb is in it's idle phase: https://pprof.me/2d2a1527503cfa10ae0a46890b2cb3a0

And here's the CPU profile when loxilb is in it's busy phase: https://pprof.me/414cf26812ea9d7d3b973563bec491ed

Iterestingly, loxilb seems to consume 100% here even thougth the benchmark is not at it's limit (I can achieve 15.5k RPS using this same setup). Therefore, maybe this behaviour is not acutally limiting the performance (at least directly).

from loxilb.