balancer by lua block: Getting errors : attempt to send data on a closed socket about lua-resty-core HOT 9 CLOSED

@rohitjoshi I've just had a closer look and I don't think that error is trigger by the balancer_by_lua* config snippet you provided.

The error message is generated in your cosocket related Lua code when you try continuing using a cosocket object after a fatal error has happened in a previous cosocket operation. How to fix:

1. ensure you have Lua code that checks the return values of all the cosocket operations that may fail and handle any errors properly in your own Lua code (like retry or give up).
2. after you have handled all the errors on the Lua land, turn off such automatic cosocket error logging by adding the following line to your nginx.conf:

lua_socket_log_errors off;

from lua-resty-core.

@rohitjoshi Will you please provide a minimal and standalone example that we can easily reproduce the issue on our side?

from lua-resty-core.

@rohitjoshi BTW, it's a bad idea to parse a JSON string in your balancer_by_lua* handler upon every request. It's very expensive.

from lua-resty-core.

@rohitjoshi For the minimal example that can reproduce the issue, I don't think you need the JSON part or the branching code testing the size of the server array (you can just use a static table, or even a static host and port pair).

from lua-resty-core.

@rohitjoshi And for the standalone part of the example, you need to provide your location definition, as well as a fake backend server in the same nginx.conf, as well as the test requests required to access the entry point of your nginx server.

from lua-resty-core.

@rohitjoshi Please see the test suite for balancer_by_lua* and ngx.balancer for an idea of what standalone and minimal examples look like:

https://github.com/openresty/lua-resty-core/blob/master/t/balancer.t

from lua-resty-core.

@rohitjoshi A fatal error in a cosocket operation closes the underlying socket immediately. So when you go ahead and try sending data on this closed socket without re-connecting, you get the error log message "attempt to send data on a closed socket" from the cosocket object when your do not turn off lua_socket_log_errors (default on).

from lua-resty-core.

@agentzh thanks. Yes, errors seem to be due to upstream simulator server closing the connection. I did not see errors reflected in my client (wrk).

I ran performance test using two scenarios using wrk.

./wrk -c15000 -t128 -d30s -H "Api-Key: rohit" "https://10.205.192.64:11111/perftest_category/success?http_status=201&body_size=100"

Based on the body_size query parameter, simulator server returns that size in the response.

I ran 3 tests with body size 100, 500 and 1000.

1. Balancer by Lua:

Response size: 100 Requests/sec:   69123.15  Avg Latency:  239.27ms
Response size: 500 Requests/sec:   68385.40  Avg Latency:  247.50ms
Response size: 1000 Requests/sec:  64240.14 Avg Latency:  254.42ms

1. Nginx upstream with nginx-upstream-dynamic-servers module (dynamic resolve)

Response size: 100 Requests/sec:   71613.66  Avg Latency:  248.93ms
Response size: 500 Requests/sec:   68367.540  Avg Latency:  251.14ms
Response size: 1000 Requests/sec:  65553.37 Avg Latency:  242.05ms

Both test are returning almost identical results with nginx upstream has slightly lower latency and higher throughput. May be due to my code where I am converting string to table using msgpack. (I replaced json with msgpack). Will optimize this.

By the way, I did not see any attempt to send data on a closed socket while using nginx upstream but did see upstream prematurely closed connection while reading response header from upstream errors which might be equivalent to what is logged when balancer by lua is used.

Thanks again.

from lua-resty-core.

Setting lua_socket_log_errors does disable these errors.

from lua-resty-core.