The haproxy from sflow

haproxy-1.5.18 tcp mode occasionally occur cD message item in haproxy.log, but backend return response quickly, and frontend service also not slow. below is the detail:

Haproxy-1.5.18 tcp mode occasionally occur cD message item in haproxy.log, but backend return response quickly, and frontend service also not slow. below is the detail:

Our message flow as below:
           172.30.49.193     172.30.49.18       172.30.49.29
            UI-Console  --->   Haproxy  ---->  Tomcat
            alias:  (A)           (B)                     (C)
       note: 
            message come from Internet Browser to UI-Console application(marked as A), then UI-Console 
       redispatch it to Haproxy(marked as B), and Haproxy transmit it to backend service who using 
       tomcat as web container(marked as C). 

We find that occasionally the request will response slow, as a result the UI-Console sometimes can not get response in time, thus can not show the response result on UI (Internet Browser). we analysis our log, and find that service C return response quickly, logs as below:

172.30.49.18 - - [29/Sep/2017:15:43:27 +0200] "GET /network/v1/publicIps HTTP/1.1" 200 3568 462
172.30.49.18 - - [29/Sep/2017:15:43:32 +0200] "GET /network/v1/publicIps HTTP/1.1" 200 3568 446
172.30.49.18 - - [29/Sep/2017:15:43:36 +0200] "GET /network/v1/publicIps HTTP/1.1" 200 3568 387
172.30.49.18 - - [29/Sep/2017:15:43:38 +0200] "GET /network/v1/publicIps HTTP/1.1" 200 3568 417
172.30.49.18 - - [29/Sep/2017:15:43:39 +0200] "GET /network/v1/publicIps HTTP/1.1" 200 3568 422
172.30.49.18 - - [29/Sep/2017:15:43:40 +0200] "GET /network/v1/publicIps HTTP/1.1" 200 3568 433
172.30.49.18 - - [29/Sep/2017:15:43:42 +0200] "GET /network/v1/publicIps HTTP/1.1" 200 3568 429
172.30.49.18 - - [29/Sep/2017:15:43:43 +0200] "GET /network/v1/publicIps HTTP/1.1" 200 3568 479
172.30.49.18 - - [29/Sep/2017:15:43:44 +0200] "GET /network/v1/publicIps HTTP/1.1" 200 3568 434
172.30.49.18 - - [29/Sep/2017:15:43:45 +0200] "GET /network/v1/publicIps HTTP/1.1" 200 3568 405
    note:
        The last column is the response time of a request.

Then we analysis haproxy.log and find cD terminal code in haproxy.log as below:
    Sep 29 15:44:10 localhost haproxy[67984]: 172.30.49.193:58298[29/Sep/2017:15:43:27.244] VPC VPC/VPC-172.30.49.29 1/0/120193 40293 cD 7/1/1/0/0 0/0

It seems the problem occur below Service A and B, considering that TCP packets should be transmitted and switched directly via A to B, the murderer seems B. Is haproxy delay the response? To check it we use tcpdump tool to capture tcp packages.

From captured tcp packages, we find Service A not close the socket immediately when A send packages over, and then after 20s Haproxy send a FIN-ACK package closed the socket. one question is that why haproxy close the socket after 20s, is there any configure item to config it.

Also we discover another problem,  when haproxy send packages over to backend service C, sometimes haproxy does not close the socket immediately, then service C will close the socket after 20s(because Tomcat connector's connectTimeout is configed as 20s in server.xml on tomcat). so another question is that why sometimes haproxy does not close the socket immediately when haproxy send packages over to backend service C.  is there any configuration could cause the delay? plus, below is our haproxy configuration:
haproxy.cfg:
    global
         log 127.0.0.1   local2 debug
         maxconn 250000
         user root
         group root
         daemon
         nbproc 4
    defaults
         log global
         mode    tcp
         option  tcplog
         option  dontlognull
         retries 3
         option redispatch
         maxconn 4000
         timeout connect 30000
         timeout client  120000
         timeout server  120000
   listen  AS-Server    172.30.49.153:8443
         mode tcp
         maxconn 4000
         balance roundrobin
         server  AS-Server-172.30.49.54  172.30.49.54:7443 check inter 500 fall 2 rise 2
         server  AS-Server-172.30.49.132  172.30.49.132:7443 check inter 500 fall 2 rise 2
   listen  VPC    172.30.49.152:8443
         mode tcp
         maxconn 4000
         balance roundrobin
         server  VPC-172.30.49.29  172.30.49.29:7443 check inter 500 fall 2 rise 2
         server  VPC-172.30.49.30  172.30.49.30:7443 check inter 500 fall 2 rise 2
         server  VPC-172.30.49.233  172.30.49.233:7443 check inter 500 fall 2 rise 2

core.sleep() ignores param and sleeps for 10 seconds in response action

When using a response action, this function - sleeps for 10 seconds, no matter what param i pass to it. Also it seems that the wait time always equals timeout connect. The sample config is:

global
daemon
log /dev/log local6
lua-load /etc/haproxy/delay.lua

defaults
mode http
timeout connect 10000ms

frontend fe
bind *:80
mode http
default_backend b_http_hosts

backend b_http_hosts
mode http
http-response lua.delay_response
server s_web1 server:80 check

and the LUA code:

function delay_response(txn)
core.msleep(1)
end

core.register_action("delay_response", {"tcp-res", "http-res" }, delay_response);

Note that if core.msleep() is commented out - everything works as expected.

I tested version 1.6 (it hangs 30 seconds there), 1.7 - matches timeout connect, and 1.8 - same as 1.7.

Any idea how to overcome this problem? All i need is to delay the responses based on information from backend header.

Thanks.

sflow / haproxy Goto Github PK

haproxy's People

Contributors

Stargazers

Watchers

Forkers

haproxy's Issues

haproxy-1.5.18 tcp mode occasionally occur cD message item in haproxy.log, but backend return response quickly, and frontend service also not slow. below is the detail:

core.sleep() ignores param and sleeps for 10 seconds in response action

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent