boytm / mproxy Goto Github PK

• 复现步骤

1. nginx 后端服务器 IP 10.1.1.100
2. mproxy 工作在 HTTP 代理模式 且 ulimit open files 1024，IP 10.1.1.200
3. 使用 tornado 和 libcurl 发起 1000 个并行请求

import requests
import time
import tornado.httpclient
from tornado import gen

chunk_size = 1024 * 512
jobs = 1000
url = 'http://10.1.1.100/iso/CentOS-7-x86_64-DVD-1708.iso'

class AsyncRequest():
    def __init__(self, ioloop):
        #self.client = tornado.httpclient.AsyncHTTPClient()
        self.client = tornado.curl_httpclient.CurlAsyncHTTPClient(max_clients=512)
        request = tornado.httpclient.HTTPRequest(url, proxy_host='10.1.1.200', proxy_port=8081, streaming_callback=self.on_chunk)

        self.client.fetch(request)
        self.total = 0
        self.count = 0

    def on_chunk(self, data):
        self.total += len(data)
        self.count += 1
        if self.count % 64 == 0:
            print 'received data len %dM' % (self.total / 1024 / 1024)

tornado.httpclient.AsyncHTTPClient.configure("tornado.curl_httpclient.CurlAsyncHTTPClient")
io_loop = tornado.ioloop.IOLoop.current()

for i in range(jobs):
    r = AsyncRequest(io_loop)

io_loop.start()

当 测试客户端 报出大量超时信息退出后，可以看到 mproxy 大量的 CLOSE_WAIT 状态 TCP 连接

ERROR:tornado.application:Future exception was never retrieved: CurlError: HTTP 599: Connection timed out after 20036 milliseconds

• 问题原因
  libevent 2.0.22 和 2.1.8 的 evdns_getaddrinfo() 作域名解析时候，如果名称是 IP 或者可以通过 /etc/hosts 找到时候，其对回调的执行是在 evdns_getaddrinfo() 里面立即执行，并不会等到下个 event loop 再去执行。
  这就导致 bufferevent_socket_connect_hostname() 的域名解析回调 bufferevent_connect_getaddrinfo_cb() 可能会在 bufferevent_socket_connect_hostname() 函数中被执行，而 bufferevent_connect_getaddrinfo_cb() 并不会将执行 bufferevent_socket_connect() 的错误（这里是 socket() 无法创建 fd 的错误）通过状态码返还给最外层的用户。 如果这时 bufferevent 的 event cb 还没有被设置，这就会导致最外层的用户看状态码是正常的，但内层的 bufferevent_connect_getaddrinfo_cb() 也无法通过 event cb 通知最外层的用户。

• 解决办法：

1. 确保在 bufferevent_socket_connect() 和 bufferevent_socket_connect_hostname() 之前设置 event cb。
2. bufferevent_socket_connect() 和 bufferevent_socket_connect_hostname() 之后的代码，要有意识到 event cb 可能因为错误已经被执行过，导致 bufferevent 或者 相关变量被释放，代码不能访问那些可能已经被释放的变量。 如果确实有需要，或者嫌这样程序逻辑太过复杂，可以通过 BEV_OPT_DEFER_CALLBACKS 让 event cb 延迟到 bufferevent_socket_connect() 和 bufferevent_socket_connect_hostname() 之外执行。
3. libevent 修复 libevent/libevent#597

...
4 warnings generated.
[ 53%] Building C object CMakeFiles/mproxy.dir/evhtp_sock_relay.c.o
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -DENABLE_SS=1 -DUSE_CRYPTO_OPENSSL=1 -g -isystem /usr/local/include -isystem /opt/local/include -I/Users/yuming/mproxy/libevhtp -I/Users/yuming/mproxy/libevhtp/compat -o CMakeFiles/mproxy.dir/evhtp_sock_relay.c.o -c /Users/yuming/mproxy/evhtp_sock_relay.c
[ 61%] Building C object CMakeFiles/mproxy.dir/lru.c.o
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -DENABLE_SS=1 -DUSE_CRYPTO_OPENSSL=1 -g -isystem /usr/local/include -isystem /opt/local/include -I/Users/yuming/mproxy/libevhtp -I/Users/yuming/mproxy/libevhtp/compat -o CMakeFiles/mproxy.dir/lru.c.o -c /Users/yuming/mproxy/lru.c
/Users/yuming/mproxy/lru.c:231:49: warning: format specifies type
'unsigned short' but the argument has type 'evhtp_error_flags'
(aka 'unsigned char') [-Wformat]
bi->parent->hostname, (int)bi->parent->port, errtype);
^~~~~~~
/Users/yuming/mproxy/log.h:102:53: note: expanded from macro 'LOGD'
#define LOGD(fmt, ...) LOG(LOG_LEVEL_DEBUG, fmt, ## VA_ARGS)
^~~~~~~~~~~
/Users/yuming/mproxy/log.h:90:124: note: expanded from macro 'LOG'
...PRIORITY_TO_STRING(prio), FILENAME, ## VA_ARGS);
^~~~~~~~~~~
1 warning generated.
[ 69%] Building C object CMakeFiles/mproxy.dir/utils.c.o
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -DENABLE_SS=1 -DUSE_CRYPTO_OPENSSL=1 -g -isystem /usr/local/include -isystem /opt/local/include -I/Users/yuming/mproxy/libevhtp -I/Users/yuming/mproxy/libevhtp/compat -o CMakeFiles/mproxy.dir/utils.c.o -c /Users/yuming/mproxy/utils.c
[ 76%] Building C object CMakeFiles/mproxy.dir/log.c.o
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -DENABLE_SS=1 -DUSE_CRYPTO_OPENSSL=1 -g -isystem /usr/local/include -isystem /opt/local/include -I/Users/yuming/mproxy/libevhtp -I/Users/yuming/mproxy/libevhtp/compat -o CMakeFiles/mproxy.dir/log.c.o -c /Users/yuming/mproxy/log.c
[ 84%] Building C object CMakeFiles/mproxy.dir/ss_connector.c.o
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -DENABLE_SS=1 -DUSE_CRYPTO_OPENSSL=1 -g -isystem /usr/local/include -isystem /opt/local/include -I/Users/yuming/mproxy/libevhtp -I/Users/yuming/mproxy/libevhtp/compat -o CMakeFiles/mproxy.dir/ss_connector.c.o -c /Users/yuming/mproxy/ss_connector.c
[ 92%] Building C object CMakeFiles/mproxy.dir/encrypt.c.o
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -DENABLE_SS=1 -DUSE_CRYPTO_OPENSSL=1 -g -isystem /usr/local/include -isystem /opt/local/include -I/Users/yuming/mproxy/libevhtp -I/Users/yuming/mproxy/libevhtp/compat -o CMakeFiles/mproxy.dir/encrypt.c.o -c /Users/yuming/mproxy/encrypt.c
[100%] Linking C executable mproxy
/usr/local/Cellar/cmake/3.3.1/bin/cmake -E cmake_link_script CMakeFiles/mproxy.dir/link.txt --verbose=1
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -g -Wl,-search_paths_first -Wl,-headerpad_max_install_names CMakeFiles/mproxy.dir/connector.c.o CMakeFiles/mproxy.dir/evhtp_proxy.c.o CMakeFiles/mproxy.dir/evhtp_sock_relay.c.o CMakeFiles/mproxy.dir/lru.c.o CMakeFiles/mproxy.dir/utils.c.o CMakeFiles/mproxy.dir/log.c.o CMakeFiles/mproxy.dir/ss_connector.c.o CMakeFiles/mproxy.dir/encrypt.c.o -o mproxy libevhtp/libevhtp.a /usr/local/lib/libevent.dylib -lrt /opt/local/lib/libcrypto.dylib
ld: library not found for -lrt
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [mproxy] Error 1
make[1]: *** [CMakeFiles/mproxy.dir/all] Error 2
make: *** [all] Error 2
yudeMacBook-Air:mproxy brite$

如何解决？谢谢你

hi.

root@wh:~/mproxy-by-boytm/build# cmake ..
CMake Error at CMakeLists.txt:36 (add_subdirectory):
The source directory

/root/mproxy-by-boytm/libevhtp

does not contain a CMakeLists.txt file.

CMake Error at CMakeLists.txt:48 (CHECK_FUNCTION_EXISTS):
Unknown CMake command "CHECK_FUNCTION_EXISTS".

-- Configuring incomplete, errors occurred!
See also "/root/mproxy-by-boytm/build/CMakeFiles/CMakeOutput.log".
root@wh:~/mproxy-by-boytm/build#

怎么解决？谢谢侬