如当前版本下面的程序在退出时，如果StopLoop是被工作线程调用的。可能出现死锁状态。

`procedure TIocpCrossSocket.StopLoop;
var
I: Integer;
begin
if (FIoThreads = nil) then Exit;

CloseAll;

while (ListensCount > 0) or (ConnectionsCount > 0) do Sleep(1);

for I := 0 to Length(FIoThreads) - 1 do
PostQueuedCompletionStatus(FIocpHandle, 0, 0, POverlapped(SHUTDOWN_FLAG));

CloseHandle(FIocpHandle);

for I := 0 to Length(FIoThreads) - 1 do
begin
FIoThreads[I].WaitFor;
FreeAndNil(FIoThreads[I]);
end;
FIoThreads := nil;
end;
`

建议改为：

`procedure TIocpCrossSocket.StopLoop;
var
I: Integer;
begin
if (FIoThreads = nil) then Exit;

CloseAll;

while (ListensCount > 0) or (ConnectionsCount > 0) do Sleep(1);

for I := 0 to Length(FIoThreads) - 1 do
begin
FIoThreads[I].Terminate;
PostQueuedCompletionStatus(FIocpHandle, 0, 0, POverlapped(SHUTDOWN_FLAG));
end;

for I := 0 to Length(FIoThreads) - 1 do
begin
while not FIoThreads[I].Finished do
begin
if GetCurrentThreadID = MainThreadID then //由于是通过Synchronize同步到主线程执行，所以调用CheckSynchronize，防止死锁
CheckSynchronize(0)
else if GetCurrentThreadID = FIoThreads[I].ThreadID then Break;
Sleep(1);
end;
FreeAndNil(FIoThreads[I]);
end;
CloseHandle(FIocpHandle);
FIoThreads := nil;
end;
`

同理epoll，kqueue

It is far more efficient to pass parameters to functions in Delphi as consts, where ever possible.

android boring ssl support or integrate pre-compile libraries

It is possible to corrupt the recv data stream on Android and Linux using the Epoll because more than one I/O thread could be processing a Read operation simultaneously. This is because a call to [TIoEvent.Read, TIoEvent.Write] is issued. This can cause OpenSSL to fail with a corrupted datastream because the order of calls to TriggerReceived() may be different than the order of calls to _HandleRead because 2 threads are processing a Read at the same time. A workaround is to have a special ReadLock in the Epoll class for _HandleRead so that a one-to-one match between calls to _HandleRead and TriggerReceived happens.

The SendBuf() routine will break apart buffers larger than 32,768 into blocks and send then in order after each IOCP write event triggers a callback. However, if the user application calls Send() as well, which in turn calls WSASend() before the entire process is completed, the order of data can become corrupted.

建议继承顺序调整为TCrossData->TAbstractCrossConnection->TCrossConnection->TCrossHttpConnection->TCrossWebSocketConnection—》TCrossMbedTlsConnection/TCrossOpenSSLConnection，目前你这样的继承顺序满足不了，我要ssl和非ssl http Server在一个程序中同时用的状况

_ListenSocket : ICrossSocket;
....
....
_ListenSocket := TCrossSocket.Create(0);
....
....
_ListenSocket := nil; // 不能自动销毁对象，是否存在循环引用？？

环境：Windows7 / RAD Studio 10.1

在win7上跑https server，运行一段时间再关闭，提示TIOListen有内存泄露。

比如我的连接方法是
aClientSocket.Connect('127.0.10.10',8888);
那么LocalAddr应该是127.10.10.1,LocalPort应该是8888(新连接: Connect 127.0.0.1:3431 [at 127.0.10.10:8888])
你的控件支持多端口同时监听，如果没有这两个属性，那么不知道客户是通过哪个端口连接上来的

And also would be cool if English would be used as main communication language, since there are many international users around.

Currently projet is little bit hard to follow.

Keep up the good work!

ICrossHttpRequest 的ContentEncoding 缺少。请帮忙公开一下。

...

在ubuntu 上，用delphi 10.2 编译例子。
用ab 测试，基本上跑不下来。

1. TMbedtls_SSL_Cache_Context = record
   chain: PMbedtls_SSL_Cache_Entry; // !< start of the chain
   timeout: Integer; // !< cache entry timeout
   max_entries: Integer; // !< maximum entries
   mutex: TTlsMutex; // !< mutex 不是pointer
   end;

2. TMbedtls_Entropy_Context = record
   accumulator_started: Integer;
   accumulator: TMbedtls_SHA512_Context;
   source_count: Integer;
   source: array [0 .. MBEDTLS_ENTROPY_MAX_SOURCES - 1]
   of TMbedtls_Entropy_Source_State;
   havege_data: TMbedtls_Havege_State;
   mutex: TTlsMutex; // !< mutex 不是pointer
   initial_entropy_run: Integer;
   end;

3. TMbedtls_CTR_DRBG_Context = record
   counter: array [0 .. 15] of Byte; // !< The counter (V).
   reseed_counter: Integer; // !< The reseed counter.
   prediction_resistance: Integer; // !< This determines whether prediction
   // resistance is enabled, that is
   // whether to systematically reseed before
   // each random generation.
   entropy_len: Size_T; // !< The amount of entropy grabbed on each
   // seed or reseed operation.
   reseed_interval: Integer; // !< The reseed interval.

   aes_ctx: TMbedtls_AES_Context; // !< The AES context.

   //
   // * Callbacks (Entropy)
   //
   f_entropy: Pointer; // !< The entropy callback function.

   p_entropy: Pointer; // !< The context for the entropy function.

   mutex: TTlsMutex; //不是pointer
   end;

建立一个空的Net.Posix.inc后，修改Net.CrossSocket.Epoll单元的TEpollCrossSocket._HandleWrite方法542行PosixSend为TSocketAPI.Send（报PosixSend函数未定义）：
// 发送数据
//LSent := PosixSend(LConnection.Socket, LSendItem.Data, LSendItem.Size);
LSent := TSocketAPI.Send(LConnection.Socket, LSendItem.Data, LSendItem.Size);
可以通过编译，但测试在Linux（Ubuntu17.04 Server+Delphi10.2）下工作不正常！

把所有weak替换为unsafe检测无泄漏，但用weak则有泄漏
我用fastmm4 release 4992检测的

hi Souledge,

thanks for your patience with me,

I have a feature request.

Guess a server with thousands of websocket connections.

Now websocket inherit from httpconnection, so every connection uses a lot of memory (many fields, methods, properties).

My proposal is, after http upgrade 101 websocket established, free the httpconnection interface, leaving only the necessary for the websocket connection.

This will make the websocket server lighter on resources so more scalable.

Thank you.

Roberto

Since there are a limited number of I/O threads, if you attempt to simultaneously SSL handshake connections totaling more than the number of available I/O threads, a dead-lock will occur. This happens because the SSL object is locked during the TriggerConnected/TriggerReceived callback. These operations should occur outside of the SSL lock to avoid the issue.

在 Net.CrossSocket.Iocp.pas中：

procedure TIocpCrossSocket.StopLoop;
var
I: Integer;
begin
if (FIoThreads = nil) then Exit;

CloseAll;

while (ListensCount > 0) or (ConnectionsCount > 0) do Sleep(1);

for I := 0 to Length(FIoThreads) - 1 do
PostQueuedCompletionStatus(FIocpHandle, 0, 0, POverlapped(SHUTDOWN_FLAG));
WaitForMultipleObjects(Length(FIoThreadHandles), Pointer(FIoThreadHandles), True, INFINITE);
CloseHandle(FIocpHandle);
for I := 0 to Length(FIoThreads) - 1 do
FreeAndNil(FIoThreads[I]);
FIoThreads := nil;
FIoThreadHandles := nil;
end;

windows的WaitForMultipleObjects好像只能处理64个等待对象，如果线程数量超过64个，可能会有问题。现在或未来会有的计算机的核心数超过64的。
https://msdn.microsoft.com/en-us/library/ms687025(VS.85).aspx

是否可以改为：
procedure TIocpCrossSocket.StopLoop;
var
I: Integer;
begin
if (FIoThreads = nil) then Exit;

CloseAll;

while (ListensCount > 0) or (ConnectionsCount > 0) do Sleep(1);

for I := 0 to Length(FIoThreads) - 1 do
PostQueuedCompletionStatus(FIocpHandle, 0, 0, POverlapped(SHUTDOWN_FLAG));
for I :=0 to Length(FIoThreads) -1 do
#WaitForSingleObjects(FIoThreadHandles, INFINITE); //并不会比Multi多等时间，阻塞在最长时间的Thread后，后面其他线程就直接过。
CloseHandle(FIocpHandle);
for I := 0 to Length(FIoThreads) - 1 do
FreeAndNil(FIoThreads[I]);
FIoThreads := nil;
FIoThreadHandles := nil;
end;

很高兴你的程序质量有大幅提升，不仅使用了很多Delphi的新语法，如匿名函数、泛型、并行库等，而且还重构了代码，采用面向接口的编程，更方便使用和扩张，同时更好的防止内存泄漏。

为了得到更高的性能，希望作者能有更多改进：
1.每个线程有自己独立的完成端口（消息队列），不访问其他线程的完成端口。不像现在每个Server都创建完成端口，而是每次启动时就固定创建好CPU个线程和完成端口。每个TCP连接或者相关的TCP连接，只由此线程处理。这样可以不用处理粘包，避免IO乱序（单Socket大量包时后到的io先被处理）问题，可以大量避免锁的使用，性能得到提升。
2.在改进第1步基础上，可再进一步改进：建立CPU亲和性，每个CPU运行一个工作线程，每个线程建立独立的内存管理器，线程启动时申请大块内存，由线程的内存管理器分配内存。

if you have time should try to adopt some enhancements to the socket base,
investigate to use zerocopy SNDBUF network stack together with vectorized I/O (scatter gather)

search for info "windows scatter gather zerocopy", you will found many source code examples
scatter gather is supported in posix too

TEpollConnection.Destroy never gets called because connection reference count never reaches zero. This leads to a leak and other unwanted behaviors on Linux.

更新48版本后64位的有问题，打不开网页，而且用NativeInt好像没有Int64合适？

https://github.com/nodejs/http-parser
这个解析器性能非常高

With EPOLL APIs the http request/response for multiple requests originating from the same IP can arrive on the same socket. What can occur is a scenario where the http response header is sent before the http response body. Consider the following scenario:

Client Socket #28 receives:
00000000: 47 45 54 20 2F 20 48 54 | GET / HT
00000008: 54 50 2F 31 2E 31 0D 0A | TP/1.1..
00000010: 48 6F 73 74 3A 20 31 39 | Host: 19
00000018: 32 2E 31 36 38 2E 31 2E | 2.168.1.
00000020: 38 35 3A 38 30 38 30 0D | 85:8080.
00000028: 0A 0D 0A 47 45 54 20 2F | ...GET /
00000030: 20 48 54 54 50 2F 31 2E | HTTP/1.
00000038: 31 0D 0A 48 6F 73 74 3A | 1..Host:
00000040: 20 31 39 32 2E 31 36 38 | 192.168
00000048: 2E 31 2E 38 35 3A 38 30 | .1.85:80
00000050: 38 30 0D 0A 0D 0A | 80....
<Note: 2 requests on the same client socket from EPOLL>

At the same moment, Client Socket #29 receives:
00000000: 47 45 54 20 2F 20 48 54 | GET / HT
00000008: 54 50 2F 31 2E 31 0D 0A | TP/1.1..
00000010: 48 6F 73 74 3A 20 31 39 | Host: 19
00000018: 32 2E 31 36 38 2E 31 2E | 2.168.1.
00000020: 38 35 3A 38 30 38 30 0D | 85:8080.
00000028: 0A 0D 0A 47 45 54 20 2F | ...GET /
00000030: 20 48 54 54 50 2F 31 2E | HTTP/1.
00000038: 31 0D 0A 48 6F 73 74 3A | 1..Host:
00000040: 20 31 39 32 2E 31 36 38 | 192.168
00000048: 2E 31 2E 38 35 3A 38 30 | .1.85:80
00000050: 38 30 0D 0A 0D 0A | 80....

This is okay, as long as #28 responds:
Header
Body
Header
Body
While socket #29 responds:
Header
Body
Header
Body

But what is actually happening is:
Header
Header
Header
Header
Body
Body
Body
Body

This causes 2 of the 4 request to fail on the client side.

D10.2 编译时,peoll_pwait 未定义(连接库里没这函数),是pwait不是wait

建议不要以内部内存管理器为依据！

今天修改部分代码时很多对象没有释放，但关闭时没有任何提示

在群里有人说接口还是存在一些问题，然后群里大牛菜根跟踪发现问题所在！

Issue:

Net.CrossSocket.Iocp
接口没有const，这样会自动引用计数+-
function TIocpCrossSocket.CreateListen(AOwner: ICrossSocket;
AListenSocket: THandle; AFamily, ASockType, AProtocol: Integer): ICrossListen;
begin
Result := TIocpListen.Create(AOwner, AListenSocket, AFamily, ASockType, AProtocol);
end;

procedure TIocpCrossSocket.Listen(const AHost: string; APort: Word;
const ACallback: TProc<ICrossListen, Boolean>);
begin

这里将对象以接口方式传输过去，返回的时候计数为0，把self释放了
LListen := CreateListen(Self, LListenSocket, LAddrInfo.ai_family,
LAddrInfo.ai_socktype, LAddrInfo.ai_protocol);

end;

Fix:

建立一个新的接口基类，我放到了Utils.Utils单元

type
TInterfacedObjectNoARC = class(TObject, IUnknown)
protected
function QueryInterface(const IId: TGUId; out Obj): HResult; stdcall;
function _AddRef: Integer; stdcall;
function _Release: Integer; stdcall;
end;

{ TObjectInterface }

function TInterfacedObjectNoARC.QueryInterface(const IId: TGUId; out Obj): HResult;
begin
if GetInterface(IId, Obj) then
Result := 0
else
Result := E_NOINTERFACE;
end;

function TInterfacedObjectNoARC._AddRef: Integer;
begin
Result := -1;
end;

function TInterfacedObjectNoARC._Release: Integer;
begin
Result := -1;
end;

将所有继承自TInterfacedObject的对象都改为继承自TInterfacedObjectNoARC

这样可以彻底解决将接口当指针使用的问题。

I have been testing under Linux and encountering a few issues. This is the stress tool I use. One issue is Broken Pipe exceptions.

To reproduce use URLStress,
https://blogs.msdn.microsoft.com/friis/2010/12/28/urlstress-a-simple-gui-tool-with-source-code-to-stress-your-favorite-web-server/
Direct download: https://msdnshared.blob.core.windows.net/media/MSDNBlogsFS/prod.evol.blogs.msdn.com/CommunityServer.Components.PostAttachments/00/10/10/96/08/URLSTRESS.zip

hello Souledge,
I propose a little modify to raise the throughput of HTTP GET files, using NT TransmitFile() async over IOCOMP.

https://docs.microsoft.com/it-it/windows/desktop/api/mswsock/nf-mswsock-transmitfile

TF_USE_KERNEL_APC and TF_WRITE_BEHIND will make sendfile completely asynchronous with a callback when completed or failed (using overlapped IOCP binding),
is the same way used in IIS https://docs.microsoft.com/en-us/previous-versions/iis/6.0-sdk/ms525816%28v%3dvs.90%29

before call transmitfile use this flag when open the file handle that the
TransmitFile function transmits. Since the operating system reads the file data sequentially, you can improve caching performance by opening the handle with FILE_FLAG_SEQUENTIAL_SCAN

perhaps you can do a DEFINE USEKERNEL, $IFDEF WIN64 $IFDEF USEKERNEL -> then use this way, we will have a lot better throughput thanks to the kernel cache and async callback

和群友聊天时，群友说openssl有问题已经把你crosssocket的ssl换成了mbedtls

procedure TCrossMbedTlsWebSocketServer.SetPrivateKey(APKeyBuf: Pointer;
APKeyBufSize: Integer);
begin
MbedCheck(mbedtls_pk_parse_key(@FPKey, APKeyBuf, APKeyBufSize + 1, nil, 0), //必须加一
'mbedtls_pk_parse_key SetPrivateKey:');

_UpdateCert;
end;

procedure TCrossMbedTlsWebSocketServer.SetCertificate(ACertBuf: Pointer;
ACertBufSize: Integer);
begin
MbedCheck(mbedtls_x509_crt_parse(@FCERT, ACertBuf, ACertBufSize + 1), //必须加一
'mbedtls_x509_crt_parse SetCertificate:');

_UpdateCert;
end;

The CryptoLib library is loaded by the SSLLib library. This means that in Net.OpenSSL.pas in procedure LoadSslLibs the CryptoLib has to be loaded first. Otherwise the SSLLib library will load the CryptoLib from the path and not from the provided _SslLibPath variable.

I encountered this when using it on Linux with my own builded SSL libraries in the application folder. The SSLLib was loading the wrong CryptoLib until I reversed the order.

参数： 创建线程数=0
测试函数：TCrossHttpResponse.SendFile(const AFileName: string。。
在文件打开成功后，增加Writeln('Create:'+LStream.FileName);
在语句FreeAndNil(LStream)前增加Writeln('Free:'+LStream.FileName);
运行，用于发布一个网页（含有图片，js 等），显示如下：
CrossHttpServer start at port:80, IO threads:9
Press enter stop
Create:E:\Myworker\html/index.html
Free:E:\Myworker\html/index.html
Create:E:\Myworker\html/iamge/ic_launcher.png
Create:E:\Myworker\html/iamge/ic_launcher.png
Create:E:\Myworker\html/iamge/guide1.png
Free:E:\Myworker\html/iamge/guide1.png
Create:E:\Myworker\html/iamge/guide2.png
Create:E:\Myworker\html/iamge/guide2.png
Create:E:\Myworker\html/iamge/guide3.png
Free:E:\Myworker\html/iamge/guide2.png
Free:E:\Myworker\html/iamge/guide2.png
Free:E:\Myworker\html/iamge/guide3.png
Free:E:\Myworker\html/iamge/guide2.png
Free:E:\Myworker\html/iamge/guide3.png
Create:E:\Myworker\html/iamge/guide5.png
Free:E:\Myworker\html/iamge/guide5.png
Free:E:\Myworker\html/iamge/ic_launcher.png
Create:E:\Myworker\html/iamge/guide4.png
Create:E:\Myworker\html/iamge/guide4.png
Create:E:\Myworker\html/iamge/guide6.png
Free:E:\Myworker\html/iamge/guide4.png
Free:E:\Myworker\html/iamge/guide6.png
感觉线程都乱套了？ 用一个线程时正确的。

1.需要多少硬件配置/带宽,内存,你的测试环境是如何的,双机什么配置,
专业交换机,/普通IT用户的无线路由器/最垃圾的电脑白丁家路由器

2.现实生产环境,BAT这流氓家服务器质量,大概到什么浏览,需要什么配置

3.同时在线多少人,需要多少内存
我测试32位程序时,20万连接,需要200M内存,包括进程自己需要的内存,Win7平台
4.每秒发起多少连接时,需要什么样的硬件配置,现实环境,速度是如何, 其它有什么产品有更高并发,如何提升
我测试下来,不管Win还是Linux,新建连接数,都只有每秒1000-2000(1000M,OpenWrt路由,双i7, Win7 64bit)
(本机IP或回路,每秒新建连接大概7K-1W)
5.每秒IO次数,如何优化
我每秒IO量(1Byte-1KByte一包),和新连接数速度,差不多,实在寒酸,基本用户都不可能满足,更不要说DDOS
6.一些测试工具推荐,瓶颈查找方法或工具等

以上是我自己遇到的情况, 用你的TestCrossSocket/CrossHttpConsole(非SSL) ,
AB,和自己写的socket多线程,异步并发程序测试得出结果

在win7上跑https server，在android和iphone手机上用idhttp发请求，在idcookmanager添加cookie信息后再发post请求，https server有时候收不到cookie信息，大概１０次中有５，６收不到。但是客户端使用win32程序，用同样的控件和代码发送请求，就没问题。后来试用过，一定要在idhttp的customheader中把cookie信息加上，https server上就都能正常收到cookie信息。
但是之前用diocp的框架来做server，用同样的控件和代码，在android和iphone手机上用idhttp发请求都是可以正常接收到cookie信息的。

FClientSocket.Connect('127.10.10.1',7000,
procedure(aSocketHandle: THandle;ASuccess: Boolean)
begin
TThread.Synchronize(nil,
procedure
begin
if ASuccess then
ShowMessage('Connect SUCCESS!! aSocketHandle:' + aSocketHandle.ToString)
else
ShowMessage('Connect FAILED!!aSocketHandle:' + aSocketHandle.ToString);
end);
end);
连接服务端成功后，
procedure TBaseSocket.TriggerConnected(AConnection: ICrossConnection);
begin
inherited;
PrintLog((Format('Connected:PeerAddr:%s;PeerPort:%d;ConnectType:%s;socket:%d',[AConnection.PeerAddr,
AConnection.PeerPort,
GetEnumName(TypeInfo(TConnectType),Ord(AConnection.ConnectType)),
AConnection.Socket])));
end;
以后事件的打印结果：
Connected:PeerAddr:127.10.10.1;PeerPort:7000;ConnectType:ctConnect;socket:820

如果获取LocalAddr与LocalPort（从服务端连接事件可以看到本地连接所使用的端口是4930）？

HTTP客户端以POST方式提交请求，在服务器端中无法访问，request 的params和query中都没有提交的数据。因此在OnHttpRequest事件中无法处理POST请求。

比如 connent，回调函数只能判断是否成功，拿不到出错信息。
还有 OnDisconnected 也是，不知道是什么原因断开。。

breaking the barrier of 100k ops/sec on my I7

https://en.delphipraxis.net/topic/711-rtl-patch-to-speedup-350-server-app-tested-on-i7-quad-core/?tab=comments#comment-5668

please tell me if you found errors or troubles in your apps

constructor TCrossHttpServer.Create(AIoThreads: Integer);
var
I: Integer;
begin
inherited Create(AIoThreads); 在Win64时, 这行运行期,内部报错, 我没细看原因,你有空看看,

您好：
_HandleRead 这个处理函数，是在线程函数调用的
LRcvd := TSocketAPI.Recv(LConnection.Socket, FRecvBuf[0], RCV_BUF_SIZE);

在处理接受数据是是不是要考虑线程安全。对这方面不太了解。能给我具体说说吗？ 谢谢。

本想测试一下是否支持断点续传，结果放上去一个6GB的iso文件，直接下载不了

内存池减少碎片
端口重用可以开多个进程处理，这样就类似iis进程池

When you exchange and receive data less than 32768 bytes, it works without any problems. However, when transferring data larger than 32768, it sends the size of the stream to the front of the stream, which results in poor data sizing.

hello,

imho your code is well written, especially the IOCOMP windows code,
however let me express doubts about the scalability and reliability of EPOLL linux wrapper, using a single thread with global locking queue, you should avoid this

best regards
Roberto

Net.CrossSocket.Base中：
procedure TAbstractCrossSocket.CloseAll;
begin
CloseAllConnections;
CloseAllListens;
end;
在执行CloseAllConnections过程中，其他线程可能会接收了新的连接进来，导致CloseAllConnections不能关闭这些新连接，如果先CloseAllListens，是不是可以避免？
是否应该改为：
procedure TAbstractCrossSocket.CloseAll;
begin
CloseAllListens;
CloseAllConnections;
end;

感觉把close放到callback后面靠谱点，现在这样callnack传递的对象好像意义不大，另外zs群主说异步发送有时不返回，具体情况不清楚

According to the OpenSSL man pages, you must call ERR_clear_error() if you are using SSL_get_error() from multiple threads, otherwise errors in the queue appearing in from one thread will trigger errors from other threads (even other connections). You can reproduce this bug by setting up an IOCP server with OpenSSL, connect with an SSL client and then attempt to connect with a non-ssl client. Both connections will be dropped and errors will be reported on both connections.

please check the problems of this library with intel tbb malloc (causing AV stack overflow)

https://en.delphipraxis.net/topic/636-heavy-bug-102-103-rio-passing-parameter-pointers-in-anonymous-methods/

do we have memory overrun or use-after-free in the code (something as out of order getmem/freemem)?