Increase MAX_WORKER_THREADS so it can be used in high traffic websites about civetweb HOT 15 CLOSED

MAX_WORKER_THREADS is set to 1024 in case it has not been set before.
In case you compile it on your own (e.g. for a Linux system), you can just set this define to whatever value you like.

Windows 32 has a limit of about 2000 threads
(http://blogs.technet.com/b/markrussinovich/archive/2009/07/08/3261309.aspx, http://blogs.msdn.com/b/oldnewthing/archive/2005/07/29/444912.aspx).
Windows 64 could do 50k threads.

Since this is a limit due to memory architecture, Linux may have similar limits (that seem configurable: http://stackoverflow.com/questions/344203/maximum-number-of-threads-per-process-in-linux)

Increasing the number of threads to increase the number of connections seems to be a huge waste of memory. The alternative is a redesign:
https://groups.google.com/forum/#!topic/civetweb/LlHAM6xCSDA
https://groups.google.com/forum/#!topic/civetweb/sFHF75IDSH0

However, I would be interested to find out how the server behaves by just increasing the number of threads. Did you test it with 20480?

from civetweb.

I just checked (with cat /proc/sys/kernel/threads-max) my Mageia linux, it is 62108, on CentOS 7, 6187514.
I am using it in a ceph radosgw, serving large number of huge files, so I need several thousands connections.
I compiled with 20480, and I am using it normally, more than 7000 connections simultaneously, but hit the interface 10G limit.

from civetweb.

By "interface 10G limit" you mean, your test already transfers 10 gigabit per second, so the number of connection does not increase anymore. You could try it with slow clients, or even just standard telnet clients that don't do anything after connecting.

If you do have good experience with a some-10-k limit, there is not reason not to increase the default maximum define, at least for 64 bit. Yes, there would be much smarter ways to handle that much connections, but they take more time to implement than to just change one line - and in the meantime it does not hurt to change this number.

I would change it to a number where you would hit the port limit (64k), after doing some tests.

from civetweb.

May I ask you to tell me a little bit about your experience with using
CivetWeb for many clients?
It may help to understand how it is used and to improve quality.

from civetweb.

Hi, I limited the client speed in my test, so I hit the 20480 connections limit I put, all working fine, there is no single error message, thank you.
But I use it with radosgw as integrated server, so I don't now how much resources it (civetweb only) uses.

Regards
Mustafa

from civetweb.

Hi,

You are using a 64 bit system - how much physical memory does it have?
Could you tell me how much memory your system takes with the 20480 threads
and how much with the default 1024 (or 50)?
Could you try what limit you hit when compiling with 102400 threads and
using slow clients?

Regards,
bel

On Thu, Nov 20, 2014 at 8:54 AM, Mustafa Muhammad [email protected]
wrote:

Hi, I limited the client speed in my test, so I hit the 20480 connections
limit I put, all working fine, there is no single error message, thank you.
But I use it with radosgw as integrated server, so I don't now how much
resources it (civetweb only) uses.

Regards
Mustafa

—
Reply to this email directly or view it on GitHub
#50 (comment).

from civetweb.

Hi,
My system has 377 GB memory, but since civetweb is build with radosgw, I can't measure the impact of civetweb alone (when downloading files), running without clients the system uses only 3 GBs, I created a load in which I use curl in loops from different machines, I hit my 20480 connection limit, returning some json, memory consumption only became 4 GBs, so the differenc is 1 GB, I'll try compiling with 102400 threads but the slow clients thing will mix radosgw usage and civetweb usage, so I'll try the curl thing again

Regards
Mustaf

from civetweb.

Please tell me when you found time to test what happens with the 102400 thread limit.
I think you might get stuck at 64k. I will than change the default limit in the source.

from civetweb.

Hi, I did compile with 100k, I am hitting a limit at about 32k, it is not number of open files nor process count, but this is where I stopped.

Regards
Mustafa

from civetweb.

In the IP protocol, there is a 16 bit port number, so the number of ports is limited to 64k.
I was just not sure if any connection uses one or two ports, so I expected you hit a limit either below 32867 or 65535. Did you test with the client at the same machine?

from civetweb.

The 64k limit is actually per IP address, so if your server has 2 ip addresses, it could have 128k connections - but this is rather theoretical, I don't think a limit above 64k would make any sense in a practical application.
I think 64k would be the highest meaningful limit.

from civetweb.

The limit is now 64k.

It is still interesting to know what number you can actually reach at maximum.
Please let us know any new record number of connections.

from civetweb.

I don't think this is related to the ports, and no, not the same machine, this is about the running threads, the maximum number of threads I got is 32K, before testing the connections, I checked the number of threads (ps aux -L | grep civetweb | wc -l).
I'll report any higher number I get (if I could find the problem)

Regards
Mustafa

from civetweb.

Civetweb now prints out how much worker threads have been created (if it were less than configured).

I tried to reach >30000 threads, but Linux is, for whatever reason, full of limits to basically everything.
And most sites you find discussing these limits are hopelessly out of date for a current 64 bit system increased the number of threads in the kernel settings to 100000 and the number of memory mappings per process to 200000, reduced the stack size. but still there is a limit around 30887.
I will keep on trying, but I might already come to the RAM limit of my machine.

from civetweb.

There are a couple of limits, I had to adjust for Debian/Ubuntu-Linux.

/etc/sysctl.conf:
kernel.threads-max = 250000
vm.max_map_count = 500000
kernel.pid_max = 999999
fs.file-max = 250000

/etc/security/limits.conf:

•           soft    nproc           200000
  

•           hard    nproc           200000
  

•           soft    as              unlimited
  

•           hard    as              unlimited
  

•           soft    nofile          200000
  

•           hard    nofile          200000
  

•           soft    rtprio          20
  

•           hard    rtprio          20
  

•           soft    sigpending      200000
  

•           hard    sigpending      200000
  

  (Not sure if I actually need the later two).

The stack limit is already set in the Makefile.

Then I can reach 65536 worker threads, and the server runs stable with it.
Memory consumption is high but stable:
VmPeak: 6143860 kB
VmSize: 5568228 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 563784 kB
VmRSS: 0 kB
VmData: 5553708 kB
VmStk: 136 kB
VmExe: 988 kB
VmLib: 3088 kB
VmPTE: 10712 kB
VmSwap: 1956356 kB
Threads: 65539

When running the clients at the same machine as the server, I reach about
28233 connections at the same time. Then creating a new connection fails
(connect: Cannot assign requested address): This might be the port limit
(or?). Anyway, that much clients on the same machine is not a reasonable
use case anyway.

On Sat, Nov 29, 2014 at 8:08 AM, Mustafa Muhammad [email protected]
wrote:

I don't think this is related to the ports, and no, not the same machine,
this is about the running threads, the maximum number of threads I got is
32K, before testing the connections, I checked the number of threads (ps
aux -L | grep civetweb | wc -l).
I'll report any higher number I get (if I could find the problem)

Regards
Mustafa

—
Reply to this email directly or view it on GitHub
#50 (comment).

from civetweb.