に影響されて自分用に検証してみます
c: connections
Language | Frameworks | RPS(c=10) | RPS(c=100) | RPS(c=1K) |
---|---|---|---|---|
Python | japronto | 35192.51 |
182209.20 |
313111.38 |
Rust | hyper | 31478.74 |
181978.18 |
250753.83 |
Nim | jester | 37875.16 |
180562.05 |
267158.10 |
Golang | fasthttp | 32896.92 |
178742.38 |
272827.53 |
Scala | colossus | 30223.88 |
153630.81 |
173208.76 |
Python | gunicorn + meinheld | 30511.41 |
142269.85 |
147096.10 |
Golang | net/http | 27543.16 |
112160.14 |
126247.31 |
Golang | echo | 27501.64 |
107365.95 |
121058.50 |
Golang | gin | 27195.70 |
102053.41 |
117794.30 |
Scala | Finatra | 25884.09 |
101039.92 |
95836.10 |
Rust | iron | 32352.37 |
79042.21 |
78359.21 |
Scala | finch | 23548.16 |
67201.73 |
67690.82 |
Elixir | cowboy | 21982.09 |
62189.22 |
58050.46 |
Scala | akka-http | 14829.31 |
60694.92 |
65393.73 |
Scala | Play Framework 2.x + netty | 21157.73 |
54774.22 |
59055.88 |
Rust | rocket | nan |
50183.59 |
nan |
Elixir | cowboy + plug | 20930.40 |
48501.17 |
42336.23 |
Scala | http4s | 20936.95 |
41011.60 |
41406.29 |
Elixir | cowboy2 + plug | 17279.92 |
33547.23 |
30558.31 |
Ballerina | Ballerina | 13055.05 |
26570.28 |
32383.85 |
Python | gunicorn + flask + meinheld | 15485.77 |
21216.30 |
20185.63 |
Elixir | cowboy + phoenix | 15972.04 |
19897.84 |
29068.48 |
Python | bjoern + bottle | 14890.77 |
17342.36 |
16298.68 |
Elixir | cowboy2 + maru | 13616.37 |
15345.62 |
14410.75 |
Python | gunicorn | 7679.21 |
11505.21 |
10999.50 |
Node.js | express | 9871.12 |
9844.02 |
9687.88 |
Python | uwsgi + flask | 6575.53 |
8327.47 |
8272.04 |
Python | gunicorn + flask | 3961.87 |
7499.01 |
7328.44 |
Python | gunicorn + django | 3459.65 |
4085.68 |
5958.89 |
Ruby | Ruby on Rails 5.x + unicorn | 3990.49 |
3748.68 |
3941.27 |
Python | bottle | 2403.29 |
2318.08 |
1752.36 |
Ruby | Ruby on Rails 5.x + puma | 3075.53 |
2134.67 |
2241.91 |
Python | flask | 1077.82 |
1207.28 |
1205.87 |
さまざまな言語実装をリクエストの処理能力に焦点を当ててベンチマークすることが目的です。
- "Hello world" の文字列をレスポンスとして返却する
機能のみを実装し、検証を行います。
言語の宣伝や批判を行うものではありません。 言語やアプリケーションサーバー、フレームワーク毎にそれぞれ優れている点があり、パフォーマンスだけに基づいて選ぶべきではないと思います。
- Erlang/OTP 21
- Elixir 1.7.4 (compiled with Erlang/OTP 21)
- Python 3.7.1
- Nim 0.19.0
- Golang 1.11.2
- Rust 1.30.1
- Scala 2.12.7
- sbt 1.2.6
- Ruby 2.5.1
- Ballerina 0.983.0
- Node.js 11.3.0
下記の環境を検証に利用。
- NIFCLOUD type: large
- Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz (4 cores)
- 4 GB
- Ubuntu 18.04 LTS
- NIFCLOUD type: large16
- Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz (4 cores)
- 16 GB
- Ubuntu 18.04 LTS
PrivateLan を作成し、アプリケーションサーバー と 負荷検証(wrk)サーバー 用に Network を構築します。
[App server] : 192.168.10.10 ----------- [Load server] : 192.168.10.5
/etc/sysctl.conf
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_dynaddr = 1
net.ipv4.tcp_rfc1337 = 1
net.ipv4.tcp_fin_timeout = 10
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_slow_start_after_idle = 0
net.core.somaxconn = 65535
net.ipv4.tcp_max_syn_backlog = 65535
/etc/security/limits.conf
* soft nofile 65536
* hard nofile 65536
root soft nofile 65536
root hard nofile 65536
/etc/pam.d/common-session
session required pam_limits.so
/etc/pam.d/common-nonsession
session required pam_limits.so
負荷ツール wrk で検証。
5回計測し、 RPS がもっとも良い結果をピックアップする。
wrk -t 4 -c 100 -d 30s --timeout 2000 http://0.0.0.0:9292
wrk による負荷検証中のモニタリング結果を取得。
dstat -tf -cm -C 0,1,2,3,total --noheaders 1
cd servers/cowboy-with-plug
MIX_ENV=prod mix do deps.get, compile
MIX_ENV=prod mix run --no-halt
# wrk -t 4 -c 100 -d30s --timeout 2000 http://192.168.10.10:4000/
Running 30s test @ http://192.168.10.10:4000/
4 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 2.23ms 1.96ms 46.55ms 90.02%
Req/Sec 12.22k 1.53k 46.68k 85.68%
1459851 requests in 30.10s, 273.14MB read
Requests/sec: 48501.17
Transfer/sec: 9.07MB
cd servers/cowboy2-with-plug
MIX_ENV=prod mix do deps.get, compile
MIX_ENV=prod mix run --no-halt
# wrk -t 4 -c 100 -d30s --timeout 2000 http://192.168.10.10:4000/
Running 30s test @ http://192.168.10.10:4000/
4 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 3.11ms 2.32ms 46.92ms 87.94%
Req/Sec 8.45k 1.05k 33.10k 87.51%
1009746 requests in 30.10s, 188.92MB read
Requests/sec: 33547.23
Transfer/sec: 6.28MB
cd servers/cowboy
MIX_ENV=prod mix do deps.get, compile
MIX_ENV=prod mix run --no-halt
# wrk -t 4 -c 100 -d30s --timeout 2000 http://192.168.10.10:4000/
Running 30s test @ http://192.168.10.10:4000/
4 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.69ms 1.55ms 45.41ms 93.23%
Req/Sec 15.66k 1.60k 50.82k 86.01%
1871909 requests in 30.10s, 230.63MB read
Requests/sec: 62189.22
Transfer/sec: 7.66MB
cd servers/cowboy-with-phoenix
MIX_ENV=prod mix do deps.get, compile
PORT=4000 MIX_ENV=prod elixir --detached -S mix phx.server
# wrk -t 4 -c 100 -d30s --timeout 2000 http://192.168.10.10:4000/
Running 30s test @ http://192.168.10.10:4000/
4 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 5.14ms 2.15ms 59.25ms 99.23%
Req/Sec 5.00k 325.16 5.42k 92.08%
596999 requests in 30.00s, 111.70MB read
Requests/sec: 19897.84
Transfer/sec: 3.72MB
cd servers/cowboy2-with-maru
MIX_ENV=prod mix do deps.get, compile
MIX_ENV=prod mix run --no-halt
# wrk -t 4 -c 100 -d30s --timeout 2000 http://192.168.10.10:4000/
Running 30s test @ http://192.168.10.10:4000/
4 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 6.54ms 2.66ms 42.51ms 73.36%
Req/Sec 3.86k 341.07 7.03k 74.75%
460403 requests in 30.00s, 86.14MB read
Requests/sec: 15345.62
Transfer/sec: 2.87MB
cd servers/flask
pipenv sync
pipenv run server
# wrk -t 4 -c 100 -d30s --timeout 2000 http://192.168.10.10:4000/
Running 30s test @ http://192.168.10.10:4000/
4 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 81.80ms 3.59ms 121.52ms 93.11%
Req/Sec 303.11 43.47 460.00 56.42%
36258 requests in 30.03s, 5.22MB read
Requests/sec: 1207.28
Transfer/sec: 178.04KB
cd servers/uwsgi-with-flask
pipenv sync
pipenv run server
# wrk -t 4 -c 100 -d30s --timeout 2000 http://192.168.10.10:4000/
Running 30s test @ http://192.168.10.10:4000/
4 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 11.42ms 2.80ms 41.85ms 79.96%
Req/Sec 2.10k 411.37 2.60k 70.38%
250659 requests in 30.10s, 18.17MB read
Socket errors: connect 0, read 250625, write 0, timeout 0
Requests/sec: 8327.47
Transfer/sec: 618.05KB
cd servers/gunicorn-with-flask
pipenv sync
pipenv run server
# wrk -t 4 -c 100 -d30s --timeout 2000 http://192.168.10.10:4000/
Running 30s test @ http://192.168.10.10:4000/
4 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 12.79ms 1.91ms 55.56ms 97.01%
Req/Sec 1.89k 158.63 1.99k 95.08%
225575 requests in 30.08s, 33.77MB read
Requests/sec: 7499.01
Transfer/sec: 1.12MB
cd servers/gunicorn-with-flask-and-meinheld
pipenv sync
pipenv run server
# wrk -t 4 -c 100 -d30s --timeout 2000 http://192.168.10.10:4000/
Running 30s test @ http://192.168.10.10:4000/
4 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 4.76ms 1.84ms 45.72ms 91.67%
Req/Sec 5.34k 518.99 20.07k 96.84%
638610 requests in 30.10s, 98.05MB read
Requests/sec: 21216.30
Transfer/sec: 3.26MB
cd servers/bottle
pipenv sync
pipenv run server
# wrk -t 4 -c 100 -d30s --timeout 2000 http://192.168.10.10:4000/
Running 30s test @ http://192.168.10.10:4000/
4 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 547.07ms 2.89s 26.72s 96.07%
Req/Sec 626.82 291.02 1.74k 75.02%
69580 requests in 30.02s, 10.02MB read
Requests/sec: 2318.08
Transfer/sec: 341.83KB
cd servers/bjoern-with-bottle
pipenv sync
pipenv run server
# wrk -t 4 -c 100 -d30s --timeout 2000 http://192.168.10.10:4000/
Running 30s test @ http://192.168.10.10:4000/
4 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 5.86ms 2.24ms 126.73ms 99.37%
Req/Sec 4.36k 214.90 6.47k 98.08%
520322 requests in 30.00s, 49.62MB read
Requests/sec: 17342.36
Transfer/sec: 1.65MB
cd servers/gunicorn
pipenv sync
pipenv run server
# wrk -t 4 -c 100 -d30s --timeout 2000 http://192.168.10.10:4000/
Running 30s test @ http://192.168.10.10:4000/
4 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 8.16ms 1.65ms 41.82ms 96.60%
Req/Sec 2.90k 446.99 3.31k 94.90%
346116 requests in 30.08s, 51.82MB read
Requests/sec: 11505.21
Transfer/sec: 1.72MB
cd servers/gunicorn-with-meinheld
pipenv sync
pipenv run server
# wrk -t 4 -c 100 -d30s --timeout 2000 http://192.168.10.10:4000/
Running 30s test @ http://192.168.10.10:4000/
4 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 714.05us 0.87ms 41.18ms 99.85%
Req/Sec 35.81k 0.97k 47.65k 93.68%
4282263 requests in 30.10s, 657.51MB read
Requests/sec: 142269.85
Transfer/sec: 21.84MB
cd servers/iron
cargo run --release
# wrk -t 4 -c 100 -d30s --timeout 2000 http://192.168.10.10:4000/
Running 30s test @ http://192.168.10.10:4000/
4 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 410.29us 746.75us 40.81ms 99.88%
Req/Sec 39.72k 7.50k 48.68k 62.96%
2379133 requests in 30.10s, 258.66MB read
Requests/sec: 79042.21
Transfer/sec: 8.59MB
cd servers/rocket
cargo run --release
# wrk -t 4 -c 100 -d30s --timeout 2000 http://192.168.10.10:4000/
Running 30s test @ http://192.168.10.10:4000/
4 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.05ms 288.93us 7.84ms 92.74%
Req/Sec 12.61k 387.69 13.32k 70.68%
1510538 requests in 30.10s, 207.44MB read
Socket errors: connect 0, read 1510538, write 0, timeout 0
Requests/sec: 50183.59
Transfer/sec: 6.89MB
cd servers/go-net-http
go build main.go
./go-net-http
# wrk -t 4 -c 100 -d30s --timeout 2000 http://192.168.10.10:4000/
Running 30s test @ http://192.168.10.10:4000/
4 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 0.91ms 739.03us 34.46ms 96.93%
Req/Sec 28.25k 1.32k 58.73k 91.59%
3375936 requests in 30.10s, 412.10MB read
Requests/sec: 112160.14
Transfer/sec: 13.69MB
cd servers/go-fasthttp
go build main.go
./go-fasthttp
# wrk -t 4 -c 100 -d30s --timeout 2000 http://192.168.10.10:4000/
Running 30s test @ http://192.168.10.10:4000/
4 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 604.77us 1.32ms 41.20ms 99.64%
Req/Sec 44.92k 1.78k 47.38k 96.10%
5380177 requests in 30.10s, 749.12MB read
Requests/sec: 178742.38
Transfer/sec: 24.89MB
cd servers/jester
nim c -d:release --threads:on src/app.nim
./src/app
# wrk -t 4 -c 100 -d30s --timeout 2000 http://192.168.10.10:4000/
Running 30s test @ http://192.168.10.10:4000/
4 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 560.12us 0.86ms 40.91ms 99.85%
Req/Sec 45.45k 1.82k 59.51k 83.94%
5434775 requests in 30.10s, 751.54MB read
Requests/sec: 180562.05
Transfer/sec: 24.97MB
cd servers/hyper
cargo run --release
# wrk -t 4 -c 100 -d30s --timeout 2000 http://192.168.10.10:4000/
Running 30s test @ http://192.168.10.10:4000/
4 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 551.41us 0.86ms 40.94ms 99.84%
Req/Sec 45.84k 1.34k 65.29k 90.26%
5477391 requests in 30.10s, 663.40MB read
Requests/sec: 181978.18
Transfer/sec: 22.04MB
cd servers/play2-netty
sbt "start -Dhttp.port=4000"
# wrk -t 4 -c 100 -d30s --timeout 2000 http://192.168.10.10:4000/
Running 30s test @ http://192.168.10.10:4000/
4 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 2.22ms 2.53ms 62.23ms 92.11%
Req/Sec 13.76k 0.96k 16.61k 82.50%
1643383 requests in 30.00s, 200.61MB read
Requests/sec: 54774.22
Transfer/sec: 6.69MB
cd servers/rails-puma
rails server --environment production
# wrk -t 4 -c 100 -d30s --timeout 2000 http://192.168.10.10:4000/
Running 30s test @ http://192.168.10.10:4000/
4 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 9.35ms 6.72ms 99.99ms 78.65%
Req/Sec 1.07k 658.39 2.24k 54.50%
64117 requests in 30.04s, 16.08MB read
Requests/sec: 2134.67
Transfer/sec: 548.26KB
cd servers/rails-puma
bundle exec unicorn_rails -c config/unicorn.rb --env production
# wrk -t 4 -c 100 -d30s --timeout 2000 http://192.168.10.10:4000/
Running 30s test @ http://192.168.10.10:4000/
4 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 26.24ms 691.46us 41.35ms 87.10%
Req/Sec 0.94k 34.01 1.01k 75.00%
112502 requests in 30.01s, 30.15MB read
Socket errors: connect 0, read 112502, write 0, timeout 0
Requests/sec: 3748.68
Transfer/sec: 1.00MB
cd servers/ballerina
ballerina build hello_service.bal
ballerina run target/hello_service.balx
# wrk -t 4 -c 100 -d30s --timeout 2000 http://192.168.10.10:4000/
Running 30s test @ http://192.168.10.10:4000/
4 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 7.18ms 10.93ms 111.40ms 88.80%
Req/Sec 6.69k 523.04 10.85k 72.80%
799758 requests in 30.10s, 108.30MB read
Requests/sec: 26570.28
Transfer/sec: 3.60MB
cd servers/echo
go build
./echo
# wrk -t 4 -c 100 -d30s --timeout 2000 http://192.168.10.10:4000/
Running 30s test @ http://192.168.10.10:4000/
4 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 0.95ms 763.11us 34.54ms 95.85%
Req/Sec 27.02k 1.25k 46.54k 85.94%
3231644 requests in 30.10s, 394.49MB read
Requests/sec: 107365.95
Transfer/sec: 13.11MB
cd servers/gin
go build
./gin
# wrk -t 4 -c 100 -d30s --timeout 2000 http://192.168.10.10:4000/
Running 30s test @ http://192.168.10.10:4000/
4 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 0.99ms 480.82us 18.88ms 80.58%
Req/Sec 25.66k 0.97k 32.58k 78.72%
3071775 requests in 30.10s, 374.97MB read
Requests/sec: 102053.41
Transfer/sec: 12.46MB
cd servers/finatra
sbt "run -http.port=:4000"
# wrk -t 4 -c 100 -d30s --timeout 2000 http://192.168.10.10:4000/
Running 30s test @ http://192.168.10.10:4000/
4 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.37ms 1.72ms 38.96ms 90.10%
Req/Sec 25.43k 1.38k 44.45k 85.77%
3041298 requests in 30.10s, 420.56MB read
Requests/sec: 101039.92
Transfer/sec: 13.97MB
cd servers/akka-http
sbt run
# wrk -t 4 -c 100 -d30s --timeout 2000 http://192.168.10.10:4000/
Running 30s test @ http://192.168.10.10:4000/
4 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 2.60ms 6.70ms 270.24ms 94.87%
Req/Sec 15.30k 2.51k 33.21k 68.67%
1826908 requests in 30.10s, 242.18MB read
Requests/sec: 60694.92
Transfer/sec: 8.05MB
cd servers/japronto
pipenv sync
pipenv run server
# wrk -t 4 -c 100 -d30s --timeout 2000 http://192.168.10.10:4000/
Running 30s test @ http://192.168.10.10:4000/
4 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 578.94us 1.21ms 41.06ms 99.73%
Req/Sec 45.90k 1.77k 70.19k 92.09%
5484442 requests in 30.10s, 475.96MB read
Requests/sec: 182209.20
Transfer/sec: 15.81MB
cd servers/colossus
sbt run
# wrk -t 4 -c 100 -d30s --timeout 2000 http://192.168.10.10:4000/
Running 30s test @ http://192.168.10.10:4000/
4 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 0.91ms 5.93ms 222.35ms 99.66%
Req/Sec 38.69k 1.97k 61.96k 86.59%
4624180 requests in 30.10s, 590.93MB read
Requests/sec: 153630.81
Transfer/sec: 19.63MB
cd servers/finch
sbt run
# wrk -t 4 -c 100 -d30s --timeout 2000 http://192.168.10.10:4000/
Running 30s test @ http://192.168.10.10:4000
4 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 3.20ms 4.34ms 48.98ms 83.35%
Req/Sec 16.91k 1.03k 31.06k 83.19%
2022737 requests in 30.10s, 246.92MB read
Requests/sec: 67201.73
Transfer/sec: 8.20MB
cd servers/http4s
sbt run
# wrk -t 4 -c 100 -d30s --timeout 2000 http://192.168.10.10:4000/
Running 30s test @ http://192.168.10.10:4000
4 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 2.74ms 3.76ms 210.31ms 96.54%
Req/Sec 10.41k 1.93k 34.89k 84.97%
1234449 requests in 30.10s, 150.69MB read
Requests/sec: 41011.60
Transfer/sec: 5.01MB
cd servers/gunicorn-django
pipenv run server
# wrk -t 4 -c 100 -d30s --timeout 2000 http://192.168.10.10:4000/
Running 30s test @ http://192.168.10.10:4000
4 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 15.92ms 7.04ms 58.66ms 70.58%
Req/Sec 1.03k 317.53 1.64k 64.30%
122854 requests in 30.07s, 21.79MB read
Requests/sec: 4085.68
Transfer/sec: 742.12KB
cd servers/express
yarn install
yarn run start
# wrk -t 4 -c 100 -d30s --timeout 2000 http://192.168.10.10:4000/
Running 30s test @ http://192.168.10.10:4000
4 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 10.19ms 1.81ms 53.62ms 98.08%
Req/Sec 2.47k 128.61 2.72k 72.58%
295459 requests in 30.01s, 60.86MB read
Requests/sec: 9844.02
Transfer/sec: 2.03MB