http://stackoverflow.com/questions/15261612/post-request-with-wrk

When the URL you supply on the command line has no resource, you get a segmentation fault (at least on Linux).

For example, this seg faults:

./wrk -c 20 -r 1000 -t 4 http://localhost:8000

This doesn't:

./wrk -c 20 -r 1000 -t 4 http://localhost:8000/

Here's the part of my GDB session that helped me find the problem:

119     request.buf  = format_request(host, port, path, headers);
(gdb) 
118     cfg.addr     = *addr;
(gdb) 
119     request.buf  = format_request(host, port, path, headers);
(gdb) p host
$1 = 0x609048 "localhost"
(gdb) p port
$2 = 0x609068 "8000"
(gdb) p path
$3 = 0x800000002245 <Address 0x800000002245 out of bounds>

The workaround is easy; just include the resource in the URL.

Thanks for a very cool tool!

Tim

Would you please provide an example? Thanks a lot.

Environment

• Ubuntu 13.04
• Linux kernel 3.8.0-26 (x86_64)

Procedures

Here's what I did.

sudo apt-get install libcurl4-openssl-dev
make
./wrk ${URL}

Then I got the following error:

zmalloc: Out of memory trying to allocate 800000040 bytes
Aborted (core dumped)

I tried a number of different combinations of options (such as -t, -c and -d) but wrk produced the same error message regardless of what options were given.

Certain https requests seem to hang wrk. It seems strange and unlikely but it appears to happen when certain URLs contain a hyphen. See the last example below:

$ wrk -t1 -c1 -d1s http://github.com/favicon.ico
Running 1s test @ http://github.com/favicon.ico
  1 threads and 1 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    32.32ms    0.86ms  33.92ms   75.00%
    Req/Sec    15.00      0.00    15.00    100.00%
  16 requests in 1.02s, 1.78KB read
Requests/sec:     15.63
Transfer/sec:      1.74KB

$ wrk -t1 -c1 -d1s https://github.com/favicon.ico
Running 1s test @ https://github.com/favicon.ico
  1 threads and 1 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    34.61ms    1.22ms  35.93ms   71.43%
    Req/Sec    27.57      0.53    28.00     57.14%
  26 requests in 1.02s, 172.88KB read
Requests/sec:     25.48
Transfer/sec:    169.44KB

$ wrk -t1 -c1 -d1s http://github.com/windows-tile.png
Running 1s test @ http://github.com/windows-tile.png
  1 threads and 1 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    33.03ms    2.29ms  38.63ms   87.50%
    Req/Sec    15.00      0.00    15.00    100.00%
  16 requests in 1.03s, 1.86KB read
Requests/sec:     15.49
Transfer/sec:      1.80KB

$ wrk -t1 -c1 -d1s https://github.com/windows-tile.png
Running 1s test @ https://github.com/windows-tile.png
  1 threads and 1 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     0.00us    0.00us   0.00us    -nan%
    Req/Sec     0.00      0.00     0.00      -nan%
  0 requests in 2.00s, 8.00KB read
Requests/sec:      0.00
Transfer/sec:      4.00KB

Throw a dog a bone.

Some times it is nice to have all (timed) results in some tabular format to be used for example with gnuplot to plot some charts.

The data is already stored i guess in the stats.c it would be nice to have an extra flag to dump it to the stdout instead of the summary.

Need version only show

$ wrk -v
wrk 3.0.3 [kqueue] Copyright (C) 2012 Will Glozer

But show all commands with version

$ wrk -v
wrk 3.0.3 [kqueue] Copyright (C) 2012 Will Glozer
Usage: wrk <options> <url>
  Options:
    -c, --connections <N>  Connections to keep open
    -d, --duration    <T>  Duration of test
    -t, --threads     <N>  Number of threads to use

    -s, --script      <S>  Load Lua script file
    -H, --header      <H>  Add header to request
        --latency          Print latency statistics
        --timeout     <T>  Socket/request timeout
    -v, --version          Print version details

  Numeric arguments may include a SI unit (1k, 1M, 1G)
  Time arguments may include a time unit (2s, 2m, 2h)

Is it somehow possible to do a POST request instead of GET?

My server uses the Host:port to decide who to serve a request.

wrk sends a Host header with only the host, leaving out the port.

This causes my server to respond with an error, as it cannot
recognize a server based on just that host alone.

It's a one-line fix. Pull Request coming.

default use HTTP/1.1, thanks

as an option would be fantastic. If you're interested I can send a pull-request

"invalid URL: http://user:[email protected]:8080/something/event/1"

It is a valid url. it would be awesome if it supported it but at least should say something like "authentication not supported" in this case.

I'm still looking into this one myself, but I'm hoping you might have a quick answer for me. We're trying out the pipeline branch for our Round 6 test and we're running into a strange issue. For the 3 different tests that I've tried so far (Netty, Spray.io, Servlet), if I run wrk without any additional headers like so, everything seems to work just fine:

wrk --pipeline 32 -d 60 -c 256 -t 8

On the other hand, when I run wrk with our full compliment of headers, I get a ton of write errors:

wrk \
-H 'Host: localhost' \
-H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' \
-H 'Accept-Language: en-US,en;q=0.5' \
-H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) Gecko/20130501 Firefox/30.0 AppleWebKit/600.00 Chrome/30.0.0000.0 Trident/10.0 Safari/600.00' \
-H 'Cookie: uid=12345678901234567890; __utma=1.1234567890.1234567890.1234567890.1234567890.12; wd=2560x1600' \
-H 'Connection: keep-alive' \
--pipeline 32 -d 5 -c 256 -t 8

Output:

Running 5s test @ http://localhost:8080/plaintext
  8 threads and 256 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     0.00us    0.00us   0.00us    -nan%
    Req/Sec     0.00      0.00     0.00      -nan%
  53 requests in 6.00s, 7.30KB read
  Socket errors: connect 0, read 0, write 31863, timeout 3
Requests/sec:      8.83
Transfer/sec:      1.22KB

On the server end of things, Netty is constantly printing this error out to the terminal.

java.io.IOException: Connection reset by peer
    at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
    at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:225)
    at sun.nio.ch.IOUtil.read(IOUtil.java:193)
    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359)
    at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:259)
    at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:885)
    at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:226)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:72)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:434)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:397)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:327)
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:109)
    at java.lang.Thread.run(Thread.java:722)

This is normal to print out at the very end of the run when connections are closed, but instead these errors are constantly popping up during the test run.

We're starting to use your wonderful benchmarking tool for our performance blog https://github.com/TechEmpower/FrameworkBenchmarks but we ran into a strange issue with one of our tests. We modified one our tests and ran wrk to see how our results changed

wrk -r 100000 -c 256 -t 8 http://localhost/json

But this seemed to just run forever. So we turned down the number of connections to the lowest possible:

wrk -r 1 http://localhost/json

But this has the same problem, wrk just continued on forever. I've included the output of running curl -v http://url which will give you an example of what the response looks like.

* About to connect() to localhost port 8080 (#0)
*   Trying localhost... connected
> GET /hello_world/json HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: localhost:8080
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Thu, 04 Apr 2013 20:38:26 GMT
< Status: 200 OK
< Connection: close
< Content-Type: application/json
<
* Closing connection #0
{"message":"Hello World!"}

A ramp-up option, like httperf's --ra would be useful. A challenge I see is taking into consideration both the number of connections and the number of threads when doing a ramp-up.

I would like to point out that identifiers like "__APRINTF_H" and "__WRK_H" do not fit to the expected naming convention of the C++ language standard.
Would you like to adjust your selection for unique names?

I Tried to compile wrk on windows mingw64 and mingw32, but they faliled. Also Cygwin64 failed (Luajit compile error). Finally I was able to compile Cygwin32 and made it to work with 3 changes.

• in stats.c change powl -> pow
• in stats.c change sqrtl -> sqrt
• in net.c add: #include <sys/socket.h>

Usually more than 1 thread will fail, but if I try many times I'm able to get 2 or 3 threads to work.

I have precompiled binaries if somebody needs them.

Wrk is the only tool I have found that is capable of testing really fast web servers, thanks!

Do you plan to add this feature? I'd like to set the maximum number of requests per connection.
Thanks for your great tool !

Hi again @wg. It has been brought to our attention that in some of the results from our frameworks benchmarks, Wrk was reporting socket timeouts that we failed to capture in our automatic scraping of results that ultimately turn into charts and tables.

For example, see the raw results we captured for Go at 256 concurrency on EC2 (scroll down to the bottom of this file):

https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/results/ec2/20130404200504/json/go/raw

Would you be able to make the timeout period configurable? A quick spot-check of the code suggests (and I am probably wrong here!) that it's set to 2,000ms. I am probably wrong because other results such as the Cake tests show maximum latency higher than 2 seconds:

https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/results/ec2/20130404200504/query/cake/raw

(Incidentally, that file has many more socket timeouts reported.)

If you think we're doing something wrong unrelated to the timeout, we'd love your advice.

Some additional questions:

1. Can you give a brief description of the difference between a read error and a timeout? Is a "read error" indicative of the server responding with a 500 response?
2. Can you double-check the Max req/sec per thread calculation? It looks to always be an integer but is rendered with decimal-point precision. E.g., within that Go results file linked above, at 128 concurrency, the max is "8.00k."
3. Related to the above, do you have any conjecture for why we see a thread's Max req/seq that is so much higher than the average req/seq across all threads?

Thanks again for all of your help and I apologize for looping you in here. Please let me know how I can return the favor.

Script: test.js

# wrk -t12 -c400 -d30s http://127.0.0.1:8080/
Running 30s test @ http://127.0.0.1:8080/
  12 threads and 400 connections
[1]    45943 segmentation fault  wrk -t12 -c400 -d30s http://127.0.0.1:8080/

#  wrk --version
wrk 2.2.2 [kqueue] Copyright (C) 2012 Will Glozer

Run wrk random crash segmentation fault

tar xf /Library/Caches/Homebrew/wrk-3.0.1.tar.gz
==> make
make 
Building LuaJit...
LUAJIT scripts/wrk.lua
/bin/sh: ./luajit: No such file or directory
make: *** [obj/bytecode.o] Error 127
make: *** Waiting for unfinished jobs....

Homebrew/legacy-homebrew#23808

First, thanks for making this tools available.

I think I found an issue with the sampling logic of wrk. Is there any explanation how the sampling logic for per-thread latencies + rps is supposed to work? IMO it can't work reliably.

Current sampling code:

    uint64_t n = rand64(&thread->rand, thread->connections);
    uint64_t elapsed_ms = (time_us() - thread->start) / 1000;
    connection *c = thread->cs + n;
    uint64_t requests = (thread->complete / elapsed_ms) * 1000;

    pthread_mutex_lock(&statistics.mutex);
    stats_record(statistics.latency,  c->latency);
    stats_record(statistics.requests, requests);
    pthread_mutex_unlock(&statistics.mutex);

The logic seems to be the following:

• for latency:
  • take a random connection from all the thread's connections
  • take the latency of the last completed request of this connection as sample
• for RPS:
  • take the complete number of completed requests and divide by the total time elapsed and take this as sample

The basic problem is that this kind of sampling doesn't take its data from the current sampling interval.

• Latency sampling will be wrong if the latency of a request is chosen which was already chosen before. Consider a sequence of two especially long lasting (>> sampling interval) requests. Then, while the second request is running, the latency of the former one can be chosen several times. Depending on how big the first value was this skews the statistics more or less.
• RPS average is wrong because always the complete interval is taken into account. This skews the statistics towards things that happened in the beginning. Consider a linearly increasing RPS rate. The actual average is then exactly half of the end rate. Sampled values, however, will range from the very low start value to the actual average at the end. Because you then average another time when calculating the statistics the value will be wrong.

One solution would be to not rely on sampling at all and just record all the latencies to be analyzed later. Another one would be to carefully design sampling to really choose from the sampling interval.

I find the --pipeline option useful when testing the raw performance of my HTTP server. Could we resurrect the --pipeline option and make it a mere shortcut to:

--script TMPFILE

where TMPFILE is something like the following:

init = function(args)
   wrk.init(args)

   local r = {}
   for i=1,pipeline_length do
      r[i] = wrk.format(...)
   end
   req = table.concat(r)
end

request = function()
   return req
end

If you like this idea, I can come up with a pull request.

P.S: When would the pipeline branch be merged?

Understanding fully the significant undertaking this would represent, I am curious if SSL support is coming to Wrk in the future. We would like to be able to include a new test that exercises SSL as part of our benchmark suite.

We briefly experimented with http-perf, which supports SSL, but its use case seems different from what we have in mind.

Thanks!

We've noticed that on our most recent run, the latency data isn't consistently showing up. Here's an example (from the Phalcon framework):

Running 15s test @ http://172.16.98.122:8080/db?queries=1
  8 threads and 256 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     0.00us    0.00us   0.00us    -nan%
    Req/Sec     0.00      0.00     0.00      -nan%
  109876 requests in 15.14s, 24.60MB read
Requests/sec:   7258.34
Transfer/sec:      1.62MB

For some frameworks, we get latency data on every run, no problem, while other frameworks we'll sometimes get latency data, while other times we won't.

Here's output with the Phalcon framework at concurrencies of 16, 32, and 128

Running 15s test @ http://172.16.98.122:8080/db
  8 threads and 16 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     0.00us    0.00us   0.00us    -nan%
    Req/Sec     0.00      0.00     0.00      -nan%
  103443 requests in 15.00s, 23.16MB read
Requests/sec:   6896.11
Transfer/sec:      1.54MB

Running 15s test @ http://172.16.98.122:8080/db
  8 threads and 32 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     4.39ms    3.53ms  48.14ms   92.19%
    Req/Sec   597.51    490.57     1.00k    59.75%
  113664 requests in 15.00s, 25.44MB read
Requests/sec:   7577.91
Transfer/sec:      1.70MB

Running 15s test @ http://172.16.98.122:8080/db
  8 threads and 128 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     0.00us    0.00us   0.00us    -nan%
    Req/Sec     0.00      0.00     0.00      -nan%
  112338 requests in 15.00s, 25.15MB read
Requests/sec:   7489.04
Transfer/sec:      1.68MB

These were run in succession, without restarting the server. 32 concurrency is the only run that had latency data.

Depending on the test that was run (JSON, database access, multi-query, database updates), more or less runs would contain latency data. It seems to correlate that the more intensive tests had fewer runs with latency data.

This seemed to be a problem on both the master branch and pipelining branch.

If wrk runs out of file descriptors, this just shows up as socket connect errors. Perhaps it would be nicer to give some other error message to let the user know that ulimit -n should be used or /etc/security/limits.conf should be edited.

I came across this because the TechEmpower FrameworkBenchmark project uses wrk and high-concurrency test runs were running out of file descriptors because the limit was 16384 and wrk was trying to open 16384 connections, but there weren't enough file descriptors when you include file descriptors like stdin, stdout, stderr and so forth.

Is there any support for testing https? When I try it just hangs.

When measuring latency, average and std dev are only revelent if the distribution is guassian in distrition. Which is unlikely.

Better to show percentile based measurements. Like 90% of all requests served in 5ms, and 99% of requests served in 15ms.

See Gil Tene's talk "How not to measure latency" [1] for more info. Also be sure you are not falling into the "Coordinated Omission" trap where you end up measureing the latency wrong.

[1] http://www.infoq.com/presentations/latency-pitfalls

Hi,
I tried make wrk, but errors happens.

/wrk-master# make
Building LuaJIT...
make[1]: Entering directory `/root/wrk-master/deps/luajit/src'
HOSTCC    host/minilua.o
HOSTLINK  host/minilua
DYNASM    host/buildvm_arch.h
HOSTCC    host/buildvm.o
HOSTCC    host/buildvm_asm.o
HOSTCC    host/buildvm_peobj.o
HOSTCC    host/buildvm_lib.o
HOSTCC    host/buildvm_fold.o
HOSTLINK  host/buildvm
BUILDVM   lj_vm.s
ASM       lj_vm.o
CC        lj_gc.o
BUILDVM   lj_ffdef.h
CC        lj_err.o
CC        lj_char.o
BUILDVM   lj_bcdef.h
CC        lj_bc.o
CC        lj_obj.o
CC        lj_str.o
CC        lj_tab.o
CC        lj_func.o
CC        lj_udata.o
CC        lj_meta.o
CC        lj_debug.o
CC        lj_state.o
CC        lj_dispatch.o
CC        lj_vmevent.o
CC        lj_vmmath.o
CC        lj_strscan.o
CC        lj_api.o
CC        lj_lex.o
CC        lj_parse.o
CC        lj_bcread.o
CC        lj_bcwrite.o
CC        lj_load.o
CC        lj_ir.o
CC        lj_opt_mem.o
BUILDVM   lj_folddef.h
CC        lj_opt_fold.o
CC        lj_opt_narrow.o
CC        lj_opt_dce.o
CC        lj_opt_loop.o
CC        lj_opt_split.o
CC        lj_opt_sink.o
CC        lj_mcode.o
CC        lj_snap.o
CC        lj_record.o
CC        lj_crecord.o
BUILDVM   lj_recdef.h
CC        lj_ffrecord.o
CC        lj_asm.o
CC        lj_trace.o
CC        lj_gdbjit.o
CC        lj_ctype.o
CC        lj_cdata.o
CC        lj_cconv.o
CC        lj_ccall.o
CC        lj_ccallback.o
CC        lj_carith.o
CC        lj_clib.o
CC        lj_cparse.o
CC        lj_lib.o
CC        lj_alloc.o
CC        lib_aux.o
BUILDVM   lj_libdef.h
CC        lib_base.o
CC        lib_math.o
CC        lib_bit.o
CC        lib_string.o
CC        lib_table.o
CC        lib_io.o
CC        lib_os.o
CC        lib_package.o
CC        lib_debug.o
CC        lib_jit.o
CC        lib_ffi.o
CC        lib_init.o
AR        libluajit.a
CC        luajit.o
BUILDVM   jit/vmdef.lua
LINK      luajit
OK        Successfully built LuaJIT
make[1]: Leaving directory `/root/wrk-master/deps/luajit/src'
CC src/wrk.c
In file included from src/wrk.c:3:
src/wrk.h:9:25: error: openssl/ssl.h: No such file or directory
src/wrk.h:10:25: error: openssl/err.h: No such file or directory
In file included from src/wrk.c:3:
src/wrk.h:51: error: expected specifier-qualifier-list before 'SSL'
In file included from src/main.h:24,
                 from src/wrk.c:4:
src/ssl.h:6: error: expected '=', ',', ';', 'asm' or '__attribute__' before '*' token
src/wrk.c:15: error: expected specifier-qualifier-list before 'SSL_CTX'
src/wrk.c: In function 'main':
src/wrk.c:113: error: 'struct config' has no member named 'ctx'
src/wrk.c:113: warning: implicit declaration of function 'ssl_init'
src/wrk.c:115: warning: implicit declaration of function 'ERR_print_errors_fp'
src/wrk.c: In function 'thread_main':
src/wrk.c:248: error: 'connection' has no member named 'ssl'
src/wrk.c:248: error: 'struct config' has no member named 'ctx'
src/wrk.c:248: warning: implicit declaration of function 'SSL_new'
src/wrk.c:248: error: 'struct config' has no member named 'ctx'
src/wrk.c:249: error: 'connection' has no member named 'request'
src/wrk.c:250: error: 'connection' has no member named 'length'
src/wrk.c: In function 'header_field':
src/wrk.c:357: error: 'connection' has no member named 'headers'
src/wrk.c:360: error: 'connection' has no member named 'headers'
src/wrk.c: In function 'header_value':
src/wrk.c:367: error: 'connection' has no member named 'headers'
src/wrk.c:370: error: 'connection' has no member named 'headers'
src/wrk.c: In function 'response_body':
src/wrk.c:376: error: 'connection' has no member named 'body'
src/wrk.c: In function 'response_complete':
src/wrk.c:389: error: 'connection' has no member named 'start'
src/wrk.c:395: error: 'connection' has no member named 'headers'
src/wrk.c:396: error: 'connection' has no member named 'headers'
src/wrk.c:397: error: 'connection' has no member named 'headers'
src/wrk.c:397: error: 'connection' has no member named 'body'
src/wrk.c: In function 'check_timeouts':
src/wrk.c:428: error: 'connection' has no member named 'start'
src/wrk.c: In function 'socket_connected':
src/wrk.c:450: error: 'connection' has no member named 'written'
src/wrk.c: In function 'socket_writeable':
src/wrk.c:467: error: 'connection' has no member named 'written'
src/wrk.c:468: error: 'connection' has no member named 'request'
src/wrk.c:468: error: 'connection' has no member named 'length'
src/wrk.c:471: error: 'connection' has no member named 'request'
src/wrk.c:471: error: 'connection' has no member named 'written'
src/wrk.c:472: error: 'connection' has no member named 'length'
src/wrk.c:472: error: 'connection' has no member named 'written'
src/wrk.c:481: error: 'connection' has no member named 'written'
src/wrk.c:481: error: 'connection' has no member named 'start'
src/wrk.c:483: error: 'connection' has no member named 'written'
src/wrk.c:484: error: 'connection' has no member named 'written'
src/wrk.c:484: error: 'connection' has no member named 'length'
src/wrk.c:485: error: 'connection' has no member named 'written'
src/wrk.c: In function 'socket_readable':
src/wrk.c:508: error: 'connection' has no member named 'buf'
make: *** [obj/wrk.o] Error 1

So people can install it via luarocks install wrk and use it integrated with other lua tools, for example busted

The interface would have to be changed so that the whole thing can be used from a require, instead of relying on global variables like request or response. I suggest something like this:

local wrk = require 'wrk'

local token = nil
local path  = "/authenticate"

local test = wrk.new({
  request = function(self)
    return wrk.format("GET", path, self.headers)
  end,
  response = function(self, status, headers, body)
    if not token and status == 200 then
      token = headers["X-Token"]
      path  = "/resource"
      self.headers["X-Token"] = token
    end
  end})

  test:start()

• Added a new function, wrk.new(options) that creates a "task". options can have options.init, options.request, options.response, options.done. Once a task is created, these functions can also be read/write via task.init, task.request etc.
• The global functions have been moved to inside a Task "metatable".
• init, request, response and done have a new parameter, self, which points to the task. This is where the values previously stored on the global var wrk will go - so changing self.headers in the response affects the request.
• The init function gets the params passed in task:start(p1, p2, p3, ...) (command line parameters are available in any Lua script via ... on the global scope).

But this is just a suggestion. Some lua rocks actually expose global variables instead of returning a local in require, but that is generally frowned upon.

wrk reports averages and standard deviations, and by doing so, assumes that the latencies of a web service fall along a normal distribution. However, almost no web services do so.

Excluding those, and replacing them with the 50th and either 95th or 99.9th percentile would be more helpful for the vast majority of uses. Means and standard deviations of distributions that are not normal end up misleading more people about how their system behaves.

I would really like to get per-request results from a run. I thought this was already available through the done callback in the Lua scripts, but this doesn't seem to be the case. After some inspection in the code, it looks like we're only passing in the stats userdata object to done.

Was wondering/hoping we could get some more raw data.

From http://tools.ietf.org/html/rfc3986#section-3.2.2:

A host identified by an Internet Protocol literal address, version 6
[RFC3513] or later, is distinguished by enclosing the IP literal
within square brackets ("[" and "]").

Depending on how strict servers are about the spec they may reject the request if the host-header is not set properly.

For example, a lua script :

counter = 0
request = function()
    path = "/"
    wrk.port = (counter % 100) + 4000
    counter = counter + 1
    return wrk.format(nil, path)

But the format function returns the same url even the script modified the wrk.port in every request.

I'm observing a strange correlation between test duration and maximum latency: max. latency is a bit lower than specified test duration in most cases. I've also added a simple function to dump statistics data into text file for further analysis with tools like ministat - its output included below (note the difference between "avg" and "median").

Client (wrk 3.0.1) and server (nginx 1.5.6) are on different physical hosts running Ubuntu 12.04.

1 second:

$ ./wrk -d 1s -t 1 -c 400 --timeout 5s http://x.x.x.x:9081/1k.bin
Running 1s test @ http://x.x.x.x:9081/1k.bin
  1 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     7.15ms   41.77ms 977.76ms   99.73%
    Req/Sec    40.47k     2.05k   48.45k    68.29%
  37835 requests in 999.94ms, 46.08MB read
Requests/sec:  37837.19
Transfer/sec:     46.08MB

l _latency
r _requests
     N           Min           Max        Median           Avg        Stddev
l 3280           150        977758          4995     7150.2098     41773.864
r   41         37916         48454         41363     40473.195     2052.1515

2 seconds:

$ ./wrk -d 2s -t 1 -c 400 --timeout 5s http://x.x.x.x:9081/1k.bin
Running 2s test @ http://x.x.x.x:9081/1k.bin
  1 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    10.15ms   47.19ms   1.85s    99.88%
    Req/Sec    39.74k     1.02k   43.69k    66.13%
  77014 requests in 2.00s, 93.79MB read
Requests/sec:  38507.50
Transfer/sec:     46.90MB

l _latency
r _requests
     N           Min           Max        Median           Avg        Stddev
l 4960           144       1852504          8384     10151.608     47193.207
r   62         37958         43695         39826     39736.274     1024.8851

3 seconds:

$ ./wrk -d 3s -t 1 -c 400 --timeout 5s http://x.x.x.x:9081/1k.bin
Running 3s test @ http://x.x.x.x:9081/1k.bin
  1 threads and 400 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    11.35ms   76.92ms   2.92s    99.84%
    Req/Sec    24.66k     1.07k   26.58k    60.73%
  69828 requests in 3.00s, 85.04MB read
Requests/sec:  23275.59
Transfer/sec:     28.35MB

l _latency
r _requests
      N           Min           Max        Median           Avg        Stddev
l 15280           117       2921245          8027      11350.47      76924.43
r   191         22615         26583         24583     24658.288     1069.3568

I have noticed that some checks for return codes are missing in your source files.

Would you like to add more error handling for return values from functions like the following?

• pthread_mutex_init ⇒ main
• socket ⇒ connect_socket

large payloads highlight bugs in ae scheduler.

Before my patch:

./wrk -d 3 http://192.168.1.149 -M POST --body  `tr '\0' '0' < /dev/zero| head -c 10000`
Running 3s test @ http://192.168.1.149
  2 threads and 10 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     0.00us    0.00us   0.00us  100.00%
    Req/Sec     0.00      0.00     0.00    100.00%
  10 requests in 4.00s, 3.20KB read
  Socket errors: connect 0, read 0, write 0, timeout 10
  Non-2xx or 3xx responses: 10
Requests/sec:      2.50
Transfer/sec:     819.93B

After the minor change to scheduler flags:

./wrk -d 3 http://192.168.1.149 -M POST --body  `tr '\0' '0' < /dev/zero| head -c 10000`
strlen(body)=10000
req.size=10063
Running 3s test @ http://192.168.1.149
  2 threads and 10 connections
wrote 8688/10063
wrote 8688/10063
wrote 8688/10063
wrote 8688/10063
wrote 8688/10063
wrote 8688/10063
wrote 8688/10063
wrote 8688/10063
wrote 8688/10063
wrote 8688/10063
wrote 1375/10063
wrote 7240/10063
wrote 2823/10063
wrote 1375/10063
wrote 1375/10063
wrote 1375/10063
wrote 1375/10063
wrote 1375/10063
wrote 1375/10063
wrote 1375/10063
wrote 1375/10063
wrote 1375/10063
wrote 8688/10063
wrote 1375/10063
wrote 8688/10063
wrote 8688/10063
wrote 1375/10063
wrote 1375/10063
wrote 8688/10063
wrote 1375/10063
wrote 8688/10063
wrote 1375/10063
wrote 8688/10063
wrote 1375/10063
wrote 8688/10063
wrote 8688/10063
wrote 1375/10063
wrote 1375/10063
wrote 7240/10063
wrote 2823/10063
wrote 8688/10063
wrote 1375/10063
wrote 8688/10063
wrote 1375/10063
wrote 8688/10063
wrote 1375/10063
wrote 8688/10063
wrote 1375/10063
wrote 8688/10063
wrote 1375/10063
wrote 8688/10063
wrote 1375/10063
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     8.30ms    4.87ms  13.23ms   27.84%
    Req/Sec   175.04    174.42   555.00     36.63%
  1101 requests in 4.00s, 352.62KB read
  Socket errors: connect 0, read 0, write 0, timeout 5
  Non-2xx or 3xx responses: 1101
Requests/sec:    275.29
Transfer/sec:     88.17KB

The following patch reveals that the bugs occur when we can't write out a full buffer to a socket..It also 'fixes' the throughput for the 10K post payload by using AE_DONT_WAIT.

However the bug is still there when using 60K post payloads. Note the server is just nginx with default debian configuration.

--- a/src/wrk.c
+++ b/src/wrk.c
@@ -131,7 +131,7 @@ int main(int argc, char **argv) {
     cfg.addr = *addr;
     req.buf  = format_request(host, port, path, headers);
     req.size = strlen(req.buf);
-
+    printf("req.size=%lu\n", req.size);
     pthread_mutex_init(&statistics.mutex, NULL);
     statistics.latency  = stats_alloc(SAMPLES);
     statistics.requests = stats_alloc(SAMPLES);
@@ -415,7 +415,9 @@ static void socket_writeable(aeEventLoop *loop, int fd, void *data, int mask) {
     }

     if (!c->written) c->start = time_us();
-
+    if (n != req.size) {
+      printf("wrote %lu/%lu\n", n, req.size);
+    }
     c->written += n;
     if (c->written == req.size) {
         c->written = 0;
@@ -541,6 +543,7 @@ static int parse_args(struct config *cfg, char **url, char **headers, int argc,
                 break;
             case 'B':
                 req.body = optarg;
+                fprintf(stderr, "strlen(body)=%lu\n", strlen(req.body));
                 break;
             case 'L':
                 cfg->latency = true;

I've just tried to use "nc -l 9999" as a server emulation for debugging request headers, and the following wrk command:

wrk -t 1 -c 1 -r 1 http://127.0.0.1:9999/

led to the endless loop of the following calls:

[..]
connect(4, {sa_family=AF_INET, sin_port=htons(9999), sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in progress)
setsockopt(4, SOL_TCP, TCP_NODELAY, [1], 4) = 0
epoll_ctl(3, EPOLL_CTL_ADD, 4, {EPOLLOUT, {u32=4, u64=4}}) = 0
write(4, "GET / HTTP/1.1\r\nHost: 127.0.0.1:"..., 40) = -1 ECONNREFUSED (Connection refused)
epoll_ctl(3, EPOLL_CTL_DEL, 4, {0, {u32=4, u64=4}}) = 0
epoll_ctl(3, EPOLL_CTL_ADD, 4, {EPOLLIN, {u32=4, u64=4}}) = 0
epoll_wait(3, {{EPOLLIN|EPOLLHUP, {u32=4, u64=4}}}, 13, 62) = 1
read(4, "", 8192) = 0
epoll_ctl(3, EPOLL_CTL_DEL, 4, {0, {u32=4, u64=4}}) = 0
close(4) = 0
socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 4
fcntl(4, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(4, F_SETFL, O_RDWR|O_NONBLOCK) = 0
[..]

By the way, "nc -l" was closed before receiving any input from wrk. I suppose it's because of connect() + close() checks in main():

[..]
for (addr = addrs; addr != NULL; addr = addr->ai_next) {
int fd = socket(addr->ai_family, addr->ai_socktype, addr->ai_protocol);
if (fd == -1) continue;
if (connect(fd, addr->ai_addr, addr->ai_addrlen) == -1) {
if (errno == EHOSTUNREACH || errno == ECONNREFUSED) {
close(fd);
continue;
}
}
close(fd);
break;
}
[..]

What is the reason(s) for these checks here? Shouldn't they be implemented in the main cycle?

I've spent some time trying to create a working patch, but unfortunately still have no luck.

--pipeline N doesn't seem to work anymore in the pipeline branch. It doesn't understand the argument.

Is there any chance when this support is added back that it could just be 1 version of Wrk?

hi, how to make a concurrency test for a web server with wrk? or theres are some output value is similar to the concurrent?

Running 10m test @ http://192.168.2.155/bench/test.php
100 threads and 100 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 7.40ms 69.92ms 901.99ms 99.25%
Req/Sec 0.87k 331.34 2.11k 65.45%
49515425 requests in 10.00m, 52.83GB read
Requests/sec: 82530.62
Transfer/sec: 90.16MB

Some time I need to set the http proxy , how ?

Thanks a lot .

Hi @wg!

Thanks again for your help with our second round of web framework benchmarking. Wrk ran like a champ and gave people the latency information they wanted.

One of the core Netty contributors, Norman Maurer, has asked us to consider using HTTP pipelining as well. Does Wrk provide that now or could it be added?

http://www.twitter.com/normanmaurer/status/320577548356571136

Assuming we did want to run a pipelining test, for the sanity of @pfalls1 who just modified all of our scripts to work with Wrk, I'd rather stick with it.

Homebrew/legacy-homebrew#11128 (comment)

I am trying to get WRK into the Homebrew collection but they need a stable download point. If you could tag a version or provide a stable download it would be much appreciated.

Thanx!
Richard

Quick pair of questions, so I can get started. The new script that updates the global 'wrk' should that one also be name wrk.lua? There is a wrk.lua(the one that handles the internals) so wouldn't that create a conflict? Finally, wrk.body specifies the parameters of the function were testing in wrk.lua or the parameters of another function else where? I'm a bit slow on the uptake, sometimes, please excuse my banal questions

wrk 3.0.0 [kqueue] Copyright (C) 2012 Will Glozer

$ sw_vers
ProductName: Mac OS X
ProductVersion: 10.8.5
BuildVersion: 12F37

$ wrk http://localhost:8080/
PANIC: unprotected error in call to Lua API (attempt to index a string value)

Such error appears on each attempt to run wrk against web server listening on localhost:8080.

Using fe4c1a6

./wrk -t1 -c1 -d10s htt

triggers it.

If you set the load to a fixed value you can get very interesting stats about your system by measuring the latency and other server side parameters during the test.

The finaly reported value of total requests send is incorrect. I have checked values reported by wrk and values reported with nginx and wrk seems to be not counting some requests.

Classical race condition?

Some relatively complex test scenarios could be reused with parameterization. What do you think about adding --script-var <name>=<value>, where the specified variables are passed via init()?

Maybe it's not the responsibility of wrk to handle this because a user can put a pre-processor/templating engine, but it might be faily useful when it's built-in?