fortio / fortio Goto Github PK

to the data directory

that way we can turn fortio server into a more secure locally running 'app'

Hi,

Could you please explain me how fortio different from wrk in terms of benchmark tool.

I am clear with the parameter of thread but can be change number of connections as a parameter in fortio.

Please explain more in detail about qps(query per second).

That would ideally generate a single "all included" html with the js and data embedded
(could then be emailed or even interacted with on a phone etc.)

fortio version: docker package->istio/fortio:latest from 01/23/2018

18:11:12 F fortio_main.go:273> Unable to json serialize result: json: unsupported value: NaN
panic: aborting...
goroutine 1 [running]:
istio.io/fortio/log.logPrintf(0x6, 0x931b21, 0x23, 0xc420205a50, 0x1, 0x1)
/go/src/istio.io/fortio/log/logger.go:160 +0x28c
istio.io/fortio/log.Fatalf(0x931b21, 0x23, 0xc420205a50, 0x1, 0x1)
/go/src/istio.io/fortio/log/logger.go:208 +0x5c
main.fortioLoad()
/go/src/istio.io/fortio/fortio_main.go:273 +0xb5c
main.main()
/go/src/istio.io/fortio/fortio_main.go:144 +0x669

assuming their start time is similar
the histograms can be merged with stats Merge/Transfer (but those methods are on Histogram not HistgramData at the moment) - if we do it at UI level only fortioResultToJsChartData could take multiple input (separately from fortioAddToMultiResult)

cc @fawadkhaliq who mentioned this first

Consider adding (optionally?)
Access-Control-Allow-Origin
headers so fortio echo/debug can be fetched using data load and the data server can serve the json to be displayed anywhere

when giving a longer duration a progress bar would both ensure the browser doesn't timeout and provide visual feedback

Hi all,

thanks for the great work done so far. As I understand, fortio is using Golang's canonical imports. This might be confusing to Golang newbies, like me. It would be great to have a section in the FAQ on how someone can fork the repo, setup remotes and start hacking away.

I just discovered fortio doesn't run on windows:

C:\Users\l\go\src\istio.io>go get -u istio.io/fortio
# istio.io/fortio/ui
fortio\ui\uihandler.go:181:3: undefined: syscall.Kill

Ignoring for a second that go should provide a kill emulation (imo) - we probably should cleanup fortio anyway as it's only to interrupt itself

(we can also remove the exit option from the UI and only interrupt a given run or maintain a stack or list of them - or interrupt all of them through a shared channel)

related to self benchmakr

https://blog.cloudflare.com/go-dont-collect-my-garbage/

For instance
https://storage.googleapis.com/fortio-data?prefix=fortio.istio.io/data/

<ListBucketResult xmlns="http://doc.s3.amazonaws.com/2006-03-01">
<Name>fortio-data</Name>
<Prefix>fortio.istio.io/data/</Prefix>
<Marker/>
<IsTruncated>false</IsTruncated>
<Contents>
<Key>
fortio.istio.io/data/2017-12-20-231208_Test_local_run_1k_qps.json
</Key>
<Generation>1515739179746249</Generation>
<MetaGeneration>2</MetaGeneration>
<LastModified>2018-01-12T06:39:39.642Z</LastModified>
<ETag>"ae826e6fe247ce558e315c1383040fff"</ETag>
<Size>4308</Size>
</Contents>
<Contents>
<Key>
fortio.istio.io/data/2017-12-20-231317_Test_local_run_max_qps.json
</Key>
<Generation>1515739179003085</Generation>
<MetaGeneration>2</MetaGeneration>
<LastModified>2018-01-12T06:39:38.949Z</LastModified>
<ETag>"c3822e0b1a105829251e4f4ac94795ca"</ETag>
<Size>5277</Size>
</Contents>
<Contents>
<Key>
fortio.istio.io/data/2017-12-20-231459_Test_local_run_max_qps_32_connections.json
</Key>
<Generation>1515739179189295</Generation>
<MetaGeneration>2</MetaGeneration>
<LastModified>2018-01-12T06:39:39.172Z</LastModified>
<ETag>"f24106de2045841f2b1f2d1f4beb2fe1"</ETag>
<Size>5425</Size>
</Contents>
<Contents>
<Key>
fortio.istio.io/data/2017-12-21-001514_Test_local_run_max_qps_32_connections.json
</Key>
<Generation>1515739180123253</Generation>
<MetaGeneration>2</MetaGeneration>
<LastModified>2018-01-12T06:39:40.105Z</LastModified>
<ETag>"ebe97645f20cb18fec4aaf79f2003075"</ETag>
<Size>5417</Size>
</Contents>
</ListBucketResult>

https://docs.aws.amazon.com/AmazonS3/latest/API/v2-RESTBucketGET.html

Some of the TODO would be good starter tasks:

Current set as of this writing:

$ git grep TODO

Makefile:# TODO: do something about cyclomatic complexity
fgrpc/grpcrunner.go:// TODO: refactor common parts between http and grpc runners
fgrpc/grpcrunner.go:            // TODO: option to use certs
fhttp/http.go:// Version is the fortio package version (TODO:auto gen/extract).
fhttp/http.go:                  Timeout: 3 * time.Second, // TODO: make configurable
fhttp/http.go:// TODO: refactor - unwiedly/ugly atm
fhttp/http.go:  // TODO: safer to start with -1 and fix ok for http 1.0
fhttp/http.go:          // TODO: need automated tests
fhttp/http.go:                  c.code = ParseDecimal(c.buffer[retcodeOffset : retcodeOffset+3]) //TODO do that only once...
fhttp/http.go:                  // TODO handle 100 Continue
fhttp/http.go:                  // TODO: keep track of list of newlines to efficiently search headers only there
fhttp/http.go:                                          // TODO: just consume the extra instead
fhttp/http.go:          // TODO: this easily lead to contention - use 'thread local'
fhttp/http.go:// TODO: switch to Duration.Round once switched to go 1.9
fhttp/httprunner.go:    // TODO 1. use std client automatically when https url
pingsrv.go:     // TODO doesn't work for ipv6 addrs etc
stats/stats.go:// TODO: consider using an interval search for the last N big buckets
stats/stats.go: // TODO potentially merge despite different offset/scale
stats/stats_test.go:    // TODO: fix the p51 (and p1...), should be 0 not 10
stats/stats_test.go:    tP := []float64{100.} // TODO: use 75 and fix bug
ui/static/js/fortio_chart.js:// TODO: object-ify
ui/static/js/fortio_chart.js:      // TODO may need updateChart() if we persist settings even the first time
ui/templates/browse.html:<p><!-- TODO: find a way to flush/not need two p to get the form visible! -->
ui/templates/main.html:{{end}}{{end}} <!-- 2 extra header lines, TODO: add a JS 'more headers' button -->
ui/uihandler.go:// TODO: auto map from (Http)RunnerOptions to form generation and/or accept
ui/uihandler.go:// TODO: unit tests, allow additional data sets.
ui/uihandler.go:        debugPath = ".." + debugpath // TODO: calculate actual path if not same number of directories

Can be as simple as a .zip file with fortio.exe and the resources

Issue is how to find the resources and what to do with path

like http://httpbin.org/status/xxx (eg http://httpbin.org/status/418)

Hi,

it would be nice to have an ignore-tls flag or something similar to test against https with an self-signed certificate. Or is there some way to add the self-signed certificate to the trusted certs?

To the latest

Fortio Improvement list:

• a request id in either headers or query arg or both, way to get cached/non cached request
• calculate bytes sent/received including headers (to be researched)
• h2 testing (#452)
• go over the TODOs
• warm up time
• payload option (exists in grpcping)
• fortio's own perf regression test (ie > 100k qps)

Done:

• curl / single request mode - #21
• ^C support to stop / option to run continuously until ^C - #5
• struct, json or other structured output for the histogram - #9
• graph (picture) generation - #14
• evaluate/use https://github.com/spf13/cobra -> no - min dependencies
• fixed number of requests mode - #82
• consider passing a context and/or using channels - #64

originally from istio/istio#453

Good source of info:

https://thenextweb.com/dd/2015/06/12/20-best-javascript-chart-libraries/

http://www.chartjs.org/ seems like a good candidate

@ymesika mentioned http://visjs.org/graph2d_examples.html too

like we have delay and status query args
let's add a count that will increment a request counter and a way to get it back

possibly a json version of debug

issue/concern: serialization/perf hit for those requests (but should be fine up to several k qps and this would be more for feature correctness testing : make 100 requests and ensure exactly 100 were received

potentially a more expensive mode would maintain a complete set of request/trace ids

to match the web ui save json using the same naming scheme (labels and date)

changing the log level doesn't work

not seeing how it's handled as an argument either?

https://github.com/istio/fortio/blob/master/fortio_main.go#L74

#62 added the ability to export/backup local json data to the cloud

let's add peer2peer / reverse capability to fetch and graph from tsv

For fortio use in functional test, being able to specify exactly 100 requests for instance would be useful
(right now if you ask for 10 qps, 10 seconds, 8 connections you get 100 requests + 8 warmup requests)

Need to be able to monitor cpu/memory/diskio for containers and pods

the persistent volume writeonce in current
istio/fortio-deployment#32
doesn't work well

switch to gsutil init dump or, after #90, fetch from backup (if I can get gcsweb to serve index.tsv equiv)

it's pretty low in some parts but can be increased fairly easily

except for the ui handler - for which some sort of recording of Webtest.sh would help

@ldemailly commented on Fri Dec 01 2017

http://localhost:8080/fortio/?labels=Fortio&url=%3Cb%3Efoo%3C%2Fb%3E&load=Start

shows foo instead of <b>foo</b>

eg using "x" doesn't cleanup the watchers properly and 'end' correctly

It'd be nice to find a lightweight way to test the UI including for working JS (simulating a browser executing the page...)

(cc @vii )

support for multiple http port listen:
the code already does but not the command line/image

multiple simple echo srv at least
and maybe 1 report and 1 full UI on different ports too

cc @ZackButcher for his health checks tests. @danehans as it relates to #91

there are random failures and it takes a long while to run the linters on circleci

For people who don't already have go installed, it would be great if someone would create a brew recipe:
https://docs.brew.sh/How-To-Open-a-Homebrew-Pull-Request.html

(apparently they don't want the main author to do it, but please update this issue if you contribute one so I can help make sure it works well)

for the basic client,

there are numerous combo of modes - chunked vs content length vs keepalive
and could use test to confirm that it works irrespective of framing
(whether we get all the bytes 1 one read or 1 byte at a time or some random chunking)

todo: see what curl and other client use to test

• allows remote trigger of load tests
• Json output option to programmatically gather the data

http://localhost:8080/fortio/sync?url=+http%3A%2F%2Ffortio.istio.io%2Fdata%2Findex.tsv

13:43:19 I uihandler.go:406> Sync: GET /fortio/sync?url=+http%3A%2F%2Ffortio.istio.io%2Fdata%2Findex.tsv HTTP/1.1 [::1]:58855 ()
13:43:19 I http.go:139> Setting regular extra header X-On-Behalf-Of: [::1]:58855
13:43:19 E http.go:150> Unable to make request for http://fortio.istio.io/data/index.tsv : parse http://fortio.istio.io/data/index.tsv: first path segment in URL cannot contain colon
13:43:19 http: panic serving [::1]:58855: runtime error: invalid memory address or nil pointer dereference
goroutine 67 [running]:
net/http.(*conn).serve.func1(0xc4202c2460)
/usr/local/Cellar/go/1.9.2/libexec/src/net/http/server.go:1697 +0xd0
panic(0x1461aa0, 0x17809b0)
/usr/local/Cellar/go/1.9.2/libexec/src/runtime/panic.go:491 +0x283
istio.io/fortio/fhttp.(*Client).Fetch(0x0, 0x0, 0xc42021acc0, 0x20, 0x17, 0x0)
/Users/ldemailly/go/src/istio.io/fortio/fhttp/http.go:186 +0x3a
istio.io/fortio/ui.SyncHandler(0x174aea0, 0xc4203800e0, 0xc4202e8300)
/Users/ldemailly/go/src/istio.io/fortio/ui/uihandler.go:591 +0x327
net/http.HandlerFunc.ServeHTTP(0x15139d8, 0x174aea0, 0xc4203800e0, 0xc4202e8300)
/usr/local/Cellar/go/1.9.2/libexec/src/net/http/server.go:1918 +0x44
net/http.(*ServeMux).ServeHTTP(0x17904c0, 0x174aea0, 0xc4203800e0, 0xc4202e8300)
/usr/local/Cellar/go/1.9.2/libexec/src/net/http/server.go:2254 +0x130
net/http.serverHandler.ServeHTTP(0xc420230750, 0x174aea0, 0xc4203800e0, 0xc4202e8300)
/usr/local/Cellar/go/1.9.2/libexec/src/net/http/server.go:2619 +0xb4
net/http.(*conn).serve(0xc4202c2460, 0x174b660, 0xc4202e0300)
/usr/local/Cellar/go/1.9.2/libexec/src/net/http/server.go:1801 +0x71d
created by net/http.(*Server).Serve
/usr/local/Cellar/go/1.9.2/libexec/src/net/http/server.go:2720 +0x288

09:44:32 I uihandler.go:449> GET /fortio/?url=http%3A%2F%2Flocalhost%3A8080%2Ffortio%2Ffetch%2Flocalhost%3A8080%2Fecho&qps=100&t=3s&c=1&p=50%2C+75%2C+99%2C+99.9&r=0.0001&H=User-Agent%3A+istio%2Ffortio-0.3.4&H=&load=Start HTTP/1.1 [::1]:65293
09:44:32 I uihandler.go:82> Starting load request from [::1]:65293 for http://localhost:8080/fortio/fetch/localhost:8080/echo
09:44:32 I http.go:101> Setting regular extra header User-Agent: istio/fortio-0.3.4
09:44:32 I httprunner.go:83> Starting http test for http://localhost:8080/fortio/fetch/localhost:8080/echo with 1 threads at 100.0 qps
09:44:32 I uihandler.go:449> GET /fortio/fetch/localhost:8080/echo HTTP/1.1 [::1]:65296
09:44:32 I http.go:634> Server wants to close connection, no keep-alive!
09:44:32 E http.go:644> More data is available but stopping after 135, increase -httpbufferkb

so when testing denial or other error rules cases it doesn't abort

As discussed previously, can we move this from being a top-level repo to being a subdirectory in istio/istio? Perhaps istio/istio/tools/fortio for example?

Thanks.

@smawson @louiscryan @rshriram

Need to check

https://www.youtube.com/watch?v=lJ8ydIuPFeU

http://psy-lob-saw.blogspot.com/2016/07/fixing-co-in-cstress.html

https://github.com/giltene/wrk2/blob/master/README.md

before switching to 1.9 let's measure that it's same or better
(in term of fortio self qps - should be > 100k qps with the fast client, benchmark against the stdclient to see if it's gotten better)

cc @bochunz want to run / compare ?

when using rewrite rule, the outside reachable url and the inside one aren't the same
(maybe the ingress should say what was the original url ?)

right now fortio even as a library doesn't export the histogram data except through an io.Writer

which makes graphing and recording into db/other systems rely on script/parsing of the text output

for instance:

      {
        "Start": 0.04,
        "End": 0.045000000000000005,
        "Percent": 38.41961852861036,
        "Count": 43
      },
      {
        "Start": 0.045000000000000005,
        "End": 0.05,
        "Percent": 43.596730245231605,
        "Count": 57
      },
      {
        "Start": 0.05,
        "End": 0.060000000000000005,
        "Percent": 51.316984559491374,
        "Count": 85
      },
      {
        "Start": 0.060000000000000005,
        "End": 0.07,
        "Percent": 59.2188919164396,
        "Count": 87
      },
      {
        "Start": 0.07,
        "End": 0.08,
        "Percent": 65.39509536784742,
        "Count": 68
      },
      {
        "Start": 0.08,
        "End": 0.09000000000000001,
        "Percent": 72.11625794732062,
        "Count": 74
      },
      {
        "Start": 0.09000000000000001,
        "End": 0.1,
        "Percent": 76.93006357856494,
        "Count": 53
      },
      {
        "Start": 0.1,
        "End": 0.2,
        "Percent": 99.09173478655768,
        "Count": 244
      },
      {
        "Start": 0.2,
        "End": 0.246923071,
        "Percent": 100,
        "Count": 10
      }
    ],

means 99.09% is only reached at 0.2 not 0.15

but if we want to have exact match for round numbers we should change the bucket definition to be inclusive instead of exclusive

Please let me know how to install fortio on MacBook.

I tried brew install fortio as well