for-get / katt Goto Github PK

View Code? Open in Web Editor NEW

117.0 117.0 18.0 6.87 MB

KATT (Klarna API Testing Tool) is an HTTP-based API testing tool for Erlang.

Home Page: https://github.com/for-GET/katt

License: Apache License 2.0

Makefile 2.55% Erlang 96.15% Elixir 0.70% Emacs Lisp 0.09% Dockerfile 0.28% Shell 0.23%

api erlang har http test

katt's People

Contributors

Stargazers

Watchers

Forkers

odobroiu pap lpgauth michalwski issuu licenser arcusfelis harenson erikgustafson-klarna tjarvstrand mitizhi faragoa f3c0 brucify sztheory danfilip kianmeng otp-bank

katt's Issues

case_clause error in KATT under HEAD request

Symptom:

I'm executing a HEAD request to a non-existing URI. KATT (c7414fe) crashes with the following backtrace:

{{case_clause,{ok,404,
                  [{<<"server">>,<<"Cowboy">>},
                   {<<"date">>,<<"Mon, 04 Apr 2016 15:25:43 GMT">>},
                   {<<"content-length">>,<<"0">>},
                   {<<"content-type">>,<<"application/json">>}]}},
 [{katt_callbacks,request,3,
                  [{file,"/Users/jlouis/P/oa/_build/test/lib/katt/src/katt_callbacks.erl"},
                   {line,163}]},
  {katt,run_transactions,5,
        [{file,"/Users/jlouis/P/oa/_build/test/lib/katt/src/katt.erl"},
         {line,211}]},
  {katt,run_blueprint,4,
        [{file,"/Users/jlouis/P/oa/_build/test/lib/katt/src/katt.erl"},
         {line,167}]},
  {katt,run,4,
        [{file,"/Users/jlouis/P/oa/_build/test/lib/katt/src/katt.erl"},
         {line,94}]}]}

Expected output:

I would have expected this test to pass without a backtrace from KATT. Maybe there is something wrong with the handling of HEAD requests.

Workaround:

s/HEAD/GET/g works.

after a failed rollercoaster in finding a load-testing tool (ab, httperf, siege, tsung, loader.io, loadimpact, ... , some others still to be investigated), I was thinking "how hard can it be" (tm) to turn KATT into a "simple" load-testing mode

new params

number of total runs
number of workers (runs in parallel)
load-test timeout

run 1 scenario, confirm that it passes
start the load-test
output statistics; for each transaction latency (min, max, average, std dev)

pinging @sstrigler @dmitriid @isakb (either to laugh at me, or to bring in some constructive criticism :) )

handle HTTP duplicate headers

discovered (and grosely patched for one specific case) in klarna/katt@e9821d7

Confusing error message when katt script file does not exist

katt foo.katt
escript: exception error: no function clause matching
katt_cli:main([],[],["foo.katt"],[]) (src/katt_cli.erl, line 51)
in function escript:run/2 (escript.erl, line 760)
in call from escript:start/1 (escript.erl, line 277)
in call from init:start_em/1
in call from init:do_boot/3

Katt version:
c418814 (1.5.4-29-gc418814)

support har files

not to be dependent on https://github.com/for-GET/katt-util - har-to-katt util

float Params get multiplied by factor 10

I experience that any float I assign to the test case get multiplied by a factor 10.

Test case:
katt:run(Scenario, [{test_float, 1.23}]).

Content of "Scenario":

POST http://httpbin.org/post
> Accept: application/json
> Content-Type: application/json
> User-Agent: KATT
{
    "my_float": "{{<my_float}}"
}
< 200
< Content-Type: application/json
{
    "json": {
        "my_float": 1.23,
    }
}

This fails with the error:
{fail,[{not_equal,{"/body/json/my_float",1.23,12.3}}]}

And the actual request being sent (sending to localhost listening with nc -l:

POST / HTTP/1.1
Accept: application/json
Content-Type: application/json
User-Agent: KATT
Host: localhost:9999
Content-Length: 24

{
    "my_float": 12.3
}

As you can see 1.23 have changed to 12.3

no error on wildcard expected but undefined ?!

it seems strange, and potentially a bug that

{
  "foo": "{{_}}",
  "{{_}}": "{{unexpected}}"
}

will not fail if the payload is actually the empty object {}

Possibility to give hackney options

Possibility to give hackney:request/5 additional options, without having to give a whole function in to replace {request, ?DEFAULT_REQUEST_FUN}.

Currently ?DEFAULT_REQUEST_FUN is hardcoded to use two specific options when calling hackney:request/5.

-define(DEFAULT_REQUEST_FUN,        fun katt_callbacks:request/3).

In katt_util.erl:

external_http_request(Url, Method, Hdrs, Body, Timeout, []) ->
  ...
  Options = [ {recv_timeout, Timeout}
            , {insecure, true}
            ],
  case hackney:request(Method, BUrl, BHdrs, Body, Options) of

unhandled {error, timeout} crash in cli

katt_util:transaction_result_to_mochijson3 cannot handle {error, timeout} (request timed out)

matching fails

GET /
200
< content-type: application/vnd.fooBAR+json{{_}}

will match content-type: application/vnd.foo+json though it shouldn't

assign parameters in between transactions

makes it easier/modular to write tests

display a diff for string errors

at times, both the expected and the actual value of a string can be super-long.

it would be nice if there was a built-in word-diff mechanism in place (or extension) that highlights the diffs between

Erlang code as a step

Hi.

I needed "smarter" KATT scenarios so I implemented a new step type that I call action. It allows you to write Erlang code as a step and it looks like this, tell me what you think:

Running this in the erlang shell:
katt:run("priv/test_action.apib", [{inparam, "invalue"}]).

POST /helloworld
> Accept: text/html
Hello {{<inparam}}
< 200
{
"param1":"{{>param1}}",
"param2":"{{>param2}}"
}

! erlang <<<
A = mymodule:dostuff("{{<param1}}"),
B = othermodule:dostuff("{{<param2}}"),
[{outparam1, A},{outparam2, B}].
>>>

POST {{<outparam1}}/{{<outparam2}}
< 200

Prepare katt for jsx 3.0

The default behaviour changed in jsx 3.0, jsx:decode will return with maps by default.
katt's special meaning tags does not work with maps returned by jsx:decode

Change jsx:decode function calls to explicitly return with proplists.
This way katt can be used with jsx 3.0

add support for application/x-www-form-urlencoded

i.e. allow for a nice DSL to send in urlencoded key-value dictionaries

possible suggestion is to base this on JSON, on the condition that the first character of the payload is {, else default to plain text:

POST /foo
> Content-Type: application/x-www-form-urlencoded
{
  "key1": "http://example.com",
  "key2": "a=b"
}

would be the equivalent of

POST /foo
> Content-Type: application/x-www-form-urlencoded
key1=http%3A%2F%2Fexample.com&key2=a%3Db

refactor tests

to get around these annoyances:

put together constructs related to one test, so that you can read assertion, mocked request, mocked apib (not jump through the file, or have 3 buffers opened just to have the full view of 1 test). Possibly go as far as 1 test module per test (more copy-paste boilerplate, but who cares)
if a test fails today, the output is truncated (ellipsis)
TBC

support base_url param

rather than specifying protocol, port, hostname, allow base_url param that would be parsed into the required parameters

WHY? makes it easier to call katt (cli)

add parallel functionality

at times, timing is less important.

several requests could be grouped with a special header, in order to have them executed in parallel, before joining (i.e. waiting for all of them to finish) and continuing with the next non-grouped request

`Actual` is unbound/undefined in `runtime_validation` type.

Thought I could use runtime_validation to work around #64 but I cannot get the actual value out of the response.

POST http://httpbin.org/post
> Accept: application/json
> Content-Type: application/json
> User-Agent: KATT
> Content-Type: application/json
{
    "myint":123
}
< 200
< Content-Type: application/json
{
    "json": {
        "myint": {
            "{{type}}": "runtime_validation",
            "erlang": "{pass, [{\"foo\", Actual}]}"
        }   
    }
}

Fails with

{fail,[{not_equal,{"/body/origin/{{runtime_validation}}",
                       {[123,112,97,115,115,44,32,91,123,34,102,111,111,98,97,
                         114,34,44,32,65,99,116,117,97,108,125,93,125,46,126,
                         110,"error",58,
                         [123,["unbound",44,"'Actual'"],125]],
                        undefined},
                       "185.20.241.138"}}]}

allow katt:run with #katt_blueprint

so that parsing can be done in a disconnected pre-step

mentioned in #34 (comment)

Cannot store response Integers in Param

I have this small test:

POST http://httpbin.org/post
> Accept: application/json
> Content-Type: application/json
> User-Agent: KATT
> Content-Type: application/json
{
    "myint":123
}
< 200
< Content-Type: application/json
{
    "json": {
        "myint": "{{>myint}}",
    }
}

It fails with the error

{fail,[{not_equal,{"/body/json/myint","{{>myint}}",123}}]}

Looks like KATT will do a comparison rather than saving the variable into myint.

string matching fails for {{_}}

matching the instance "foo": "bar baz" works against "foo": "{{>var}} baz" but fails against "foo": "{{_}} baz"

katt_util:is_valid does not allow additional properties

After using the set validation type I found an issue where set items would not match if the Actual content had additional properties. I looked through the documentation and the tests and could not find that this behaviour was intended

This problem can be reproduce by using the APIB specification

"someList": {
"{{type}}": "set",
"value": [{"foo": "bar"}]
}

And try to match against the response

"someList": [
{
"foo", "bar",
"foo2":"bar"
}
]

After some investigation the problem seems to be in katt_util/is_valid

If you want to I can create a failing test case for this issue

Triple backtick in blueprint

I'd like to put a request block inside a triple backtick block so that a blueprint file can be rendered as markdown. However, if I have more than two of those blocks, katt will fail to parse the blueprint:

{{badmatch,{fail,{expected,{no_match,<<"`">>},{{line,33},{column,1}}}}},
 [{katt_blueprint_parse,file,1,
                        [{file,"src/katt_blueprint_parse.erl"},{line,55}]},
  {katt,run,4,[{file,"src/katt.erl"},{line,89}]}]}.

Katt compile error due to get_stacktrace error in meck

Katt version: 1.5.4-29-gc418814

When compiling Katt on our build servers the build fails:

[10:12:37][/src Deps] Compiled src/meck_cover.erl
[10:12:37][/src Deps] /src/support/katt/deps/meck/src/meck_code_gen.erl:185: erlang:get_stacktrace/0: deprecated; use the new try/catch syntax for retrieving the stack backtrace
[10:12:37][/src Deps] Compiling /src/support/katt/deps/meck/src/meck_code_gen.erl failed:
[10:12:37][/src Deps] ERROR: compile failed while processing /src/support/katt/deps/meck: rebar_abort
[10:12:37][/src Deps] make[1]: *** [Makefile:111: compile] Error 1

NB: For some reason I cannot reproduce the problem on my Mac. Compiling meck works fine.

Here is a longer excerpt from the log file from the build server.

[10:11:50][/src Deps] ==> katt (get-deps)
[10:11:50][/src Deps] Pulling hackney from {git,"git://github.com/benoitc/hackney.git",
[10:11:50][/src Deps] {tag,"1.6.0"}}
[10:11:50][/src Deps] Cloning into 'hackney'...
[10:11:54][/src Deps] Pulling jsx from {git,"git://github.com/talentdeficit/jsx.git",{tag,"2.8.0"}}
[10:11:54][/src Deps] Cloning into 'jsx'...
[10:11:58][/src Deps] Pulling tdiff from {git,"git://github.com/tomas-abrahamsson/tdiff.git",
[10:11:58][/src Deps] {tag,"0.1"}}
[10:11:58][/src Deps] Cloning into 'tdiff'...
[10:11:59][/src Deps] Pulling meck from {git,"git://github.com/eproxus/meck.git",{tag,"0.8.4"}}
[10:11:59][/src Deps] Cloning into 'meck'...
[10:12:01][/src Deps] Pulling neotoma from {git,"git://github.com/seancribbs/neotoma.git",
[10:12:01][/src Deps] {tag,"1.7.3"}}
[10:12:01][/src Deps] Cloning into 'neotoma'...
[10:12:03][/src Deps] ==> hackney (get-deps)
[10:12:03][/src Deps] Pulling idna from {git,"https://github.com/benoitc/erlang-idna",{tag,"1.2.0"}}
[10:12:03][/src Deps] Cloning into 'idna'...
[10:12:09][/src Deps] Pulling mimerl from {git,"https://github.com/benoitc/mimerl",{tag,"1.0.2"}}
[10:12:09][/src Deps] Cloning into 'mimerl'...
[10:12:13][/src Deps] Pulling certifi from {git,"https://github.com/certifi/erlang-certifi",
[10:12:13][/src Deps] {tag,"0.4.0"}}
[10:12:13][/src Deps] Cloning into 'certifi'...
[10:12:16][/src Deps] Pulling metrics from {git,"https://github.com/benoitc/erlang-metrics",
[10:12:16][/src Deps] {tag,"1.0.1"}}
[10:12:16][/src Deps] Cloning into 'metrics'...
[10:12:17][/src Deps] Pulling ssl_verify_fun from {git,"https://github.com/deadtrickster/ssl_verify_fun.erl",
[10:12:17][/src Deps] {tag,"1.1.0"}}
[10:12:17][/src Deps] Cloning into 'ssl_verify_fun'...
[10:12:22][/src Deps] ==> idna (get-deps)
[10:12:22][/src Deps] ==> mimerl (get-deps)
[10:12:22][/src Deps] ==> certifi (get-deps)
[10:12:22][/src Deps] ==> metrics (get-deps)
[10:12:22][/src Deps] ==> ssl_verify_fun (get-deps)
[10:12:22][/src Deps] ==> jsx (get-deps)
[10:12:22][/src Deps] ==> tdiff (get-deps)
[10:12:22][/src Deps] ==> meck (get-deps)
[10:12:22][/src Deps] ==> neotoma (get-deps)
[10:12:22][/src Deps] ==> idna (compile)
[10:12:23][/src Deps] Compiled src/idna.erl
[10:12:23][/src Deps] Compiled src/idna_ucs.erl
[10:12:23][/src Deps] Compiled src/idna_unicode.erl
[10:12:24][/src Deps] Compiled src/punycode.erl
[10:12:24][/src Deps] Compiled src/idna_unicode_data2.erl
[10:12:31][/src Deps] Compiled src/idna_unicode_data1.erl
[10:12:31][/src Deps] ==> mimerl (compile)
[10:12:32][/src Deps] Compiled src/mimerl.erl
[10:12:32][/src Deps] ==> certifi (compile)
[10:12:34][/src Deps] Compiled src/certifi.erl
[10:12:35][/src Deps] Compiled src/certifi_cacerts.erl
[10:12:35][/src Deps] Compiled src/certifi_weakcerts.erl
[10:12:35][/src Deps] ==> metrics (compile)
[10:12:35][/src Deps] Compiled src/metrics_dummy.erl
[10:12:35][/src Deps] Compiled src/metrics_exometer.erl
[10:12:35][/src Deps] Compiled src/metrics.erl
[10:12:35][/src Deps] Compiled src/metrics_folsom.erl
[10:12:35][/src Deps] ==> ssl_verify_fun (compile)
[10:12:35][/src Deps] Compiled src/ssl_verify_fingerprint.erl
[10:12:36][/src Deps] Compiled src/ssl_verify_hostname.erl
[10:12:36][/src Deps] ==> hackney (compile)
[10:12:36][/src Deps] Compiled src/hackney_app.erl
[10:12:36][/src Deps] Compiled src/hackney_bstr.erl
[10:12:36][/src Deps] Compiled src/hackney_cookie.erl
[10:12:36][/src Deps] Compiled src/hackney_connect.erl
[10:12:36][/src Deps] Compiled src/hackney_date.erl
[10:12:36][/src Deps] Compiled src/hackney.erl
[10:12:36][/src Deps] Compiled src/hackney_headers.erl
[10:12:36][/src Deps] Compiled src/hackney_http_connect.erl
[10:12:36][/src Deps] Compiled src/hackney_http.erl
[10:12:36][/src Deps] Compiled src/hackney_manager.erl
[10:12:36][/src Deps] Compiled src/hackney_pool_handler.erl
[10:12:36][/src Deps] /src/support/katt/deps/hackney/src/hackney_multipart.erl:271: Warning: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
[10:12:36][/src Deps] Compiled src/hackney_multipart.erl
[10:12:36][/src Deps] /src/support/katt/deps/hackney/src/hackney_pool.erl:213: Warning: call to crypto:rand_bytes/1 will fail, since it was removed in 20.0; use crypto:strong_rand_bytes/1
[10:12:36][/src Deps] Compiled src/hackney_pool.erl
[10:12:36][/src Deps] Compiled src/hackney_socks5.erl
[10:12:36][/src Deps] Compiled src/hackney_ssl_transport.erl
[10:12:36][/src Deps] Compiled src/hackney_response.erl
[10:12:36][/src Deps] Compiled src/hackney_sup.erl
[10:12:36][/src Deps] Compiled src/hackney_tcp_transport.erl
[10:12:36][/src Deps] Compiled src/hackney_request.erl
[10:12:36][/src Deps] /src/support/katt/deps/hackney/src/hackney_stream.erl:35: Warning: erlang:get_stacktrace/0: deprecated; use the new try/catch syntax for retrieving the stack backtrace
[10:12:36][/src Deps] Compiled src/hackney_stream.erl
[10:12:36][/src Deps] Compiled src/hackney_trace.erl
[10:12:36][/src Deps] Compiled src/hackney_util.erl
[10:12:36][/src Deps] Compiled src/hackney_url.erl
[10:12:36][/src Deps] ==> jsx (compile)
[10:12:36][/src Deps] Compiled src/jsx_consult.erl
[10:12:36][/src Deps] Compiled src/jsx.erl
[10:12:36][/src Deps] Compiled src/jsx_encoder.erl
[10:12:36][/src Deps] Compiled src/jsx_config.erl
[10:12:36][/src Deps] Compiled src/jsx_to_json.erl
[10:12:36][/src Deps] Compiled src/jsx_to_term.erl
[10:12:36][/src Deps] Compiled src/jsx_verify.erl
[10:12:36][/src Deps] Compiled src/jsx_parser.erl
[10:12:37][/src Deps] Compiled src/jsx_decoder.erl
[10:12:37][/src Deps] ==> tdiff (compile)
[10:12:37][/src Deps] /src/support/katt/deps/tdiff/src/tdiff_benchmark.erl:22: Warning: export_all flag enabled - all functions will be exported
[10:12:37][/src Deps] /src/support/katt/deps/tdiff/src/tdiff_benchmark.erl:34: Warning: random:seed0/0: the 'random' module is deprecated; use the 'rand' module instead
[10:12:37][/src Deps] /src/support/katt/deps/tdiff/src/tdiff_benchmark.erl:76: Warning: random:uniform/0: the 'random' module is deprecated; use the 'rand' module instead
[10:12:37][/src Deps] /src/support/katt/deps/tdiff/src/tdiff_benchmark.erl:90: Warning: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
[10:12:37][/src Deps] /src/support/katt/deps/tdiff/src/tdiff_benchmark.erl:96: Warning: random:uniform/1: the 'random' module is deprecated; use the 'rand' module instead
[10:12:37][/src Deps] /src/support/katt/deps/tdiff/src/tdiff_benchmark.erl:153: Warning: random:seed/3: the 'random' module is deprecated; use the 'rand' module instead
[10:12:37][/src Deps] Compiled src/tdiff_benchmark.erl
[10:12:37][/src Deps] Compiled src/tdiff.erl
[10:12:37][/src Deps] Compiled src/tdiff_debug.erl
[10:12:37][/src Deps] ==> meck (compile)
[10:12:37][/src Deps] Compiled src/meck_args_matcher.erl
[10:12:37][/src Deps] Compiled src/meck_code.erl
[10:12:37][/src Deps] Compiled src/meck_cover.erl
[10:12:37][/src Deps] /src/support/katt/deps/meck/src/meck_code_gen.erl:185: erlang:get_stacktrace/0: deprecated; use the new try/catch syntax for retrieving the stack backtrace
[10:12:37][/src Deps] Compiling /src/support/katt/deps/meck/src/meck_code_gen.erl failed:
[10:12:37][/src Deps] ERROR: compile failed while processing /src/support/katt/deps/meck: rebar_abort
[10:12:37][/src Deps] make[1]: *** [Makefile:111: compile] Error 1
[10:12:37][/src Deps] make: *** [Makefile:79: deps] Error 2
[10:12:37][/src Deps] *** Unmount system ***

add sleep functionality

at times, timing is important. a special header e.g. X-KATT-delay could dictate that the request is only to be executed after X seconds

allow chained APIBs for CLI katt

either by reading from stdin cat foo.apib bar.apib | katt ...
either by separating params from APIBs katt param1=1 param2=2 -- foo.apib bar.apib

Errors should not be printed to stdout

Use case: Piping output to json will produce confusing error messages in case of e.g. a parse error in the katt script.

Errors should be printed to stderr instead of stdout.

Katt version:
c418814 (1.5.4-29-gc418814)

detect redundant make

running make rebuilds every time

integrate/link to dev.erldocs.com

https://dev.erldocs.com/github.com/for-get/katt/

Update katt to avoid deprecated function call, in preparation for OTP25.

When compiling katt with the latest Erlang I get this warning:

src/katt.erl:161: http_uri:parse/1 is deprecated and will be removed in OTP 25; use use uri_string functions instead

I am not skilled enough in Erlang to create a pull request but I assume that the line:

{ok, {Protocol0, _ , Hostname, Port, Path0, _ }} = http_uri:parse(BaseUrl),

ought to be perhaps be rewritten into something like:

{ _ , Hostname, Path0, Port, _ , Protocol0, _ } = uri_string:parse(BaseUrl)

Using BARE_MODE

Hi,

I realized that BARE_MODE in it's current form doesn't in fact solve our problem. Since katt is a rebar dependency of our project and hackney is a subdependency it will get pulled down during the get-deps step. There is no time to create the marker file since the rebar command is atomic as far as make is concerned.

It would be nice if there were an environment variable that could be set.

broken with rebar 3.14

most probably due to erlang/rebar3#2211

// reported by @sstrigler

Add support for asserting [] response body

Currently, if you try to assert [] in the response body, katt will crash.

The reason is that katt interprets the [] in the spec file as empty list, but the same thing from the response is interpreted as {array, []}, which fails in tdiff:diff/3.

Support {{_}} when Content-Type application/json is set

Given the next apib file:

--- Test ---

---

---

# Test

GET http://www.mocky.io/v2/56dda46d110000910ad08673
< 200
{{_}}

GET http://www.mocky.io/v2/56dda46d110000910ad08673
< 200
< Content-Type: application/json
{{_}}

And running it, we get the next:

== PROGRESS REPORT parsing ==
{{2016,3,8},{14,2,55}}

"example.apib"


== PROGRESS REPORT parsed ==
{{2016,3,8},{14,2,55}}

"example.apib"


== PROGRESS REPORT run_transaction ==
{{2016,3,8},{14,2,55}}

<<"# Test\n">>


== PROGRESS REPORT transaction_result ==
{{2016,3,8},{14,2,55}}

{<<"# Test\n">>,
 [{"hostname","127.0.0.1"},
  {"port",80},
  {"protocol","http:"},
  {"request_timeout",20000},
  {"scenario_timeout",120000}],
 {katt_request,"GET","http://www.mocky.io/v2/56dda46d110000910ad08673",[],
               null},
 {katt_response,200,
                [{"Via","1.1 vegur"},
                 {"Date","Tue, 08 Mar 2016 13:02:55 GMT"},
                 {"Content-Type","application/json; charset=utf-8"},
                 {"Connection","close"},
                 {"Server","Cowboy"}],
                <<"[]">>,
                {array,[]}},
 pass}


== PROGRESS REPORT run_transaction ==
{{2016,3,8},{14,2,55}}

null

** exception exit: {case_clause,{start_object,{decoder,null,2,1,3,key}}}
     in function  mochijson3:decode_object/3 (src/mochijson3.erl, line 293)
     in call from mochijson3:json_decode/2 (src/mochijson3.erl, line 270)
     in call from katt_callbacks_json:parse_json/1 (src/katt_callbacks_json.erl, line 153)
     in call from katt:make_katt_response/3 (src/katt.erl, line 263)
     in call from katt:run_transactions/6 (src/katt.erl, line 199)
     in call from katt:run_scenario/5 (src/katt.erl, line 155)
     in call from katt:run/4 (src/katt.erl, line 91)
(kred@llenguado)26> 
=ERROR REPORT==== 8-Mar-2016::14:02:55 ===
Error in process <0.7751.0> on node 'kred@llenguado' with exit value: {{case_clause,{start_object,{decoder,null,2,1,3,key}}},[{mochijson3,decode_object,3,[{file,"src/mochijson3.erl"},{line,293}]},{mochijson3,json_decode,2,[{file,"src/mochijson3.erl"},{line,270}]},{katt_callbacks_json,parse_json...

So the first call works just fine, but the second crashes because katt tries to parse {{_}} as if it was json and it fails.

I think we should be able to assert on the content type and still ignore the whole body. Is there any other way to do that?

Storing key value will always succeed even if key does not exists.

GET http://httpbin.org/cache
> User-Agent: KATT
< 200
< Content-Type: application/json
{
    "foo": "{{>bar}}"
}

The key foo does not exist so I would have assumed this would fail. It does not and bar will get the value undefined

So is this an expected behaviour or a bug?

CLI flags

For big scenarios the output of KATT CLI can be big and not so helpful

Add flag to show only the failing response
Add flag to not show the (possibly big) body, just the errors

Allow secret params that are not logged

Allow some way of specifying secret params that should not be printed to stdout, e.g. params starting with underscore.

The benefits would be:

Avoid having secrets such as passwords or tokens appearing in logs.
Avoid having logs flooded with huge tokens.

Regarding benefit 2: In our case we we specify 4 different jwt tokens as params to KATT. They are each 940 chars and they appear 3 times each in the log (also in final_params and transaction_results). That is roughly 12K chars printed for each KATT file. We have roughly 10 KATT files so that is 120K chars in the log files for each test run.

Support complex/multiple store

Use case: Set-Cookie: cookie_name=cookie_value; path=/;

How do you extract cookie_value or cookie_name=cookie_value in order to use in future requests?

Support arbitrary transformations in request and response record fields

Would it be of interest to support arbitrary transformations in requests and response?

The gist of this proposal is to perform additional computation in katt:make_katt_request/3 and katt:make_katt_response/3 which evaluates transformations parametrized with the current Params set, substituting the result of these transformations into the appropriate request / response fields.

Background

Given a set of transactions in a blueprint, the response of one transaction may contain a value that is stored via {{>var}}. In one or more of the following transactions, rather than substituting the contents of var using {{<var}}, it may be more interesting to substitute a transformation of the contents of var - loosely, something like {{{ "type": "erlang", "eval": "mod:transform/1" }}}, where mod:transform/1 is parametrized over Params, and thus accessing var using proplists:get_value/2.

Example

A very made up example could be a client requesting a nonce for challenge, then answering it:

GET /challenge
> Accept: application/json
> Host: localhost
< 200
< Content-Type: application/json; charset=utf-8
{
    "nonce": "{{>nonce}}"
}

POST /challenge
> Content-Type: application/json
> Accept: application/json
{
    "response": "{{{ "type": "erlang", "eval": "challenge_response:answer_from_nonce/1" }}}"
}
< 200
< Content-Type: application/json; charset=utf-8
{
    "response_valid": true
}

However, the evaluation shouldn't necessary be limited to the body, but also be allowed in url and headers (think JWT).

Remarks

The runtime_value validation computation supports some type of arbitrary evaluation, but is only available in the (expected) body of the response. The substitution of params values is performed by katt_callbacks:recall/4, before the contents of the erlang or shell field is evaluated. Although there is some overlap between the expressiveness of this feature and runtime_value validation, this feature is different as it is run before the validation step.

Also, in relation to the discussion in #29 and #33, the type/eval field construction would allow for language independence (in principle), although there are open questions as to how to pass the Params array to the shell function, assuming we are restricted to for instance POSIX sh(1).

travis to github actions

wait for for-GET/jesse#100 for-GET/jesse#99 and use the same workflow configuration as jesse

Not possible to validate response as json if no content-type in response header

Specifying a response content-type in KATT to e.g. "application/foobar.object-v1+json" will:

cause Katt to validate the response as json
check that the response header specifies that content-type.

If the server does not respond with a content-type it is not possible to validate the response as json. This would be nice.

nicer message when scenario file doesn't exist

Not possible to specify multiple http response codes

Use case:
We are using KATT for api testing an http endpoint. But for some tests we first need to make sure that a resource exists (in this case a "customer"). Therefore we need to make a POST request to create the resource. If it already exists we will get 409 back, if not 201.

As far as I understand it is not possible to specify that more than response code is allowed and it is not possible to ignore the response code, only the header and body contents.

It would be very convenient to use KATT also for the POST call and not have to use another tool like Curl.

Use Rebar3 to publish. See publishing guide on http://www.rebar3.org/docs/publishing-packages

duplicate json-object properties doesn't throw parsing error

this is treated as valid json

{
  "a": "b",
  "a": "c"
}
`
``

proper json in actual, when run with --json

when there are errors, erlang structs get converted to "json", but still some things like actual get converted into e.g. "{array, ...}" rather than [...]