vdobler / ht Goto Github PK

View Code? Open in Web Editor NEW

24.0 24.0 4.0 3.38 MB

Http Testing made easy

License: BSD 3-Clause "New" or "Revised" License

Go 93.80% HTML 4.28% Shell 1.77% JavaScript 0.01% Makefile 0.07% Dockerfile 0.07%

ht's People

Contributors

Stargazers

Watchers

Forkers

schumacherfm gophersgang golint-fixer

ht's Issues

Load/Performance tests should pull data from an analytics platform

As an idea: ht might read the top n most requested URLs provided by an analytics platform like Google Analytics (GA).

Provide the authentication credentials for GA to ht.
ht logs in to GA pulls out the top n URLs
ht runs the performance test with those URLs

If you can show me the entry point where the benchmarks/performance tests will be started I might implement that feature myself, if you are interested in it.

Not resolvable variable is not reported

Assuming PRODUCTURL is not defined (i.e. not listed in "Variables" section during test execution), incorrect request is made:

GET https://xxxx:9002/.%7B%7BPRODUCTURL%7D%7D

Expected: Error message should be displayed.

Replace Otto with goja

https://github.com/dop251/goja

Full ECMAScript 5.1 support (yes, including regex and strict mode).
Passes nearly all tc39 tests tagged with es5id. The goal is to pass all of them.
On average 6-7 times faster than otto. Also uses considerably less memory.

;-)

Run all suites recursivly by providing only a folder name

$ $ ./ht/ht_darwin exec -Dfile=./env/local.json -v mobile

The above command runs all suites in alphabetical order in all folders.

Bug in filename extension in report.html

See the file extension .nil which triggers a download.

Suggestion: if response body is nil do not print it in the report

Use case: I would like to test if an ecommerce website has between Friday and Saturday for certain URLs (products) new prices and from Sunday on restored the old prices. Those prices are in the product listing and on the product detail page, etc.
If the tests fail I got notified via email, slack, twitter, carrier pigeon ...

The old command to execute such a test:

$ ht_darwin exec -output ./system-integration/reports/ -Dfile ./system-integration/env/local.json ./system-integration/product_listing/...

The new command could be for example

$ ht_darwin cron -config mycron.hjson -Dfile ./system-integration/env/local.json &

Starts ht_darwin in the background and keeps it running until killed or handled via supervisord.

The mycron.hjson contains the cron expression when to execute which test suite, a path to the test suite)

For example:

{
    "Tests": [
        {
            "Name": "cron test new prices",
            "CronExpression": "0 2 * * 5,6",
            "Suite": "./system-integration/product_listing/...",
            "Variables": {
                "SKU_321654987_PRICE": 324.45,
            },
            "report": ["email1","slackA"]
        },
        {
            "Name": "cron test old prices",
            "CronExpression": "0 2 * * 7",
            "Suite": "./system-integration/product_listing/...",
            "Variables": {
                "SKU_321654987_PRICE": 399.95,
            },
            "report": "email2",
        },
    ],
    "email1": {
        "type": "email",
        "smtp": "smtp.gmail.com",
        "user": "hans",
        "password": "uhu34hifhqief",
    },
    "email2": {
        "type": "email",
        "smtp": "smtp.gmail.com",
        "user": "hans",
        "password": "uhu34hifhqief",
    },
    "slackA": {
        "type": "slack",
        "TOKEN": "30205ead7bbe0b88c56d4005edb8ceff"
    },
}

Might there be another way to achieve this, an easier via e.g. Unix command line tools?
Any other ideas or just crap?

go test ./... fails -> phantomjs, Rscript not found

I know I have those programs not installed ... but I have a suggestion:
Check if they are installed, and if not skip the test with a simple message:

if false == binariesInstalled() {
    t.Skip("phantomjs not found, skipping ...")
}

So am I'm able to run the test suite and see if it all compiles but in-deeper testing is for you.

$ go test ./...
?   	github.com/vdobler/ht/cmd/ht	[no test files]
ok  	github.com/vdobler/ht/cookiejar	0.013s
ok  	github.com/vdobler/ht/fingerprint	1.544s
--- FAIL: TestRTHistogram (0.00s)
	graphics_test.go:26: Unexpected error fork/exec /usr/bin/Rscript: no such file or directory
Hello from testdata/custom.js
--- FAIL: TestScreenshotPass (0.01s)
	phantomjs_test.go:348: 0. Basic Screenshot of Home, Fail: exec: "phantomjs": executable file not found in $PATH
	phantomjs_test.go:348: 1. Greet Anonymous (white bg), Fail: exec: "phantomjs": executable file not found in $PATH
	phantomjs_test.go:348: 3. Greet Red user (red bg), Fail: exec: "phantomjs": executable file not found in $PATH
	phantomjs_test.go:348: 6. Greet RT user (olive bg), Fail: exec: "phantomjs": executable file not found in $PATH
	phantomjs_test.go:348: 8. Greet RT user (lime bg), Fail: exec: "phantomjs": executable file not found in $PATH
	phantomjs_test.go:348: 10. Greet Anonymous (different sizes), Fail: exec: "phantomjs": executable file not found in $PATH
	phantomjs_test.go:348: 11. Greet Bob, ignoring name, Fail: exec: "phantomjs": executable file not found in $PATH
	phantomjs_test.go:348: 12. Greet Bob, tollerating difference, Fail: exec: "phantomjs": executable file not found in $PATH
--- FAIL: TestRenderedHTMLPassing (0.00s)
	phantomjs_test.go:508: 1. Welcome Anonymous, rendered body, Fail: exec: "phantomjs": executable file not found in $PATH
	phantomjs_test.go:508: 3. Welcome Joe, Fail: exec: "phantomjs": executable file not found in $PATH
--- FAIL: TestRenderingTime2 (0.00s)
	phantomjs_test.go:603: Want status=Pass and nil error, got Fail, exec: "phantomjs": executable file not found in $PATH <*exec.Error>
--- FAIL: TestFancyScreenshotPass (0.00s)
	phantomjs_test.go:697: Want status=Pass and nil error, got Fail, exec: "phantomjs": executable file not found in $PATH <*exec.Error>
FAIL

Feature: Connection Timing information

I would like to see a report which states the timing information of a network request.

Hopefully this feature fits into your sprint planning process ;-)

https://twitter.com/davecheney/status/779269890918146048

Printscreen from the Chrome Network Panel

Problem with parsing two variables

Error message:

2016/02/26 14:08:10 ERROR failed preparing request: parse https://localhost:9002{{PRODUCTURL}}: invalid character "{" in host name ["xxxx"]
2016/02/26 14:08:10 Suite "xxxx", cannot prepare main test "xxxx": failed preparing request: parse https://localhost:9002{{PRODUCTURL}}: invalid character "{" in host name

Version affected:

$ ~/code/ht/ht_windows.exe version
ht version v1.0.0beta1

and also linux version:

ht version v0.9.0

Steps to reproduce:

create test with following parameters:

Request: {
    URL: "{{FRONTEND}}{{PRODUCTURL}}",
},

Trying to find workaround:

Request: {
    URL: "{{FRONTEND}}/.{{PRODUCTURL}}",
},

More verbose error messages

My program of two files:

package main

import (
	"net/http"
	"time"

	"github.com/vdobler/ht/ht"
)

var page01 = &ht.Test{
	Name: "Load page01 with a GET request",
	Request: ht.Request{
		Method: "GET",
		URL:    "{{PROTOCOL}}://{{HOST}}/page01.html",
		Header: http.Header{
			"Accept":          []string{"text/html"},
			"Accept-Encoding": []string{"gzip, deflate, br"},
		},
		Timeout: 1 * time.Second,
	},
	Checks: ht.CheckList{
		ht.StatusCode{Expect: 200},
		&ht.Header{
			Header: "Etag",
			Condition: ht.Condition{
				Min: 10}},
		&ht.None{
			Of: ht.CheckList{
				&ht.HTMLContains{
					Text: []string{"<esi:"},
				}}},
		&ht.Body{
			Contains: "demo-store.shop/autumn-pullie.html",
			Count:    1,
		},
	},
}

package main

import (
	"os"
	"runtime"

	"github.com/vdobler/ht/cookiejar"
	"github.com/vdobler/ht/ht"
)

func main() {
	jar, err := cookiejar.New(&cookiejar.Options{})
	if err != nil {
		panic(err)
	}
	c := ht.Collection{
		Tests: []*ht.Test{
			page01,
		},
	}
	if err := c.ExecuteConcurrent(runtime.NumCPU(), jar); err != nil {
		panic(err)
	}
	if c.Error != nil {
		panic(c.Error)
	}

	for _, test := range c.Tests {
		if err := test.PrintReport(os.Stdout); err != nil {
			panic(err)
		}
	}
}

Generates this error message:

$ go run ht/*.go
panic: malformed check: expected selector, found EOF instead

goroutine 1 [running]:
main.main()
	/Users/xxxxxx/src/github.com/SchumacherFM/caddyesi/ht/main.go:22 +0x296
exit status 2

Which check? Where? Would love to see a bit of more verbosity.

Usability error when a file in a suite cannot be found

I have a suite:

{
    Name: " Shop Basic Functionality",
    KeepCookies: true,

    Main: [
        {File: "homepage-basics.ht"}
    ]

    Teardown: [
        {File: "get-admin-token.ht"}
  ],
}

The file homepage-basics.ht exists but the file get-admin-token.ht does not exists.

When I run now:

$ ./ht/ht_darwin exec -output ./reports/ -Dfile ./env/local.json ./frontend/index.suite

The result is:

$ ./ht/ht_darwin exec -output ./reports/ -Dfile ./env/local.json ./frontend/index.suite
Seeding random number generator with 1507875229665009841.
Resetting global counter to 1.
Using "phantomjs" as PhantomJS executable.

Total 0,  Passed 0,  Skipped 0,  Errored 0,  Failed 0,  Bogus 0
NOTRUN

There is no indication that a file is missing or spelt wrong. You can search all over the suite and test files to look for the bug ... bit time consuming.

Would be nice to have an appropriate error message. 🙌

ht version: latest

RepsonseBody with correct file extension

ht/ht/report.go

Line 557 in f4a52f6

err = ioutil.WriteFile(path.Join(dir, tr.SeqNo+".ResponseBody"), body, 0666)

Are you open for the feature to extend the written file MainTest-XX.ResponseBody with the correct file extension depending on the content type?

E.g. MainTest-04.ResponseBody.json for pure JSON responses; MainTest-04.ResponseBody.html for HTML docs ... would be easier to open the files with the correct tools in an operating system.

Copy as CURL in report html

See printscreen

Idea to convert http.Request into curl: https://github.com/moul/http2curl/blob/master/http2curl.go

Master branch breaks build

user at machine in ~/GoPro/src/github.com/vdobler/ht/cmd/ht on master
$ go build .
# github.com/vdobler/ht/cmd/ht
./load.go:87: scenarios[i].Log undefined (type suite.Scenario has no field or method Log)

I'm using the master branch for integration testing in a travis CI build but now my build is breaking.

Would you please fix it.