Code Monkey home page Code Monkey logo

spectest's Introduction

Go Reference LinuxUnitTest MacUnitTest WindowsUnitTest UnitTestExampleCodes reviewdog Gosec Coverage

What is spectest?

A simple and extensible behavioral testing library. Supports mocking external http calls and renders sequence diagrams on completion. In behavioral tests the internal structure of the app is not known by the tests. Data is input to the system and the outputs are expected to meet certain conditions.

This project is forked from steinfletcher/apitest apitest was functionally complete. However, I wanted more features, so I decided to fork it to actively develop it further. I will mainly enhance document generation and integration with AWS. There are no plans for compatibility between apitest and spectest. Therefore, future development will include BREAKING CHANGES.

The spectest has its own unique use cases, Use Cases of spectest. Please refer to this document for more information.

Supported OS

  • Linux
  • Mac
  • Windows (Original apitest does not support Windows)

Installation

go get -u github.com/nao1215/spectest

Demo

animated gif

Examples

Framework and library integration examples

Example Comment
gin popular martini-like web framework
graphql using gqlgen.com to generate a graphql server
gorilla the gorilla web toolkit
iris (broken) iris web framework
echo High performance, extensible, minimalist Go web framework
fiber Express inspired web framework written in Go
httprouter High performance HTTP request router that scales well
mocks example mocking out external http calls
sequence diagrams generate sequence diagrams from tests
Ginkgo Ginkgo BDD test framework
plantuml wxample generating plantuml

Companion libraries (Side projects)

In the original apitest repository, side projects were managed in separate repositories. However, in spectest, these side projects are managed within the same repository. However, the difflib, which operates independently from spectest, and the malfunctioning aws package, are managed in separate repositories.

Library Comment
JSON Path JSON Path assertion addons
JOSN Schema JSON Schema assertion addons
CSS Selectors CSS selector assertion addons
PlantUML Export sequence diagrams as plantUML
DynamoDB (broken) Add DynamoDB interactions to sequence diagrams

Credits

This library was influenced by the following software packages:

  • YatSpec for creating sequence diagrams from tests
  • MockMVC and superagent for the concept and behavioral testing approach
  • Gock for the approach to mocking HTTP services in Go
  • Baloo for API design

Code snippets

JSON body matcher

func TestApi(t *testing.T) {
	spectest.New().
		Handler(handler).
		Get("/user/1234").
		Expect(t).
		Body(`{"id": "1234", "name": "Tate"}`).
		Status(http.StatusOK).
		End()
}

JSONPath

For asserting on parts of the response body JSONPath may be used. A separate module must be installed which provides these assertions - go get -u github.com/nao1215/spectest/jsonpath. This is packaged separately to keep this library dependency free.

Given the response is {"a": 12345, "b": [{"key": "c", "value": "result"}]}

func TestApi(t *testing.T) {
	spectest.Handler(handler).
		Get("/hello").
		Expect(t).
		Assert(jsonpath.Contains(`$.b[? @.key=="c"].value`, "result")).
		End()
}

and jsonpath.Equals checks for value equality

func TestApi(t *testing.T) {
	spectest.Handler(handler).
		Get("/hello").
		Expect(t).
		Assert(jsonpath.Equal(`$.a`, float64(12345))).
		End()
}

Custom assert functions

func TestApi(t *testing.T) {
	spectest.Handler(handler).
		Get("/hello").
		Expect(t).
		Assert(func(res *http.Response, req *http.Request) error {
			assert.Equal(t, http.StatusOK, res.StatusCode)
			return nil
		}).
		End()
}

Assert cookies

func TestApi(t *testing.T) {
	spectest.Handler(handler).
		Patch("/hello").
		Expect(t).
		Status(http.StatusOK).
		Cookies(spectest.Cookie("ABC").Value("12345")).
		CookiePresent("Session-Token").
		CookieNotPresent("XXX").
		Cookies(
			spectest.Cookie("ABC").Value("12345"),
			spectest.Cookie("DEF").Value("67890"),
		).
		End()
}

Assert headers

func TestApi(t *testing.T) {
	spectest.Handler(handler).
		Get("/hello").
		Expect(t).
		Status(http.StatusOK).
		Headers(map[string]string{"ABC": "12345"}).
		End()
}

Mocking external http calls

var getUser = spectest.NewMock().
	Get("/user/12345").
	RespondWith().
	Body(`{"name": "jon", "id": "1234"}`).
	Status(http.StatusOK).
	End()

var getPreferences = spectest.NewMock().
	Get("/preferences/12345").
	RespondWith().
	Body(`{"is_contactable": true}`).
	Status(http.StatusOK).
	End()

func TestApi(t *testing.T) {
	spectest.New().
		Mocks(getUser, getPreferences).
		Handler(handler).
		Get("/hello").
		Expect(t).
		Status(http.StatusOK).
		Body(`{"name": "jon", "id": "1234"}`).
		End()
}

Generating sequence diagrams from tests

func TestApi(t *testing.T) {
	spectest.New().
		Report(spectest.SequenceDiagram()).
		Mocks(getUser, getPreferences).
		Handler(handler).
		Get("/hello").
		Expect(t).
		Status(http.StatusOK).
		Body(`{"name": "jon", "id": "1234"}`).
		End()
}

It is possible to override the default storage location by passing the formatter instance Report(spectest.NewSequenceDiagramFormatter(".sequence-diagrams")). If you want to change the report file name , you use CustomReportName("file name is here") . By default, the hash value becomes the report file name.

You can bring your own formatter too if you want to produce custom output. By default a sequence diagram is rendered on a html page.

The spectest checks the Content Type of the response. If it's an image-related MIME type, the image will be displayed in the report. In the apitest, binary data was being displayed.

DiagramSample

One feature that does not exist in the apitest fork is the ability to output reports in markdown format. The below code snippet is an example of how to output a markdown report.

func TestApi(t *testing.T) {
	spectest.New().
		CustomReportName("markdow_report").
		Report(spectest.SequenceReport(spectest.ReportFormatterConfig{
			Path: "doc",
			Kind: spectest.ReportKindMarkdown,
		})).
		Handler(handler).
		Get("/image").
		Expect(t).
		Body(string(body)).
		Header("Content-Type", "image/png").
		Header("Content-Length", fmt.Sprint(imageInfo.Size())).
		Status(http.StatusOK).
		End()
}

MarkdownReportSample

Debugging http requests and responses generated by api test and any mocks

func TestApi(t *testing.T) {
	spectest.New().
		Debug().
		Handler(handler).
		Get("/hello").
		Expect(t).
		Status(http.StatusOK).
		End()
}

Provide basic auth in the request

func TestApi(t *testing.T) {
	spectest.Handler(handler).
		Get("/hello").
		BasicAuth("username", "password").
		Expect(t).
		Status(http.StatusOK).
		End()
}

Pass a custom context to the request

func TestApi(t *testing.T) {
	spectest.Handler(handler).
		Get("/hello").
		WithContext(context.TODO()).
		Expect(t).
		Status(http.StatusOK).
		End()
}

Provide cookies in the request

func TestApi(t *testing.T) {
	spectest.Handler(handler).
		Get("/hello").
		Cookies(spectest.Cookie("ABC").Value("12345")).
		Expect(t).
		Status(http.StatusOK).
		End()
}

Provide headers in the request

func TestApi(t *testing.T) {
	spectest.Handler(handler).
		Delete("/hello").
		Headers(map[string]string{"My-Header": "12345"}).
		Expect(t).
		Status(http.StatusOK).
		End()
}

Provide query parameters in the request

Query, QueryParams and QueryCollection can all be used in combination

func TestApi(t *testing.T) {
	spectest.Handler(handler).
		Get("/hello").
		QueryParams(map[string]string{"a": "1", "b": "2"}).
		Query("c", "d").
		Expect(t).
		Status(http.StatusOK).
		End()
}

Providing {"a": {"b", "c", "d"} results in parameters encoded as a=b&a=c&a=d. QueryCollection can be used in combination with Query

func TestApi(t *testing.T) {
	spectest.Handler(handler).
		Get("/hello").
		QueryCollection(map[string][]string{"a": {"b", "c", "d"}}).
		Expect(t).
		Status(http.StatusOK).
		End()
}

Provide a url encoded form body in the request

func TestApi(t *testing.T) {
	spectest.Handler(handler).
		Post("/hello").
		FormData("a", "1").
		FormData("b", "2").
		FormData("b", "3").
		FormData("c", "4", "5", "6").
		Expect(t).
		Status(http.StatusOK).
		End()
}

Provide a multipart/form-data

func TestApi(t *testing.T) {
	spectest.Handler(handler).
		Post("/hello").
		MultipartFormData("a", "1", "2").
		MultipartFile("file", "path/to/some.file1", "path/to/some.file2").
		Expect(t).
		Status(http.StatusOK).
		End()
}

Capture the request and response data

func TestApi(t *testing.T) {
	spectest.New().
		Observe(func(res *http.Response, req *http.Request, specTest *spectest.SpecTest) {
			// do something with res and req
		}).
		Handler(handler).
		Get("/hello").
		Expect(t).
		Status(http.StatusOK).
		End()
}

Intercept the request

This is useful for mutating the request before it is sent to the system under test.

func TestApi(t *testing.T) {
	spectest.Handler(handler).
		Intercept(func(req *http.Request) {
			req.URL.RawQuery = "a[]=xxx&a[]=yyy"
		}).
		Get("/hello").
		Expect(t).
		Status(http.StatusOK).
		End()
}

Contributing

View the contributing guide.

spectest's People

Contributors

anthonyfong100 avatar aqyuki avatar billbell73 avatar charlesworth avatar chrisjharding avatar codacy-badger avatar de-perotti avatar dependabot-preview[bot] avatar dependabot[bot] avatar emmanuelay avatar enrico5b1b4 avatar fergusstrange avatar github-actions[bot] avatar haraldmosh avatar hatsunemiku3939 avatar ikawaha avatar jessbellon avatar joncfoo avatar maranqz avatar mikacodes avatar nao1215 avatar seabornlee avatar steinfletcher avatar w32blaster avatar webbgeorge avatar ytausch avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar

Forkers

ikawaha aqyuki

spectest's Issues

Support more http methods mock

Is your feature request related to a problem? Please describe.

Related #10

We need to add the following HTTP method mock methods to the Mock structure.

  • HEAD
  • CONNECT
  • OPTIONS
  • TRACE

Modifying Data Structures

Spectest has many fields within a single struct. I believe these fields can be encapsulated into smaller structs. By creating new structs and associating methods with them, the code readability and organization can be significantly improved.

Fix broken example of github.com/kataras/iris

Is your feature request related to a problem? Please describe.

The apitest package, which is the source of the fork, provided an example at github.com/kataras/iris. However, they were broken and did not work. We will re-create the examples again.

Combine the documents (HTML) into one.

Currently, we are generating separate HTML files for each test case, but I would like to generate a single HTML document that consolidates documentation for each endpoint.

Output error messages in red text 

Is your feature request related to a problem? Please describe.

If the expected value does not match the obtained value, the following message appears. We want to display this message in red text to make it easier for users to understand the error content.

                Error:          Not equal: 
                                expected: true
                                actual  : false
                Test:           TestNewApp/Width_test
                Messages:       mismatched values for header 'Content-Type'. Expected image/png but received text/plain; charset=utf-8
                Name:           width,height is zero

Use fatih/color to change text color for cross-platform support.

Feature: markdown indexing

Is your feature request related to a problem? Please describe.

The spectest produces reports in Markdown format on a per-test-case basis. It is hard for us to find the desired report among many Markdowns.

Therefore, we will add a function to index markdowns located under the specified directory.

Describe the solution you'd like

jbrown1618/markdown-index is close to the function we are looking for. However, some features are missing.

For example,

  • change generating file name.
  • change index markdown title
  • change url text (now, it is same as file name)

Additional context

This feature will implement as cli command.
In addition, add the ability to generate indexes with go generate.

Feature to Improve Readability of Test Results Output

Is your feature request related to a problem? Please describe.

A feature has been added to change the output of test results when using spectest ./... instead of go test ./.... For instance, successful tests will be displayed as green dots, while failed tests will be represented by red dots. This functionality is inspired by ideas from onsi/ginkgo and shellspec/shellspec.

Describe the solution you'd like

WIP

Describe alternatives you've considered

WIP

Additional context

WIP

Fix broken example of github.com/99designs/gqlgen

Is your feature request related to a problem? Please describe.

The apitest package, which is the source of the fork, provided an example at github.com/99designs/gqlgens. However, they were broken and did not work. We will re-create the examples again.

Supported windows

Is your feature request related to a problem? Please describe.

I modify the code so that unit tests pass in a Windows environment.
I know that the difflib package is not compatible with windows. I think we should export for difflib package as an external repository.

Return error instead of using panic

Is your feature request related to a problem? Please describe.

Due to heavy use of method chaining, most of the error handling in apitest relies on panic(). However, even with method chaining, errors can be returned to the calling function or evaluated later.

In spectest, the error handling approach will be changed.

Improving unit test coverage.

To proceed with refactoring, it is necessary to ensure the current behavior through unit tests. First, create unit tests for functions that are not covered by test coverage. Since unit tests are created incrementally, it is expected that multiple pull requests will be made.

You can check the unit test coverage with the following command.

$ make test
$ open coverage.html 

Creating Function/Method-Based Sequence Diagrams

Is your feature request related to a problem? Please describe.

"Spectest" has the functionality to create sequence diagrams. However, due to the coarse granularity of the generated sequence diagrams, I want to create sequence diagrams based on method calls.

When creating function/method-based sequence diagrams, it is straightforward to perform static analysis of the code. This method eliminates the need to execute it every time during testing, addressing concerns about slow performance.

Therefore, a command-based sequence diagram generation feature is developed to facilitate verification.

Introduce gosec

Introduce gosec. For example;

name: Gosec
on:
  workflow_dispatch:
  push:
    branches: [main]
  pull_request:
    branches: [main]
jobs:
  tests:
    runs-on: ubuntu-latest
    env:
      GO111MODULE: on
    steps:
      - name: Checkout Source
        uses: actions/checkout@v4
      - name: Run Gosec Security Scanner
        uses: securego/gosec@master
        with:
          args: ./...

Adding AWS SDK Mock to go-spectest/aws

Is your feature request related to a problem? Please describe.

We are implementing mocks for the official AWS package (SDK) in go-spectest/aws. The package was severely broken at the time of forking, and there was also an issue with the limited number of supported AWS resources.

Therefore, we believe it would be best to start fresh by adding AWS SDK mocks anew. With over 100 types of AWS resources, manually adding mocks would be daunting. We are considering adopting a method to automatically generate mocks from the interfaces exposed by the AWS SDK.

Additional context

Eventually, go-spectest/aws is planned to be integrated into go-spectest/spectest.

Output sequence diagram with custom name instead of hash

Is your feature request related to a problem? Please describe.

The spectest package generates sequence diagrams.
This diagram has a hash as the file name. I want to store sequence diagrams in a document directory to share them with developers. However, using hash names for file titles is a bit inconvenient.

So, I wanted to give custom names to the sequence diagrams.

Describe the solution you'd like

The solution is to add CustomeSequenceDiagramName().
(Method name may not be appropriate)

example

				Report(spectest.SequenceDiagram()).
				CustomeSequenceDiagramName("diagram_name_is_here").
				// ...
				End()

CustomSequenceDiagramName set APITest.meta["diagramName"]. If meta['diagramName'] is set, the hash name will not be used when outputting the sequence diagram.
The problem with this approach is that if you execute Meta() after executing CustomSequenceDiagramName(), the settings from CustomSequenceDiagramName() are lost.

Describe alternatives you've considered

The alternative way is implementing new ReportFormatter interface.
This method has a large modification scale. However, it has the advantage of being able to be implemented outside of the spectest package (I would adopt this idea if no additional features were added).

Additional context

nothing

Adding Documentation Comments and Reducing Function Complexity

Is your feature request related to a problem? Please describe.

Some functions in spectest are hard to understand at a glance. To address this, I will add documentation. Additionally, due to the lengthy lines of code within the functions, I will refactor them into smaller, more manageable functions.

Display response data in the document according to Content Type

Is your feature request related to a problem? Please describe.

The spectest display response data in the document according to Content Type.
For example, spectest will display binary data in the document if the response data is an image. If possible, i would like to display images in the document.

Describe the solution you'd like

Determine Content-Type from HTTP headers and create HTML template generation process according to Content-Type.

Describe alternatives you've considered

The method I suggested has the potential to significantly rewrite the HTML template.

Additional context

nothing

update spectest/x/db

Is your feature request related to a problem? Please describe.

The spectest/x/db package is a wrapper for the database/sql package. However, there are wrappers for deprecated methods, and some parts do not work as wrappers for the current database/sql package.

So, update spectest/x/db.

`spectest index` command standard outputs the index markdown file path after generated it.

Is your feature request related to a problem? Please describe.

After generating the index markdown, spectest index does not output anything. I think it is difficult for the user to understand what happened with this specification. Therefore, the generated index file path will be outputted to the standard output.

Describe the solution you'd like

If markdown.GenerateIndex() completes successfully, it will output the index file path to the standard output. If an error occurs, it will directly return the error to the higher-level function.
https://github.com/go-spectest/spectest/blob/26de85c47a740f596035376cf4b84e09ac9c7fc2/cmd/spectest/sub/index.go#L59-L62

Switching line feed by OS

Is your feature request related to a problem? Please describe.

It seems that "\n" is hard-coded as a newline code, so use "\n" on Linux and Mac, and "\r\n" on Windows.

SPECTEST ROADMAP

Introduction

The spectest is a package forked from steinfletcher/apitest. The apitest has the functionality to generate HTML documents from end-to-end test results, and the development of spectest was initiated to enhance this feature.

One month after the fork, spectest has added the capability to output documents in markdown format (I have also conducted refactoring). While there are several feature ideas, developing them alone would be time-consuming.

Therefore, I have decided to clearly outline spectest's roadmap and record feature ideas as issues. If you are interested in spectest, feel free to implement some of these issues.

Goal

  1. Generate Markdown documents from E2E test results to reduce the time developers spend on document creation.
  2. Support BDD-style testing methods.
  3. Become a versatile library that allows easy writing of unit tests for entities other than APIs.
  4. Provide CLI commands to enhance the developer experience during unit test execution.

Roadmap

  • Roadmap 1. Refactoring
  • Roadmap 2. Adding Functionality to Simplify Unit Testing
  • Roadmap 3. Enhancement of Documentation Feature
  • Roadmap 4. Creating Mocks for External Systems Such as AWS
  • Roadmap 5. Making Spectest Usable for Simple Unit Tests
  • Roadmap 6. Creating a Website to Explain How to Use Spectest

Roadmap 1. Refactoring

Roadmap 2. Adding Functionality to Simplify Unit Testing

Roadmap 3. Enhancement of Documentation Feature

Roadmap 4. Creating Mocks for External Systems Such as AWS

  • #143
  • ??? Adding another service mock such as firebase.

Roadmap 5. Making Spectest Usable for Simple Unit Tests

Roadmap 6. Creating a Website to Explain How to Use Spectest

Feature: generate golden file

Is your feature request related to a problem? Please describe.

We often manually create expectations for response data. This process is very tedious. Therefore, we will add a function to generate the expected value data in a specified path when the test is executed with a specific procedure.

Describe the solution you'd like

We would add the Golden() method. This method writes response body data to the specified file path. However, the write operation is performed only when certain flags are enabled.

The method to enable the flags is under consideration. For example, we can create a spectest cli command, and the Golden() method will generate a file if the --golden option of this command is enabled.

Merge projects managed by git submodule into spectest.

Is your feature request related to a problem? Please describe.

Many projects managed by git submodule are tightly integrated with spectest. Therefore, whenever code refactoring is completed within each repository, it is incorporated into spectest.

Side project for spectest: client/server application

Why we need side project

I believe that the best way to improve the spectest package is to actually use it. Since spectest is a testing framework, it requires something to test. However, it doesn't make sense to use it in very small projects.

Therefore, I plan to develop a client-server application and test its server-side components using spectest. The goal is to reach a state where spectest is used for all unit tests (currently, spectest provides functionality only for E2E testing).

The server and client will be developed in separate repositories. The client is intended for Android (kotlin) or Web (elm) platforms.

First Decisions

Deciding on the application concept is the first step.
We plan to create an app for upscaling images and videos, but this application will require Deep Learning training. Additionally, we need to consider other app ideas that do not require as much development effort.

Support more http methods

Now, spectest support below method;

  • GET
  • POST
  • PUT
  • DELETE
  • PATCH

These method are defined in RFC 7231 section 4.3. The spectest is not supported them.

  • HEAD
  • CONNECT
  • OPTIONS
  • TRACE

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.