Code Monkey home page Code Monkey logo

pacto's Introduction

Gem Version Build Status Code Climate Dependency Status Coverage Status

Pacto is currently INACTIVE. Unfortunately none of the maintainers have had enough time to maintain it. While we feel that Pacto had good ideas, we also feel that a lot has changed since Pacto was first conceived (including the OpenAPIs initiative) and that too much work would be required to maintain & update Pacto. Instead, we encourage others to use other projects that have focused on Consumer-Driven Contracts, like Pact, or to write their own Pacto-inspired project.

[INACTIVE] Pacto

Pacto is a judge that arbitrates contract disputes between a service provider and one or more consumers. In other words, it is a framework for Integration Contract Testing, and helps guide service evolution patterns like Consumer-Driven Contracts or Documentation-Driven Contracts.

Pacto considers two major terms in order decide if there has been a breach of contract: the request clause and the response clause.

The request clause defines what information must be sent by the consumer to the provider in order to compel them to render a service. The request clause often describes the required HTTP request headers like Content-Type, the required parameters, and the required request body (defined in json-schema) when applicable. Providers are not held liable for failing to deliver services for invalid requests.

The response clause defines what information must be returned by the provider to the consumer in order to successfully complete the transaction. This commonly includes HTTP response headers like Location as well as the required response body (also defined in json-schema).

Test Doubles

The consumer may also enter into an agreement with test doubles like WebMock, vcr or mountebank. The services delivered by the test doubles for the purposes of development and testing will be held to the same conditions as the service the final services rendered by the parent provider. This prevents misrepresentation of sample services as realistic, leading to damages during late integration.

Pacto can provide a test double if you cannot afford your own.

Due Diligence

Pacto usually makes it clear if the consumer or provider is at fault, but if a contract is too vague Pacto cannot assign blame, and if it is too specific the matter may become litigious.

Pacto can provide a contract writer that tries to strike a balance, but you may still need to adjust the terms.

Implied Terms

  • Pacto only arbitrates contracts for JSON services.
  • Pacto requires Ruby 1.9.3 or newer, though you can use Polyglot Testing techniques to support older Rubies and non-Ruby projects.

Roadmap

  • The test double provided by Pacto is only semi-competent. It handles simple cases, but we expect to find a more capable replacement in the near future.
  • Pacto will provide additional Contract Writers for converting from apiblueprint, WADL, or other documentation formats in the future. It's part of our goal to support Documentation-Driven Contracts
  • Pacto reserves the right to consider other clauses in the future, like security and compliance to industry specifications.

Usage

See also: http://thoughtworks.github.io/pacto/usage/

Pacto can perform three activities: generating, validating, or stubbing services. You can do each of these activities against either live or stubbed services.

You can also use Pacto Server if you are working with non-Ruby projects.

Configuration

In order to start with Pacto, you just need to require it and optionally customize the default Configuration. For example:

require 'pacto'

Pacto.configure do |config|
  config.contracts_path = 'contracts'
end

Generating

The easiest way to get started with Pacto is to run a suite of live tests and tell Pacto to generate the contracts:

Pacto.generate!
# run your tests

If you're using the same configuration as above, this will produce Contracts in the contracts/ folder.

We know we cannot generate perfect Contracts, especially if we only have one sample request. So we do our best to give you a good starting point, but you will likely want to customize the contract so the validation is more strict in some places and less strict in others.

Contract Lists

In order to stub or validate a group of contracts you need to create a ContractList. A ContractList represent a collection of endpoints attached to the same host.

require 'pacto'

default_contracts = Pacto.load_contracts('contracts/services', 'http://example.com')
authentication_contracts = Pacto.load_contracts('contracts/auth', 'http://example.com')
legacy_contracts = Pacto.load_contracts('contracts/legacy', 'http://example.com')

Validating

Once you have a ContractList, you can validate all the contracts against the live host.

contracts = Pacto.load_contracts('contracts/services', 'http://example.com')
contracts.simulate_consumers

This method will hit the real endpoint, with a request created based on the request part of the contract.
The response will be compared against the response part of the contract and any structural difference will generate a validation error.

Running this in your build pipeline will ensure that your contracts actually match the real world, and that you can run your acceptance tests against the contract stubs without worries.

Stubbing

To generate stubs based on a ContractList you can run:

contracts = Pacto.load_contracts('contracts/services', 'http://example.com')
contracts.stub_providers

This method uses webmock to ensure that whenever you make a request against one of contracts you actually get a stubbed response, based on the default values specified on the contract, instead of hitting the real provider.

You can override any default value on the contracts by providing a hash of optional values to the stub_providers method. This will override the keys for every contract in the list

contracts = Pacto.load_contracts('contracts/services', 'http://example.com')
contracts.stub_providers(request_id: 14, name: "Marcos")

Pacto Server (non-Ruby usage)

See also: http://thoughtworks.github.io/pacto/patterns/polyglot/

We've bundled a small server that embeds pacto so you can use it for non-Ruby projects. If you want to take advantage of the full features, you'll still need to use Ruby (usually rspec) to drive your API testing. You can run just the server in order to stub or to write validation issues to a log, but you won't have access to the full API fail your tests if there were validation problems.

Command-line

The command-line version of the server will show you all the options. These same options are used when you launch the server from within rspec. In order to see the options: bundle exec pacto-server --help

Some examples:

$ bundle exec pacto-server -sv --generate
# Runs a server that will generate Contracts for each request received
$ bundle exec pacto-server -sv --stub --validate
# Runs the server that provides stubs and checks them against Contracts
$ bundle exec pacto-server -sv --live --validate --host
# Runs the server that acts as a proxy to http://example.com, validating each request/response against a Contract

RSpec test helper

You can also launch a server from within an rspec test. The server does start up an external port and runs asynchronously so it doens't block your main test thread from kicking off your consumer code. This gives you an externally accessable server that non-Ruby clients can hit, but still gives you the full Pacto API in order to validate and spy on HTTP interactions.

Example usage of the rspec test helper:

require 'rspec'
require 'pacto/rspec'
require 'pacto/test_helper'

describe 'my consumer' do
  include Pacto::TestHelper

  it 'calls a service' do
    with_pacto(
      :port => 5000,
      :directory => '../spec/integration/data',
      :stub => true) do |pacto_endpoint|
      # call your code
      system "curl #{pacto_endpoint}/echo"
      # check results
      expect(Pacto).to have_validated(:get, 'https://localhost/echo')
    end
  end
end

Rake Tasks

Pacto includes a few Rake tasks to help with common tasks. If you want to use these tasks, just add this top your Rakefile:

require 'pacto/rake_task'

This should add several new tasks to you project:

rake pacto:generate[input_dir,output_dir,host]  # Generates contracts from partial contracts
rake pacto:meta_validate[dir]                   # Validates a directory of contract definitions
rake pacto:validate[host,dir]                   # Validates all contracts in a given directory against a given host

The pacto:generate task will take partially defined Contracts and generate the missing pieces. See Generate for more details.

The pacto:meta_validate task makes sure that your Contracts are valid. It only checks the Contracts, not the services that implement them.

The pacto:validate task sends a request to an actual provider and ensures their response complies with the Contract.

Contracts

Pacto works by associating a service with a Contract. The Contract is a JSON description of the service that uses json-schema to describe the response body. You don't need to write your contracts by hand. In fact, we recommend generating a Contract from your documentation or a service.

A contract is composed of a request that has:

  • Method: the method of the HTTP request (e.g. GET, POST, PUT, DELETE);
  • Path: the relative path (without host) of the provider's endpoint;
  • Headers: headers sent in the HTTP request;
  • Params: any data or parameters of the HTTP request (e.g. query string for GET, body for POST).

And a response has that has:

  • Status: the HTTP response status code (e.g. 200, 404, 500);
  • Headers: the HTTP response headers;
  • Body: a JSON Schema defining the expected structure of the HTTP response body.

Below is an example contract for a GET request to the /hello_world endpoint of a provider:

{
  "request": {
    "method": "GET",
    "path": "/hello_world",
    "headers": {
      "Accept": "application/json"
    },
    "params": {}
  },

  "response": {
    "status": 200,
    "headers": {
      "Content-Type": "application/json"
    },
    "body": {
      "description": "A simple response",
      "type": "object",
      "properties": {
        "message": {
          "type": "string"
        }
      }
    }
  }
}

Constraints

  • Pacto only works with JSON services
  • Pacto requires Ruby 1.9.3 or newer (though you can older Rubies or non-Ruby projects with a Pacto Server)
  • Pacto cannot currently specify multiple acceptable status codes (e.g. 200 or 201)

Contributing

Read the CONTRIBUTING.md file.

pacto's People

Contributors

cv avatar danny-andrews avatar davidbegin avatar fernando-alves avatar filipesperandio avatar jmmercadom avatar marcosccm avatar maxlinc avatar raquelguimaraes avatar srikar avatar taisediass avatar ticolucci avatar tmattia avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

pacto's Issues

Do something about headers...

One of the advantages I see about generating contracts as opposed to recording HTTP interactions like a traditional self-initializing fake is we avoid saving either:

  • Ephemeral data (data that is likely to change when you re-record, even if it's only been a few minutes)
  • Sensitive data

Pacto is good at this for request/response bodies. It doesn't care about changing timestamps, storing passwords, etc., as long as the structure doesn't change. It will only record the actual values if you ask it to. Even then, I'm hoping to store it in a separate file so version control will maintain a clear separation between "changing schema" and "changing data".

Headers are more problematic. Pacto does throw out some ephemeral headers... but it stores the exact data for whatever headers it keeps. The GitHub samples, for example, store Etags:

"Etag": ""59455ce089e99a07e0b12b9273cd6c8d""

I would like to keep info about the structure but not data. There are a few ways this could bone done. One might be to just indicate if the header should be present at all (true/false values?). Another option is to allow a subschema for each header. This is powerful, but could make the contracts even harder to read. A third option is to specify a type, which could be basic primitives (string, int, etc), but could also probably include predefined formats URI, email address, UUID, ETag, etc. Another option is to use regexes.

Any creative suggestions?

Better response header filtering

Opening a new issue. Attaching a PR failed, because I was branched from the pre-release branch instead of master...

Generator should drop these by default:

  • Headers starting with "X-"
  • Cache validation headers
    • Last-Modified
    • Etag
    • MAYBE NOT cache policy headers (Cache-Control), sometimes the age is dynamic, though...
  • Connection controls
    • Transfer-Encoding

Contributor guidelines

Please contribute and update the todo list with your ideas

As a Pacto contributor
I want to contribute following some guidelines
So I can participate more effectively in the project

It was previously discussed that we need some guidelines for contributors so everybody proposes ways to contribute and agrees. We have to take more actions about this like on #28 that was a starting point to have a more consistent codebase,

These is a initial list of things to be addressed:

  • Create a page on the project that explains the best way to contribute on the project
  • Have a guideline on the best way for us to write specs
  • Have a guideline on the best way for us to write production code

Request/Response - handle more classes

We use request and response objects for a few things: generation, validating, looking up contracts.

I've been able to wire Pacto up to WebMock, VCR, Goliath (RackRequest-like), etc. However, I keep having to write (relatively small) methods to transform from one set of HTTP Request/Response classes to another fairly similar one.

I don't think this is as horrible as it sounds. We might not need to write adapters for every possible library. We could investigate:

  • Can WebMock handle this (it already has adapters for many HTTP libs)
  • Is there a least-common-denomator we can use?
  • Can we use some simple duck typing? E.g.: parse path from uri if it exists, check if headers are a string or already parsed into a hash.

Request#execute fails with a bad message when you don't provide a scheme for the host

If you call execute on a Pacto::Request that was initialized with a host that doesn't contain a scheme you get a cryptic Faraday exception.

request = Pacto::Request.new("localhost", { the_rest: "of the data" })
request.execute #boom

This happens for instance when you try to use the pacto:validate rake task with a "bad" uri

rake 'pacto:validate[localhost,/contracts]'
#boom

The Request class actually has some uri methods that normalize the scheme, but they are not used by the execute method.

PR coming soon =)

Proposal: Consumer Actors

Consumer Actor Adapter

The roadmap for Pacto v0.4.0 will shift towards a more hexagonal or "Ports and Adapters" architecture. One of my planned adapters is Actors - in the sense of "courtroom re-enactments".

The Consumer Actor adaptor type will drive the consumer during testing.

Responsibility

The Consumer Actor is responsible for creating a request for Pacto to validate. Since the creation of a request kicks off the test workflow, this adapter is fairly open ended. It converts something into a Pacto Request, but the type of something will probably vary with each Consumer Actor.

We already have three likely candidates for Consumer Actors:

  • JSONGeneratorConsumerActor - creates a request based on the JSON schema. This is current behavior used by the validate rake task
  • ExampleConsumerActor - Creates a request based on an example. I propose this as the new default behavior for the validate rake task in v0.4.0, taking advantage of the example section of the new Contract schema.
  • WebMockConsumerActor - Forwards a request detected via WebMock to Pacto. This is usually used when driving tests via RSpec. See the WebMockAdapter.
  • ProxyConsumerActor - Forwards a request received by a server to Pacto. This, combined with the WebMockConsumerActor, is how the Pacto Server works.

Proposed Adapter Interface

# Generates a request for the service described by the contract
# @param contract [Pacto::Contract] the contract describing the service.
# @params data [Object] data to convert to a request (expected type varies by actor)
# @return [Request]
def reenact(contract, data)
  # Do stuff
end

I think the returned type can be a Faraday::Request. There is an argument for having a Pacto::Request class for encapsulation, but Faraday::Request a generic HTTP interface that supports many HTTP libraries, and having concepts of both Pacto::Request and Pacto::RequestClause has caused confusion in the past. So I support Faraday::* for objects representing real HTTP transactions, and Pacto::* for abstract definitions and rules for those transactions.

Proposed Port Locations

The main location where this needs to be plugged in is RequestClause#execute.

There should be a default Consumer Actor class set at either the Pacto::Configuration or perhaps as part of each "prosecutor" that drives validation (e.g. the rake, rspec, or pacto-server APIs).

Simplify the user journey test that needs a external server running

For every tests (user journey) that needs a external sever running we add a lot of complexity to the code, therefore to the tests: need to have a Gemfile and run bundler install --local first, add a gem excon into the Pacto Gemfile, put the source code for the service into the test steps.
Too much code just for an acceptance test and taking into consideration that will be also part of the documentation all this will lead to confusion for other people that wants to learn Pacto features.

I would like to rewrite those tests and make it more readable and also use the mechanisms that Pacto already have, like the DummyServer class.

Probably we will need to re-introduce again the cucumber tag @needs_server for the tests that interact with an external service.

Start documenting Pacto

As a Pacto user
I want to have the documentation needed
So that I can easy understand how to use all Pacto features

As a Pacto contributor
I want to have the documentation needed
So that I can understand easy Pacto implementation
And contribute without too much hassle

Let's discuss on the better way to document Pacto for users and contributors.

Proposal: Provider Actors

Consumer Actor Adapter

The roadmap for Pacto v0.4.0 will shift towards a more hexagonal or "Ports and Adapters" architecture. One of my planned adapters is Actors - in the sense of "courtroom re-enactments".

The Provider Actor adaptor type will drive the provider responses.

Responsibility

The Provider Actor is responsible for creating responses to Requests which have been generated by the Consumer Actor.

This is less open-ended than the consumer actor. It will always receive a Faraday::Request and is expected to create a Faraday::Response. See the notes on Consumer Actor about why I think we should use Faraday::* for objects that represent actual HTTP transactions rather than abstract templates and rules for the service.

Concrete Implementations

Currently we have two concrete implementations that should be refactored to match the pattern:

  • JSONGeneratorProviderActor - creates a response based on the JSON schema. This is current behavior when Pacto is stubbing.
  • ExampleProviderActor - Creates a response based on an example. I propose this as the new default behavior for Pacto stubbing, taking advantage of the example section of the new Contract schema.
  • LiveProviderActor - Proxies the request to a real service endpoint, either the actual provider or an out of process stub server like Mountebank or Mimic.

Proposed Adapter Interface

# Generates a response for the service
# @param contract [Pacto::Contract] the contract describing the service.
# @params request [Faraday::Request] a request ot respond to
# @return [Faraday::Response]
def react(contract, request)
  # respond
end

Proposed Port Locations

This should take the place of Contract#stub_contract! and Contract#provider_response.

There should be a default Provider Actor set at the top-level (Pacto or Pacto::Configuration) or "prosecutor" level, just like with Consumer Actors.

Clarify the concepts or pre-processor, post-processor and callbacks/hooks.

Apparently the idea of pre-processor, post-processor, hook and callback are a little mixed up on the codebase.
The first two existed before in order to use ERBProcessor and the HashMergeProcessor.
The terms Hook and Callback are used interchangeably, these use the ERBHook that is only a caller to ERBProcessor.
In practice the one that matters/works now is the callback (ERBHook).
From previous conversations I know that we wanted to get rid of HashMergeProcessor since a long time.
I would like to:

  • Remove the code related with pre-processor and post-processor.
  • Remove the HashMergeProcessor
  • Use only one term for the hook/callback.

If we agree on these proposals, I will create the PR with the corresponding changes.

Support 1.8.7

Consider support ruby 1.8.7. There are projects that may want to use Pacto that still supports 1.8.7.

Workaround necessary for live validation

There's an issue where RequestPatterns are not associated with a Contract until they are stubbed. This means you can't do validation against a live service without stubbing and then unstubbing the service. This shouldn't be necessary.

Workaround for live testing:

contracts = Pacto.build_contracts(File.join(Pacto.configuration.contracts_path, host), "https://#{host}")
Pacto.validate!

# Workaround
contracts.stub_all
WebMock.reset! # contracts.unstub_all would be nicer :)
WebMock.allow_net_connect!

Generation: record examples

There should at least be an option for recording samples while generating contracts. Especially since JSON::Generator-based stubbing only supports json-schema draft3 (#10), but generation and example-based stubbing supports draft3 or draft4.

Cucumber aruba steps missing

When I run bundle exec cucumber features/<any_folder> the tests fail, it says that have aruba steps not implemented.
But when I run bundle exec cucumber everything is good.
Anybody has the same issue?

End to end test

We need a test (or tests) that executes the two main features: validate a contract and stub a response based on a contract.
It might be interesting take a look on Aruba to do the task, but that will mean start using cucumber. I would like to keep all the tests suite in RSpec including these end to end test, probably there is tool like Aruba but for RSpec, we have to research.

Rubocop locked to 0.14.0

Rubocop 0.15.0 just came out and lots of things suddenly started showing up as violations. I'm locking it to 0.14.0 for now, because the generator_update needs to get merged.

Proposal: store host in Contract, but make it overridable via ContractList

I'd like contracts to contain site info. Conceptually, I believe the site is part of the fully-qualified description of a service. For example, since Pacto does not store the site, what service is at:

{
  "path": "/user"
  // (I wish you could actually put comments in JSON) ...
}

That's a lot of different than something like https://api.twitter.com/user vs https://api.github.com/user. What makes it even worse is that many APIs have a root service, so you get a path of just /.

We do need the site info (in Addressable::URI site is scheme, user, password, host, and port) in order to deal with services, but we're forcing users to use out-of-band knowledge to get it (like using folder names on the file system as hosts, assuming the scheme, and no user/password/port) in order to load them. It usually looks like this:

hosts = Dir["#{Pacto.configuration.contracts_path}/*"].each do |hostdir|
  puts "Loading #{hostdir}"
  host = File.basename hostdir
  contracts = Pacto.build_contracts(hostdir, "https://#{host}").stub_all
end

I propose that instead we store full sites in the Contract like this:

{
  "site": "https://api.github.com",
  "path": "/user"
}

And support an API like this:

# Stub the same site in the contract, e.g. https://api.github.com
contracts = Pacto.build_contracts(contracts_dir).stub_all

# Or override the site to stub something else
contracts = Pacto.build_contracts(contracts_dir, :site => 'https://beta.api.github.com').stub_all

Another option is to remove the assumption that ContractLists need to hold contracts for the same host, and accept a map for replacements. For example:

# Stub Rackspace London endpoints instead of Dallas/Fort-Worth
contracts = Pacto.build_contracts('rackspace_services', :replace_sites => {
  'https://dfw.servers.api.rackspacecloud.com' => 'https://lon.loadbalancers.api.rackspacecloud.com',
'https://dfw.identity.api.rackspacecloud.com' => 'https://lon.identity.api.rackspacecloud.com',
  'https://dfw.loadbalancers.api.rackspacecloud.com' => 'https://lon.loadbalancers.api.rackspacecloud.com'
  # etc...
}).stub_all

NB: Originally wrote it about "hosts", but changed to "sites" so we wouldn't need separate options for scheme, port, etc.

Final note: we currently support rfc6570 uri templates, and there is an open PR to put rfc6570 support in WebMock, so this should also be a valid site:
https://{region}.servers.api.rackspacecloud.com

Different code coverage reports

We have different numbers for test coverage, not sure if is bad or good, but it's confusing.

Running rake unit the code coverage is around 99%
Running rake integration the code coverage is around 86%
Running rake journeys the code coverage is around 87%

The last is the one that is sent to Corveralls.

At the end which coverage we should take into account?
Should we have different reports for different test levels?

auto-stubbing arrays?

I was trying to generate stubs with a schema that contains an array type. I couldn't get anything working, though I'm not sure if the issue is with my schema, json-schema, or pacto.

Should add a simple array sample.

ERB postprocessing choked on base64 encoded content

I ran a test that fetches the Pacto README from the GitHub API. The response is in JSON, but includes a base64 encoded "content" field. Pacto didn't like this. I think it was a character encoding mixup, but there's also the chance of base64 encoded data that looks like ERB.

This choked on a live response. I think we should only run the ERB processing on stubbed responses. It wouldn't completely fix this issue, but it would avoid a lot of surprises.

I, [2014-02-12T13:37:31.812524 #89323] INFO -- : get https://api.github.com/repos/thoughtworks/pacto/readme
D, [2014-02-12T13:37:31.812612 #89323] DEBUG -- request: Accept: "application/vnd.github.beta+json"
Accept-Encoding: "gzip;q=1.0,deflate;q=0.6,identity;q=0.3"
D, [2014-02-12T13:37:31.982011 #89323] DEBUG -- Pacto: Generating Contract for GET https://api.github.com/repos/thoughtworks/pacto/readme with headers {'Accept'=>'application/vnd.github.beta+json', 'Accept-Encoding'=>'gzip;q=1.0,deflate;q=0.6,identity;q=0.3', 'User-Agent'=>'Ruby'}, #WebMock::Response:0x007fb2bacb0f10
/opt/boxen/rbenv/versions/2.1.0/lib/ruby/2.1.0/erb.rb:850:in eval': (erb):15: Invalid char\x15' in expression (SyntaxError)
(erb):16: Invalid char \x05' in expression (erb):16: Invalid char\x14' in expression
(erb):16: syntax error, unexpected tIDENTIFIER, expecting keyword_do or '{' or '('
�p��^�W...��D�oN������#��;�*f������>[
^
(erb):16: Invalid char \x06' in expression (erb):16: Invalid char\x06' in expression
(erb):17: Invalid char \x1D' in expression (erb):17: Invalid char\x0E' in expression
(erb):17: syntax error, unexpected tCONSTANT, expecting '}'
��
AG8W�ΟD�����|�@4S�Ԅ��T���e�D�PD���...
^
from /opt/boxen/rbenv/versions/2.1.0/lib/ruby/2.1.0/erb.rb:850:in `result'

Pacto.generate! and Pacto.validate!

I'd like to make it so you can do this:

WebMock.allow_net_connect!

Pacto.configure do | config |
  config.contracts_path = 'my_contracts'
end

Pacto.generate!
# ... run your tests against live services and Pacto will generate contracts in  ./my_contracts

Pacto.load_all 'my_contracts', 'http://www.example.com'
Pacto.validate!
# ... run your tests again and Pacto will validate all the live services w/ the new cont

WebMock.disable_net_connect!
Pacto.use :default
# ... run your tests a third time, it's stubbed now

The hooks that make generate and validate work are pretty easy. I've got most of the code on the modes branch. I think this also solves most of #49. I'm sure there is a better way than I was storing the mode (generating, validating, ...) on that branch, though.

I think if we add this we can easily finish the README and release a new version. What do you guys think?

Contract generation not working

The current version 0.3.0 the rake task generate is failing.

Apparently on this commit a8b4f59 the code related with this task was removed. This change leaved the test features/generate/generation.feature empty =/. By the way, the README is still showing this feature as available.

Why did we decided to remove this feature?

Simplify contract's specification

The specification (of a contract) right now is very verbose. For anything non-trivial, the specification is basically non-human readable. We could use the yaml representation of json-schema as that is more compact and then write a simple DSL that represents the request-response bits and keep the structure of the actual response body still in yaml representation of json schema.

Replace WebMock with a server instance

Instead of using WebMock and stub the requests on the same application we could create a server instance that reads the contracts and generates the end point based on the contracts.

I wouldn't like to reinvent the wheel so I did a small research and found these two possible candidates:

Both are stub servers that are configurable through a entry point.

Any thoughts about this?

Record a contract by calling a URL

You should be able to put Pacto in record mode, so it will attempt to generate a draft contract by calling the live service.

This is similar to VCR's record mode, or how
json_schema_generator allows users to generate a schema from a URL that returns json. A simple tool is possible for contracts that use simple GET requests, but for services with more complex authentication or POST bodies it would be better to record by running a test suite.

API documentation generation

I'm currently using https://github.com/zipmark/rspec_api_documentation and mostly happy with it. The documentation is well formatted and I get the bonus of the documentation when I write my rspec acceptance tests.

This works really well for cases where my codebase is the server.

However I am wanting documentation for cases where my codebase is the client. I wrote some contract tests but they don't work with the documentation because the tool assumes I'm only testing my own endpoints and controllers.

I searched https://www.ruby-toolbox.com/categories/documentation_tools and none of them seem to have what I'm looking for. Since pacto is all about contract testing, I was hoping that generating API documentation could be one of the outputs from the contract tests. (Here is an example output: http://rad-example.herokuapp.com/docs/orders/getting_a_list_of_orders)

Proposal: new Contract Schema for 0.4.0

We've already talked about a bunch of new features, and have talked about the contract changes that they would require. I think I have a good enough idea of the changes required that it makes sense to put a proposal here rather than dealing with it separately through each PR.

Ideally the items under breaking changes should still load contracts that use the old format, but will display a deprecation warning.

The additions are non-breaking as long as they have a default value or handler.

Proposed Contract changes for v0.4.0

Breaking changes

  • Rename path to uriTemplate (WebMock 0.18.0 has URI Template support now, and calling out that these aren't paths but templates will make it clearer)

    Before: "path": "/hello_world"
    After: "uriTemplate": "/hello_world{?lang}"

  • Require site (see #84):

    Before: N/A
    After: "site": "https://api.github.com"

  • Multiple status codes (#96)

    Before: "status": 200
    After: "status": [200, 201]

Additions

  • Add request examples (#118). These can be used for simulating consumers:

    Before: N/A
    After: See below

Example section

The examples section should contain named examples. Each example may include:

  • A request, containing headers and/or body
  • A response, containing a status, headers, and/or body

This section will be used both for simulating consumers (based on the request examples) and stubbing providers (based on response examples).

Meta-validating of a contract should include confirming that all examples match the contract rules. So, for example, if a contract requires a request body, then an example request that lacks a body will cause the contract to fail meta-validation.

{
"examples": {
  "en_US": {
    "request": {
      "headers": {
        "Accept-Language": "en-US"
      }
    },
    "response": {
      "body": {
        "Hello, world!"
      }
    }
  },
  "pt_BR": {
    "request": {
      "headers": {
        "Accept-Language": "pt-BR"
      }
    },
    "response": {
      "body": {
        "Olá, Mundo!"
      }
    }
  }
}

Rake task for contract meta-validation

Add a rake task to make sure the contracts are a valid format (i.e. the have the right request/response structure, and the response body is a json-schema)

Terminology & structure

I want to get the codebase a bit more organized and consistent. Several of these ideas have been discussed before and we seemed to have some consensus, but they didn't sneak into PRs yet.

My proposal:

  • rename build_contract/build_contracts to load_contract/load_contracts
  • rename validate_consumer to validate_response and validate_provider to simulate_request (I've realized that validate_consumer validates stubbed responses while the validate_provider simulates a consumer making valid requests)
  • deprecated Pacto::Core::Hook, Pacto::Hooks::ERBHook, and Pacto::ERBProcessor for middleware.
  • rename Pacto::Stubs::BuiltIn to Pacto::Stubs::WebMockAdapter
  • rename stub_all to simulate_consumers and stub_all to stub_providers

I'd also like to outline a structure, similar to http://fog.io/about/structure.html, where we outline a few central concepts that deserve their own folders (mostly around extension points - validators, stubs, generators, ...). I think "core" should be used for essential but non-extensible parts of Pacto... i.e. things that load and control the flow of the extensible pieces.

Support multi_json

We should support multi_json. From their README:

Instead of choosing a single JSON coder and forcing users of your library to be stuck with it, you can use MultiJSON instead, which will simply choose the fastest available JSON coder.

We can either put multi_json as an explicit dependency, or use json-schema's approach (attempt multi_json, fallback to json).

Pacto templating

We believe Pacto needs a templating system. This is useful both because:

  1. Some requests manipulation may need to be done to playback requests. For example, an authentication token or nonce is not replayable, so there needs to be a mechanism to insert this data during replay.
  2. While stubbing, it may be useful to map request data to response data. For example, if you have a "Create" service, then you might want to make the name in the request match the name in the response.
  3. Similarly, while stubbing it is sometimes desirable to give the stubs a more dynamic feel by using random data. This idea is implemented by the Faker gem (random names, text, etc.) and by Fog's Fog::Mock class (random IP addresses, base64 encoding data, etc).

Extract DEBUG_PACTO debugging logic into an actual logger

Instead of the current

if ENV['DEBUG_PACTO']
  #do something
end

we could put the logging behavior into a class. Then we can either pass an actual logger or a noop logger to anyone wanting to log debug information, no ugly ifs needed.

Add static analysis for the source code

As a Pacto developer
I want to automatic execute static analysis on the code I write
In order to keep the codebase consistent among all Pacto collaborators

It was previously discussed that we need some style/lint/static analysis for Pacto, I am doing some research on the tools available and a good candidate is this project https://github.com/bbatsov/rubocop.

Running this checker with the default configuration on the Pacto current source code raises more than 500 "offenses" =/, that are basically 500 static analysis rules broken.

To start we can disable most of these rules and then gradually enable and fix each "offense". Of course we need to customize rubocop to fit in some code style that we (everybody) agrees.

Meta-validate: validate all examples

#118 adds examples, but does not include this requirement from #113:

Meta-validating of a contract should include confirming that all examples match the contract rules. So, for example, if a contract requires a request body, then an example request that lacks a body will cause the contract to fail meta-validation.

Rake task pacto:validate is not working

The rake task pacto:validate is not working properly, Apparently is running code that does not exist anymore: Pacto.build_from_file.
Even the fix is relatively easy to do, I would like to have an automated test that will prevent this in the future.

All contributors: We must cover critical cases with user journey tests, otherwise only our users will find these severe issues.

Dealing with Location response header

How do you guys think we should deal with the Location header? RESTful create services often return a Location Header.

It may be a required part of the response, and so it does make sense to include it in the Response Headers in the contract (and is currently included when by Contract generation). However, the value changes so we can't validate an exact value. Since Location should be a URL, should we apply the same matching rules as for the contract itself?

Perhaps we should support URI templates in Location, like account & server_id in this example?

"Location": "https://dfw.servers.api.rackspacecloud.com/v2/:account/servers/:server_id"

We could also support regex matching for Headers. It has the advantage of solving similar issues with non-URI headers, but it's uglier, especially embedded within json. the equivalent would be something like:

"Location": "/^https\:\/\/dfw\.servers\.api\.rackspacecloud\.com\/v2\/\w+\/servers\/[\w-]+$/"

Proposed rspec matchers

The Pacto validation mode is pretty weak right now, because it just prints out validation messages. Having it throw exceptions may be even worse in some cases.

What would really help is being able to set expectations in my tests. I think Pacto should provide some rspec matchers. Not included by default, but easily added by requiring pacto/rspec.

How about?

Basic:

expect(Pacto).to have_validated_all_requests
expect(Pacto).to have_matched_all_requests_to_contracts

Advanced:

The idea here is to borrow mostly from WebMock (which borrowed from fakeweb-matcher):

expect(Pacto).to have_validated(:get, "www.example.com").
  with(:body => "abc", :headers => {'Content-Length' => 3}).twice

expect(Pacto).to_not have_validated(:get, "www.something.com")

expect(Pacto).to have_validated(:post, "www.example.com").with { |req| req.body == "abc" }

expect(Pacto).to have_validated(:get, "www.example.com").with(:query => {"a" => ["b", "c"]})

expect(Pacto).to have_validated(:get, "www.example.com").
  with(:query => hash_including({"a" => ["b", "c"]}))

expect(Pacto).to have_validated(:get, "www.example.com").
  with(:body => {"a" => ["b", "c"]}, :headers => {'Content-Type' => 'application/json'})

but add an extra piece to the fluent api: against_contract:

expect(Pacto).to have_validated(:get, "www.something.com").against_contract('SomethingService')

I've also come across json_spec. I'm trying to figure out how I can mix that in as well.

Malpractice: discovery and checks

The HTTP 1.1 specification says "Field names are case-insensitive" about HTTP headers. Yet I've run across several implementations that are case sensitive. I've thought for a while about having Pacto help detect these sorts of problems - improper use of HTTP.

I think we can add two different categories of "malpractice" checks:

  • Malpractice judgments: Pacto detects and reports issues directly. This is a linting approach, similar to Rack::Lint or RedBot, though those only detect provider response issue, and Pacto should also be able to lint consumer requests.
  • Malpractice discovery: Pacto cannot directly discover things like a consumer doing case-sensitive matching on HTTP header field names, but it could help detect it by altering headers names to make sure the consumer (or producer) isn't making a flawed assumption. This is more of a heckling approach to make sure consumers/produces aren't tightly coupled and/or are robust enough to handle transient errors, similar to heckle itself, Netflix's simian army or Michael Nygard's evil test ideas.

Pacto doesn't need to implement a lot of these checks, but it should provide an extension point for them so users and/or other gems can provide custom malpractice judgment or discovery.

P.s. Judgment has some odd spelling so it might need an alias.

Actors: Props and Mustaches

Actors w/ Props & Mustaches

I want to be able to hire Actors (Consumer or Provider Actors) with mustaches and props. Cause it's funny :D.

Mustaches

I want to be able to use templates with actors, at least the ones that use the examples section in the new Contract Schema. We have used ERB in the past, but I did a quick test with Mustache and I think it could be better.

Props

Props are extras that can be used in the actor templates. Basically, it's the template variable binding. We already bind values that are set on the contract or extracted using URI templates (rfc 6570). You could think of those as the "standard props".

An actor could be given "extra props" to make additional data to the template. This can be simple variables, but Mustache also supports lambas, so it could be a function. Useful if you want a sample that returns random data.

Possible API & Example

Suppose you want to use Faker::Internet in your actors. The actor could be constructed to have a mustache and Faker::Internet props:

actor = Pacto::ExampleProviderActor.new(:mustached => true, :props => [Faker::Internet])
# That should enable mustache processing, and add Faker::Internet to the binding used to render the mustache template.

So, if it processes a template like this:

{
  "server": {
    "name": "{{ domain_name }}",
    "public_ip_address": "{{ ip_v4_address }}",
    "private_ip_address": "{{ ip_v4_address }}",
  }
}

Should produce something like:

{
  "server": {
    "name": "effertz.info",
    "public_ip_address": "24.29.18.175",
    "private_ip_address": "10.10.25.96",
  }
}

Advantages over ERB

ERB could still be possible via other plugins, but I think Mustache may be a better fit to integrate with the Actors by default.

I like mustache because:

  • It is polyglot friendly. That's a general goal for Pacto, since I use it for non-Ruby projects, even if I don't have any plans to port Pacto or Pacto contracts to other languages in the near future. It may also help port templates to/from other projects like Mountebank or Guzzle Service Descriptions (where you could use PHP's Faker).
  • The modular "prop" system and the logic-less nature of Mustache will hopefully encourage more readability and reuse of template functions across services and projects. So you shouldn't see things like <%= %w(ubuntu windows centos).sample %>. You'd have {{ random_os }} instead.
  • Mustache has basic support sections based on a lists, processing a section only if a list is empty, processing a section only if the not empty, or processing a section for each item in the list. This limited support make it easy to create templates that model conventional RESTful collections without overcomplicated or hard to read templates (once you get used to mustache). Example below.

Disadvantages

The {{ }} syntax might not be great for embedding in JSON, but this can be customized. You could, for example, use <% %> as the delimiter.

I'm pretty close to being able to model a collection cleanly, but commas), are a bit tricky. This is almost right:

    {{^servers}}
      204 No Content - No servers exist
    {{/servers}}
    [
      {{#servers}}
      {
          "name": "{{ name }}",
          "ram": "{{ ram }}"
      }
      {{/servers}}
    ]

Simplifying our common use case

Imagine the most simple Pacto use case:

  • I have a service A that depends on service B. I want to use Pacto to run the tests on service A against the contracts relating to service B and I want to validate those contracts against service B, but in two different points in my timeline.

I think the current state of the gem is not servicing this simple use case

We have some very advanced functionality, like modes, tags, generating contracts automatically, auto validation and whatnot. These things are too complicated when you just want to start toying with the gem, to see if it will help your project or not.

They are all valid functionality, but I think they should be build on top the simplest use case.

I have some proposals for a syntax:

Focusing on individual contracts

# for stubbing
Pacto.build_contract("path/to/file", host).stub

# for validating
Pacto.build_contract("path/to/file", host).validate

Focusing on collection of contracts

contracts = Pacto.build_contracts("path/to/file/or/dir", host)

# for stubbing
contracts.stub_all

# for validating
contracts.validate_all

I prefer the second one. What do you guys think?

Support multiple status codes

This has been in the README for a while, under constraints:

Pacto cannot currently specify multiple acceptable status codes (e.g. 200 or 201)

I think it'll get more attention as an issue.

Issue with colorization

After the switch to colored, I'm having an issue using the rake tasks in pacto-demo.

If I use the task as is, I get:

Generating contracts from partial contracts in requests and recording to contracts

rake aborted!
undefined method green' for "Successfully generated all contracts":String /Users/Thoughtworker/repos/opensource/pacto/lib/pacto/rake_task.rb:114:ingenerate_contracts'
/Users/Thoughtworker/repos/opensource/pacto/lib/pacto/rake_task.rb:40:in `block in generate_task'

I can work around this by adding

require 'colored'

to the top of my Rakefile.

However, if I try to put that require in Pacto, then I get:

rake aborted!
wrong number of arguments (1 for 0)
/opt/boxen/rbenv/versions/2.0.0-p247/lib/ruby/gems/2.0.0/gems/rainbow-1.1.4/lib/ansi_color.rb:47:in to_i' /opt/boxen/rbenv/versions/2.0.0-p247/lib/ruby/gems/2.0.0/gems/rainbow-1.1.4/lib/ansi_color.rb:47:inrgb_from_html'
/opt/boxen/rbenv/versions/2.0.0-p247/lib/ruby/gems/2.0.0/gems/rainbow-1.1.4/lib/ansi_color.rb:41:in code_from_html' /opt/boxen/rbenv/versions/2.0.0-p247/lib/ruby/gems/2.0.0/gems/rainbow-1.1.4/lib/ansi_color.rb:26:incode'
/opt/boxen/rbenv/versions/2.0.0-p247/lib/ruby/gems/2.0.0/gems/rainbow-1.1.4/lib/rainbow.rb:33:in foreground' /opt/boxen/rbenv/versions/2.0.0-p247/lib/ruby/gems/2.0.0/gems/colored-1.2/lib/colored.rb:69:incolorize'
/opt/boxen/rbenv/versions/2.0.0-p247/lib/ruby/gems/2.0.0/gems/colored-1.2/lib/colored.rb:38:in block (2 levels) in <module:Colored>' /Users/Thoughtworker/repos/opensource/pacto/lib/pacto/rake_task.rb:114:ingenerate_contracts'
/Users/Thoughtworker/repos/opensource/pacto/lib/pacto/rake_task.rb:40:in `block in generate_task'

Handling 100-Continue

Sometimes a client requests the server to send back HTTP 100-Continue before continuing with a post request. The HTTP spec states:

The purpose of the 100 (Continue) status (see section 10.1.1) is to allow a client that is sending a request message with a request body to determine if the origin server is willing to accept the request (based on the request headers) before the client sends the request body. In some cases, it might either be inappropriate or highly inefficient for the client to send the body if the server will reject the message without looking at the body.

This causes issues with Pacto, which will try to validate the Continue response rather than the final response. So it raises some questions... should we ignore the the first request/response (returning 100-Continue) and only validate the final response? Or are there still some partial validations that should be done?

.NET does this by default for all POST requests. Other HTTP libraries may do this for requests with large bodies. As a wokraround, the .NET behavior can be disabled with:

System.Net.ServicePointManager.Expect100Continue = false;

Variables in schemas?

I had a thought... so I captured it. It's not something I plan to implement any time soon, but an idea I wanted to record for future discussion.

I remember talking with @JesusMercado or @marcosccm about templating, and how I only planned it for the "default" values in json-schema. For example

{
  "server_type": {
    "type": "string",
    "required": true,
    "default": "<%= %w(osx windows linux).sample) %>",
  }
}

I thought of some other scenarios where you might want variables in the schema. I'm not sure this is a completely necessary feature, or even the best way to support this - but it's got me thinking about the challenges and advantages.

Pagination limits

It's pretty common to have a "limit" or similar request parameter that defines the maximum expected size for a hash or array in the response.

So given a request:

GET /api/things?limit=5 HTTP/1.1

We might want a contract that has a response schema like:

{
  "things": {
    "type": "array",
    "required": true,
    "maxItems": <%= params['limit'] %>
  }
}

That way the actual use of the limit is expressed in the contract, and Pacto dynamically adjusts expectations as you send 1, 2, 5 or 100.

Searching

Another possible use is with searching. Suppose you send:

GET /api/things?q=max HTTP/1.1

You might want a schema like:

{
  "things": {
    "type": "array",
    "required": true,
    "items" {
      "type": "object",
      "properties": {
        "name": {
          "type": "string",
          "pattern": "<%= params['q'] %>"
        }
      }
    }
  }
}

Content negotiation

Or, there's possible examples with common request headers. You can probably do some powerful stuff with this, but it gets tricky pretty quick:

GET /api/things HTTP/1.1
Accept-Language: pt-Br
Accept-Language: en-US
{
  "$schema": "http://json-schema.org/draft-03/schema#",
  "type": "object",
  "properties": {
    "essays": {
      "type": "array",
      "required": true,
      "items": {
        "type": "object",
        "properties": {
          "name": {
            "type": "string"
          },
          "language": {
            "type": "string",
            "required": true,
            "enum": <%= headers['Accept-Language'] %>
          }
        }
      }
    }
  }
}

Summary

The tricky part here is that the Contract won't even load, because it can't parse the templates. It works okay when the template is inside quotes, but that doesn't work for json-schema keywords that expect a non-string value (e.g. integer for maxItems). Not only is it invalid json-schema, but it's not even valid JSON until we can process the template, which isn't really possible until we process an HTTP interaction.

I don't think we should support any of these scenarios right now, but:

  • It should be possible to write custom validation rules in Ruby that achieve similar results
  • If these scenarios are common enough, we might want to look into how we could parse the schemas above, or other ways to define similar behavior in the schemas.

Pacto::Request#method conflicts with Object#method

The Pacto::Request has a getter named "method", which can cause confusion with Object#method. In some situations, you may get this error:

ArgumentError:
       wrong number of arguments (0 for 1)
     # ./lib/pacto/generator.rb:44:in `method'

I'm planning to rename it http_method.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.