Describing our APIs using OpenAPI is great, but if we want the description to remain complete and accurate, we need to test it! The good news is that if youโre using the api test helper, you get that testing for free.
Take this test, for example:
data = api :post, "/repositories/#{@repo.id}/check-runs", {},
input: {
name: "coverage-stuff",
head_sha: @sha,
status: "in_progress",
started_at: Time.now.utc,
}
assert_equal 200, last_response.status, last_response.body
The api method makes a call to our API and returns the data returned by the resource. By default every api test helper call goes through an OpenAPI validation middleware. A router takes the path you provide, matches the operation, and uses that operation to validate everything about that request (query parameters, response schemas, and more).
If a mismatch is detected by the middleware, it will raise, making your test fail with an helpful error. Imagine, for example, you added a new property called my_new_property to a serializer but forgot to add it to the OpenAPI description. If you have a test returning that new property, it would fail with an error like the following:
2021-03-09 00:00:00 - OpenApi::Validation::Error
Property my_new_property
is not described by the response 200
for operation my/operation
. This either means you forgot to describe that property in the operation description located at app/description/operations/my/operation.yml
or that the test is wrong.
More info: https://thehub.github.com/engineering/development-and-ops/public-apis/openapi/contract-testing
There are plenty of validations like this one running when you use the api test method. Thankfully, each error should have enough information on how to fix it. If it doesnโt, open an issue or let us know in #ecosystem-api. Over time this means the OpenAPI description always reflects what happens at runtime, which means our documentation, SDKs, and any tooling that depends on OpenAPI will remain accurate!
##Legacy Behavior
Sometimes tests will fail due to a behavior thatโs not described in OpenAPI but used in the test. Generally the failure indicates that we should document it in OpenAPI, but in some cases weโd rather not expose it externally. For example, some obscure and deprecated use case may fail a contract test, but weโd rather not expose it to our documentation and SDKs through the OpenAPI description.
In these cases we still want to describe that behavior in OpenAPI, but make sure it is not exposed anywhere else than in our own internal description. To do so, you can use the x-internal-only OpenAPI extension:
parameters:
- name: legacy_param
in: query
schema:
- type: string
x-internal-only: true
This will satisfy the contract tests, but wonโt leak the existence of this parameter outside of github/github.
Skipping Contract Testing
On certain occasions you may find it necessary to skip OpenAPI contract testing
Skipping a specific test
Itโs possible you may want to test an endpoint and disable the OpenAPI validation middleware. Sometimes a test is sending a wrong value, but you want to test the behavior of the endpoint, disregarding the extra OpenAPI contract testing layer. To do so, you can use skip_openapi_validation when calling the api helper:
data = api :get, "/users",
my_param: "not_supported",
skip_openapi_validation: "Skipping because we want to test the 422 returned when passing an unsupported param"
Skipping entire endpoints/resources
Additionally to the โper-testโ OpenAPI skipping, Itโs also possible to opt out an entire path from OpenAPI contract testing. Although we strive to have a 100% accurate description, sometimes internal or deprecated endpoints need to be ignored.
Add a regex for the paths you want to exclude from this OpenAPI validation in our IgnoreList.