Code Monkey home page Code Monkey logo

openapi-modern's Introduction

NAME

OpenAPI::Modern - Validate HTTP requests and responses against an OpenAPI v3.1 document

VERSION

version 0.067

SYNOPSIS

my $openapi = OpenAPI::Modern->new(
  openapi_uri => '/api',
  openapi_schema => YAML::PP->new(boolean => 'JSON::PP')->load_string(<<'YAML'));
openapi: 3.1.0
info:
  title: Test API
  version: 1.2.3
paths:
  /foo/{foo_id}:
    parameters:
    - name: foo_id
      in: path
      required: true
      schema:
        pattern: ^[a-z]+$
    post:
      operationId: my_foo_request
      parameters:
      - name: My-Request-Header
        in: header
        required: true
        schema:
          pattern: ^[0-9]+$
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              properties:
                hello:
                  type: string
                  pattern: ^[0-9]+$
      responses:
        200:
          description: success
          headers:
            My-Response-Header:
              required: true
              schema:
                pattern: ^[0-9]+$
          content:
            application/json:
              schema:
                type: object
                required: [ status ]
                properties:
                  status:
                    const: ok
YAML

say 'request:';
my $request = POST '/foo/bar',
  'My-Request-Header' => '123', 'Content-Type' => 'application/json', Host => 'example.com',
  Content => '{"hello": 123}';
my $results = $openapi->validate_request($request);
say $results;
say ''; # newline
say JSON::MaybeXS->new(convert_blessed => 1, canonical => 1, pretty => 1, indent_length => 2)->encode($results);

say 'response:';
my $response = Mojo::Message::Response->new(code => 200, message => 'OK');
$response->headers->content_type('application/json');
$response->headers->header('My-Response-Header', '123');
$response->body('{"status": "ok"}');
$results = $openapi->validate_response($response, { request => $request });
say $results;
say ''; # newline
say JSON::MaybeXS->new(convert_blessed => 1, canonical => 1, pretty => 1, indent_length => 2)->encode($results);

prints:

request:
'/request/body/hello': got integer, not string
'/request/body': not all properties are valid

{
  "errors" : [
    {
      "absoluteKeywordLocation" : "https://example.com/api#/paths/~1foo~1%7Bfoo_id%7D/post/requestBody/content/application~1json/schema/properties/hello/type",
      "error" : "got integer, not string",
      "instanceLocation" : "/request/body/hello",
      "keywordLocation" : "/paths/~1foo~1{foo_id}/post/requestBody/content/application~1json/schema/properties/hello/type"
    },
    {
      "absoluteKeywordLocation" : "https://example.com/api#/paths/~1foo~1%7Bfoo_id%7D/post/requestBody/content/application~1json/schema/properties",
      "error" : "not all properties are valid",
      "instanceLocation" : "/request/body",
      "keywordLocation" : "/paths/~1foo~1{foo_id}/post/requestBody/content/application~1json/schema/properties"
    }
  ],
  "valid" : false
}

response:
valid

{
  "valid" : true
}

DESCRIPTION

This module provides various tools for working with an OpenAPI Specification v3.1 document within your application. The JSON Schema evaluator is fully specification-compliant; the OpenAPI evaluator aims to be but some features are not yet available. My belief is that missing features are better than features that seem to work but actually cut corners for simplicity.

CONSTRUCTOR ARGUMENTS

If construction of the object is not successful, for example the document has a syntax error, the call to new() will throw an exception. Be careful about examining this exception, for it might be a JSON::Schema::Modern::Result object, which has a boolean overload of false when it contains errors! But you never do if ($@) { ... }, right?

openapi_uri

The URI that identifies the OpenAPI document. Ignored if "openapi_document" is provided.

If it is not absolute, it is resolved at runtime against the request's Host header (when available) and scheme (e.g. https).

openapi_schema

The data structure describing the OpenAPI v3.1 document (as specified at https://spec.openapis.org/oas/v3.1.0). Ignored if "openapi_document" is provided.

openapi_document

The JSON::Schema::Modern::Document::OpenAPI document that holds the OpenAPI information to be used for validation. If it is not provided to the constructor, then both "openapi_uri" and "openapi_schema" MUST be provided, and "evaluator" will also be used if provided.

evaluator

The JSON::Schema::Modern object to use for all URI resolution and JSON Schema evaluation. Ignored if "openapi_document" is provided. Optional.

ACCESSORS/METHODS

openapi_uri

The URI that identifies the OpenAPI document.

openapi_schema

The data structure describing the OpenAPI document. See "https://spec.openapis.org/oas/v3.1.0" in the specification.

openapi_document

The JSON::Schema::Modern::Document::OpenAPI document that holds the OpenAPI information to be used for validation.

document_get

my $parameter_data = $openapi->document_get('/paths/~1foo~1{foo_id}/get/parameters/0');

Fetches the subschema at the provided JSON pointer. Proxies to "get" in JSON::Schema::Modern::Document::OpenAPI. This is not recursive (does not follow $ref chains) -- for that, use $openapi->recursive_get(Mojo::URL->new->fragment($json_pointer)), see "recursive_get".

evaluator

The JSON::Schema::Modern object to use for all URI resolution and JSON Schema evaluation.

validate_request

$result = $openapi->validate_request(
  $request,
  # optional second argument can contain any combination of:
  my $options = {
    path_template => '/foo/{arg1}/bar/{arg2}',
    operation_id => 'my_operation_id',
    path_captures => { arg1 => 1, arg2 => 2 },
    method => 'get',
  },
);

Validates an HTTP::Request, Plack::Request, Catalyst::Request or Mojo::Message::Request object against the corresponding OpenAPI v3.1 document, returning a JSON::Schema::Modern::Result object.

The second argument is an optional hashref that contains extra information about the request, corresponding to the values expected by "find_path" below. It is populated with some information about the request: save it and pass it to a later "validate_response" (corresponding to a response for this request) to improve performance.

validate_response

$result = $openapi->validate_response(
  $response,
  {
    path_template => '/foo/{arg1}/bar/{arg2}',
    request => $request,
  },
);

Validates an HTTP::Response, Plack::Response, Catalyst::Response or Mojo::Message::Response object against the corresponding OpenAPI v3.1 document, returning a JSON::Schema::Modern::Result object.

The second argument is an optional hashref that contains extra information about the request corresponding to the response, as in "find_path".

request is also accepted as a key in the hashref, representing the original request object that corresponds to this response (as not all HTTP libraries link to the request in the response object).

find_path

$result = $self->find_path($options);

Uses information in the request to determine the relevant parts of the OpenAPI specification. request should be provided if available, but additional data can be used instead (which is populated by earlier "validate_request" or "find_path" calls to the same request).

The single argument is a hashref that contains information about the request. Possible values include:

  • request: the object representing the HTTP request. Should be provided when available.

  • path_template: a string representing the request URI, with placeholders in braces (e.g. /pets/{petId}); see https://spec.openapis.org/oas/v3.1.0#paths-object.

  • operation_id: a string corresponding to the operationId at a particular path-template and HTTP location under /paths

  • path_captures: a hashref mapping placeholders in the path to their actual values in the request URI

  • method: the HTTP method used by the request (used case-insensitively)

All of these values are optional (unless request is omitted), and will be derived from the request URI as needed (albeit less efficiently than if they were provided). All passed-in values MUST be consistent with each other and the request URI.

When successful, the options hash will be populated with keys path_template, path_captures, method, and operation_id, and the return value is true. When not successful, the options hash will be populated with key errors, an arrayref containing a JSON::Schema::Modern::Error object, and the return value is false. Other values may also be populated if they can be successfully calculated.

In addition, these values are populated in the options hash (when available):

You can find the associated operation object by using either operation_uri, or by calling $openapi->openapi_document->get_operationId_path($operation_id) (see "get_operationId_path" in JSON::Schema::Modern::Document::OpenAPI) (note that the latter will be removed in a subsequent release, in order to support operations existing in other documents).

Note that the /servers section of the OpenAPI document is not used for path matching at this time, for either scheme and host matching nor path prefixes. For now, if you use a path prefix in servers entries you will need to add this to the path templates under `/paths`.

recursive_get

Given a uri or uri-reference, get the definition at that location, following any $refs along the way. Include the expected definition type (one of schema, response, parameter, example, request-body, header, security-scheme, link, callbacks, or path-item) for validation of the entire reference chain.

Returns the data in scalar context, or a tuple of the data and the canonical URI of the referenced location in list context.

If the provided location is relative, the main openapi document is used for the base URI. If you have a local json pointer you want to resolve, you can turn it into a uri-reference by prepending #.

my $schema = $openapi->recursive_get('#/components/parameters/Content-Encoding', 'parameter');

# starts with a JSON::Schema::Modern object (TODO)
my $schema = $js->recursive_get('https:///openapi_doc.yaml#/components/schemas/my_object')
my $schema = $js->recursive_get('https://localhost:1234/my_spec#/$defs/my_object')

canonical_uri

An accessor that delegates to "canonical_uri" in JSON::Schema::Modern::Document.

schema

An accessor that delegates to "schema" in JSON::Schema::Modern::Document.

get_media_type

An accessor that delegates to "get_media_type" in JSON::Schema::Modern.

add_media_type

A setter that delegates to "add_media_type" in JSON::Schema::Modern.

CACHING

Very large OpenAPI documents may take a noticeable time to be loaded and parsed. You can reduce the impact to your preforking application by loading all necessary documents at startup, and impact can be further reduced by saving objects to cache and then reloading them (perhaps by using a timestamp or checksum to determine if a fresh reload is needed).

sub get_openapi (...) {
  my $serialized_file = Path::Tiny::path($serialized_filename);
  my $openapi_file = Path::Tiny::path($openapi_filename);
  my $openapi;
  if ($serialized_file->stat->mtime < $openapi_file->stat->mtime)) {
    $openapi = OpenAPI::Modern->new(
      openapi_uri => '/api',
      openapi_schema => decode_json($openapi_file->slurp_raw), # your openapi document
    );
    $openapi->evaluator->add_schema(decode_json(...));  # any other needed schemas
    my $frozen = Sereal::Encoder->new({ freeze_callbacks => 1 })->encode($openapi);
    $serialized_file->spew_raw($frozen);
  }
  else {
    my $frozen = $serialized_file->slurp_raw;
    $openapi = Sereal::Decoder->new->decode($frozen);
  }

  # add custom format validations, media types and encodings here
  $openapi->evaluator->add_media_type(...);

  return $openapi;
}

See also "CACHING" in JSON::Schema::Modern.

ON THE USE OF JSON SCHEMAS

Embedded JSON Schemas, through the use of the schema keyword, are fully draft2020-12-compliant, as per the spec, and implemented with JSON::Schema::Modern. Unless overridden with the use of the jsonSchemaDialect keyword, their metaschema is https://spec.openapis.org/oas/3.1/dialect/base, which allows for use of the OpenAPI-specific keywords (discriminator, xml, externalDocs, and example), as defined in "https://spec.openapis.org/oas/v3.1.0#schema-object" in the specification. Format validation is turned on, and the use of content* keywords is off (see "validate_content_schemas" in JSON::Schema::Modern).

References (with the $ref) keyword may reference any position within the entire OpenAPI document; as such, json pointers are relative to the root of the document, not the root of the subschema itself. References to other documents are also permitted, provided those documents have been loaded into the evaluator in advance (see "add_schema" in JSON::Schema::Modern).

Values are generally treated as strings for the purpose of schema evaluation. However, if the top level of the schema contains "type": "number" or "type": "integer", then the value will be (attempted to be) coerced into a number before being passed to the JSON Schema evaluator. Type coercion will not be done if the type keyword is omitted. This lets you use numeric keywords such as maximum and multipleOf in your schemas. It also resolves inconsistencies that can arise when request and response objects are created manually in a test environment (as opposed to being parsed from incoming network traffic) and can therefore inadvertently contain perlish numbers rather than strings.

LIMITATIONS

All message validation is done using Mojolicious objects (Mojo::Message::Request and Mojo::Message::Response). If messages of other types are passed, conversion is done on a best-effort basis, but since different implementations have different levels of adherence to the RFC specs, some validation errors may occur e.g. if a certain required header is missing on the original. For best results in validating real messages from the network, parse them directly into Mojolicious messages (see "parse" in Mojo::Message).

Only certain permutations of OpenAPI documents are supported at this time:

  • for path parameters, only style: simple and explode: false is supported

  • for query parameters, only style: form and explode: true is supported, only the first value of each parameter name is considered, and allowEmptyValue and allowReserved are not checked

  • cookie parameters are not checked at all yet

  • application/x-www-form-urlencoded and multipart/* messages are not yet supported

SEE ALSO

SUPPORT

Bugs may be submitted through https://github.com/karenetheridge/OpenAPI-Modern/issues.

I am also usually active on irc, as 'ether' at irc.perl.org and irc.libera.chat.

You can also find me on the JSON Schema Slack server and OpenAPI Slack server, which are also great resources for finding help.

AUTHOR

Karen Etheridge <[email protected]>

COPYRIGHT AND LICENCE

This software is copyright (c) 2021 by Karen Etheridge.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.

Some schema files have their own licence, in share/oas/LICENSE.

openapi-modern's People

Contributors

karenetheridge avatar

Stargazers

 avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

openapi-modern's Issues

include operation_path in validate_request/response $options hash

We provide the path_template and method in the options hash after validating, but it would be helpful to provide the json pointer directly to the operation as operation_path, allowing the user to $openapi->get($operation_path) to access the operation schema directly.

Use request's host in error result's absoluteKeywordLocation

If the openapi document is constructed with a relative URI as its identifier, there is an opportunity to resolve this URI at runtime when generating errors so the stated location of the document uses the same host as that used in the original request, so the caller has an absolute URI as a reference for the document used to generate the error.

  • JSM::evaluate() needs to accept a base_uri at runtime, as a config_override (3rd argument)
  • capture $request->uri->host when $request and $response initializes its $state, and pass this value to JSM::evaluate

'servers' information in the document should be taken into account, e.g.:

  • servers: specifies a base uri of "/v1"
  • path-item: specifies an operation with path "/foo/bar"
  • openapi document is configured with a canonical uri of "/api"

therefore an incoming request with uri "http://example.com/v1/foo/bar" is routed to this operation, and errors will use absoluteKeywordLocation: "http://example.com/v1/api#/path/to/error"

use of readOnly, writeOnly violates the spec (?) and usage with PUT etc

I have implemented readOnly such that a property with that annotation cannot be used in requests (and similarly for writeOnly in responses). However, this prevents the labelling of a property as readOnly if it is to be used in a PUT, where (by REST convention) all properties are provided and the data overwrites the original idempotently -- here, the meaning is "this property may be present, but it can't change from its original value". As such it should only be treated as an annotation, and the application should figure out what to do with it, rather than generating a validation error itself.

Therefore:

  • remove the checking of readOnly, writeOnly in validate_request and validate_response
  • provide guidance (in cookbook documentation?) for getting these annotations out of the result and determining if the referenced properties are changing values.
  • provide guidance for how to indicate that a particular propertly should not be present at all:
allOf:
  not:
    required: [id]
  not:
    required: [created]
  not:
    required: [updated]

or:

properties:
  id: false
  created: false
  updated: false

or:

$ref: '#/components/schemas/model.foo'
additionalProperties: false
properties:
  foo: true # property in model.foo
  bar: true # property in model.foo
  blah: true # property in model.foo

Tests have started to fail (with JSON::Schema::Modern 0.561?)

Sample fail report: http://www.cpantesters.org/cpan/report/a98e9330-93d6-11ed-810b-43cad168a89d

Statistics suggest that JSON::Schema::Modern 0.561 is a strong candidate for a blame:

****************************************************************
Regression 'mod:JSON::Schema::Modern'
****************************************************************
Name                   Theta          StdErr     T-stat
[0='const']           1.0000          0.0000    32264009803194304.00
[1='eq_0.558']        0.0000          0.0000       5.40
[2='eq_0.559']        0.0000          0.0000       0.62
[3='eq_0.560']        0.0000          0.0000       1.88
[4='eq_0.561']       -1.0000          0.0000    -5533232032271860.00

R^2= 1.000, N= 154, K= 5
****************************************************************

prototype feature: "no readonly values in requests, no writeonly values in responses"

It is convenient to use a common schema definition to describe a particular model, for both GET and POST (fetch and create) operations. However, some properties are readOnly, such as primary keys and timestamps, and should not be included in POST bodies. If we decorate these properties with readOnly: true, we can check for annotations resulting from these properties being present in the payload and issue an error if they are present.

This could be enabled with x-no-read-only: true somewhere in the openapi operation spec or the schema itself.

If in the schema itself, that keyword itself would result in an annotation in the validation result. Otherwise, it would simply be another keyword in the operation data. After calling $js->evaluate and getting a true result, we can check for the resulting annotations and create an appropriate error.

(similarly we can have a "no write-only values" check for properties returned in a GET operation.)

chunked encoding

...breaks the usual rules on trusting Content-Length. handle this.

speed up schema traversal

  • skip boolean schemas, or hashref schemas with no keys
  • instead of calling Path::Tiny::subsumes, inline the logic here (we can skip object inflation, checking is_absolute, is_rootdir etc)
  • unshift instead of push to -@real_json_schemas - the next entry may subsume us so check this entry first. (i.e. check real paths in reverse order.)

strict/lax options

Some validation checks are more strict than they perhaps should be, e.g.

Add a strict or lax option to twiddle this behaviour.

optimize 'traverse' phase

Instead of calling 'traverse' on all schemas, we can collect $schema, $id and $anchor keywords when we evaluate the document against its metaschema.

This solution can also be extended to JSM itself to remove the traverse phase (or rather, restructure it so there is no traverse method in JSM nor _traverse_keyword_* methods in the vocabulary classes, and instead restructure JSM::Document::BUILD to evaluating the document against its metaschema with keyword callbacks (for the purpose of populating the resource index).

This may or may not save time; some benchmarking is needed. Fortunately we have an openapi document that takes ~30 seconds to fully load (on test hardware), so we should be able to get some measurable impact.

handle duck-typed requests and responses

Currently we assume, and are only testing with, HTTP::Request/HTTP::Response/HTTP::Headers/URI objects. Also add tests for Mojo::Message::Request/Mojo::Message::Response/Mojo::Headers/Mojo::URL and see what needs to be changed to make these work with the existing code.

gracefully handle references to other documents

Any $ref could be to a totally different document. We can't assume we're in the same document or that our base URI is the same. This also means that things like the operationId index might be buggily holding a reference to another document.

authorization

given a request object (and operationId or path_template), determine authentication for the request.

  • for each of the security schemes itemized in the operation object, evaluate the individual auth implementation against the request object.
  • separate implementations required for: "apiKey", "http", "mutualTLS", "oauth2", "openIdConnect":
    • HTTP authentication
    • an API key (either as a header, a cookie parameter or as a query parameter)
    • mutual TLS (use of a client certificate)
    • OAuth2’s common flows (implicit, password, client credentials and authorization code) as defined in RFC6749 - Please note that as of 2020, the implicit flow is about to be deprecated by OAuth 2.0 Security Best Current Practice. Recommended for most use case is Authorization Code Grant flow with PKCE.
    • OpenID Connect Discovery.

validation and parsing of link objects

as in https://spec.openapis.org/oas/v3.1.0#link-object

validation:

  • operationId or operationRef referenced in 'parameters' must exist
  • expression in 'parameters' or 'requestBody' must match the ABNF
  • parameter names in 'parameters' must exist in the referenced operation
  • all the referenced operation's required parameters must be defined in the link 'parameters'
  • for expression $request.path.foo, path parameter foo must exist in the referenced operation

runtime stuff:

  • for a given http request and operation (use a json pointer), produce the corresponding evaluated links (name => evaluated parameters, ...) -- error if the evaluated link parameters do not match the schema in its operation (you can use karenetheridge/Mojolicious-Plugin-OpenAPI-Modern#17 to turn this into an actual HTTP request).

PS. we can probably enhance the specification metaschema with more validation of link expressions, using the provided ABNF as a guide.

Inclusion of Path Item Objects

Hello Karen,
I think I found bug

I have 2 openapi YAML files which are the following:

data/openapi.yaml

openapi: "3.1.0"
info:
  version: "1.0.0"
  title: "Sample API"
  description: Buy or rent spacecrafts
  license:
    name: "Apache 2.0"
    url: "https://blabla.com/mojo/"
    
paths:
  /spacecrafts/{id}:
    $ref: components/path_items/spacecraft.yaml
components:
  securitySchemes:
    ApiKey:
      type: apiKey
      in: header
      name: X-Api-Key
security:
  - ApiKey: []

data/components/path_items/spacecraft.yaml

parameters:
 - name: id
   description: The unique identifier of the spacecraft
   in: path
   required: true
   schema:
     description: The unique identifier of a resource
     type: string
     pattern: '^\d+$'
get:
 summary: Read a spacecraft
 operationId: getspacecraftById
 responses:
   "200":
     description: The spacecraft corresponding to the provided `id`
     content:
       application/json:
         schema:
           type: object
           required:
             - message
           properties:
             message:
               description: A human readable error message
               type: string
   404:
     description: No spacecraft found for the provided `spacecraftId`
     content:
       application/json:
         schema:
           type: object
           required:
             - message
           properties:
             message:
               description: A human readable error message
               type: string

When I try to load the app I get the following compilation error

Cannot load OpenAPI document: '/paths/~1spacecrafts~1{id}/$ref': additional property not permitted
'/paths/~1spacecrafts~1{id}': not all additional properties are valid
'/paths': not all properties are valid
'': not all properties are valid at /opt/perlbrew/perls/perl-5.26.1/lib/site_perl/5.26.1/Mojolicious/Plugin/OpenAPI/Modern.pm line 61.

If the inclusion of a file happens inside a schema for example:

openapi: "3.1.0"
info:
 version: "1.0.0"
 title: "Sample API"
 description: Buy or rent spacecrafts
 license:
   name: "Apache 2.0"
   url: "https://admin.adzuna.co.uk/mojo/"

paths:
 /spacecrafts/{id}:
   parameters:
     - name: id
       description: The unique identifier of the spacecraft
       in: path
       required: true
       schema:
         $ref: components/schemas/id.yaml

there is no compilation error in this case.

Any help will be much appreciated!
Again, thanks for your time

properly handle unicode characters in the path template

At present, ascii sorting is used when considering path templates to match against a concrete URI. Because of {'s placement in byte representation, all literal path segments are considered for templated path segments (with the exception of ~), but segments starting with a unicode character will take lower precedence.

Use a custom sorting algorithm so that all templated segments are considered after literal segments.

(note that path templates are not URL-encoded.)

extract a subset of the document based on a list of paths

Needed for a proper implementation of #27.

Given a list of json pointers (document paths), find all the $refs included by this subset of the document, and recursively include all the sections so referenced, recursively.

This is pretty straightforward with a %seen hash and a while loop, if we have already built up a dependency tree of references ahead of time (which we can do with callbacks on $ref during traverse).

various types of parameter serialization and types

https://spec.openapis.org/oas/v3.1.0#parameter-object
https://swagger.io/docs/specification/describing-parameters/
https://swagger.io/docs/specification/serialization/

If there is a type: string at the top level of the schema, single parameters get sent as strings and multiple parameters of the same name get sent as arrays (which will cause the validation to fail against a type: string). e.g. use @values = $uri->query_form($param_obj->{name}); $missing = !@values; $values = (@values > 1 ? \@values : $values[0];

If there is a type: integer or type: number at the top level of the schema, consider also converting string values to numbers (if possible) so numeric valiation keywords can be used e.g. minimum, multipleOf.

if there is a type: array at the top level of the schema, send single values as an arrayref, e.g. @values = $uri->query_form($param_obj->{name}); $missing = !@values; $values = \@values;

when explode=false, parse style=form parameters foo=1,2,3 as the array [ 1, 2, 3 ].

cookbook for how to test for various unintuitive scenarios

e.g.

  • request body requiredness can be specified directly with required: true.

  • to indicate that a request body is NOT included, specify a Content-Length header that is required=false, with schema type: integer, const: 0. This will result in no error if the header is missing, and an error if the header is present and its value is anything but 0. You can also specify a requestBody with media-type */* and schema false which will produce a failure if there is a non-zero-length body, just in case the Content-Length header lies.

  • to indicate that a response body is mandatory, specify that the Content-Length header is required=true, with schema type: integer, minimum: 1. this can be abstracted as /components/headers/require_body_payload.

  • to indicate that a response body is NOT included, follow the recipe for request bodies above.

  • to test a header value, of which there can only be one: use schema type: string type: array maxItems: 1

  • to test a header value, of which there may be one or more than one: use schema type: array and test each item with items, maxItems, minItems. explode must be false

  • to test a single header value as a number: use schema type: number

  • to test a header value, of which there may be one or more than one, as numbers: use schema type: array; items: { type: number }.

(some of these not implemented yet, see #8, #14)

path specs in trie form

generate a trie of all the path specs (keys under /paths). This can be useful for route generation and finding the operation that corresponds to an actual request URI.

Any time a path parameter appears, it must be in its own segment.

optionally populate a provided state object, for logic reuse

validate_request does a lot of work in parsing the request to figure out the corresponding path-item and operation. We don't want to redo this work in validate_response -- it can pass back a state object which captures this information.

maybe it should also store a checksum of the $request object to make sure we don't mix it up with another one.

cache operationIds

..by json-pointer to the corresponding operation, and possibly by path_spec + method (see note below).

and error on duplication

Provide a lookup method for getting the operation structure by operationId.

Autoload external $ref files

Hello again,

Considering the following openapi schema

openapi: "3.1.0"
info:
  version: "1.0.0"
  title: "Sample API"
  description: Buy or rent spacecrafts
  license:
    name: Apache 2.0
paths:
  /spacecraft/{id}:
    parameters:
      - $ref: components/parameters/path/id.yaml
    get:
      summary: Read a spacecraft
      operationId: getResourceById
      responses:
        "200":
          description: The spacecraft corresponding to the provided `id`
          content:
            application/json:
              schema:
                $ref: components/schemas/spacecraft.yml

This passes compilation but when trying to call validate_request inside the controller module

$controller->openapi->validate_request( $controller->req );

I get this:

$VAR1 = 'EXCEPTION: unable to find resource /<full_path>/components/parameters/path/id.yaml';

Is this a bug?
or
Do I need to do this in order to add external files in general?

$controller->openapi->evaluator->add_schema('/spacecraft/{id}' => 'components/schemas/spacecraft.yml');

during traverse, check that all local $refs are resolvable

This is the stupid way of doing it. but we can do better by adding a callback to the $defs keyword for "reference", and also hooking on $id and $anchor.

subtest 'ensure that all $refs are resolvable' => sub {
    my @nodes = ($doc);
    my ( @refs, @anchors );
    while (@nodes) {
        my $node = shift @nodes;
        push @nodes, @$node if ref $node eq 'ARRAY';
        if ( ref $node eq 'HASH' ) {
            push @nodes, values %$node;
            push @refs,  $node->{'$ref'} if exists $node->{'$ref'};
            # TODO: we should actually be asking the OpenAPI document for all the identifiers found within.
            push @anchors, $node->{'$anchor'} if exists $node->{'$anchor'};
        }
    }

    my $mjp = Mojo::JSON::Pointer->new($doc);
    foreach my $ref ( uniq @refs ) {
        next if $ref eq $openapi_schema->{'$id'};
        my $uri = Mojo::URL->new($ref);
        ok(
            (
                $uri->clone->fragment(undef) eq ''
                  && (
                    ( $uri->fragment // '' ) =~ m!^/!
                    ? $mjp->contains( $uri->fragment )
                    : grep $_ eq $uri->fragment, @anchors
                  )
            ),
            '$ref to "' . $ref . '" is resolvable in the local document',
        );
    }
};

prohibit request bodies for GET/HEAD, unless explicitly specified for the operation

Unless the operation explicitly specifies a requestBody, GET and HEAD requests should error if there is a request body present, as this is considered a smuggling vector.

"having a request body" should be interpreted as: "Content-Type" header with any value other than 0, or "Transfer-Encoding" being present at all.

e.g. see https://www.imperva.com/learn/application-security/http-request-smuggling/

This may need to be a configurable setting, but let's see.

prohibit "identical" paths, where only the placeholder names differ

/users/{id} and /users/{name} cannot both exist. However, this is not checked in the metaschema (due to an inability to express this in JSON Schema), and we don't check for it in code either.

We can check for this by replacing all placeholders in paths with a token that would be illegal in a URI (say a control character) and then checking for uniqueness (use bits of the PP implementation of uniq, that uses a %seen hash for counting occurrences).

prohibit $refs to invalid locations

This covers two types of $refs:

  • $refs in an openapi keyword. These can only be in specific locations and must have a target of a specific type. Use the existing $ref callbacks to manage these, as that's how we tell when we're in a specific location in the document.

  • $refs in embedded schemas, which can only go to other embedded schemas (or subschemas). depends on karenetheridge/JSON-Schema-Modern#51.

keep track of the location of all resources in the document, for later validation when use by a $ref

We already use callbacks on $ref to track the locations of all operations. We could do this in a more universal way - track the locations of all entities we encounter (using their $defs name), and then we can verify that $refs in the document (not in a json schema, but the openapi part of the doc) are pointing to the right location.

We can do this at runtime, in _resolve_ref, by passing in the name of the entity we expect to be at the other end, and comparing that to our entity index.

We could also do this at initialization time by also tracking the location of all $refs and then resolving the target and comparing against our list of entities. This may not work for $refs to other documents though, as those may not have been loaded yet. This would also incur a startup penalty that may not be worth it.

Fail installing JSON::Schema::Modern::Document::OpenAPI dependency

Hello Karen

First of all thanks for this great plug in. I really want to start using it.
But I am currently facing an installation issue when trying to install Mojolicious::Plugin::OpenAPI::Modern through cpanm
This is the build.log file. Can you help?

Checking dependencies from MYMETA.json ...
Checking if you have open 0 ... Yes (1.11)
Checking if you have YAML::PP 0.005 ... Yes (0.036)
Checking if you have List::Util 0 ... Yes (1.62)
Checking if you have Scalar::Util 0 ... Yes (1.62)
Checking if you have File::ShareDir 0 ... Yes (1.118)
Checking if you have URI::Escape 0 ... Yes (5.10)
Checking if you have Test::JSON::Schema::Acceptance 1.014 ... Yes (1.019)
Checking if you have strictures 2 ... Yes (2.000006)
Checking if you have utf8 0 ... Yes (1.19)
Checking if you have Ref::Util 0 ... Yes (0.204)
Checking if you have Test::More 0.96 ... Yes (1.302190)
Checking if you have Test::Fatal 0 ... Yes (0.016)
Checking if you have Test::File::ShareDir 0 ... Yes (1.001002)
Checking if you have URI 0 ... Yes (5.10)
Checking if you have namespace::clean 0 ... Yes (0.27)
Checking if you have Encode 2.89 ... Yes (3.18)
Checking if you have JSON::Schema::Modern::Vocabulary 0 ... Yes (0.567)
Checking if you have MooX::TypeTiny 0.002002 ... Yes (0.002003)
Checking if you have Math::BigInt 0 ... Yes (1.999806)
Checking if you have experimental 0 ... Yes (0.016)
Checking if you have JSON::Schema::Modern 0.565 ... Yes (0.567)
Checking if you have lib 0 ... Yes (0.64)
Checking if you have File::Spec 0 ... Yes (3.67)
Checking if you have Module::Metadata 0 ... Yes (1.000033)
Checking if you have HTTP::Request 0 ... Yes (6.37)
Checking if you have JSON::Schema::Modern::Utilities 0.531 ... Yes (0.567)
Checking if you have HTTP::Status 0 ... Yes (6.37)
Checking if you have HTTP::Response 0 ... Yes (6.37)
Checking if you have Safe::Isa 0 ... Yes (1.000010)
Checking if you have Test::Memory::Cycle 0 ... Yes (1.06)
Checking if you have if 0 ... Yes (0.0606)
Checking if you have Mojo::Message::Response 0 ... Yes (undef)
Checking if you have Test::Deep 0 ... Yes (1.130)
Checking if you have Path::Tiny 0 ... Yes (0.122)
Checking if you have Mojo::Message::Request 0 ... Yes (undef)
Checking if you have constant 0 ... Yes (1.33)
Checking if you have Carp 0 ... Yes (1.42)
Checking if you have Types::Standard 0 ... Yes (2.004000)
Checking if you have warnings 0 ... Yes (1.37)
Checking if you have MooX::HandlesVia 0 ... Yes (0.001009)
Checking if you have strict 0 ... Yes (1.11)
Checking if you have JSON::Schema::Modern::Document 0 ... Yes (0.567)
Checking if you have Moo 0 ... Yes (2.005004)
Checking if you have Feature::Compat::Try 0 ... Yes (0.05)
Building and testing OpenAPI-Modern-0.044
cp lib/JSON/Schema/Modern/Vocabulary/OpenAPI.pm blib/lib/JSON/Schema/Modern/Vocabulary/OpenAPI.pm
cp share/strict-schema.json blib/lib/auto/share/dist/OpenAPI-Modern/strict-schema.json
cp share/oas/meta/base.schema.json blib/lib/auto/share/dist/OpenAPI-Modern/oas/meta/base.schema.json
cp share/oas/dialect/base.schema.json blib/lib/auto/share/dist/OpenAPI-Modern/oas/dialect/base.schema.json
cp share/oas/LICENSE blib/lib/auto/share/dist/OpenAPI-Modern/oas/LICENSE
cp lib/OpenAPI/Modern.pm blib/lib/OpenAPI/Modern.pm
cp share/oas/schema.json blib/lib/auto/share/dist/OpenAPI-Modern/oas/schema.json
cp share/oas/schema-base.json blib/lib/auto/share/dist/OpenAPI-Modern/oas/schema-base.json
cp lib/JSON/Schema/Modern/Document/OpenAPI.pm blib/lib/JSON/Schema/Modern/Document/OpenAPI.pm
cp share/strict-dialect.json blib/lib/auto/share/dist/OpenAPI-Modern/strict-dialect.json
# 
# Versions for all modules listed in MYMETA.json (including optional ones):
# 
# === Configure Requires ===
# 
#     Module               Want     Have
#     ------------------- ----- --------
#     Module::Build::Tiny 0.034    0.039
#     perl                5.020 5.026001
# 
# === Test Requires ===
# 
#     Module                            Want     Have
#     ------------------------------ ------- --------
#     File::Spec                         any     3.67
#     HTTP::Request                      any     6.37
#     HTTP::Response                     any     6.37
#     HTTP::Status                       any     6.37
#     Module::Metadata                   any 1.000033
#     Test::Deep                         any    1.130
#     Test::Fatal                        any    0.016
#     Test::File::ShareDir               any 1.001002
#     Test::JSON::Schema::Acceptance   1.014    1.019
#     Test::Memory::Cycle                any     1.06
#     Test::More                        0.96 1.302190
#     URI                                any     5.10
#     YAML::PP                         0.005    0.036
#     lib                                any     0.64
#     open                               any     1.11
#     perl                           v5.20.0 5.026001
#     utf8                               any     1.19
# 
# === Test Recommends ===
# 
#     Module         Want     Have
#     ---------- -------- --------
#     CPAN::Meta 2.120900 2.150010
# 
# === Runtime Requires ===
# 
#     Module                               Want     Have
#     -------------------------------- -------- --------
#     Carp                                  any     1.42
#     Encode                               2.89     3.18
#     Feature::Compat::Try                  any     0.05
#     File::ShareDir                        any    1.118
#     JSON::Schema::Modern                0.565    0.567
#     JSON::Schema::Modern::Document        any    0.567
#     JSON::Schema::Modern::Utilities     0.531    0.567
#     JSON::Schema::Modern::Vocabulary      any    0.567
#     List::Util                            any     1.62
#     Math::BigInt                          any 1.999806
#     Mojo::Message::Request                any    undef
#     Mojo::Message::Response               any    undef
#     Moo                                   any 2.005004
#     MooX::HandlesVia                      any 0.001009
#     MooX::TypeTiny                   0.002002 0.002003
#     Path::Tiny                            any    0.122
#     Ref::Util                             any    0.204
#     Safe::Isa                             any 1.000010
#     Scalar::Util                          any     1.62
#     Types::Standard                       any 2.004000
#     URI::Escape                           any     5.10
#     constant                              any     1.33
#     experimental                          any    0.016
#     if                                    any   0.0606
#     namespace::clean                      any     0.27
#     perl                              v5.20.0 5.026001
#     strict                                any     1.11
#     strictures                              2 2.000006
#     warnings                              any     1.37
# 
# === Runtime Suggests ===
# 
#     Module            Want    Have
#     ----------------- ---- -------
#     Class::XSAccessor  any    1.19
#     Cpanel::JSON::XS   any    4.30
#     Ref::Util::XS      any   0.117
#     Type::Tiny::XS     any missing
# 
# === Other Modules ===
# 
#     Module                 Have
#     ---------------- ----------
#     Cpanel::JSON::XS       4.30
#     Encode                 3.18
#     File::Temp           0.2304
#     JSON::MaybeXS      1.004005
#     JSON::PP         2.27400_02
#     JSON::XS               4.03
#     Module::Runtime       0.016
#     Mojolicious            9.31
#     Pod::Coverage          0.23
#     Sub::Name              0.26
#     YAML                   1.30
#     autodie                2.29
# 
t/00-report-prereqs.t .... ok
t/dialects.t ............. ok
t/document-paths.t ....... ok

    #   Failed test 'subschema resources are correctly identified in the document'
    #   at t/document-schemas.t line 170.
    # Compared $data->{"http\:\/\/localhost\:1234\/pathItem0_get_requestBody_id"}{"vocabularies"}[1]
    #    got : 'JSON::Schema::Modern::Vocabulary::Validation'
    # expect : 'JSON::Schema::Modern::Vocabulary::Applicator'
    # Looks like you failed 1 test of 2.

#   Failed test 'identify subschemas'
#   at t/document-schemas.t line 248.
# Looks like you failed 1 test of 2.
t/document-schemas.t ..... 
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/2 subtests 

    #   Failed test 'the document itself is recorded as a resource'
    #   at t/document-toplevel.t line 37.
    # Compared $data->{"http\:\/\/localhost\:1234\/api"}{"vocabularies"}[1]
    #    got : 'JSON::Schema::Modern::Vocabulary::Validation'
    # expect : 'JSON::Schema::Modern::Vocabulary::Applicator'
    # Looks like you failed 1 test of 1.

#   Failed test 'basic construction'
#   at t/document-toplevel.t line 51.

    #   Failed test 'resources are properly stored on the evaluator'
    #   at t/document-toplevel.t line 389.
    # Compared $data->{"http\:\/\/localhost\:1234\/api"}{"vocabularies"}[1]
    #    got : 'JSON::Schema::Modern::Vocabulary::Validation'
    # expect : 'JSON::Schema::Modern::Vocabulary::Applicator'
    # Looks like you failed 1 test of 21.

#   Failed test 'top level document fields'
#   at t/document-toplevel.t line 447.
# Looks like you failed 2 tests of 2.
t/document-toplevel.t .... 
Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/2 subtests 
t/find_path.t ............ ok
# 
# 
# Results using Test::JSON::Schema::Acceptance 1.019
# specification version: draft2020-12
# using custom test directory: t/oas-vocabulary
# optional tests included: no
# 
# filename                                  pass  todo-fail  fail
# ---------------------------------------------------------------
# discriminator.json                          12          0     0
# formats.json                                37          0     0
# ---------------------------------------------------------------
# TOTAL                                       49          0     0
# 
# Congratulations, all non-optional tests are passing!
# 
t/oas-vocabulary.t ....... ok
t/openapi-constructor.t .. ok
t/operationIds.t ......... ok
t/parameters.t ........... ok
t/validate_request.t ..... ok
t/validate_response.t .... ok

Test Summary Report
-------------------
t/document-schemas.t   (Wstat: 256 Tests: 2 Failed: 1)
  Failed test:  2
  Non-zero exit status: 1
t/document-toplevel.t  (Wstat: 512 Tests: 2 Failed: 2)
  Failed tests:  1-2
  Non-zero exit status: 2
Files=12, Tests=96, 23 wallclock secs ( 0.15 usr  0.03 sys + 21.40 cusr  0.84 csys = 22.42 CPU)
Result: FAIL
-> FAIL Installing JSON::Schema::Modern::Document::OpenAPI failed.

linting options for documents

We can provide access to a number of different lint checks through a separate method that might be helpful.

Some of these are areas for improvement, and some of them are things that are not strictly wrong but are obviously buggy.

  • identify $refs to local document locations (i.e. fragment-only uri-references) that do not exist (will be an error at runtime)
  • unused definitions in /components/* (i.e. no $refs point to them) - not an error as some other document might reference them

provide openapi schema data, filtered by tags, authorization roles, or list of operationIds

use the security requirement data for each operation (https://spec.openapis.org/oas/v3.1.0#security-requirement-object) to produce a sanitized version of $openapi_doc->schema that only contains information the indicated role is authorized to know about.

  • need to filter all top-level items, not just /paths -- e.g. stuff in /components may need to be shown, so identify which $refs are being used in /paths etc and only include those referenced items. This will require building a dependency tree of references, either up front when we traverse, or on demand (which can also use traverse with callbacks, and cache the resulting data).

Error open (<:unix) on '.../JSON/Schema/Modern/Document/OpenAPI/oas/schema.json': No such file or directory

The test suite fails on some of my smoker systems:

    #   Failed test 'no exception when the document itself is provided'
    #   at t/openapi-constructor.t line 56.
    #          got: 'Error open (<:unix) on '/home/cpansand/.cpan/build/2021112623/JSON-Schema-Modern-Document-OpenAPI-0.003-0/blib/lib/auto/JSON/Schema/Modern/Document/OpenAPI/oas/schema.json': No such file or directory at /home/cpansand/.cpan/build/2021112623/JSON-Schema-Modern-Document-OpenAPI-0.003-0/blib/lib/JSON/Schema/Modern/Document/OpenAPI.pm line 151.
    # '
    #     expected: undef
...
t/openapi-constructor.t .. 
Dubious, test returned 1 (wstat 256, 0x100)
Failed 1/1 subtests 
    # No tests run!

#   Failed test 'No tests run for subtest "error handling"'
#   at t/validate_request.t line 620.
Error open (<:unix) on '/home/cpansand/.cpan/build/2021112623/JSON-Schema-Modern-Document-OpenAPI-0.003-0/blib/lib/auto/JSON/Schema/Modern/Document/OpenAPI/oas/dialect/base.schema.json': No such file or directory at /home/cpansand/.cpan/build/2021112623/JSON-Schema-Modern-Document-OpenAPI-0.003-0/blib/lib/JSON/Schema/Modern/Document/OpenAPI.pm line 151.
# Tests were run but no plan was declared and done_testing() was not seen.
# Looks like your test exited with 255 just after 1.
t/validate_request.t ..... 
Dubious, test returned 255 (wstat 65280, 0xff00)
Failed 1/1 subtests 

Pass:fail ratio on my smokers is currently 30:4. The failing configurations look like to be system perls on Debian (or Debian-like) systems.

deprecate handling of HTTP::Request, HTTP::Response

we're supporting HTTP::* objects everywhere at the same time as Mojo objects, which makes the code a bit messy.

Instead, we can convert HTTP::* objects to Mojo objects early on and just deal with one type system.
We also need to cope with some incompatibilities, e.g. HTTP::* Request objects not filling in all the headers that should be there.

bundle all the schema files together

let the update-schemas script bundle all these together, and then _add_vocab_and_default_schemas will know what file to add to get all the resources.

provide option to collect and return all annotations

An application might want to see all "extra" keywords included in a schema (say x-controller). At present we generate a internal JSM::Result object for each subschema we evaluate (for parameters, body etc), but throw them away if the final validation result is successful. Instead, we should collect all annotations together on the final result object that we return to the caller (in validate_request, validate_response).

This can be enabled via an optional config value so we don't waste resources when not wanted.

security_for_operation

see https://spec.openapis.org/oas/v3.1.0#fixed-fields-7 under 'security' --
respect document-scoped 'security' requirements, unless removed with a [], fetch the list of security requirements (by name) for this operation.

Either as part of this call, or as a separate call, translate a security scheme name to its underlying object at /components/securitySchemes/.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.