Code Monkey home page Code Monkey logo

protoc-gen-jsonschema's Introduction

Protobuf to JSON-Schema compiler

This takes protobuf definitions and converts them into JSONSchemas, which can be used to dynamically validate JSON messages.

Useful for people who define their data using ProtoBuf, but use JSON for the "wire" format.

"Heavily influenced" by Google's protobuf-to-BigQuery-schema compiler.

Generated Schemas

  • One JSONSchema file is generated for each root-level proto message and ENUM. These are intended to be stand alone self-contained schemas which can be used to validate a payload derived from their source proto message
  • Nested message schemas become referenced "definitions". This means that you know the name of the proto message they came from, and their schema is not duplicated (within the context of one JSONSchema file at least)

Logic

  • For each proto file provided
    • Generates schema for each ENUM
      • JSONSchema filename deried from ENUM name
    • Generates schema for each Message
      • Builds a list of every nested message and converts them to JSONSchema
      • Recursively converts attributes and nested messages within the root message
        • Optionally makes all fields required
        • Optionally allows NULL values
        • Optionally allows additional properties
        • Optionally marks all fields required
        • Specially marked fields are labelled required (options.proto)
        • Specially marked fields are omitted (options.proto)
        • Special handling for "OneOf"
        • Special handling for arrays
        • Special handling for maps
      • Injects references to nested messages
      • JSONSchema filename derived from Message name
    • Bundles these into a protoc generator response

Installation

Note: This tool requires Go 1.11+ to be installed.

Install this plugin using Go:

go install github.com/chrusty/protoc-gen-jsonschema/cmd/protoc-gen-jsonschema@latest

Usage

Note: This plugin requires the protoc CLI to be installed.

protoc-gen-jsonschema is designed to run like any other proto generator. The following examples show how to use options flags to enable different generator behaviours (more examples in the Makefile too).

protoc \ # The protobuf compiler
--jsonschema_out=. \ # jsonschema out directory
--proto_path=testdata/proto testdata/proto/ArrayOfPrimitives.proto # proto input directories and folders

Configuration Parameters

The following configuration parameters are supported. They should be added to the protoc command and can be combined as a comma-delimited string. Some examples are included in the following Examples section.

Options can also be provided in this format (which is easier on the eye):

protoc \
  --plugin=${HOME}/go/bin/protoc-gen-jsonschema \
  --jsonschema_opt=enforce_oneof
  --jsonschema_opt=file_extension=schema.json \
  --jsonschema_opt=disallow_additional_properties \
  --jsonschema_out=schemas \
  --proto_path=proto
CONFIG DESCRIPTION
all_fields_required Require all fields in schema
allow_null_values Allow null values in schema
debug Enable debug logging
disallow_additional_properties Disallow additional properties in schema
disallow_bigints_as_strings Disallow big integers as strings
enforce_oneof Interpret Proto "oneOf" clauses
enums_as_strings_only Only include strings in the allowed values for enums
file_extension Specify a custom file extension for generated schemas
json_fieldnames Use JSON field names only
prefix_schema_files_with_package Prefix the output filename with package
proto_and_json_fieldnames Use proto and JSON field names
type_names_with_no_package When generating type names and refs, do not include the full package in the type name

Custom Proto Options

If you don't want to use the configuration parameters (admittedly quite a nasty cli syntax) then some of the generator behaviour can be controlled using custom proto options. These are defined in options.proto, and your protoc command will need to include this file. See the sample protos and generator commands in the Makefile.

Enum Options

These apply to specifically marked enums, giving you more finely-grained control than with the CLI flags.

Field Options

These apply to specifically marked fields, giving you more finely-grained control than with the CLI flags.

  • ignore: Ignore (omit) a specific field
  • required: Mark a specific field as being REQUIRED

File Options

These options apply to an entire proto file.

  • ignore: Ignore (skip) a specific file
  • extension: Specify a custom file-extension for the generated schema for this file

Message Options

These options apply to a specific proto message.

Validation Options

We are also beginning to support validation options from protoc-gen-validate.

At the moment the following are supported (but export more in the future):

  • Arrays
    • MaxItems
    • MinItems
  • Strings
    • MaxLength
    • MinLength
    • Pattern

Examples

Require all fields

Because proto3 doesn't accommodate this.

protoc \
--jsonschema_out=all_fields_required:. \
--proto_path=testdata/proto testdata/proto/ArrayOfPrimitives.proto

Allow NULL values

By default, JSONSchemas will reject NULL values unless we explicitly allow them

protoc \
--jsonschema_out=allow_null_values:. \
--proto_path=testdata/proto testdata/proto/ArrayOfPrimitives.proto

Enable debug logging

protoc \
--jsonschema_out=debug:. \
--proto_path=testdata/proto testdata/proto/ArrayOfPrimitives.proto

Disallow additional properties

JSONSchemas won't validate JSON containing extra parameters

protoc \
--jsonschema_out=disallow_additional_properties:. \
--proto_path=testdata/proto testdata/proto/ArrayOfPrimitives.proto

Disallow permissive validation of big-integers as strings

(eg scientific notation)

protoc \
--jsonschema_out=disallow_bigints_as_strings:. \
--proto_path=testdata/proto testdata/proto/ArrayOfPrimitives.proto

Prefix generated schema files with their package name (as a directory)

protoc \
--jsonschema_out=prefix_schema_files_with_package:. \
--proto_path=testdata/proto testdata/proto/ArrayOfPrimitives.proto

Target specific messages within a proto file

# Generates MessageKind10.jsonschema and MessageKind11.jsonschema
# Use this to generate json schema from proto files with multiple messages
# Separate schema names with '+'
protoc \
--jsonschema_out=messages=[MessageKind10+MessageKind11]:. \
--proto_path=testdata/proto testdata/proto/TwelveMessages.proto

Generate fields with JSON names

protoc \
--jsonschema_out=json_fieldnames:. \
--proto_path=testdata/proto testdata/proto/ArrayOfPrimitives.proto

Custom schema file extension

The default file extension is json. You can override that with the "file_extension" parameter.

protoc \
--jsonschema_out=file_extension=jsonschema:. \
--proto_path=internal/converter/testdata/proto internal/converter/testdata/proto/ArrayOfPrimitives.proto

Generate type names without fully qualified package

By default, referenced type names will be generated using the fully qualified package and type name. e.g packageName.TypeName. Setting this option will generate type names and their references only as TypeName

protoc \
--jsonschema_out=type_names_with_no_package:. \
--proto_path=internal/converter/testdata/proto internal/converter/testdata/proto/ArrayOfPrimitives.proto

Sample protos (for testing)

Links

protoc-gen-jsonschema's People

Contributors

akihiroueda35 avatar andrewzurn avatar ballbag22 avatar blakesmith avatar ceason avatar chrusty avatar faroceann avatar felixjung avatar fleker avatar grant avatar ianvdhsbm avatar lologarithm avatar maxtibs avatar mier85 avatar mrozycki-tink avatar norbjd avatar rpbritton avatar schigh avatar wk8 avatar zbiljic avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

protoc-gen-jsonschema's Issues

Use .json, not .jsonschema for JSON schemas

Tooling is generally better with the .json extension with JSON schemas, and the spec is indifferent with file extension. We should rename .jsonschema to .json everywhere in this repo.

I do not know/remember if this proposed change would just be internal or would be external.

Sometimes $ref, sometimes embed

Source:
model.proto.txt

Why we don't get reference generated in 100% cases.
From Party.jsonschema -- characteristic.items is ref, while external_id.items is embedded.

        "characteristic": {
            "items": {
                "$schema": "http://json-schema.org/draft-04/schema#",
                "$ref": "ru.beeline.tmf.model.Characteristic"
        ....
        "external_id": {
            "items": {
                "$schema": "http://json-schema.org/draft-04/schema#",
                "properties": {
                    "external_identification_type_id": {

I feel it should always be a ref.
currently external_identification_type_id field is generated in 5 files, while file ExternalIdentification.jsonschema is not referenced at all.

Schema name is lost in nested definition

Consider:

syntax = "proto2";

package tutorial;

message Person {
  optional string name = 1;
  optional int32 id = 2;
  optional string email = 3;

  enum PhoneType {
    MOBILE = 0;
    HOME = 1;
    WORK = 2;
  }

  message PhoneNumber {
    optional string number = 1;
    optional PhoneType type = 2 [default = HOME];
  }

  repeated PhoneNumber phones = 4;
}

message AddressBook {
  repeated Person people = 1;
}

This is the generated json:

{
    "$schema": "http://json-schema.org/draft-04/schema#",
    "properties": {
        "people": {
            "items": {
                "$schema": "http://json-schema.org/draft-04/schema#",
                "properties": {
                    "name": {
                        "type": "string"
                    },
                    "id": {
                        "type": "integer"
                    },
                    "email": {
                        "type": "string"
                    },
                    "phones": {
                        "items": {
                            "$schema": "http://json-schema.org/draft-04/schema#",
                            "properties": {
                                "number": {
                                    "type": "string"
                                },
                                "type": {
                                    "enum": [
                                        "MOBILE",
                                        0,
                                        "HOME",
                                        1,
                                        "WORK",
                                        2
                                    ],
                                    "oneOf": [
                                        {
                                            "type": "string"
                                        },
                                        {
                                            "type": "integer"
                                        }
                                    ]
                                }
                            },
                            "additionalProperties": true,
                            "type": "object"
                        },
                        "type": "array"
                    }
                },
                "additionalProperties": true,
                "type": "object"
            },
            "type": "array"
        }
    },
    "additionalProperties": true,
    "type": "object"
}

There is no way to know that AddressBook.people is of type Person. It is an array with the the fields that are in the json, but knowing that it's part of the Person schema is useful.

Render protobuf oneof as json schema oneOf

When converting the following proto file:

syntax="proto3";

package org.example;

message Foo {
	oneof choice {
		Bar bar = 1;
		Baz baz = 2;
	}
	
	message Bar {
		int32 foo = 1;
	}
	
	message Baz {
		string foo = 1;
	}
}

then the resulting json schema is:

{
    "$schema": "http://json-schema.org/draft-04/schema#",
    "properties": {
        "bar": {
            "properties": {
                "foo": {
                    "type": "integer"
                }
            },
            "additionalProperties": true,
            "type": "object"
        },
        "baz": {
            "properties": {
                "foo": {
                    "type": "string"
                }
            },
            "additionalProperties": true,
            "type": "object"
        }
    },
    "additionalProperties": true,
    "type": "object"
}

This would allow a Foo instance to contain both the bar and baz attributes, which kind of defeats the purpose of the oneof.

Would it be possible to generate a schema like this instead?

{
    "$schema": "http://json-schema.org/draft-04/schema#",
    "properties": {
        "bar": {
            "properties": {
                "foo": {
                    "type": "integer"
                }
            },
            "additionalProperties": true,
            "type": "object"
        },
        "baz": {
            "properties": {
                "foo": {
                    "type": "string"
                }
            },
            "additionalProperties": true,
            "type": "object"
        }
    },
    "oneOf": [{
            "required" : ["bar"]
        }, {
            "required" : ["baz"]
        }
    ],    
    "additionalProperties": true,
    "type": "object"
}

Cannot compile when two messages have the same name

I've defined two protos with exactly the same content, but in different folders as such:

proto/
โ”œโ”€ a/
โ”‚  โ”œโ”€ pet.proto
โ”œโ”€ b/
โ”‚  โ”œโ”€ pet.proto

As I expect the output should be in the same structure as shown above just like when using --python_out or other options, protoc throws Tried to write the same file twice.

Even if I rename one of the messages to message Pet1, the output structure is flat, like so:

schema/
โ”œโ”€ Pet.jsonschema
โ”œโ”€ Pet1.jsonschema

Should that be fixed? Cuz reading and writing files recursively should be a feature.

stack overflow with gnmi.proto

protoc -I . --jsonschema_out=. -I/root/dev_ubuntu/go/src/github.com/google/protobuf/src -I/root/dev_ubuntu/go/src gnmi.proto
WARN[0000] protoc-gen-jsonschema will create multiple MESSAGE schemas from one proto file proto_filename=gnmi.proto schemas=24
WARN[0000] protoc-gen-jsonschema will create multiple ENUM schemas from one proto file proto_filename=gnmi.proto schemas=24
INFO[0000] Generating JSON-schema for MESSAGE jsonschema_filename=Update.jsonschema msg_name=Update proto_filename=gnmi.proto
runtime: goroutine stack exceeds 1000000000-byte limit
fatal error: stack overflow

runtime stack:
runtime.throw(0x85c72f, 0xe)
/usr/local/go/src/runtime/panic.go:774 +0x72
runtime.newstack()
/usr/local/go/src/runtime/stack.go:1046 +0x6e9
runtime.morestack()
/usr/local/go/src/runtime/asm_amd64.s:449 +0x8f

goroutine 1 [running]:
runtime.mapaccess1_fast32(0x7dd280, 0x0, 0x1000000000bd1e0, 0x0)

========================
//
// Copyright 2016 Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//
syntax = "proto3";

import "google/protobuf/any.proto";
import "google/protobuf/descriptor.proto";
import "github.com/openconfig/gnmi/proto/gnmi_ext/gnmi_ext.proto";

// Package gNMI defines a service specification for the gRPC Network Management
// Interface. This interface is defined to be a standard interface via which
// a network management system ("client") can subscribe to state values,
// retrieve snapshots of state information, and manipulate the state of a data
// tree supported by a device ("target").
//
// This document references the gNMI Specification which can be found at
// http://github.com/openconfig/reference/blob/master/rpc/gnmi
package gnmi;

// Define a protobuf FileOption that defines the gNMI service version.
extend google.protobuf.FileOptions {
// The gNMI service semantic version.
string gnmi_service = 1001;
}

// gNMI_service is the current version of the gNMI service, returned through
// the Capabilities RPC.
option (gnmi_service) = "0.7.0";

service gNMI {
// Capabilities allows the client to retrieve the set of capabilities that
// is supported by the target. This allows the target to validate the
// service version that is implemented and retrieve the set of models that
// the target supports. The models can then be specified in subsequent RPCs
// to restrict the set of data that is utilized.
// Reference: gNMI Specification Section 3.2
rpc Capabilities(CapabilityRequest) returns (CapabilityResponse);
// Retrieve a snapshot of data from the target. A Get RPC requests that the
// target snapshots a subset of the data tree as specified by the paths
// included in the message and serializes this to be returned to the
// client using the specified encoding.
// Reference: gNMI Specification Section 3.3
rpc Get(GetRequest) returns (GetResponse);
// Set allows the client to modify the state of data on the target. The
// paths to modified along with the new values that the client wishes
// to set the value to.
// Reference: gNMI Specification Section 3.4
rpc Set(SetRequest) returns (SetResponse);
// Subscribe allows a client to request the target to send it values
// of particular paths within the data tree. These values may be streamed
// at a particular cadence (STREAM), sent one off on a long-lived channel
// (POLL), or sent as a one-off retrieval (ONCE).
// Reference: gNMI Specification Section 3.5
rpc Subscribe(stream SubscribeRequest) returns (stream SubscribeResponse);
}

// Notification is a re-usable message that is used to encode data from the
// target to the client. A Notification carries two types of changes to the data
// tree:
// - Deleted values (delete) - a set of paths that have been removed from the
// data tree.
// - Updated values (update) - a set of path-value pairs indicating the path
// whose value has changed in the data tree.
// Reference: gNMI Specification Section 2.1
//message Notification {
//int64 timestamp = 1; // Timestamp in nanoseconds since Epoch.
//Path prefix = 2; // Prefix used for paths in the message.
// An alias for the path specified in the prefix field.
// Reference: gNMI Specification Section 2.4.2
//string alias = 3;
//repeated Update update = 4; // Data elements that have changed values.
//repeated Path delete = 5; // Data elements that have been deleted.
// This notification contains a set of paths that are always updated together
// referenced by a globally unique prefix.
//bool atomic = 6;
//}

// Update is a re-usable message that is used to store a particular Path,
// Value pair.
// Reference: gNMI Specification Section 2.1
message Update {
Path path = 1; // The path (key) for the update.
Value value = 2 [deprecated=true]; // The value (value) for the update.
TypedValue val = 3; // The explicitly typed update value.
uint32 duplicates = 4; // Number of coalesced duplicates.
}

// TypedValue is used to encode a value being sent between the client and
// target (originated by either entity).
message TypedValue {
// One of the fields within the val oneof is populated with the value
// of the update. The type of the value being included in the Update
// determines which field should be populated. In the case that the
// encoding is a particular form of the base protobuf type, a specific
// field is used to store the value (e.g., json_val).
oneof value {
string string_val = 1; // String value.
int64 int_val = 2; // Integer value.
uint64 uint_val = 3; // Unsigned integer value.
bool bool_val = 4; // Bool value.
bytes bytes_val = 5; // Arbitrary byte sequence value.
float float_val = 6; // Floating point value.
Decimal64 decimal_val = 7; // Decimal64 encoded value.
ScalarArray leaflist_val = 8; // Mixed type scalar array value.
google.protobuf.Any any_val = 9; // protobuf.Any encoded bytes.
bytes json_val = 10; // JSON-encoded text.
bytes json_ietf_val = 11; // JSON-encoded text per RFC7951.
string ascii_val = 12; // Arbitrary ASCII text.
// Protobuf binary encoded bytes. The message type is not included.
// See the specification at
// github.com/openconfig/reference/blob/master/rpc/gnmi/protobuf-vals.md
// for a complete specification.
bytes proto_bytes = 13;
}
}

// Path encodes a data tree path as a series of repeated strings, with
// each element of the path representing a data tree node name and the
// associated attributes.
// Reference: gNMI Specification Section 2.2.2.
message Path {
// Elements of the path are no longer encoded as a string, but rather within
// the elem field as a PathElem message.
repeated string element = 1 [deprecated=true];
string origin = 2; // Label to disambiguate path.
repeated PathElem elem = 3; // Elements of the path.
string target = 4; // The name of the target
// (Sec. 2.2.2.1)
}

// PathElem encodes an element of a gNMI path, along with any attributes (keys)
// that may be associated with it.
// Reference: gNMI Specification Section 2.2.2.
message PathElem {
string name = 1; // The name of the element in the path.
map<string, string> key = 2; // Map of key (attribute) name to value.
}

// Value encodes a data tree node's value - along with the way in which
// the value is encoded. This message is deprecated by gNMI 0.3.0.
// Reference: gNMI Specification Section 2.2.3.
message Value {
option deprecated = true;
bytes value = 1; // Value of the variable being transmitted.
Encoding type = 2; // Encoding used for the value field.
}

// Encoding defines the value encoding formats that are supported by the gNMI
// protocol. These encodings are used by both the client (when sending Set
// messages to modify the state of the target) and the target when serializing
// data to be returned to the client (in both Subscribe and Get RPCs).
// Reference: gNMI Specification Section 2.3
enum Encoding {
JSON = 0; // JSON encoded text.
BYTES = 1; // Arbitrarily encoded bytes.
PROTO = 2; // Encoded according to out-of-band agreed Protobuf.
ASCII = 3; // ASCII text of an out-of-band agreed format.
JSON_IETF = 4; // JSON encoded text as per RFC7951.
}

// Error message previously utilised to return errors to the client. Deprecated
// in favour of using the google.golang.org/genproto/googleapis/rpc/status
// message in the RPC response.
// Reference: gNMI Specification Section 2.5
message Error {
option deprecated = true;
uint32 code = 1; // Canonical gRPC error code.
string message = 2; // Human readable error.
google.protobuf.Any data = 3; // Optional additional information.
}

// Decimal64 is used to encode a fixed precision decimal number. The value
// is expressed as a set of digits with the precision specifying the
// number of digits following the decimal point in the digit set.
message Decimal64 {
int64 digits = 1; // Set of digits.
uint32 precision = 2; // Number of digits following the decimal point.
}

// ScalarArray is used to encode a mixed-type array of values.
message ScalarArray {
// The set of elements within the array. Each TypedValue message should
// specify only elements that have a field identifier of 1-7 (i.e., the
// values are scalar values).
repeated TypedValue element = 1;
}

// SubscribeRequest is the message sent by the client to the target when
// initiating a subscription to a set of paths within the data tree. The
// request field must be populated and the initial message must specify a
// SubscriptionList to initiate a subscription. The message is subsequently
// used to define aliases or trigger polled data to be sent by the target.
// Reference: gNMI Specification Section 3.5.1.1
message SubscribeRequest {
oneof request {
SubscriptionList subscribe = 1; // Specify the paths within a subscription.
Poll poll = 3; // Trigger a polled update.
AliasList aliases = 4; // Aliases to be created.
}
// Extension messages associated with the SubscribeRequest. See the
// gNMI extension specification for further definition.
repeated gnmi_ext.Extension extension = 5;
}

// Poll is sent within a SubscribeRequest to trigger the device to
// send telemetry updates for the paths that are associated with the
// subscription.
// Reference: gNMI Specification Section Section 3.5.1.4
message Poll {
}

// SubscribeResponse is the message used by the target within a Subscribe RPC.
// The target includes a Notification message which is used to transmit values
// of the path(s) that are associated with the subscription. The same message
// is to indicate that the target has sent all data values once (is
// synchronized).
// Reference: gNMI Specification Section 3.5.1.4
message SubscribeResponse {
oneof response {
//Notification update = 1; // Changed or sampled value for a path.
// Indicate target has sent all values associated with the subscription
// at least once.
bool sync_response = 3;
// Deprecated in favour of google.golang.org/genproto/googleapis/rpc/status
Error error = 4 [deprecated=true];
}
// Extension messages associated with the SubscribeResponse. See the
// gNMI extension specification for further definition.
repeated gnmi_ext.Extension extension = 5;
}

// SubscriptionList is used within a Subscribe message to specify the list of
// paths that the client wishes to subscribe to. The message consists of a
// list of (possibly prefixed) paths, and options that relate to the
// subscription.
// Reference: gNMI Specification Section 3.5.1.2
message SubscriptionList {
Path prefix = 1; // Prefix used for paths.
repeated Subscription subscription = 2; // Set of subscriptions to create.
// Whether target defined aliases are allowed within the subscription.
bool use_aliases = 3;
QOSMarking qos = 4; // DSCP marking to be used.
// Mode of the subscription.
enum Mode {
STREAM = 0; // Values streamed by the target (Sec. 3.5.1.5.2).
ONCE = 1; // Values sent once-off by the target (Sec. 3.5.1.5.1).
POLL = 2; // Values sent in response to a poll request (Sec. 3.5.1.5.3).
}
Mode mode = 5;
// Whether elements of the schema that are marked as eligible for aggregation
// should be aggregated or not.
bool allow_aggregation = 6;
// The set of schemas that define the elements of the data tree that should
// be sent by the target.
repeated ModelData use_models = 7;
// The encoding that the target should use within the Notifications generated
// corresponding to the SubscriptionList.
Encoding encoding = 8;
// An optional field to specify that only updates to current state should be
// sent to a client. If set, the initial state is not sent to the client but
// rather only the sync message followed by any subsequent updates to the
// current state. For ONCE and POLL modes, this causes the server to send only
// the sync message (Sec. 3.5.2.3).
bool updates_only = 9;
}

// Subscription is a single request within a SubscriptionList. The path
// specified is interpreted (along with the prefix) as the elements of the data
// tree that the client is subscribing to. The mode determines how the target
// should trigger updates to be sent.
// Reference: gNMI Specification Section 3.5.1.3
message Subscription {
Path path = 1; // The data tree path.
SubscriptionMode mode = 2; // Subscription mode to be used.
uint64 sample_interval = 3; // ns between samples in SAMPLE mode.
// Indicates whether values that have not changed should be sent in a SAMPLE
// subscription.
bool suppress_redundant = 4;
// Specifies the maximum allowable silent period in nanoseconds when
// suppress_redundant is in use. The target should send a value at least once
// in the period specified.
uint64 heartbeat_interval = 5;
}

// SubscriptionMode is the mode of the subscription, specifying how the
// target must return values in a subscription.
// Reference: gNMI Specification Section 3.5.1.3
enum SubscriptionMode {
TARGET_DEFINED = 0; // The target selects the relevant mode for each element.
ON_CHANGE = 1; // The target sends an update on element value change.
SAMPLE = 2; // The target samples values according to the interval.
}

// QOSMarking specifies the DSCP value to be set on transmitted telemetry
// updates from the target.
// Reference: gNMI Specification Section 3.5.1.2
message QOSMarking {
uint32 marking = 1;
}

// Alias specifies a data tree path, and an associated string which defines an
// alias which is to be used for this path in the context of the RPC. The alias
// is specified as a string which is prefixed with "#" to disambiguate it from
// data tree element paths.
// Reference: gNMI Specification Section 2.4.2
message Alias {
Path path = 1; // The path to be aliased.
string alias = 2; // The alias value, a string prefixed by "#".
}

// AliasList specifies a list of aliases. It is used in a SubscribeRequest for
// a client to create a set of aliases that the target is to utilize.
// Reference: gNMI Specification Section 3.5.1.6
message AliasList {
repeated Alias alias = 1; // The set of aliases to be created.
}

// SetRequest is sent from a client to the target to update values in the data
// tree. Paths are either deleted by the client, or modified by means of being
// updated, or replaced. Where a replace is used, unspecified values are
// considered to be replaced, whereas when update is used the changes are
// considered to be incremental. The set of changes that are specified within
// a single SetRequest are considered to be a transaction.
// Reference: gNMI Specification Section 3.4.1
message SetRequest {
Path prefix = 1; // Prefix used for paths in the message.
repeated Path delete = 2; // Paths to be deleted from the data tree.
repeated Update replace = 3; // Updates specifying elements to be replaced.
repeated Update update = 4; // Updates specifying elements to updated.
// Extension messages associated with the SetRequest. See the
// gNMI extension specification for further definition.
repeated gnmi_ext.Extension extension = 5;
}

// SetResponse is the response to a SetRequest, sent from the target to the
// client. It reports the result of the modifications to the data tree that were
// specified by the client. Errors for this RPC should be reported using the
// https://github.com/googleapis/googleapis/blob/master/google/rpc/status.proto
// message in the RPC return. The gnmi.Error message can be used to add additional
// details where required.
// Reference: gNMI Specification Section 3.4.2
message SetResponse {
Path prefix = 1; // Prefix used for paths.
// A set of responses specifying the result of the operations specified in
// the SetRequest.
repeated UpdateResult response = 2;
Error message = 3 [deprecated=true]; // The overall status of the transaction.
int64 timestamp = 4; // Timestamp of transaction (ns since epoch).
// Extension messages associated with the SetResponse. See the
// gNMI extension specification for further definition.
repeated gnmi_ext.Extension extension = 5;
}

// UpdateResult is used within the SetResponse message to communicate the
// result of an operation specified within a SetRequest message.
// Reference: gNMI Specification Section 3.4.2
message UpdateResult {
// The operation that was associated with the Path specified.
enum Operation {
INVALID = 0;
DELETE = 1; // The result relates to a delete of Path.
REPLACE = 2; // The result relates to a replace of Path.
UPDATE = 3; // The result relates to an update of Path.
}
// Deprecated timestamp for the UpdateResult, this field has been
// replaced by the timestamp within the SetResponse message, since
// all mutations effected by a set should be applied as a single
// transaction.
int64 timestamp = 1 [deprecated=true];
Path path = 2; // Path associated with the update.
Error message = 3 [deprecated=true]; // Status of the update operation.
Operation op = 4; // Update operation type.
}

// GetRequest is sent when a client initiates a Get RPC. It is used to specify
// the set of data elements for which the target should return a snapshot of
// data. The use_models field specifies the set of schema modules that are to
// be used by the target - where use_models is not specified then the target
// must use all schema models that it has.
// Reference: gNMI Specification Section 3.3.1
message GetRequest {
Path prefix = 1; // Prefix used for paths.
repeated Path path = 2; // Paths requested by the client.
// Type of elements within the data tree.
enum DataType {
ALL = 0; // All data elements.
CONFIG = 1; // Config (rw) only elements.
STATE = 2; // State (ro) only elements.
// Data elements marked in the schema as operational. This refers to data
// elements whose value relates to the state of processes or interactions
// running on the device.
OPERATIONAL = 3;
}
DataType type = 3; // The type of data being requested.
Encoding encoding = 5; // Encoding to be used.
repeated ModelData use_models = 6; // The schema models to be used.
// Extension messages associated with the GetRequest. See the
// gNMI extension specification for further definition.
repeated gnmi_ext.Extension extension = 7;
}

// GetResponse is used by the target to respond to a GetRequest from a client.
// The set of Notifications corresponds to the data values that are requested
// by the client in the GetRequest.
// Reference: gNMI Specification Section 3.3.2
message GetResponse {
//repeated Notification notification = 1; // Data values.
Error error = 2 [deprecated=true]; // Errors that occurred in the Get.
// Extension messages associated with the GetResponse. See the
// gNMI extension specification for further definition.
repeated gnmi_ext.Extension extension = 3;
}

// CapabilityRequest is sent by the client in the Capabilities RPC to request
// that the target reports its capabilities.
// Reference: gNMI Specification Section 3.2.1
message CapabilityRequest {
// Extension messages associated with the CapabilityRequest. See the
// gNMI extension specification for further definition.
repeated gnmi_ext.Extension extension = 1;
}

// CapabilityResponse is used by the target to report its capabilities to the
// client within the Capabilities RPC.
// Reference: gNMI Specification Section 3.2.2
message CapabilityResponse {
repeated ModelData supported_models = 1; // Supported schema models.
repeated Encoding supported_encodings = 2; // Supported encodings.
string gNMI_version = 3; // Supported gNMI version.
// Extension messages associated with the CapabilityResponse. See the
// gNMI extension specification for further definition.
repeated gnmi_ext.Extension extension = 4;
}

// ModelData is used to describe a set of schema modules. It can be used in a
// CapabilityResponse where a target reports the set of modules that it
// supports, and within the SubscribeRequest and GetRequest messages to specify
// the set of models from which data tree elements should be reported.
// Reference: gNMI Specification Section 3.2.3
message ModelData {
string name = 1; // Name of the model.
string organization = 2; // Organization publishing the model.
string version = 3; // Semantic version of the model.
}

all_fields_required fails on nil pointer

All fields required ~/bin/protoc/bin/protoc --jsonschema_out=all_fields_required:. config.proto fails on nil pointer for https://github.com/hypertrace/agent-config/blob/main/config.proto

~/bin/protoc/bin/protoc --version                                                                                                                                                                            1 โ†ต ploffay@Pavols-MacBook-Pro
libprotoc 3.14.0
 ~/bin/protoc/bin/protoc --jsonschema_out=all_fields_required:.  config.proto                                                                                                                                              ploffay@Pavols-MBP
WARN[0000] protoc-gen-jsonschema will create multiple MESSAGE schemas from one proto file  proto_filename=config.proto schemas=5
INFO[0000] Generating JSON-schema for MESSAGE            jsonschema_filename=AgentConfig.jsonschema msg_name=AgentConfig proto_filename=config.proto
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x134e978]

goroutine 1 [running]:
github.com/iancoleman/orderedmap.(*OrderedMap).Keys(...)
	/Users/ploffay/projects/golang/src/github.com/iancoleman/orderedmap/orderedmap.go:94
github.com/chrusty/protoc-gen-jsonschema/internal/converter.(*Converter).convertField(0xc00021ff48, 0xc000266300, 0xc00024e800, 0xc00022d500, 0xc000266780, 0xc00026ad90, 0x0, 0x0)
	/Users/ploffay/projects/golang/src/github.com/chrusty/protoc-gen-jsonschema/internal/converter/types.go:278 +0x10d8
github.com/chrusty/protoc-gen-jsonschema/internal/converter.(*Converter).recursiveConvertMessageType(0xc00021ff48, 0xc000266300, 0xc00022d500, 0xc000251218, 0x4, 0xc000266780, 0x0, 0x203001, 0x30, 0x203000)
	/Users/ploffay/projects/golang/src/github.com/chrusty/protoc-gen-jsonschema/internal/converter/types.go:471 +0x41d
github.com/chrusty/protoc-gen-jsonschema/internal/converter.(*Converter).convertField(0xc00021ff48, 0xc000266300, 0xc00024e300, 0xc00022d100, 0xc000266780, 0xc000254d20, 0x1, 0x2)
	/Users/ploffay/projects/golang/src/github.com/chrusty/protoc-gen-jsonschema/internal/converter/types.go:226 +0xbe5
github.com/chrusty/protoc-gen-jsonschema/internal/converter.(*Converter).recursiveConvertMessageType(0xc00021ff48, 0xc000266300, 0xc00022d100, 0x0, 0x0, 0xc000266780, 0xc000266700, 0x0, 0x0, 0x0)
	/Users/ploffay/projects/golang/src/github.com/chrusty/protoc-gen-jsonschema/internal/converter/types.go:471 +0x41d
github.com/chrusty/protoc-gen-jsonschema/internal/converter.(*Converter).convertMessageType(0xc00021ff48, 0xc000266300, 0xc00022d100, 0x1, 0x1, 0xc00026a1c0)
	/Users/ploffay/projects/golang/src/github.com/chrusty/protoc-gen-jsonschema/internal/converter/types.go:309 +0xba
github.com/chrusty/protoc-gen-jsonschema/internal/converter.(*Converter).convertFile(0xc00021ff48, 0xc00022d000, 0xc00021fb40, 0x1, 0x1, 0xc00026a000, 0xc00021faf0)
	/Users/ploffay/projects/golang/src/github.com/chrusty/protoc-gen-jsonschema/internal/converter/converter.go:186 +0x759
github.com/chrusty/protoc-gen-jsonschema/internal/converter.(*Converter).convert(0xc00021ff48, 0xc0001d4380, 0xc00021fdf8, 0x1, 0x1)
	/Users/ploffay/projects/golang/src/github.com/chrusty/protoc-gen-jsonschema/internal/converter/converter.go:234 +0x767
github.com/chrusty/protoc-gen-jsonschema/internal/converter.(*Converter).ConvertFrom(0xc00021ff48, 0x146e160, 0xc000010010, 0x1, 0x1, 0x2)
	/Users/ploffay/projects/golang/src/github.com/chrusty/protoc-gen-jsonschema/internal/converter/converter.go:61 +0x285
main.main()
	/Users/ploffay/projects/golang/src/github.com/chrusty/protoc-gen-jsonschema/cmd/protoc-gen-jsonschema/main.go:34 +0x1b0
--jsonschema_out: protoc-gen-jsonschema: Plugin failed with status code 2.

Steps to reproduce:

  1. clone https://github.com/hypertrace/agent-config
  2. add package org.hypertrace; to the config.proto
  3. run ~/bin/protoc/bin/protoc --jsonschema_out=all_fields_required:. config.proto

Installation Problems

Hi @chrusty,

With the new changes in #14, I am confused about the new way of installing protoc-gen-jsonschema.

Before the change, running go get git.enova.com/cashnet/protoc-gen-go worked well since main.go was still in the top level directory. Now, there doesn't seem to exist a main.go or any main package at all that can be built.

Running make build or go install github.com/chrusty/protoc-gen-jsonschema/cmd/protoc-gen-jsonschema as instructed by the README leads to errors regarding github.com/chrusty/protoc-gen-jsonschema/cmd not existing. Is there something I am missing here?

I appreciate your help!

Transient fields

Idea would be to have an proto option to mark a field to not be added to the json schema, example:

message User {

  optional string name = 0;
  optional string email = 1;

  optional string type = 3 [(schemaTransient) = ""];
  repeated Address addresses = 4 [(schemaTransient) = ""];

}

which would output a json schema with only fields numbers 0 and 1, fields number 3 and 4 are not present in the json schema.

Goal of this:
use a message for the back end and allow to use the same message for the front end while being able to hide some fields because they are technical or soft managed and there's no point to pollute the schema with those

$ref: broken?

(thanks a lot for a superb plugin!)

Source:
model.proto.txt
e

I'm a bit confused about $ref codegeneration.
It get's generated "just.name.with.package.of.referred.entity".
This is not valid JSON reference, as I understand it.

Example of "wrong" $ref from Party.jsonschema:

                "$ref": "ru.beeline.tmf.model.Characteristic"

I feel this should have been:

                "$ref": "Characteristic.jsonschema#/definitions/ru.beeline.tmf.model.Characteristic"

Duplicated sample step in Makefile

Hi, it looks like the lines below (currently 58 and 59) are repeated in the Makefile, except for the echo command on the end.

@PATH=./bin:$$PATH; protoc --jsonschema_out=jsonschemas -I. --proto_path=${PROTO_PATH {PROTO_PATH}/OptionRequiredMessage.proto || echo "No messages found (OptionRequiredMessage.proto)"

@PATH=./bin:$$PATH; protoc --jsonschema_out=jsonschemas -I. --proto_path=${PROTO_PATH} ${PROTO_PATH}/OptionRequiredMessage.proto || echo "No messages found (Proto3Required.proto)"

Question: OneOf json schemas

I have a question regarding the limitations of protoc-gen-jsonschema or protobuf as a whole.

Basically I intend to write a proto file that could generate a structure similar to this:

{
    "description" : "schema validating people and vehicles",
    "type" : "object",
    "oneOf" : [
       {
        "type" : "object",
        "properties" : {
            "firstName" : {
                "type" : "string"
            },
            "lastName" : {
                "type" : "string"
            },
            "sport" : {
                "type" : "string"
            }
          }
      }, 
      {
        "type" : "object",
        "properties" : {
            "vehicle" : {
                "type" : "string"
            },
            "price" : {
                "type" : "integer"
            }
        },
        "additionalProperties":false
     }
    ]
}

Because if I try to create a similar thing in proto:

message SomeMessage {
    oneof oneof_type {
        TypeOne foo = 1;
        TypeTwo bar = 2;
    }
}

The generated json schema results into:

{
  "properties": {
    "foo": {
        // properties inside
    },
    "bar": {
        // properties inside
    }
  }
}

which to be fair isn't the same as having a true oneOf that doesn't depend on the keys.

Adding titles to generated JSON Schema

I would be happy to implement it myself, but it seems like a bigger change, so I wanted to consult first.

Our main use-case for the JSON Schema is generating a form, which can then be filled out by users. Something like this: https://rjsf-team.github.io/react-jsonschema-form/. This means every field has a title and a description to explain what the thing is.

Descriptions are already handled by this generator, but the titles are not. Right now we're trying to auto-generate these from field names, which works well 90% of the time, but can look wonky, e.g. when the name contains an acronym. So it would be great if there was a way to include the title in the proto and have it included appropriately in the schema.

What I was thinking was something like this:

// Thing
//
// This is a thing.
message Thing {
  ...
}

If the comment has a line separated by an empty comment line, it could be split into title (the first line) and description (the rest of it). Otherwise just use the whole thing as a description like it is now. So the above would generate:

{
  "Thing": {
    "title": "Thing",
    "description": "This is a thing.",
    ...
  },
  ...
}

Now that we have a generous allocation of protobuf options numbers we should use them!

I've recently secured numbers 1125-1129 for protoc-gen-jsonschema. This allows us to define options which control the behaviour of the generator to a finer degree, without having to mess about with CLI flags.

These could be used for the following:

  • FieldOptions
    • Required
    • Ignore
  • MessageOptions
    • AllFieldsRequired
    • DisallowAdditionalProperties
    • AllowNulls
    • Ignore
  • FileOptions
    • Ignore
    • Extention
  • EnumOptions
  • OneOfOptions
  • The first thing to do is to slightly change the name of the fieldoptions to accommodate this

Generator fails if proto has no package

I was attempting to generate a JSON schema for a new proto file. Even debug output only hinted at the problem:

$ protoc --jsonschema_out=debug:./jsonschema/ test.proto 
DEBU[0000] Converting input                             
DEBU[0000] Loading a message                             msg_name=Timestamp package_name=google.protobuf
DEBU[0000] Loading a message                             msg_name=TestMessage package_name=
DEBU[0000] Converting file                               filename=test.proto
DEBU[0000] Serializing code generator response          
WARN[0000] Failed to process code generator but successfully sent the error to protoc 
--jsonschema_out: protoc-gen-jsonschema: Plugin failed with status code 1.

Once I add a package to my proto, the generator works. I probably should have done this with my proto anyway, but since package is optional, the generator shouldn't die, or should at least have a more clear error message.

Licence

Thank you for your library
Can you specify the MIT license, Apache2 licence ... in a file LICENCE. txt
Thank

When specifying the disallow_bigints_as_strings, oneOf is still used instead of type

I like the ability to have big ints in scientific notation as strings, however with the option turned on, the type could be string or integer, that makes sense.

I am proposing that when the option is turned off, meaning it can only be a integer, what we revert to using the type field in JSON schema to denote the type of the property.

For instance:

"response_start": {
  "oneOf": [{
    "type": "integer"
  }]
},

should be

"response_start": {
  "type": "integer"
}

Allow Only JSON Field Names

There is no option to only allow JSON Field names, only an option to allow proto field names or proto AND json field names

Proposal

  • Current config: proto_and_json_fieldnames (includes both proto and json field names)
  • Requested config: json_fieldnames (excludes proto field names)

Relevant IF

jsonSchemaType.Properties.Set(fieldDesc.GetName(), recursedJSONSchemaType)
if c.UseProtoAndJSONFieldnames && fieldDesc.GetName() != fieldDesc.GetJsonName() {
jsonSchemaType.Properties.Set(fieldDesc.GetJsonName(), recursedJSONSchemaType)
}

Rough code

		if !c.UseJSONFieldnamesOnly {
			jsonSchemaType.Properties.Set(fieldDesc.GetName(), recursedJSONSchemaType)
		} else if c.UseJSONFieldnamesOnly || (c.UseProtoAndJSONFieldnames && fieldDesc.GetName() != fieldDesc.GetJsonName()) {
			jsonSchemaType.Properties.Set(fieldDesc.GetJsonName(), recursedJSONSchemaType)
		}

Add custom option (similar to hidden) that lets user specify specific required fields

Currently required is all or nothing for proto3 syntax. Would be really helpful to have a custom option that let's us specify if a single attirbute is required in json.

message RequiredFields {
    string not_required = 1 [(protoc.gen.jsonschema.required) = false];
    string required = 2 [(protoc.gen.jsonschema.required) = true];
    string default_not_required = 3; // by default proto3 is not required
}

Put oneOf Types Update Behind Feature Flag

If possible, could we put the oneOf feature (#57, #58) behind a feature flag? I believe the oneOf type is going to be a breaking change going forward, and isn't ideal for everyone. I'm running some language generators and the results aren't as great with this addition.

That might mean going back two commits and adding a forward commit with the feature behind a flag.

In general, it would be awesome if major features are behind a flag, such that any client can use the latest version of this tool and not see their generated schemas changing. Thank you!!

Misspelling of "extension"

I believe the word "extention", used throughout the repository, is a misspelling of "extension". I found this out when trying the "file_extention" option. It took me a few minutes to realize that it was written that way.

Thanks for this repository, by the way. Really useful!

Enums are not generated when file contains messages

When a file has both enum and message type definitions, the converter does not create files for the enum definitions.

The converter outputs a message:

WARN[0000] protoc-gen-jsonschema will create multiple ENUM schemas from one proto file  proto_filename=shared.proto schemas=2

However, when looking at the files written, there are no corresponding files for the enums.

Having a brief look at the converter, it seems like this is due to the conditional here, https://github.com/chrusty/protoc-gen-jsonschema/blob/master/internal/converter/converter.go#L143, which checks if the number of MessageTypes is zero and generates the enum types. Otherwise, it generates the message types.

Expected functionality would be that both the message and enum types in a file have corresponding output schema files.

Happy to provide more info, and possibly a MRE if needed.

Cheers!

No such package nor message in package

Im trying to understand the purpose of this message.

afaict this is triggered on conversion of every proto file where a package is specified (not specifying the package results in a warning) - and also on any "parent" packages if the package has more.than.one.part

i guess this can be safely ignored, but it creates a lot of noise when generating schemas from a large protocol set, and makes it diffficult to track any actual problems

Handle google.protobuf.Struct

Currently the generator doesn't handle the special type google.protobuf.Struct that well:

ACTUAL

message AuditLog {
  // Deprecated: Use `metadata` field instead.
  // Other service-specific data about the request, response, and other
  // activities.
  // When the JSON object represented here has a proto equivalent, the proto
  // name will be indicated in the `@type` property.
  google.protobuf.Struct service_data = 15;
}
...
"serviceData": {
  "properties": {
    "fields": {
      "additionalProperties": {
        "oneOf": [
          {
            "type": "array"
          },
          {
            "type": "boolean"
          },
          {
            "type": "number"
          },
          {
            "type": "object"
          },
          {
            "type": "string"
          }
        ],
        "description": "`Value` represents a dynamically typed value which can be either\n null, a number, a string, a boolean, a recursive struct value, or a\n list of values. A producer of value is expected to set one of that\n variants, absence of any variant indicates an error.\n\n The JSON representation for `Value` is JSON value."
      },
      "type": "object",
      "description": "Unordered map of dynamically typed values."
    }
  },
  "additionalProperties": true,
  "type": "object",
  "description": "Deprecated: Use `metadata` field instead.\n Other service-specific data about the request, response, and other\n activities.\n When the JSON object represented here has a proto equivalent, the proto\n name will be indicated in the `@type` property."
}
...

EXPECTED

...
"serviceData": {
  "type": "object",
  "description": "Deprecated: Use `metadata` field instead.\n Other service-specific data about the request, response, and other\n activities.\n When the JSON object represented here has a proto equivalent, the proto\n name will be indicated in the `@type` property."
}
...

I think we clean this up in postgen, but we should try to clean up the schema upstream.


case descriptor.FieldDescriptorProto_TYPE_GROUP, descriptor.FieldDescriptorProto_TYPE_MESSAGE:

Add comments to Enum fields

Hello, first thanks a lot for this plugin, really useful at no cost, great stuff !

However in when getting completion and associated documentation strings in an editor, Enum
fields do not have their associated comment. The comment presented is always the one for
the root Enum type name (eg. enum Channel {}).
I would gladly fix this if I could, but I don't quite have the time to get into the codebase, and it
seems to be quite specialized stuff. At the same time I feel that this won't take long to fix (having
written console applications, I might have a sixth sense for this), and this is why I'd like to request
this feature.
I will be immensely grateful for this, as for a project I rely on Enum fields that have important associated
documentation comments, which are absolutely paramount to users that will edit those files.

Thanks a lot for any news !

Support of optional fields in proto3

I am stuck using protoc-gen-jsonschema with optional fields. Here's a minimal example:

syntax = "proto3";

package myPackage;

message myMessage {
    optional float myFloat = 1.0;
}

When compiling this file using protoc --jsonschema_out... (protobuf-compiler 3.19.1), it tells:

$ protoc --jsonschema_out=. --proto_path=src src/minimalOptional.proto 
minimalOptional.proto: is a proto3 file that contains optional fields, but code generator protoc-gen-jsonschema hasn't been updated to support optional fields in proto3. Please ask the owner of this code generator to support proto3 optional.--jsonschema_out: 
$ 

So here I am :).

Are there any plans on supporting optional fields in proto3? See also: Protobuf v3.15.0

Remove new lines in descriptions.

Generated JSON schema descriptions contain new lines chars which are meaningless in a description, caused mainly by the parsed I think. For example in https://github.com/envoyproxy/envoy/blob/9d5627a0879b0a029e90515137c108e1d2884bfc/api/envoy/config/accesslog/v3/accesslog.proto#L33-L35

  // The name of the access log extension to instantiate.
  // The name must match one of the compiled in loggers.
  // See the :ref:`extensions listed in typed_config below <extension_category_envoy.access_loggers>` for the default list of available loggers.
  string name = 1;

generates something like:

"properties": {
    "name": {
        "type": "string",
        "description": "The name of the access log extension to instantiate.\n The name must match one of the compiled in loggers.\n See the :ref:`extensions listed in typed_config below \u003cextension_category_envoy.access_loggers\u003e` for the default list of available loggers."
    },

If not default case I believe it deserves an option.

I am happy to work it out myself.

Do not allow nullable fields

The schema generation for https://github.com/hypertrace/agent-config/blob/main/config.proto#L17 allows null values for all properties. Is there way to not allow nullable values?

protoc --jsonschema_out=:. config.proto

{
    "$schema": "http://json-schema.org/draft-04/schema#",
    "properties": {
        "serviceName": {
            "oneOf": [
                {
                    "type": "null"
                },
                {
                    "type": "string"
                }
            ]
        },
        "reporting": {
            "properties": {
                "address": {
                    "oneOf": [
                        {
                            "type": "null"
                        },
                        {
                            "type": "string"
                        }
                    ]
                },
                "secure": {
                    "oneOf": [
                        {
                            "type": "null"
                        },
                        {
                            "type": "boolean"
                        }
                    ]
                },
                "token": {
                    "oneOf": [
                        {
                            "type": "null"
                        },
                        {
                            "type": "string"
                        }
                    ]
                },
                "opa": {
                    "properties": {
                        "address": {
                            "oneOf": [
                                {
                                    "type": "null"
                                },
                                {
                                    "type": "string"
                                }
                            ]
                        },
                        "pollPeriodSeconds": {
                            "oneOf": [
                                {
                                    "type": "null"
                                },
                                {
                                    "type": "integer"
                                }
                            ]
                        }
                    },
                    "additionalProperties": true,
                    "type": "object",
                    "description": "opa describes the setting related to the Open Policy Agent"
                }
            },
            "additionalProperties": true,
            "type": "object",
            "description": "reporting holds the reporting settings for the agent"
        },
        "dataCapture": {
            "properties": {
                "httpHeaders": {
                    "additionalProperties": true,
                    "type": "object",
                    "description": "httpHeaders enables/disables the capture of the request/response headers in HTTP"
                },
                "httpBody": {
                    "additionalProperties": true,
                    "type": "object",
                    "description": "httpBody enables/disables the capture of the request/response body in HTTP"
                },
                "rpcMetadata": {
                    "additionalProperties": true,
                    "type": "object",
                    "description": "rpcMetadata enables/disables the capture of the request/response metadata in RPC"
                },
                "rpcBody": {
                    "additionalProperties": true,
                    "type": "object",
                    "description": "rpcBody enables/disables the capture of the request/response body in RPC"
                }
            },
            "additionalProperties": true,
            "type": "object",
            "description": "dataCapture describes the data being captured by instrumentation"
        },
        "propagationFormats": {
            "items": {
                "oneOf": [
                    {
                        "type": "string"
                    },
                    {
                        "type": "integer"
                    }
                ]
            },
            "type": "array",
            "description": "propagationFormats list the supported propagation formats"
        }
    },
    "additionalProperties": true,
    "type": "object",
    "description": "AgentConfig covers the config for agents.\n The config uses wrappers for primitive types to allow nullable values.\n The nullable values are used for instance to explicitly disable data capture or secure connection.\n Since the wrappers change structure of the objects the marshalling and unmarshalling\n have to be done via protobuf marshallers.",
    "definitions": {
        "org.hypertrace.agent.config.Message": {
            "$schema": "http://json-schema.org/draft-04/schema#",
            "properties": {
                "request": {
                    "oneOf": [
                        {
                            "type": "null"
                        },
                        {
                            "type": "boolean"
                        }
                    ]
                },
                "response": {
                    "oneOf": [
                        {
                            "type": "null"
                        },
                        {
                            "type": "boolean"
                        }
                    ]
                }
            },
            "additionalProperties": true,
            "type": "object",
            "description": "Message describes what message should be considered for certain DataCapture option",
            "id": "org.hypertrace.agent.config.Message"
        }
    }
}

google.protobuf.Timestamp needs to be converted to string in schema instead of an object

In https://developers.google.com/protocol-buffers/docs/proto3#json , the correct type for a Timestamp in JSON is string instead of an object. When using protoc-gen-jsonschema, google.protobuf.Timestamp will create:

"timestamp": {
    "properties": {
        "nanos": {
            "type": "integer"
        },
        "seconds": {
            "oneOf": [
                {
                    "type": "integer"
                },
                {
                    "type": "string"
                }
            ]
        }
    },
    "additionalProperties": true,
    "type": "object"
},

instead, it should be string:

"timestamp": {
    "type": "string"
},

please can you fix, thank you very much.

type orderedmap.OrderedMap issue while installing

I am getting an error when installing the package

# github.com/chrusty/protoc-gen-jsonschema/internal/converter
../../github.com/chrusty/protoc-gen-jsonschema/internal/converter/types.go:68:3: cannot use make(map[string]*jsonschema.Type) (type map[string]*jsonschema.Type) as type *orderedmap.OrderedMap in field value
../../github.com/chrusty/protoc-gen-jsonschema/internal/converter/types.go:226:49: invalid operation: recursedJSONSchemaType.Properties["value"] (type *orderedmap.OrderedMap does not support indexing)
../../github.com/chrusty/protoc-gen-jsonschema/internal/converter/types.go:231:83: invalid operation: recursedJSONSchemaType.Properties["value"] (type *orderedmap.OrderedMap does not support indexing)
../../github.com/chrusty/protoc-gen-jsonschema/internal/converter/types.go:265:3: cannot use make(map[string]*jsonschema.Type) (type map[string]*jsonschema.Type) as type *orderedmap.OrderedMap in field value
../../github.com/chrusty/protoc-gen-jsonschema/internal/converter/types.go:298:28: invalid operation: jsonSchemaType.Properties[fieldDesc.GetName()] (type *orderedmap.OrderedMap does not support indexing)
../../github.com/chrusty/protoc-gen-jsonschema/internal/converter/types.go:300:29: invalid operation: jsonSchemaType.Properties[fieldDesc.GetJsonName()] (type *orderedmap.OrderedMap does not support indexing)

I checked the code in types.go file, and found that it is related to jsonschema project as Properties definition has changed from "map[string]*Type{}" to "*orderedmap.OrderedMap" as per this commit.

Is there any plan for a new release to support this change?

One big schema

Didn't find a way to generate not separate .jsonschema files, but one.

Maybe there is one?

google.protobuf.Duration Should Be String

The type google.protobuf.Duration should be a string, not an object:

// Build event data for Google Cloud Platform API operations.
message BuildEventData {
  // The TTL starts ticking from create_time.
  google.protobuf.Duration queue_ttl = 1;
}

Expected

    "queueTtl": {
      "type": "string",
      "description": "TTL in queue for this build. If provided and the build is enqueued longer\n than this value, the build will expire and the build status will be\n `EXPIRED`.\n\n The TTL starts ticking from create_time."
    },

Actual

    "queueTtl": {
      "properties": {
        "seconds": {
          "type": "integer",
          "description": "Signed seconds of the span of time. Must be from -315,576,000,000\n to +315,576,000,000 inclusive. Note: these bounds are computed from:\n 60 sec/min * 60 min/hr * 24 hr/day * 365.25 days/year * 10000 years"
        },
        "nanos": {
          "type": "integer",
          "description": "Signed fractions of a second at nanosecond resolution of the span\n of time. Durations less than one second are represented with a 0\n `seconds` field and a positive or negative `nanos` field. For durations\n of one second or more, a non-zero value for the `nanos` field must be\n of the same sign as the `seconds` field. Must be from -999,999,999\n to +999,999,999 inclusive."
        }
      },
      "additionalProperties": true,
      "type": "object",
      "description": "..."
    },

References

Chrusty code

Protobuf code comment

https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/duration.proto#L92

// In JSON format, the Duration type is encoded as a string rather than an
// object, where the string ends in the suffix "s" (indicating seconds) and
// is preceded by the number of seconds, with nanoseconds expressed as
// fractional seconds. For example, 3 seconds with 0 nanoseconds should be
// encoded in JSON format as "3s", while 3 seconds and 1 nanosecond should
// be expressed in JSON format as "3.000000001s", and 3 seconds and 1
// microsecond should be expressed in JSON format as "3.000001s".

With NestedMessage, syntaxe for $ref (URI-reference) should start with #

Your tool is great, very helpfull.
When I take the Enumception.jsonschema it seems that the syntaxe for the reuse of structure PayloadMessage seems incorrect (at least in the faker tool that I used...) the line is :
"$ref": "samples.PayloadMessage",
and should be
"$ref": "#/samples.PayloadMessage",

Otherwise the reference to the definition located in the same file is not found.
Here is the doc explaining this :
https://json-schema.org/understanding-json-schema/structuring.html

proto3 "bytes" field -> "contentEncoding=base64 string", not just "string"

It would be nice if a proto3 bytes field stated in the JSON schema that the string is base64 encoded.

Reference

Expected

  • Add "contentEncoding": "base64" to byte fields.

Split up the proto type cases to something like this:

	case descriptor.FieldDescriptorProto_TYPE_STRING:
		if c.AllowNullValues {
			jsonSchemaType.OneOf = []*jsonschema.Type{
				{Type: gojsonschema.TYPE_NULL},
				{Type: gojsonschema.TYPE_STRING},
			}
		} else {
			jsonSchemaType.Type = gojsonschema.TYPE_STRING
		}

	case descriptor.FieldDescriptorProto_TYPE_BYTES:
		if c.AllowNullValues {
			jsonSchemaType.OneOf = []*jsonschema.Type{
				{Type: gojsonschema.TYPE_NULL},
				{Type: gojsonschema.TYPE_STRING},
			}
		} else {
			jsonSchemaType.Type = gojsonschema.TYPE_STRING
			jsonSchemaType.Format = "binary"
		}

https://github.com/chrusty/protoc-gen-jsonschema/blob/master/internal/converter/types.go#L125-L134

google.protobuf.Value causes stack overflow

Hello, thanks for this great work!

When protoc-gen-jsonschema processes a proto file with the well-known type Value, it causes a stack overflow:

syntax = "proto3";
package overflower; 

import "google/protobuf/struct.proto"; 

option go_package = "..."; 

message Overflower {
  google.protobuf.Value arg = 1;
}

Is this something you plan to support?

panic:

 $ protoc -I=proto --jsonschema_out=tmp proto/killer.proto                                                                   
INFO[0000] Generating JSON-schema for MESSAGE            jsonschema_filename=Overflower.jsonschema msg_name=Overflower proto_filename=killer.proto                                    
runtime: goroutine stack exceeds 1000000000-byte limit                                                                                                                                fatal error: stack overflow                                                                                                                                                           
                                                                                                                                                                                      runtime stack:                                                                                                                                                                        
runtime.throw(0x902408, 0xe)                                                                                                                                                                  /usr/lib/go-1.13/src/runtime/panic.go:774 +0x72                                                                                                                               
runtime.newstack()                                                                                                                                                                    
        /usr/lib/go-1.13/src/runtime/stack.go:1046 +0x6e9                                                                                                                             runtime.morestack()                                                                                                                                                                   
        /usr/lib/go-1.13/src/runtime/asm_amd64.s:449 +0x8f                                                                                                                                                                                                                                                                                                                  
goroutine 1 [running]:                                                                                                                                                                runtime.ifaceeq(0x9afc80, 0xc0001ce000, 0xc0001ce000, 0x0)                                                                                                                            
        /usr/lib/go-1.13/src/runtime/alg.go:234 +0x111 fp=0xc02c514300 sp=0xc02c5142f8 pc=0x4042f1                                                                                    
google.golang.org/protobuf/internal/impl.(*MessageInfo).checkField(0xc0000f64e0, 0x9afc80, 0xc0001ce000, 0xc02c514468, 0xc02c514468, 0x5c4a1b)                                                /home/jeff/go/pkg/mod/google.golang.org/[email protected]/internal/impl/message_reflect.go:340 +0xc1 fp=0xc02c514428 sp=0xc02c514300 pc=0x6361d1                               
google.golang.org/protobuf/internal/impl.(*messageState).Has(0xc0001ec100, 0x9afc80, 0xc0001ce000, 0xc0001ce000)                                                                              /home/jeff/go/pkg/mod/google.golang.org/[email protected]/internal/impl/message_reflect_gen.go:66 +0x6b fp=0xc02c514478 sp=0xc02c514428 pc=0x63a0bb                            
github.com/golang/protobuf/proto.(*textWriter).writeMessage(0xc0271abe60, 0x9ac620, 0xc0001ec100, 0x0, 0x0)                                                                                   /home/jeff/go/pkg/mod/github.com/golang/[email protected]/proto/text_encode.go:278 +0x925 fp=0xc02c5145f8 sp=0xc02c514478 pc=0x650d75                                           
github.com/golang/protobuf/proto.(*textWriter).writeSingularValue(0xc0271abe60, 0x8e6aa0, 0xc0001ec100, 0x0, 0x9afc80, 0xc00011f2e0, 0x0, 0x0)                                        
        /home/jeff/go/pkg/mod/github.com/golang/[email protected]/proto/text_encode.go:390 +0x664 fp=0xc02c514690 sp=0xc02c5145f8 pc=0x6515e4                                           github.com/golang/protobuf/proto.(*textWriter).writeMessage(0xc0271abe60, 0x9ac620, 0xc0001e0200, 0x0, 0x0)                                                                           
        /home/jeff/go/pkg/mod/github.com/golang/[email protected]/proto/text_encode.go:288 +0x2dd fp=0xc02c514810 sp=0xc02c514690 pc=0x65072d                                           github.com/golang/protobuf/proto.(*TextMarshaler).marshal(0xcdb16c, 0x9a3f20, 0xc0001e0200, 0xc0002044d8, 0xcdc300, 0xc0000bb087, 0xc0000d11c8, 0x8)                                  
        /home/jeff/go/pkg/mod/github.com/golang/[email protected]/proto/text_encode.go:86 +0x18a fp=0xc02c514870 sp=0xc02c514810 pc=0x64f18a                                            github.com/golang/protobuf/proto.(*TextMarshaler).Text(...)                                                                                                                           
        /home/jeff/go/pkg/mod/github.com/golang/[email protected]/proto/text_encode.go:44                                                                                               
github.com/golang/protobuf/proto.MarshalTextString(...)                                                                                                                               
        /home/jeff/go/pkg/mod/github.com/golang/[email protected]/proto/text_encode.go:100
github.com/chrusty/protoc-gen-jsonschema/internal/converter.(*Converter).recursiveConvertMessageType(0xc04c513f38, 0xc000216240, 0xc0001e0200, 0xc000212260, 0x10, 0xc000216[288/3963]
0x203001, 0x30, 0x203009)
        /home/jeff/go/pkg/mod/github.com/chrusty/[email protected]/internal/converter/types.go:463 +0x2b1 fp=0xc02c514db8 sp=0xc02c514870 pc=0x
8055e1
github.com/chrusty/protoc-gen-jsonschema/internal/converter.(*Converter).convertField(0xc04c513f38, 0xc000216240, 0xc0001ecf00, 0xc0001e0600, 0xc0002163f0, 0xc018059400, 0x0, 0x0)
        /home/jeff/go/pkg/mod/github.com/chrusty/[email protected]/internal/converter/types.go:225 +0x868 fp=0xc02c5150a0 sp=0xc02c514db8 pc=0x
8025b8
github.com/chrusty/protoc-gen-jsonschema/internal/converter.(*Converter).recursiveConvertMessageType(0xc04c513f38, 0xc000216240, 0xc0001e0600, 0xc000212260, 0x10, 0xc0002163f0, 0x0, 
0x203001, 0x30, 0x203009)
        /home/jeff/go/pkg/mod/github.com/chrusty/[email protected]/internal/converter/types.go:465 +0x41c fp=0xc02c5155e8 sp=0xc02c5150a0 pc=0x
80574c
github.com/chrusty/protoc-gen-jsonschema/internal/converter.(*Converter).convertField(0xc04c513f38, 0xc000216240, 0xc0001ec900, 0xc0001e0500, 0xc0002163f0, 0xc02820f010, 0x0, 0x1)
        /home/jeff/go/pkg/mod/github.com/chrusty/[email protected]/internal/converter/types.go:225 +0x868 fp=0xc02c5158d0 sp=0xc02c5155e8 pc=0x
8025b8
...

Thanks!

*structpb.Value type fields are not correctly interpreted to JSON Schema

Problem

With newer protobuf packages the struct fields that are of type *structpb.Value are outputted to JSON schema only as object and it does not allow other types that *structpb.Value supports (null, a number, a string, a boolean, a recursive struct value, or a list of values). This results in validation errors when the field value in JSON is for example string.

Protobuf package versions:

  • google.golang.org/protobuf v1.25.0
  • github.com/golang/protobuf v1.4.3

In these newer versions there is custom UnmarshalJSON/MarshalJSON methods for reading/writing the value correctly.

Fix suggestion

The *structpb.Value fields should be written with "oneOf" field containing all supported types by structpb.Value.

Resulting field:

"value": {
"oneOf": [
{
"type": "number"
},
{
"type": "string"
},
{
"type": "boolean"
},
{
"type": "object"
},
{
"type": "array"
}
]
},

(sorry about the formatting of json)

'null' is not included because that can be controlled with allow nulls flag.

Suggested fix to internal/converter/types.go:430

  case "Value":
  	t := &jsonschema.Type{
  		OneOf: []*jsonschema.Type{
  			{Type: gojsonschema.TYPE_NUMBER},
  			{Type: gojsonschema.TYPE_STRING},
  			{Type: gojsonschema.TYPE_BOOLEAN},
  			{Type: gojsonschema.TYPE_OBJECT},
  			{Type: gojsonschema.TYPE_ARRAY},
  		},
  	}

  	if c.AllowNullValues {
  		t.OneOf = append(t.OneOf, &jsonschema.Type{
  			Type: gojsonschema.TYPE_NULL,
  		})
  	}

  	return t, nil
  }

google.protobuf.Any converted into value/binaryEncoding: base64

Hello,
I am doing some tests with this nice library, but for what I have seen so far, google.protobuf.Any is associated to a base64 field in the json schema. I now that this is what happens when you try to serialize to json a protobuf including any, using classical json.Marshal(X). But, doing in this way you don't have anymore a readable file.
Instead, there is another way to serialize, using jsonpb.Unmarshal, that is able to serialize keeping all in clear.

It would be useful to have a similar feature here, that is not forcing google.protobuf.Any elements to be associated to a base64, maybe with a flag to describe how we want to handle similar situations. I don't think to have seen anything similar.

Could you have a look?

Thanks,
Marco

GoogleValue type descriptions aren't added to JSON Schema

(Creating this issue for tracking purposes)

The descriptions for well known types are not added to the JSON schema. Example:

message GoogleValue {
  // My desc.
  google.protobuf.Value arg = 1;
}

We should expect arg to be a property and have a description.

"properties": {
    "arg": {
        "description": "My desc."
        ...
    }
}

I think there is maybe something missing with:

func formatDescription(sl *descriptor.SourceCodeInfo_Location) string {

https://github.com/chrusty/protoc-gen-jsonschema/blob/1.0.0/internal/converter/types.go#L527-L541

Int64 JSON Mapping Should Be String, Not OneOf

All int64 types (and variants) should be mapped to the string type, not a oneOf: [ {type: 'string'},{type: 'integer'}].

This should be done because the JSON mapping for int64 is type string.

References

JSON Mapping:

https://developers.google.com/protocol-buffers/docs/proto3#json

Code:

case descriptor.FieldDescriptorProto_TYPE_INT64,

Solutions

Related Issues

Mapping to a oneOf is causing a related issue for a jsonschema 2 golang file (using Quicktype): googleapis/google-cloudevents-go#22 (comment)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.