salsify / avro-builder Goto Github PK

View Code? Open in Web Editor NEW

102.0 102.0 6.0 166 KB

Ruby DSL to create Avro schemas

License: MIT License

Ruby 99.91% Shell 0.09%

avro gem hacktoberfest

avro-builder's People

Contributors

Stargazers

Watchers

Forkers

anujbiyani krule xuwupeng2000 kemper olleolleolle

avro-builder's Issues

Avro::Builder::DSL#lookup_named_type should not be public

See #12 for an initial implementation to hide this method.

how to share code in the DSL

Is there a way to import (or otherwize) bring in shared fields into a record? (if not..this would be soooo great)

e.g, like this

record :foobar do
  required :highlight_id, :string
  import '../shared/all_events' 
end

byte storage for dates

which would be the most efficient in terms of byte storage (and by how much) for dates in avro/kafka? is there anything else w/ how to store the date time?

(1) timestamp-millis (long).
(2) rfc3339 (string) for a datetime.

Thanks.

Include dsl file/line numbers in stack trace

Errors in DSL files can result in stack traces like this:

ArgumentError: wrong number of arguments (given 1, expected 2..3)
/Users/jturkel/.rvm/gems/ruby-2.3.1@alerts-service/gems/avro-builder-0.8.0/lib/avro/builder/dsl.rb:31:in `instance_eval'
(eval):11:in `block in initialize'
/Users/jturkel/.rvm/gems/ruby-2.3.1@alerts-service/gems/avro-builder-0.8.0/lib/avro/builder/type_factory.rb:38:in `instance_eval'
/Users/jturkel/.rvm/gems/ruby-2.3.1@alerts-service/gems/avro-builder-0.8.0/lib/avro/builder/type_factory.rb:38:in `block in create_and_configure_builtin_type'
/Users/jturkel/.rvm/gems/ruby-2.3.1@alerts-service/gems/avro-builder-0.8.0/lib/avro/builder/type_factory.rb:35:in `tap'
/Users/jturkel/.rvm/gems/ruby-2.3.1@alerts-service/gems/avro-builder-0.8.0/lib/avro/builder/type_factory.rb:35:in `create_and_configure_builtin_type'
/Users/jturkel/.rvm/gems/ruby-2.3.1@alerts-service/gems/avro-builder-0.8.0/lib/avro/builder/dsl.rb:86:in `create_named_type'
/Users/jturkel/.rvm/gems/ruby-2.3.1@alerts-service/gems/avro-builder-0.8.0/lib/avro/builder/dsl.rb:36:in `record'
(eval):3:in `initialize'
/Users/jturkel/.rvm/gems/ruby-2.3.1@alerts-service/gems/avro-builder-0.8.0/lib/avro/builder/dsl.rb:31:in `instance_eval'
/Users/jturkel/.rvm/gems/ruby-2.3.1@alerts-service/gems/avro-builder-0.8.0/lib/avro/builder/dsl.rb:31:in `initialize'
/Users/jturkel/.rvm/gems/ruby-2.3.1@alerts-service/gems/avro-builder-0.8.0/lib/avro/builder.rb:10:in `new'
/Users/jturkel/.rvm/gems/ruby-2.3.1@alerts-service/gems/avro-builder-0.8.0/lib/avro/builder.rb:10:in `build'
/Users/jturkel/.rvm/gems/ruby-2.3.1@alerts-service/gems/salsify_avro-0.21.0/lib/salsify_avro/rake/avro_generate_task.rb:37:in `block (3 levels) in define'
/Users/jturkel/.rvm/gems/ruby-2.3.1@alerts-service/gems/salsify_avro-0.21.0/lib/salsify_avro/rake/avro_generate_task.rb:34:in `each'
/Users/jturkel/.rvm/gems/ruby-2.3.1@alerts-service/gems/salsify_avro-0.21.0/lib/salsify_avro/rake/avro_generate_task.rb:34:in `block (2 levels) in define'
/Users/jturkel/.rvm/gems/ruby-2.3.1@alerts-service/gems/rake-11.2.2/exe/rake:27:in `<top (required)>'
/Users/jturkel/.rvm/gems/ruby-2.3.1@alerts-service/bin/ruby_executable_hooks:15:in `eval'
/Users/jturkel/.rvm/gems/ruby-2.3.1@alerts-service/bin/ruby_executable_hooks:15:in `<main>'
Tasks: TOP => avro:generate
(See full trace by running task with --trace)

It would be helpful if calls to instance_eval included the source file to more easily track errors back to a line in a DSL file e.g. /Users/jturkel/salsify/alerts-service/avro/dsl/com/salsify/alerts/events/alert_created_event_value.rb rather than (eval).

Generated schema doesn't pretty print

#56 introduced mutli_json which broke pretty printing of generated schemas in applications that include Rails but not oj orYajl. There appears to be something about the Rails dependency that causes the MultiJson JSON gem adapter not to support pretty printing. Here's a reproducible test case:

require 'bundler/inline'

gemfile(true) do
  source 'https://rubygems.org'
  # gem 'oj'
  # gem 'yajl-ruby'
  gem 'multi_json'
  gem 'rspec'
  # Tests pass when Rails is not present
  gem 'rails'
end

require 'rspec/autorun'
require 'multi_json'

describe do
  let(:data) do
    { foo: { bar: 'baz' } }
  end

  let(:pretty_data) do
    "{\n  \"foo\": {\n    \"bar\": \"baz\"\n  }\n}"
  end

  it "pretty prints" do
    puts "Using adapter #{MultiJson.adapter}"
    expect(MultiJson.dump(data, pretty: true)).to eq(pretty_data)
  end
end

Allow types to be marked as "abstract"

This would be used to skip generating an Avro JSON schema file (.avsc) for abstract types.

Validate required arguments to DSL functions

Attempting to generate a schema for the following DSL (note the values option isn't specified for the map) raises a NoMethodError

record :my_event do
  required :my_map, :map
end

The stack trace is:

NoMethodError: undefined method `serialize' for nil:NilClass
/Users/jturkel/.rvm/gems/ruby-2.2.4@dandelion/gems/avro-builder-0.3.1/lib/avro/builder/types/map_type.rb:20:in `serialize'
/Users/jturkel/.rvm/gems/ruby-2.2.4@dandelion/gems/avro-builder-0.3.1/lib/avro/builder/field.rb:80:in `serialized_type'
/Users/jturkel/.rvm/gems/ruby-2.2.4@dandelion/gems/avro-builder-0.3.1/lib/avro/builder/field.rb:67:in `serialize'
/Users/jturkel/.rvm/gems/ruby-2.2.4@dandelion/gems/avro-builder-0.3.1/lib/avro/builder/types/record_type.rb:57:in `block (2 levels) in to_h'
/Users/jturkel/.rvm/gems/ruby-2.2.4@dandelion/gems/avro-builder-0.3.1/lib/avro/builder/types/record_type.rb:57:in `map'
/Users/jturkel/.rvm/gems/ruby-2.2.4@dandelion/gems/avro-builder-0.3.1/lib/avro/builder/types/record_type.rb:57:in `block in to_h'
/Users/jturkel/.rvm/gems/ruby-2.2.4@dandelion/gems/avro-builder-0.3.1/lib/avro/builder/schema_serializer_reference_state.rb:22:in `definition_or_reference'
/Users/jturkel/.rvm/gems/ruby-2.2.4@dandelion/gems/avro-builder-0.3.1/lib/avro/builder/types/record_type.rb:50:in `to_h'
/Users/jturkel/.rvm/gems/ruby-2.2.4@dandelion/gems/avro-builder-0.3.1/lib/avro/builder/dsl.rb:80:in `to_h'
/Users/jturkel/.rvm/gems/ruby-2.2.4@dandelion/gems/avro-builder-0.3.1/lib/avro/builder/dsl.rb:85:in `to_json'
/Users/jturkel/.rvm/gems/ruby-2.2.4@dandelion/gems/avro-builder-0.3.1/lib/avro/builder.rb:9:in `build'
/Users/jturkel/.rvm/gems/ruby-2.2.4@dandelion/gems/salsify_avro-0.3.0/lib/salsify_avro/rake/avro_generate_task.rb:37:in `block (3 levels) in define'
/Users/jturkel/.rvm/gems/ruby-2.2.4@dandelion/gems/salsify_avro-0.3.0/lib/salsify_avro/rake/avro_generate_task.rb:34:in `each'
/Users/jturkel/.rvm/gems/ruby-2.2.4@dandelion/gems/salsify_avro-0.3.0/lib/salsify_avro/rake/avro_generate_task.rb:34:in `block (2 levels) in define'

Support "order" option for a field

See http://avro.apache.org/docs/current/spec.html#order

Support for `union`

Optional fields are supported as a union of null and another type.

An open question is how union should interact with required/optional.

Not able to use same abstract record for multiple fields

I have a schema file, in which I have two fields having the same schema, array of records

    optional :categories, :array, items: :name_image_hash
    optional :cart_images, :array, items: :name_image_hash

name_image_hash looks like

record "name_image_hash" do
  required :url, :string
  required :name, :string
  abstract true
end

and it is generating schema like

{
    "name": "cart_images",
    "type": [
        "null",
        {
            "type": "array",
            "items": {
                "type": "record",
                "name": "name_image_hash",
                "fields": [
                    {
                        "name": "url",
                        "type": "string"
                    },
                    {
                        "name": "name",
                        "type": "string"
                    }
                ]
            }
        }
    ],
    "default": null
},
{
    "name": "categories",
    "type": [
        "null",
        {
            "type": "array",
            "items": "name_image_hash"
        }
    ],
    "default": null
}

cart_images rendered properly, but categories doesn't.

I am doing something wrong, or is it a bug ??

Earlier, I wanted to do this inline, but I am not able to get the right syntax.

can certain file extensions be skipped in the dsl rake task generation

Is there a way to have avro-builder skip certain file name masks (like *_swagger.rb) when it's generating avsc files?

I ask because i'd like to keep other (related) files side by side w/ the foobar.rb avro dsl file. like foobar_swagger.rb)

thanks!

Support nested records

Support for logical types

Currently, I don't think these are supported in the avro gem.

Allow Avro generate rake task to run for a single DSL file

It would be useful to optionally accept a single DSL file (or list of files) to regenerate instead of reprocessing all files.

Additional validations

By default generated schemas are validated by attempting to parse using the avro gem.

This is a placeholder for now to collect other validations that would be useful to give better feedback for disallowed situations.

Support `error` in addition to `record`

Is this useful outside protocols?

rake task exception when building dsl

Rake task fails to build avro files from dsl. Using ruby-2.6.5.

Building this DSL

# <rails_root>/avro/dsl/example.rb
namespace 'com.example'

fixed :password, 8

enum :user_type, :ADMIN, :REGULAR

record :user do
  required :id, :long
  required :user_name, :string
  required :type, :user_type, default: :REGULAR
  required :pw, :password
  optional :full_name, :string
  required :nicknames, :array, items: :string
  required :permissions, :map, values: :bytes
end

running be rake avro:generate generates this error

Generating Avro schema from /Users/steveburkett/openstax/event-capture-api/avro/dsl/nudged.rb
rake aborted!
ArgumentError: wrong number of arguments (given 1, expected 0)
/Users/steveburkett/.rvm/gems/ruby-2.6.5@event-capture-api/gems/avro-builder-0.17.0/lib/avro/builder/dsl.rb:74:in `to_h'
/Users/steveburkett/.rvm/gems/ruby-2.6.5@event-capture-api/gems/avro-builder-0.17.0/lib/avro/builder/dsl.rb:74:in `to_h'
/Users/steveburkett/.rvm/gems/ruby-2.6.5@event-capture-api/gems/avro-builder-0.17.0/lib/avro/builder/dsl.rb:79:in `to_json'
/Users/steveburkett/.rvm/gems/ruby-2.6.5@event-capture-api/gems/avro-builder-0.17.0/lib/avro/builder/rake/avro_generate_task.rb:48:in `block (3 levels) in define'
/Users/steveburkett/.rvm/gems/ruby-2.6.5@event-capture-api/gems/avro-builder-0.17.0/lib/avro/builder/rake/avro_generate_task.rb:38:in `each'
/Users/steveburkett/.rvm/gems/ruby-2.6.5@event-capture-api/gems/avro-builder-0.17.0/lib/avro/builder/rake/avro_generate_task.rb:38:in `block (2 levels) in define'
/Users/steveburkett/.rvm/gems/ruby-2.6.5@event-capture-api/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
/Users/steveburkett/.rvm/gems/ruby-2.6.5@event-capture-api/bin/ruby_executable_hooks:24:in `eval'
/Users/steveburkett/.rvm/gems/ruby-2.6.5@event-capture-api/bin/ruby_executable_hooks:24:in `<main>'

Issue with Ruby 2.6.5

The following basic example works with Ruby 2.6.2 but not with 2.6.5

require 'avro/builder'

Avro::Builder.build do
	namespace 'com.example'

	fixed :password, 8

	enum :user_type, :ADMIN, :REGULAR

	record :user do
	  required :id, :long
	  required :user_name, :string
	  required :type, :user_type, default: :REGULAR
	  required :pw, :password
	  optional :full_name, :string
	  required :nicknames, :array, items: :string
	  required :permissions, :map, values: :bytes
	end
end

Error:

Traceback (most recent call last):
 2: from schemas/example.rb:3:in `<main>'
 1: from /Users/sushma/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/avro-builder-0.17.0/lib/avro/builder.rb:17:in `build'
/Users/sushma/.asdf/installs/ruby/2.6.5/lib/ruby/gems/2.6.0/gems/avro-builder-0.17.0/lib/avro/builder/dsl.rb:80:in `to_json': 
uninitialized constant Avro::Builder::DSL::JSON (NameError)

array of records?

I'm trying to write a record where one of the fields is an array where each item is a record. I can do this if I don't inline the inner-most record, e.g.

record :record_in_array do
  required :name, :string
end

record :top_level do
  required :array_of_records, :array, items: :record_in_array
end

Is it possible to inline the record_in_array record? The following errors:

record :top_level do
  required :array_of_records, :array, items: :record do
    required :name, :string
  end
end

#> NoMethodError: undefined method `required' for #<Avro::Builder::Field:0x007fdd3437d278>

how to make avro-builder and avro_turf cooperate?

When i build a dsl like

namespace 'org.openstax.interactions'

record :nudged do
  required :user_uuid, :string 
  ...

it creates avsc (in ./avro/schema/nudged.avsc) like this

{
  "type": "record",
  "name": "nudged",
  "namespace": "org.openstax.interactions",
  "fields": [
    {
      "name": "user_uuid",

However, using the schema store from avro_turf, setting the schema store:

        AvroTurf::Messaging.new(
          registry_url: 'http://localhost:8081/',
          schemas_path: 'avro/schema'
        )

This fails to work when i try to encode:

AvroTurf::Messaging.new(...).encode(data, schema_name: "org.openstax.interactions.nudged")

due to this error

    "exception": "#<AvroTurf::SchemaNotFoundError: could not find Avro schema at `avro/schema/org/openstax/interactions/nudged.avsc'>",

Is there a way to tell avro-builder to build the schema in directory structure that avroturf expects, or to tell avroturf to look for schemas in the way avro-builder outputs?

thanks!!

how to customize location of dsls for the rake task?

Hi there,

How do i get the rake task to find DSLs under a Rails.root directory called 'oxavro'? i cant get this to work.

e.g., i put this

Avro::Builder.add_load_path('oxavro/dsl')

inside an initializer, but still not working. i think this gem is still using Rails.root/avro as the top level dir.

Infer namespace from directory structure

When loading a DSL from a file infer the namespace from the file path relative to a configured load path.

The namespace should still be overridable in the DSL.

Fully document the DSL

This should include all the methods in the DSL, where they can be called, and options accepted.

Perhaps this should be a separate document instead of making the README extremely long.