The elitzur from spotify

Failing AvroConverter conversion for optional-wrapped validation type

Upgrading to scio 0.95 and running into a previously-worked-around AvroConverter issue.

I have a validation type:

  case class Country(data: String) extends BaseValidationType[String] {
    override def checkValid: Boolean = ...
  }

And a case class that uses it:

case class MyClass(
  ...
  country: Option[Country],
  ...
)

And an attempt to convert:

implicitly[AvroConverter[MyClass]].toAvro(a, AvroMyClass.getClassSchema).asInstanceOf[AvroMyClass]

To this schema:


{
    {
      "name": "country",
      "type": [
        "null",
        "string"
      ],
      "default": null
    },

And deep in the internals of DerivedConverter I’m seeing Country being evaluated against a schema of "string", which should work but which instead is passed to GenericRecordBuilder and results in:

[info]   org.apache.avro.AvroRuntimeException: Not a record: "string"
[info]   at org.apache.avro.Schema.getFields(Schema.java:220)
[info]   at org.apache.avro.data.RecordBuilderBase.<init>(RecordBuilderBase.java:54)
[info]   at org.apache.avro.generic.GenericRecordBuilder.<init>(GenericRecordBuilder.java:38)
[info]   at com.spotify.elitzur.converters.avro.DerivedConverter.toAvro(AvroConverter.scala:319)
[info]   at com.spotify.elitzur.converters.avro.AvroOptionConverter.toAvro(AvroConverter.scala:138)
[info]   at com.spotify.elitzur.converters.avro.AvroOptionConverter.toAvro(AvroConverter.scala:125)
[info]   at com.spotify.elitzur.converters.avro.DerivedConverter.toAvro(AvroConverter.scala:333)

I had previously worked around this by round-tripping to bytes which no longer works:

    def toAvro[To <: SpecificRecordBase](toSchema: Schema)(implicit ac: AvroConverter[From]): To = {
      val decoder = new BinaryMessageDecoder[To](new SpecificData(), toSchema)

      val rec: GenericData.Record = ac.toAvro(instance, toSchema).asInstanceOf[GenericData.Record]
      val bytes: ByteBuffer = new BinaryMessageEncoder[GenericData.Record](new GenericData(), toSchema).encode(rec)
      val ret: To = decoder.decode(bytes)
      ret
    }

My suspicion is that the Option wrapper is breaking an otherwise working path

spotify / elitzur Goto Github PK

elitzur's People

Contributors

Stargazers

Watchers

Forkers

elitzur's Issues

Failing AvroConverter conversion for optional-wrapped validation type

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent