Doubt regarding deserialization about schema_registry_converter HOT 9 CLOSED

Sounds a bit similar like how for Java you can use the specific.avro.reader option. With which it will use reflection with the full name, to get the schema from the classpath.
But Rust doesn't have reflection. What if there was a SchemaProvidedDecoder where you could put in schema's before using it to decode, and it will try to use the schema with the same name each time? In this case giving an error when there is no schema with the correct name, or the schema is incompatible?

from schema_registry_converter.

@PauloAOliveira could you please let me know what you think of the proposed solution? In December I probably do some work for a next release and I could add this.

from schema_registry_converter.

Seems like apache/avro#1379 might be able to solve this, and the only change needed is updating the Avro dependency once it has been fixed and released? What do you think?

from schema_registry_converter.

I'm trying to debug this as we speak since the PR you linked above does break the reader tests...

Unfortunately, all this is still obscure to me so I'm wasting a lot of time trying to understand why this happens.

What I'm quite not sure to understand @PauloAOliveira is that this crate is using the schema id of every message to deserialize it, so you should never be in the position I describe in the apache/avro PR : reading data encoded with an older schema version

from schema_registry_converter.

@ultrabug seems like you get into problems when serializing to a struct, when the data contains new fields. This might indeed be a bit different from the problem you are trying to solve. I think you should try to make sure if the tests are indeed valid, or failing because of wrong assumptions, but this is not always easy. I have some time in December to look at some of those edge cases.

from schema_registry_converter.

Sorry for the delay,
We got stuck on this problem when updating our schemas given that we were suffering with deserialization problems after adding new fields, using an "old" version of schema worked, however after our first interaction we found this very PR apache/avro#1379, we just finished testing with it and seems working pretty well, we forked schema_registry_converter to update the avro version until this issue is not fixed

from schema_registry_converter.

Nice, I guess it's fine to close this issue?
I'll hope a new release with the PR is available somewhere in December, as I can put out a maintenance release by than.

from schema_registry_converter.

Yeah,
Thank you and sorry for the trouble

from schema_registry_converter.

@PauloAOliveira may I kindly ask for your report on your problem + confirmation that my PR fixes your issue to on apache/avro#1379?

Thanks

from schema_registry_converter.