[proposal] Provide a way to inject String deduplication into the deserialization pipeline about scalapb HOT 2 OPEN

Thanks for posting. Can you add to the proposal the following:

• how the duplicator will be selected?
• Would having a deduper require the field to be lazy evaled (as in #1373)?
• Can deduping made optional in the generator?
• Should it impact instantiation outside binary deserialization: fromPMessage (JSON, Spark)

from scalapb.

how the duplicator will be selected?

One of the options - provide a name of class, that implements DeduplicatorProvider interface:

trait DeduplicatorProvider {
  def getOrCreateDeduplicator[T: ByteStringCodec]: Deduplicator[T]
}

This name may be provided on generator, package, file or even field level.

Would having a deduper require the field to be lazy evaled (as in #1373)?

Actually they are related, but not tight coupled. You may want to reduce memory footprint of common objects even if they are eagerly parsed.
So, if you deserialize some string or message type, you may pass your already delimeted ByteString to Deduplicator and check the cache presence by ByteString hashing / comparision.

Can deduping made optional in the generator?

Sure, it may be done completely optional on any configuration level.
Also, I also found useful to disable it at for some field - like we have a case, where there is some string description field, which doesn't contribute anything for business logic, but will pollute the strings cache if we will try to deduplicate it. Ideally I would prefer to keep this field lazy & non-cacheable.

This may be achieved by checking the presense of deduplicate field / message option and disable it for field which shouldn't be cached.

Should it impact instantiation outside binary deserialization: fromPMessage (JSON, Spark)

I think so.

from scalapb.