String length expressed as byte or character count for bencode about faq HOT 3 OPEN

Following up on this, my problem ended up being with an implementation expressing string length as a the count of UTF-16 code units used to represent the string. Pretty removed from the standard implementation, yet it exists.
In the end, given my urgency and other implementation problems concerning Bencode I found in faq and reported #93 I threw myself in the deep end of the pool and wrote a modular Bencode decoder/encoder for jq, implementing alternative string length algorithms, which proved interesting although pretty mind-wracking (or wrecking even).
To anyone who might need it, the code in question is here.

from faq.

Hey, that's a pretty interesting problem that I haven't personally run into, even having worked on a pretty widely deployed bencode implementation (on a completely unrelated project).

Do you know of a way to reliably determine whether a file should be interpreted as the variant interpretation and when it should not? Also, do you have any examples of implementations of bencode that support this (even if they're in other langauges)?

from faq.

Hello @jzelinskie
Well, apart from falling back on the variant format and viceversa if the encoding/decoding using the other fails, I don't thinks there's a reliable way to distinguish the two. After all they contain the same data and they diverged due to a different/mistaken interpretation of the format.

As for a practical example of a software using the interpretation I was referring to, you can look https://github.com/Zimbra/zm-mailbox/blob/develop/common/src/java/com/zimbra/common/util/BEncoding.java here for the serialization functions used by the Zimbra Communication Suite in its Java code, i.e. the source of my annoyance ;D .
As to a non-internal implementation dealing with such a variation on the theme, I've used some Perl module to deal with it, but I trace it working, I think, to the "flexible" way Perl can allow you to see a string scalar variable as if you don't specifically force it to be a byte-string.

from faq.