Bytes are read from stream to String incorrectly about byte-streams HOT 8 CLOSED

Well, they use different code paths (the String constructor for byte arrays, and CharsetDecoder for streams), but obviously I'd expect them to be equivalent. Can you provide a failing test case?

from byte-streams.

I'm afraid I'm struggling to create a minimal example here. I have 4k of text that demonstrates the problem that I can't share. I have so far failed to minimize this further or find a good random string that demonstrates the same problem.

Could you list the conversion steps from ByteArrayInputStream to String? I'm having a hard time understanding the high-level conversions that are made using the graph by step debugging byte-streams. Maybe I can take my private example through these manually to see where the errors arise.

from byte-streams.

The InputStream is turned into a ByteSource [1], then the ByteSource is turned into a CharSequence [2]. Hope that helps, let me know if you have any other questions.

[1] https://github.com/ztellman/byte-streams/blob/master/src/byte_streams.clj#L526
[2] https://github.com/ztellman/byte-streams/blob/master/src/byte_streams/char_sequence.clj#L81

from byte-streams.

My hunch is that this is an issue of single characters spanning a chunk-size boundary in byte-streams.char-sequence/lazy-char-buffer-sequence causing a unicode replacement character to be used by the decoder.

from byte-streams.

My understanding is that the CharsetDecoder should handle that properly, but if so then we'd expect the malformed characters to show up at the 4096th byte, since that's the default chunk size. Maybe to make a more minimal test case you can try specifying {:chunk-size 16} or something in the convert call?

from byte-streams.

I need to sleep now, but yes, I can easily create a minimal test case now as I'm pretty certain that the problem is as described above. I confirmed this using the method you described. If I simply create a string of 3-byte chars I can see the error on the 4096th byte boundary. If I reduce the chunk size I see a lot more errors.

from byte-streams.

Okay, I'll see if I can track down what's happening.

from byte-streams.

Fixed by 29f50f7

from byte-streams.