UTF-8 characters aren’t properly encoded in URLs about rdiscount HOT 4 CLOSED

And on my computer, that code gives:

<p><a href="http://example.com/?%9F">Test</a></p>

Which is still incorrect. Looks like this will require some more investigation.

from rdiscount.

It appears that RDiscount 2.0.7 is not actually using Discount 2.0.7, or at least not the final version. Apparently the original pull request to pull in the Discount 2.0.7 sources wasn't correct.

sigh... Look like a bugfix release will be coming soon.

FWIW, Discount 2.0.7 does not have the bug you mention. Thus neither should the bugfixed RDiscount 2.0.7.

from rdiscount.

Ugh what a hideous bug...

Both Discount and RDiscount are calling mkd_string using exactly the same input - byte for byte. Yet are returning different results. How on earth can this be?

I traced the altered return data to puturl, which is responsible for percent-encoding special characters (ex: ß -> %C3%9F). It wasn't encoding %C3. Why not? Its decision to do this is based on isalnum(0xC3), which determines whether a character is alpha-numeric. And apparently this function changes its behavior based on the current global locale. Argh!

Ruby uses the locale en_US.UTF-8 on my machine, where isalpha(0xC3) => true. Everybody else uses the default locale of C (ASCII), where isalpha(0xC3) => false.

[rant] I can't believe this behavior of isalnum... UTF-8 codepoints cannot fit into a single 8-bit unsigned char value, so it is foolhardy for isalpha to try to process an 8-bit value as if it were an UTF-8 codepoint. [/rant]

Summary: The behavior of isalpha is ill-defined for any locale with codepoints requiring more than 8 bits - notably UTF-8 or any Unicode encoding.

from rdiscount.

Fixed on master branch.

from rdiscount.