Unicode support for links about python-textile HOT 10 CLOSED

Hello hupf,

Perhaps I'm missing something here - but wouldn't this suggestion also violate RFC3986? AFAIK, that RFC says to percent encode unicode characters. More info in the current standard section on Wikipedia. I could be wrong in my interpretation though.

from python-textile.

Hi netcarver,

That may be right. So the correct way to do it, would be to parse the domain part and the rest of the URL separately, then encode the domain part appropriately (xn--...) and percent encode the rest of the URL...

from python-textile.

I found out, that the autoLink() method already violates RFC3986. It matches URLs with Unicode characters, but it doesn't percent encode them, so the produced links may contain Unicode characters.

Maybe this should be fixed too?

from python-textile.

Hi hupf,

You are probably right re autoLink(). I was just flagging RFC3986 for consideration as this issue has come up for me in PHP Textile and I'll be looking to work on it for that project soon too.

from python-textile.

Thanks for this, I'll take a deeper look as well.

from python-textile.

hupf, Chris,

some of this Textpattern forum post may be of interest as may the new unicode-links branch on PHP Textile which is my first (and incomplete) attempt at this for non-restricted mode links, though I am still having a little trouble getting Japanese chars through the regex.

from python-textile.

yeah, I suck at regex, which makes me a terrible maintainer for this project ;)

from python-textile.

In the commit above, I used and fixed an implementation I found on Stack Overflow to encode the URLs correctly (host part as IDNA ant the path part as per RFC 3986): http://stackoverflow.com/questions/804336/best-way-to-convert-a-unicode-url-to-ascii-utf-8-percent-escaped-in-python/804380#804380

from python-textile.

Here is my branch with a working implementation of the URL encoding: https://github.com/atizo/python-textile/tree/unicode-links

from python-textile.

Fixed in textile/python-textile == version on pip

from python-textile.