Presence of umlauts causes problems when parsing searchindex.js about sphinx-gallery HOT 9 CLOSED

I can't get enough information from your issue report. And how you get to
the error.
I have nevertheless tested using Unicode in the examples and it worked. My
name in the license comment has also non ascii characters. Maybe making
sure your files have Unicode encoding and you have on top of the file the
encoding comment.
I remember there was already an issue or pr for Unicode support #18 #19.
But for python 2. Which is not your case.
There was also the discussion of enforcing ascii characters in code.

The case for searchindex.js is when resolving the links for the modules of
your project. Meaning you have the non ascii characters in your modules,
and sphinx is not saving the index in Unicode(which is unexpected, even
more in python 3) but this is out of our scope of control.

from sphinx-gallery.

This problem is originated by me.

I'm running sphinx with Python 3.4.1

I have an authors name with the umlaut ü in a readme.rst (utf-8 encoded).

I don't knwo why .. but sphinxgallery/docs_resolv.py:48

with open(url, 'r') as fid:

opens the file searchindex.js as:
<_io.TextIOWrapper name='searchindex.js' mode='r' encoding='ANSI_X3.4-1968'>

With this encoding the ü umlaut raise the mentioned UnicodeDecodeError.

If I force sphinxgallery/docs_resolv.py:48 to open it with UTF-8 encoding

with open(url, 'r', encoding='utf-8') as fid:

fid is <_io.TextIOWrapper name='searchindex.js' mode='r' encoding='utf-8'>

and the error disapears.

I cannot estimate if this utf8 default encoding leads to further problems or not. But if its safe, it would be nice you can add this setting.

Cheers

from sphinx-gallery.

Could it not be a problem with your locale?

What's the output of locale in your shell?
Also what do you get when doing python -c 'import sys; print(sys.stdout.encoding)'?

from sphinx-gallery.

Locale: LANG=C
python -c 'import sys; print(sys.stdout.encoding)' says ANSI_X3.4-1968

Hmm .. interesting point.
I have to dig out where these encoding setting come from.
Probably, an utf8 encoding will solve this issue.

Thank you.

from sphinx-gallery.

@carsten-forty2 did you manage to fix your problem eventually ?

from sphinx-gallery.

@lesteve no .. I had not yet the time to dig in why my sys.stdout.encoding is no UTF8

I live with a local patched version of sphinxgallery/docs_resolv.py

from sphinx-gallery.

my LC_CTYPE was set to 'C' .. that causes python to take these ANSI encoding.

If I set my LC_CTYPE to 'de_DE.UTF-8' .. everything runs fine without the mentioned patch.

The issue can be closed for me.

from sphinx-gallery.

I live with a local patched version of sphinxgallery/docs_resolv.py

OK can you just paste your patch in case this is useful for later or for someone else?

from sphinx-gallery.

sphinxgallery/docs_resolv.py:48

-with open(url, 'r') as fid:
+with open(url, 'r', encoding='utf-8') as fid:

from sphinx-gallery.