drekin / win-unicode-console Goto Github PK

A Python package to enable Unicode support when running Python from Windows console.

License: MIT License

Python 100.00%

win-unicode-console's Introduction

win-unicode-console

A Python package to enable Unicode input and display when running Python from Windows console.

The package is not needed on Python 3.6 and newer since the underlying issue has been resolved (see https://docs.python.org/3/whatsnew/3.6.html#whatsnew36-pep528).

General information

When running Python (3.5 and older) in the standard console on Windows, there are several problems when one tries to enter or display Unicode characters. The relevant issue is http://bugs.python.org/issue1602. This package solves some of them.

First, when you want to display Unicode characters in Windows console, you have to select a font able to display them. Similarly, if you want to enter Unicode characters, you have to have you keyboard properly configured. This has nothing to do with Python, but is included here for completeness.
The standard stream objects (sys.stdin, sys.stdout, sys.stderr) are not capable of reading and displaying Unicode characters in Windows console. This has nothing to do with encoding, since even sys.stdin.buffer.raw.readline() returns b"?\n" when entering α and there is no encoding under which sys.stdout.buffer.raw.write displays α.

The streams module provides several alternative stream objects. stdin_raw, stdout_raw, and stderr_raw are raw stream objects using WinAPI functions ReadConsoleW and WriteConsoleW to interact with Windows console through UTF-16-LE encoded bytes. The stdin_text, stdout_text, and stderr_text are standard text IO wrappers over standard buffered IO over our raw streams, and are intended to be primary replacements to sys.std* streams. Unfortunately, other wrappers around std*_text are needed (see below), so there are more stream objects in streams module.

The function streams.enable installs chosen stream objects instead of the original ones. By default, it chooses appropriate stream objects itself. The function streams.disable restores the original stream objects (these are stored in sys.__std*__ attributes by Python).

After replacing the stream objects, also using print with a string containing Unicode characters and displaying Unicode characters in the interactive loop works. For input, see below.
Python interactive loop doesn't use sys.stdin to read input so fixing it doesn't help. Also the input function may or may not use sys.stdin depending on whether sys.stdin and sys.stdout have the standard filenos and whether they are interactive. See http://bugs.python.org/issue17620 for more information.

To solve this, we install a custom readline hook. Readline hook is a function which is used to read a single line interactively by Python REPL. It may also be used by input function under certain conditions (see above). On Linux, this hook is usually set to GNU readline function, which provides features like autocompletion, history, …

The module readline_hook provides our custom readline hook, which uses sys.stdin to get the input and is (de)activated by functions readline_hook.enable, readline_hook.disable.

As we said, readline hook can be called from two places – from the REPL and from input function. In the first case the prompt is encoded using sys.stdin.encoding, but in the second case sys.stdout.encoding is used. So Python currently makes an assumption that these two encodings are equal.
Python tokenizer, which is used when parsing the input from REPL, cannot handle UTF-16 or generally any encoding containing null bytes. Because UTF-16-LE is the encoding of Unicode used by Windows, we have to additionally wrap our text stream objects (std*_text). Thus, streams module contains also stream objects stdin_text_transcoded, stdout_text_transcoded, and stderr_text_transcoded. They basically just hide the underlying UTF-16-LE encoded buffered IO, and sets encoding to UTF-8. These transcoding wrappers are used by default by streams.enable.

There are additional issues on Python 2.

Since default Python 2 strings correspond to bytes rather than unicode, people are usually calling print with bytes argument. Therefore, sys.stdout.write and sys.stderr.write should support bytes argument. That is why we add stdout_text_str and stderr_text_str stream objects to streams module. They are used by default on Python 2.
When we enter a Unicode literal into interactive interpreter, it gets processed by the Python tokenizer, which is bytes-based. When we enter u"\u03b1" into the interactive interpreter, the tokenizer gets essentially b'u"\xce\xb1"' plus the information that the encoding used is UTF-8. The problem is that the tokenizer uses the encoding only if sys.stdin is a file object (see https://hg.python.org/cpython/file/d356e68de236/Parser/tokenizer.c#l797). Hence, we introduce another stream object streams.stdin_text_fileobj that wraps stdin_text_transcoded and also is structurally compatible with Python file object. This object is used by default on Python 2.
The check for interactive streams done by raw_input unfortunately requires that both sys.stdin and sys.stdout are file objects. Besides stdin_text_fileobj for stdin we could use also stdout_text_str_fileobj for stdout. Unfortunately, that breaks print.

Using print statement or function leads to calling PyFile_WriteObject with sys.stdout as argument. Unfortunately, its generic write method is used only if it is not a file object. Otherwise, PyObject_Print is called, and this function is file-based, so it ends with a fprintf call, which is not something we want. In conclusion, we need stdout not to be a file object.

Given the situation described, the best solution seems to be reimplementing raw_input and input builtin functions and monkeypatching __builtins__. This is done by our raw_input module on Python 2.
Similarly to the input from from sys.stdin the arguments in sys.argv are also bytes on Python 2 and the original ones may not be reconstructable. To overcome this we add unicode_argv module. The function unicode_argv.get_unicode_argv returns Unicode version of sys.argv obtained by WinAPI functions GetCommandLineW and CommandLineToArgvW. The function unicode_argv.enable monkeypatches sys.argv with the Unicode arguments.

Installation

Install the package from PyPI via pip install win-unicode-console (recommended), or download the archive and install it from the archive (e.g. pip install win_unicode_console-0.x.zip), or install the package manually by placing directory win_unicode_console and module run.py from the archive to the site-packages directory of your Python installation.

Usage

The top-level win_unicode_console module contains a function enable, which install various fixes offered by win_unicode_console modules, and a function disable, which restores the original environment. By default, custom stream objects are installed as well as a custom readline hook. On Python 2, raw_input and input functions are monkeypatched. sys.argv is not monkeypatched by default since unfortunately some Python 2 code strictly assumes str instances in sys.argv list. Use enable(use_unicode_argv=True) if you want the monkeypathcing. For further customization, see the sources. The logic should be clear.

Generic usage of the package is just calling win_unicode_console.enable() whenever the fixes should be applied and win_unicode_console.disable() to revert all the changes. Note that it should be a responsibility of a Python user on Windows to install win_unicode_console and fix his Python environment regarding Unicode interaction with console, rather than of a third-party developer enabling win_unicode_console in his application, which adds a dependency. Our package should be seen as an external patch to Python on Windows rather than a feature package for other packages not directly related to fixing Unicode issues.

Different ways of how win_unicode_console can be used to fix a Python environment on Windows follow.

Python patch (recommended). Just call win_unicode_console.enable() in your sitecustomize or usercustomize module (see https://docs.python.org/3/tutorial/appendix.html#the-customization-modules for more information). This will enable win_unicode_console on every run of the Python interpreter (unless site is disabled). Doing so should not break executed scripts in any way. Otherwise, it is a bug of win_unicode_console that should be fixed.
Opt-in runner. You may easily run a script with win_unicode_console enabled by using our runner module and its helper run script. To do so, execute py -i -m run script.py instead of py -i script.py for interactive mode, and similarly py -m run script.py instead of py script.py for non-interactive mode. Of course you may provide arguments to your script: py -i -m run script.py arg1 arg2. To run the bare interactive interpreter with win_unicode_console enabled, execute py -i -m run.
Opt-out runner. In case you are using win_unicode_console as Python patch, but you want to run a particular script with win_unicode_console disabled, you can also use the runner. To do so, execute py -i -m run --init-disable script.py.
Customized runner. To move arbitrary initialization (e.g. enabling win_unicode_console with non-default arguments) from sitecustomize to opt-in runner, move it to a separate module and use py -i -m run --init-module module script.py. That will import a module module on startup instead of enabling win_unicode_console with default arguments.

Compatibility

win_unicode_console package was tested on Python 3.4, Python 3.5, and Python 2.7 (it is not needed on Python 3.6+). 32-bit or 64-bit shouldn't matter. It also interacts well with the following packages:

colorama package (https://pypi.python.org/pypi/colorama) makes ANSI escape character sequences (for producing colored terminal text and cursor positioning) work under MS Windows. It does so by wrapping sys.stdout and sys.stderr streams. Since win_unicode_console replaces the streams in order to support Unicode, win_unicode_console.enable has to be called before colorama.init so everything works as expected.

As of colorama v0.3.3, there was an early binding issue (tartley/colorama#32), so win_unicode_console.enable has to be called even before importing colorama. Note that is already the case when win_unicode_console is used as Python patch or as opt-in runner. The issue was already fixed.
pyreadline package (https://pypi.python.org/pypi/pyreadline/2.0) implements GNU readline features on Windows. It provides its own readline hook, which actually supports Unicode input. win_unicode_console.readline_hook detects when pyreadline is active, and in that case, by default, reuses its readline hook rather than installing its own, so GNU readline features are preserved on top of our Unicode streams.
IPython (https://pypi.python.org/pypi/ipython) can be also used with win_unicode_console.

As of IPython 3.2.1, there is an early binding issue (ipython/ipython#8669), so win_unicode_console.enable has to be called even before importing IPython. That is the case when win_unicode_console is used as Python patch.

There was also an issue that IPython was not compatible with the builtin function raw_input returning unicode on Python 2 (ipython/ipython#8670). If you hit this issue, you can make win_unicode_console.raw_input.raw_input return bytes by enabling it as win_unicode_console.enable(raw_input__return_unicode=False). This was fixed in IPython 4.

Backward incompatibility

Since version 0.4, the signature of streams.enable has been changed because there are now more options for the stream objects to be used. It now accepts a keyword argument for each stdin, stdout, stderr, setting the corresponding stream. None means “do not set”, Ellipsis means “use the default value”.

A function streams.enable_only was added. It works the same way as streams.enable, but the default value for each parameter is None.

Functions streams.enable_reader, streams.enable_writer, and streams.enable_error_writer have been removed. Example: instead of streams.enable_reader(transcode=True) use streams.enable_only(stdin=streams.stdin_text_transcoding).

There are also corresponding changes in top-level enable function.
Since version 0.3, the custom stream objects have the standard filenos, so calling input doesn't handle Unicode without custom readline hook.

Acknowledgements

The code of streams module is based on the code submitted to http://bugs.python.org/issue1602.
The idea of providing custom readline hook and the code of readline_hook module is based on https://github.com/pyreadline/pyreadline.
The code related to unicode_argv.get_full_unicode_argv is based on http://code.activestate.com/recipes/572200/.
The idea of using path hooks and the code related to unicode_argv.argv_setter_hook is based on https://mail.python.org/pipermail/python-list/2016-June/710183.html.

win-unicode-console's People

Contributors

Stargazers

Watchers

Forkers

tjguk jonitis anthrotype zed takluyver anymaster iamtutu throwable-one datakortet j-j-4 mlin26 nasingfaund

win-unicode-console's Issues

Add init.py for better compatibility

Mark Summerfield reports that the package wouldn't freeze with cx_freeze because of missing __init__.py (see http://bugs.python.org/issue1602#msg223945).

Using atexit() apparently breaks win-unicode-console

If win-unicode-console is installed as "Python patch" (in usersitecustomize, I mean) and you run this code on Windows (tested with Python 3.5 x64):

import atexit
atexit.register(lambda: input('Press ENTER to continue...'))
print(__file__)
exit()

then the following error occurs (I'm including full output of the program):

C:\[CENSORED]\test.py
Press ENTER to continue...Intenal win_unicode_console error
Traceback (most recent call last):
  File "C:\Program Files\Python35\lib\site-packages\win_unicode_console\readline_hook.py", line 84, in readline_wrapper
    line = self.readline_hook(prompt)
  File "C:\Program Files\Python35\lib\site-packages\win_unicode_console\readline_hook.py", line 59, in stdio_readline
    return sys.stdin.readline()
  File "C:\Program Files\Python35\lib\site-packages\win_unicode_console\streams.py", line 197, in readline
    return self.base.readline(size)
ValueError: I/O operation on closed file.

That's all. If you need more information or want me to carry any test, just tell.

Thanks a lot for win-unicode-console, it's awesome and it should be included with Python by default on Windows...

Raúl

EOFError() when using raw_input() on certain input in Windows

Hi,

I'm using Windows and Python 2.7. with win_unicode_console 0.5

Some input will raise EOFError, when other doesn't.

Sample code

import sys
reload(sys)
sys.setdefaultencoding("utf-8")

import win_unicode_console
win_unicode_console.enable()

try:
    input = raw_input('input?')

    print input
except:
    print sys.exc_info()

Sample Input

多々良小傘 = raise EOFError
たたらこがさ = OK

Any way to disable readline hook warning?

My sitecustomize.py file:

import win_unicode_console
win_unicode_console.enable()

content of foo.py:

import sys
for line in sys.stdin:
    sys.stdout.write(line)

running without PYTHONIOENCODING:

(dev) go|c:\srv\tmp\wuc> set PYTHONIOENCODING
Environment variable PYTHONIOENCODING not defined

(dev) go|c:\srv\tmp\wuc> type foo.py | python foo.py
c:\srv\venv\dev\lib\site-packages\win_unicode_console\__init__.py:31: RuntimeWarning: sys.stdin.encoding == None, whereas sys.stdout.encoding == 'utf-8', readline hook consumer may assume they are the same
  readline_hook.enable(use_pyreadline=use_pyreadline)
import sys
for line in sys.stdin:
    sys.stdout.write(line)

(dev) go|c:\srv\tmp\wuc>

and with PYTHONIOENCODING=utf-8:

(dev) go|c:\srv\tmp\wuc> set PYTHONIOENCODING=utf-8

(dev) go|c:\srv\tmp\wuc> type foo.py | python foo.py
c:\srv\venv\dev\lib\site-packages\win_unicode_console\__init__.py:31: RuntimeWarning: sys.stdin.encoding == None, whereas sys.stdout.encoding == 'utf-8', readline hook consumer may assume they are the same
  readline_hook.enable(use_pyreadline=use_pyreadline)
import sys
for line in sys.stdin:
    sys.stdout.write(line)

The warning is coming from sitecustomize.py, so I'm assuming I have to check if sys.stdin is coming from a pipe? (I first tried to check if sys.stdin.encoding is None -- but both it and stdout.encoding are always None in sitecustomize).

This is my latest attempt:

import sys
if sys.stdin.isatty():
    import win_unicode_console
    win_unicode_console.enable()

This works for the foo.py from above, but fails with mojibake if I change foo.py to read:

# -*- coding: utf-8 -*-
import sys
for line in sys.stdin:
    sys.stdout.write(line)
print u'✓'

i.e. adding an encoding comment, and a unicode literal (checkmark on last line).

If I change the code page to 65001 it does work:

(dev) go|c:\srv\tmp\wuc> type foo.py | python foo.py
# -*- coding: utf-8 -*-
import sys
for line in sys.stdin:
    sys.stdout.write(line)
print u'Γ£ô'
Γ£ô

(dev) go|c:\srv\tmp\wuc> chcp 65001
Active code page: 65001

(dev) go|c:\srv\tmp\wuc> type foo.py | python foo.py
# -*- coding: utf-8 -*-
import sys
for line in sys.stdin:
    sys.stdout.write(line)
print u'✓'
✓

(dev) go|c:\srv\tmp\wuc> type foo.py | python foo.py > bar.py

(dev) go|c:\srv\tmp\wuc> type bar.py
# -*- coding: utf-8 -*-
import sys
for line in sys.stdin:
    sys.stdout.write(line)
print u'✓'
✓

..but win-unicode-console is probably not involved in any of that, correct?

Unicode filename gets crippled when drag-and-dropped

When a Python script is run by drag-and-dropping another file, its path gets crippled in sys.argv if it contains Unicode characters, and our unicode_argv module doesn't resolve this. Note that this is an issue even in Python 3. Since even calling GetCommandLineW returns a crippled path, this may be issue with Windows itself. I have started an issue in Python bug tracker: http://bugs.python.org/issue27469.

Replacing the streams breaks IPython

In the IPython interactive console, if I type

import win_unicode_console.streams
win_unicode_console.streams.enable()

I get an infinite stream of errors, all of which appear to be as follows:

ArgumentError                             Traceback (most recent call last)
C:\Users\uk03306\AppData\Local\Continuum\Anaconda3\lib\site-packages\IPython\terminal\interactiveshell.py in raw_input(self, prompt)
    588
    589         try:
--> 590             line = py3compat.str_to_unicode(self.raw_input_original(prompt))
    591         except ValueError:
    592             warn("\n********\nYou or a %run:ed script called sys.stdin.close()"

C:\Users\uk03306\AppData\Local\Continuum\Anaconda3\lib\site-packages\win_unicode_console\streams.py in write(self, b)
     84                 code_units_written = c_int()
     85
---> 86                 retval = WriteConsoleW(self.handle, buffer, code_units_to_be_written, byref(code_units_written), None)
     87                 bytes_written = 2 * code_units_written.value
     88

ArgumentError: argument 4: <class 'TypeError'>: expected LP_c_ulong instance instead of pointer to c_long

I'm sure this counts as just another case of "the module doesn't play nicely with the interactive REPL" (that same enable call fails in a normal Python console, or one with just pyreadline installed, as well, but not quite as badly...), but I thought I'd report it here just in case there was something else going on. I have pyreadline installed with IPython, as recommended, and that does some fairly complicated low-level console hacking which might interact badly with this module.

If unicode Functionality is Required under Windows then Developer Responsible to Load it

"... Note that it should be a responsibility of a Python user on Windows to install win_unicode_console and fix his Python environment regarding Unicode interaction with console, rather than of a third-party developer enabling win_unicode_console in his application, which adds a dependency. Our package should be seen as an external patch to Python on Windows rather than a feature package for other packages not directly related to fixing Unicode issues."

Sounds like another typical comment from a *nix user - far removed from reality! Windows users expect these issues to be sorted out as part of the installation package.

Broken on Python 3.6

Python 3.6.0 (v3.6.0:41df79263a11, Dec 23 2016, 08:06:12) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import win_unicode_console
>>> win_unicode_console.enable()
>>> import sys

Python has stopped working Windows error message and interpreter closes.

Unable to print kanji or other asian language characters

Using win-unicode-console removed the exceptions raised by python when printing Unicode to cmd-prompt. But I noticed that cmd-prompt is unable to display characters from languages like Japanese, Mandarin or Korean. It is able to display most European languages. I tried printing German, Italian, Greek, French, Hungarian, Spanish etc and they all worked. My code looked like this:

# -*- coding: UTF-8 -*-

from __future__ import unicode_literals
import win_unicode_console

win_unicode_console.enable()

print('光り射す場所へ')

I tried it on Python 2.7.11 and Python 3.5.1 on Windows7 (all are 64 bit) with cmd-prompt set to Consolas font.
Is this an encoding problem with win-unicode-console or a limitation of font?

Colorama compatibility

The readme claims "win_unicode_console package [...] interacts well with [...] colorama package[s]". The following code snippet shows that the output changes depending on whether you first call win_unicode_console.enable() or colorama.init() (see screenshot). progress and test_progress are from here: https://github.com/verigak/progress.

In the first case, ansi escape sequences are displayed (win_unicode_console.enable "overrides" colorama. In the second case, unicode characters are displayed as "\u" (colorama overrides win_unicode_console.enable)

import colorama, win_unicode_console

colorama.init()
win_unicode_console.enable()

import progress, test_progress

Error installing package under Python2

thorsten@rednails[F:\...\script] tcc> pip2 install win-unicode-console
Collecting win-unicode-console
  Using cached win_unicode_console-0.3.1.zip
    Traceback (most recent call last):
      File "<string>", line 20, in <module>
      File "c:\users\thorsten\appdata\local\temp\pip-build-pcloii\win-unicode-console\setup.py", line 30, in <module>
        long_description=read(README),
      File "c:\users\thorsten\appdata\local\temp\pip-build-pcloii\win-unicode-console\setup.py", line 9, in read
        with open(path.join(HERE, *relative_path_parts), encoding=ENCODING) as f:
    TypeError: 'encoding' is an invalid keyword argument for this function
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

      File "<string>", line 20, in <module>

      File "c:\users\thorsten\appdata\local\temp\pip-build-pcloii\win-unicode-console\setup.py", line 30, in <module>

        long_description=read(README),

      File "c:\users\thorsten\appdata\local\temp\pip-build-pcloii\win-unicode-console\setup.py", line 9, in read

        with open(path.join(HERE, *relative_path_parts), encoding=ENCODING) as f:

    TypeError: 'encoding' is an invalid keyword argument for this function

    ----------------------------------------
    Command "python setup.py egg_info" failed with error code 1 in c:\users\thorsten\appdata\local\temp\pip-build-pcloii\win-unicode-console

Error importing package on Cygwin and Linux

win-unicode-console claims compatibility with colorama. It would be nice if you could import win-unicode-console on other platforms than Windows without an exception; just like colorama:
"On other platforms, Colorama does nothing."

thorsten@rednails zsh> python3                                                                                                                     [~]Python 3.2.5 (default, Jul 25 2014, 14:13:17)
[GCC 4.8.3] on cygwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import win_unicode_console
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.2/site-packages/win_unicode_console-0.3.1-py3.2.egg/win_unicode_console/__init__.py", line 2, in <module>
    from win_unicode_console import streams, console, readline_hook
  File "/usr/lib/python3.2/site-packages/win_unicode_console-0.3.1-py3.2.egg/win_unicode_console/streams.py", line 2, in <module>
    from ctypes import byref, windll, c_ulong
ImportError: cannot import name windll

Windows error 6 when subprocess' stdout is redirected to os.devnull

Background: I'm using version 0.4. I have win-unicode-console installed as a Python patch in sitecustomize.py. I only have one Python installed, and it's Anaconda Python 2.7.10.

I have a complicated piece of code that breaks when run with win-unicode-console enabled. I can reproduce the problem with a small test case (see attachments). On their own, the test cases are strange. Please keep in mind that they're a snippet of a much larger piece of code. 😃

I think the part of my scenario that confuses win-unicode-console is that I'm redirecting a subprocess' stdout to os.devnull. This is key to the problem, because if I redirect it to a temp file instead, the problem goes away. I guess os.devnull does not mimic a real file in some respect.

Attached are hello_world.py, fails.py, and succeeds.py. The last two scripts invoke hello_world which just prints to the console. The only difference between fails and succeeds is that fails pipes its subprocess' stdout to os.devnull, while succeeds uses a temp file.

The attachments have a .txt file extension to keep github happy. Remove that before running the files.

can not handle the output from python build-in lib(while using argparse)

I have below code :

from __future__ import unicode_literals, absolute_import

......

def main():
    global ROOT_PATH
    parser = argparse.ArgumentParser()
    parser.add_argument('-p','--path', default='__here__')
    print 1
    args = parser.parse_args()
    print 2
    p = unicode(args.path, 'gbk')
    if p == '__here__':
        ROOT_PATH = os.getcwd()
    else:
        ROOT_PATH = p

    for sf in get_sub_folders(ROOT_PATH):
        process_folder(sf)

My windows7 cmd encoding is gbk(a Chinese encoding)

Chinese work fine

E:\[Sync]\project\auto_shift>python exe.py -p E:\[Sync]\【垃圾箱】
1
2

But Japanese go wrong.

E:\[Sync]\project\auto_shift>python exe.py -p E:\同人本\シュート・ザ・ムーン (フエタキシ)
1
usage: exe.py [-h] [-p PATH]
Traceback (most recent call last):
  File "exe.py", line 196, in <module>
    main()
  File "exe.py", line 183, in main
    args = parser.parse_args()
  File "D:\Python27\lib\argparse.py", line 1704, in parse_args
    self.error(msg % ' '.join(argv))
  File "D:\Python27\lib\argparse.py", line 2374, in error
    self.exit(2, _('%s: error: %s\n') % (self.prog, message))
  File "D:\Python27\lib\argparse.py", line 2361, in exit
    self._print_message(message, _sys.stderr)
  File "D:\Python27\lib\argparse.py", line 2354, in _print_message
    file.write(message)
  File "D:\Python27\lib\site-packages\win_unicode_console-0.4-py2.7.egg\win_unicode_console\streams.py", line 217, in wr
ite
    s = s.decode(self.encoding)
  File "D:\Python27\lib\encodings\utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xa5 in position 40: invalid start byte

WinError 6, the handle is invalid

On some system, the value get from msvcrt.get_osfhandle is wrong and cause OSError. Here is what my machine get:

Python 3.5.1 (v3.5.1:37a07cee5969, Dec  6 2015, 01:38:48) [MSC v.1900 32 bit (In
tel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import msvcrt
>>> print(msvcrt.get_osfhandle(1))
7
>>>
>>> from ctypes import windll
>>> print(windll.kernel32.GetStdHandle(-11))
7
>>>

And here is another user's:

Python 3.4.3 (v3.4.3:9b73f1c3e601, Feb 24 2015, 22:44:40) [MSC v.1600 64 bit (AMD64)] on win32
Type "copyright", "credits" or "license()" for more information.
>>> import msvcrt
>>> print(msvcrt.get_osfhandle(1))
18446744073709551614
>>> from ctypes import windll
>>> print(windll.kernel32.GetStdHandle(-11))
0
>>>

Version 0.4 works fine on his computer.

File warning instead of fail if stdout does not have encoding

According to Python manual sys.stdout should have encoding.
Unfortunatelly, nose monekypatches it with StringIO with out of encoding.

See: nose-devs/nose#1065

Its nose fault, not win-unicode-console fault. But can we please do

def check_encodings():
	try:
		if sys.stdin.encoding != sys.stdout.encoding:
			# raise RuntimeError("sys.stdin.encoding != sys.stdout.encoding, readline hook doesn't know, which one to use to decode prompt")

			warnings.warn("sys.stdin.encoding == {!r}, whereas sys.stdout.encoding == {!r}, readline hook consumer may assume they are the same".format(sys.stdin.encoding, sys.stdout.encoding),
				RuntimeWarning, stacklevel=3)
	except AttributeError:
		warnings.warn("stdin or stdout does not have encoding")

ValueError: Procedure probably called with too many arguments (12 bytes in excess)

Hello,

I installed the package via "pip install win_unicode_console". When I enable it inside an interactive session, I get this error below and the python interpreter crashes. I am using the vanilla python 3.4.2 distribution for win32, downloaded from the Python.org, from within the Windows Command Prompt (on Windows 7). I did not change the cmd.exe codepage through "chcp" command, it's stays the default "cp850" on my machine.
Here is the output of my interactive session followed by traceback:

Python 3.4.2 (v3.4.2:ab2c023a9432, Oct  6 2014, 22:15:05) [MSC v.1600 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import win_unicode_console
>>> win_unicode_console
<module 'win_unicode_console' from 'C:\\Python34\\lib\\site-packages\\win_unicode_console\\__init__.py'>
>>> win_unicode_console.enable()
>>> print('hello')
Intenal win_unicode_console error
Traceback (most recent call last):
  File "C:\Python34\lib\site-packages\win_unicode_console\readline_hook.py", line 52, in readline_wrapper
    return new_zero_terminated_string(line.encode(sys.stdin.encoding))
  File "C:\Python34\lib\site-packages\win_unicode_console\readline_hook.py", line 20, in new_zero_terminated_string
    strncpy(cast(p, c_char_p), b, len(b) + 1)
ValueError: Procedure probably called with too many arguments (12 bytes in excess)
Traceback (most recent call last):
  File "C:\Python34\lib\site-packages\win_unicode_console\readline_hook.py", line 52, in readline_wrapper
    return new_zero_terminated_string(line.encode(sys.stdin.encoding))
  File "C:\Python34\lib\site-packages\win_unicode_console\readline_hook.py", line 20, in new_zero_terminated_string
    strncpy(cast(p, c_char_p), b, len(b) + 1)
ValueError: Procedure probably called with too many arguments (12 bytes in excess)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 277, in 'calling callback function'
  File "C:\Python34\lib\site-packages\win_unicode_console\readline_hook.py", line 57, in readline_wrapper
    return new_zero_terminated_string(b"\n")
  File "C:\Python34\lib\site-packages\win_unicode_console\readline_hook.py", line 20, in new_zero_terminated_string
    strncpy(cast(p, c_char_p), b, len(b) + 1)
ValueError: Procedure probably called with too many arguments (12 bytes in excess)

Thanks for your support.

Cosimo

Installation Issue

"Different ways of how win_unicode_console can be used to fix a Python environment on Windows follow.

Python patch (recommended). Just call win_unicode_console.enable() in your sitecustomize or usercustomize module (see https://docs.python.org/3/tutorial/appendix.html#the-customization-modules for more information). This will enable win_unicode_console on every run of the Python interpreter (unless site is disabled). Doing so should not break executed scripts in any way."

import site
site.getusersitepackages()

This returned a path that did not exist. I created the path and sitecustomize.py file with a single line:
win_unicode_console.enable()

I got an error so I changed it to the two lines:
import win_unicode_console
win_unicode_console.enable()

Please confirm correct installation and provide a simple validation test.

Running Win7-32

PyInstaller compatibility

I just tried win-unicode-console with the latest PyInstaller3 branch. There is no exception (like with Py2exe) but importing win-unicode-console simply has no effect. Is this a known issue? Any workaround possible?

execfile() rejects valid Python 2

Thanks for the Python 2 support and the module in general!

I'm seeing a problem when launching a Python 2 script with the runner.

I have win-unicode-console installed as a Python patch in sitecustomize.py. I only have 1 Python installed, and it's Python 2.

I have a simple Python 2 script called foo.py which contains this:

print "hello world"

According to the documentation, I should be able to temporarily disable win-unicode-console for my script by running it this way:

python -m run --init-disable foo.py

However, when I do so, I get this:

  File "foo.py", line 1
    print "hello world"
                      ^
SyntaxError: invalid syntax

The error disappears if I change foo.py to contain this (which is also valid Python 2):

print("hello world")

My guess is that the from __future__ import print_function in runner.py is leaking through to modules run via the execfile call.

Py2exe compatibility

It looks like win-unicode-console is not compatible with py2exe (see below). Is this a known issue and is there any workaround?

   import _init, docopt
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1191, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1161, in _load_backward_compatible
  File "F:\PortableApps\Python3x\lib\site-packages\zipextimporter.py", line 86, in load_module
    return zipimport.zipimporter.load_module(self, fullname)
  File "F:\Documents\dev\python\script\_init.py", line 5, in <module>
    try: import win_unicode_console
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1191, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1161, in _load_backward_compatible
  File "F:\PortableApps\Python3x\lib\site-packages\zipextimporter.py", line 86, in load_module
    return zipimport.zipimporter.load_module(self, fullname)
  File "F:\PortableApps\Python3x\lib\site-packages\win_unicode_console\__init__.py", line 2, in <module>
    from win_unicode_console import streams, console, readline_hook
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1191, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1161, in _load_backward_compatible
  File "F:\PortableApps\Python3x\lib\site-packages\zipextimporter.py", line 86, in load_module
    return zipimport.zipimporter.load_module(self, fullname)
  File "F:\PortableApps\Python3x\lib\site-packages\win_unicode_console\streams.py", line 4, in <module>
    from win_unicode_console.buffer import get_buffer
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1191, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1161, in _load_backward_compatible
  File "F:\PortableApps\Python3x\lib\site-packages\zipextimporter.py", line 86, in load_module
    return zipimport.zipimporter.load_module(self, fullname)
  File "F:\PortableApps\Python3x\lib\site-packages\win_unicode_console\buffer.py", line 8, in <module>
    PyObject_GetBuffer = pythonapi.PyObject_GetBuffer
  File "F:\PortableApps\Python3x\lib\ctypes\__init__.py", line 364, in __getattr__
    func = self.__getitem__(name)
  File "F:\PortableApps\Python3x\lib\ctypes\__init__.py", line 369, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: function 'PyObject_GetBuffer' not found

getpass warns that echo free input is not available

when win_unicode_console is enabled, and use getpass.getpass() I get:

C:\Python27\lib\getpass.py:92: GetPassWarning: Can not control echo on the terminal.
  return fallback_getpass(prompt, stream)
Warning: Password input may be echoed.
Password: password

For now I resorted to disabling it immediately before I call getpass and reenabling it afterwards:

def prompt_password():
    if HAS_WIN_UNICODE_CONSOLE:
        win_unicode_console.disable()
    password = getpass.getpass()
    if HAS_WIN_UNICODE_CONSOLE:
        win_unicode_console.enable()
    return password

Do you know any better workaround or fix?

Also, what is the preferred way to check from code if win_unicode_console has been previously enabled? It would be nice if there was a function returning a bool which tells whether win_unicode_console is already enabled or not. My global
HAS_WIN_UNICODE_CONSOLE in the code above is True whenever win_unicode_console can be imported, but that doesn't mean it is also enabled. I don't want to re-enable it if it wasn't before running getpass.

Thanks!

Cosimo

Package is not compatible with Python2

If the package doesn't support Python2, that should be stated

thorsten@rednails[F:\...\script] tcc> python2 script.py
Traceback (most recent call last):
  File "script.py", line 17, in <module>
    import win_unicode_console
  File "F:\PortableApps\Python2x\lib\site-packages\win_unicode_console\__init__.py", line 7
    def enable(*,
                ^
SyntaxError: invalid syntax

UnicodeDecodeError when printing result of glob.glob('xxx') .

I have a function like this:

import glob

def get_test_file_paths(dirs, ext='*.xls*'):
    file_paths = []
    for d in dirs:
        file_paths.extend(glob.glob(d + ext))
    return file_paths

I use glob to get a paths list which may contain Chinese.
Then I got error:

E:\Project\yq_analyze>python merge_e_files.py
data/new\d1.xls
data/new\d2_no_e.xlsx
data/new\[email protected]
Traceback (most recent call last):
  File "merge_e_files.py", line 66, in <module>
    adf = merge_e_files(file_paths)
  File "merge_e_files.py", line 16, in merge_e_files
    print path
  File "C:\Python27\lib\site-packages\win_unicode_console\streams.py", line 217, in write
    s = s.decode(self.encoding)
  File "C:\Python27\lib\encodings\utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xd6 in position 26: invalid continuation byte

win-unicode-console breaks an executed script in Python 3.5

I am using Windows 7 64 bit with the 64 bit version of Python 3.5 .
I installed win-unicode-console as my usercustomize.

Now another python program called livestreamer (https://github.com/chrippa/livestreamer/) does not work anymore.
Starting livestreamer shows this trace:

  File "\python\python35\lib\site-packages\livestreamer_cli\compat.py", line 19,
 in <module>
    stdout = sys.stdout.buffer
AttributeError: 'TextTranscodingWrapper' object has no attribute 'buffer'

You can find the code of livestreamer_cli/compat.py here https://github.com/chrippa/livestreamer/blob/develop/src/livestreamer_cli/compat.py .

According to the readme Doing so should not break executed scripts in any way. Otherwise, it is a bug of win_unicode_console that should be fixed..
I know I can circumvent this issue but it would still be nice to have it fixed.

Can't run IDLE after enabling win-unicode-console

As soon as I put

import win_unicode_console


win_unicode_console.enable()

in sitecustomize.py, IDLE won't run anymore (without showing any error message).

If I comment out the #win_unicode_console.enable() function call (only keeping the import statement) IDLE works again.

If I open an IDLE shell, reenable the win_unicode_console.enable(), then try to restart the shell, the following error shows up:

Subprocess Startup Error

IDLE's subprocess didn't make connection. Either IDLE can't start a subprocess or personal firewall software is blocking the connection.

OK

Another interesting observation is that calling
py "C:\Users\a\AppData\Local\Programs\Python\Python35\Lib\idlelib\idle.pyw" in cmd works fine and IDLE gets launched. But if I call idle.pyw using pyw.exe (the way the IDLE shortcut works by default) then nothing happens:

C:\Users\a>pyw "C:\Users\a\AppData\Local\Programs\Python\Python35\Lib\idlelib\idle.pyw"

C:\Users\a>C:\Users\a\AppData\Local\Programs\Python\Python35\Lib\idlelib\idle.pyw

C:\Users\a>

I'm running

Python 3.5.2 (v3.5.2:4def2a2901a5, Jun 25 2016, 22:18:55) [MSC v.1900 64 bit (AMD64)] on win32
Windows 10 64-bit
win-unicode-console (0.5)

Does anyone know why this is happening?