pexpect / ptyprocess Goto Github PK

Run a subprocess in a pseudo terminal

Home Page: https://ptyprocess.readthedocs.io/en/latest/

License: Other

Python 100.00%

ptyprocess's Introduction

Launch a subprocess in a pseudo terminal (pty), and interact with both the process and its pty.

Sometimes, piping stdin and stdout is not enough. There might be a password prompt that doesn't read from stdin, output that changes when it's going to a pipe rather than a terminal, or curses-style interfaces that rely on a terminal. If you need to automate these things, running the process in a pseudo terminal (pty) is the answer.

Interface:

from ptyprocess import PtyProcessUnicode
p = PtyProcessUnicode.spawn(['python'])
p.read(20)
p.write('6+6\n')
p.read(20)

ptyprocess's People

Contributors

Stargazers

Watchers

Forkers

anwilli5 tomspur mistrymj takluyver detly reynir blink1073 nad2000 sccolbert liubinhit njwhite seasar mitizhi wayaya masteret oathops minrk satyakrish asmodehn shmaix snowwalkerj lophocinalis jdemeyer ryanpetrello n0k0 jishac eugpermar kolyshkin sn-donbenjamin petesh wilvk fouriersignal peterusa123 yongfuxie cykutw ossdev07 coderclaudechan third-party-oneoffs pkfec ra2003 jk1ng pacman-syu liuxuehao craigphicks ut-382v-ashkan-and-david rwaycachedlibs mingsxs cclauss dong-zeyu bobrik dwoz saiprasad16 dustinreed-info alvistack strogo alancoding xtherry ibm-z-oss-oda hugovk arpitjain799 hroncok sysfce2 gwolfgit hellobrobro andrews239 wolfi-chainguard-demo johncaged keith-rollin zhangjie1969 yanorepuser4 chainguard-wolfi-bites-back

ptyprocess's Issues

test_spawn_sh occasionally fails

When I started testing ptyprocess with Python 3.11 I found out that test_spawn_sh fails with:

E       AssertionError: assert b'echo $ENV_KEY; exit 0' in b'\rjkulik@kuldavm-sparc-6717:...process/ptyprocess-0.7.0/tests$ echo $ENV \x08_KEY; exit 0\r\nenv_value\r\n'                                                
E        +  where b'echo $ENV_KEY; exit 0' = <built-in method strip of bytes object at 0x1fffc6b2a9f8b0>()                                                                                                              
E        +    where <built-in method strip of bytes object at 0x1fffc6b2a9f8b0> = b'echo $ENV_KEY; exit 0\n'.strip

I did some testing and it seems that this is caused by some kind of a race condition / it's speed dependent. When I run the test suite for the second time, it passes. But then when I delete the __pycache__ directory, it fails again.

I also found out that when I add a sleep after the spawn, it fails every time in every Python (and not just this test, test_spawn_sh_unicode as well):

--- ptyprocess-0.7.0/tests/test_spawn.py
+++ ptyprocess-0.7.0/tests/test_spawn.py
@@ -21,6 +21,7 @@ class PtyTestCase(unittest.TestCase):
     def _spawn_sh(self, ptyp, cmd, outp, env_value):
         # given,
         p = ptyp.spawn(['sh'], env=self.env)
+        time.sleep(1)
         p.write(cmd)
 
         # exercise,

ptyprocess 0.5.1 and 0.5.2 appear to be missing the setup.py file.

pypi tar.gz has pycache/.pyc files in tests directory

Getting https://pypi.python.org/packages/source/p/ptyprocess/ptyprocess-0.4.tar.gz to ./ptyprocess-0.4.tar.gz
$ tar tf ./ptyprocess-0.4.tar.gz
ptyprocess-0.4/
ptyprocess-0.4/ptyprocess/
ptyprocess-0.4/ptyprocess/util.py
ptyprocess-0.4/ptyprocess/ptyprocess.py
ptyprocess-0.4/ptyprocess/_fork_pty.py
ptyprocess-0.4/ptyprocess/__init__.py
ptyprocess-0.4/setup.py
ptyprocess-0.4/PKG-INFO
ptyprocess-0.4/tests/
ptyprocess-0.4/tests/test_preexec_fn.py
ptyprocess-0.4/tests/test_spawn.py
ptyprocess-0.4/tests/test_invalid_binary.py
ptyprocess-0.4/tests/__pycache__/
ptyprocess-0.4/tests/__pycache__/__init__.cpython-34.pyc
ptyprocess-0.4/tests/__pycache__/test_spawn.cpython-27-PYTEST.pyc
ptyprocess-0.4/tests/__pycache__/test_preexec_fn.cpython-34-PYTEST.pyc
ptyprocess-0.4/tests/__pycache__/test_preexec_fn.cpython-27-PYTEST.pyc
ptyprocess-0.4/tests/__pycache__/test_spawn.cpython-34-PYTEST.pyc
ptyprocess-0.4/tests/__pycache__/test_invalid_binary.cpython-27-PYTEST.pyc
ptyprocess-0.4/tests/__pycache__/test_invalid_binary.cpython-34-PYTEST.pyc
ptyprocess-0.4/tests/__init__.pyc
ptyprocess-0.4/tests/__init__.py
ptyprocess-0.4/README.rst

This results in build errors on EL7:

+ py.test
============================= test session starts ==============================
platform linux2 -- Python 2.7.5 -- pytest-2.3.5
collected 0 items / 3 errors
==================================== ERRORS ====================================
________________ ERROR collecting tests/test_invalid_binary.py _________________
import file mismatch:
imported module 'tests.test_invalid_binary' has this __file__ attribute:
  /home/takluyver/Code/ptyprocess/tests/test_invalid_binary.py
which is not the same as the test file we want to collect:
  /builddir/build/BUILD/ptyprocess-0.4/tests/test_invalid_binary.py
HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules
__________________ ERROR collecting tests/test_preexec_fn.py ___________________
import file mismatch:
imported module 'tests.test_preexec_fn' has this __file__ attribute:
  /home/takluyver/Code/ptyprocess/tests/test_preexec_fn.py
which is not the same as the test file we want to collect:
  /builddir/build/BUILD/ptyprocess-0.4/tests/test_preexec_fn.py
HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules
_____________________ ERROR collecting tests/test_spawn.py _____________________
import file mismatch:
imported module 'tests.test_spawn' has this __file__ attribute:
  /home/takluyver/Code/ptyprocess/tests/test_spawn.py
which is not the same as the test file we want to collect:
  /builddir/build/BUILD/ptyprocess-0.4/tests/test_spawn.py
HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules
=========================== 3 error in 0.04 seconds ============================

Document PC_MAX_CANON limitations

From pexpect/pexpect#79

Release?

I see that pr #49 got merged, but no release was ever made, so this isnt available on pypi. Could you create a release for this and update pypi so that (among other things) pexpect/pexpect#546 can be completed and made available?

PtyProcess.spawn (and thus pexpect) slowdown in close() loop

The following code in ptyprocess

ptyprocess/ptyprocess/ptyprocess.py

Lines 260 to 269 in 3931cd4

    
           # Do not allow child to inherit open file descriptors from parent, 
        
           # with the exception of the exec_err_pipe_write of the pipe 
        
           # and pass_fds. 
        
           # Impose ceiling on max_fd: AIX bugfix for users with unlimited 
        
           # nofiles where resource.RLIMIT_NOFILE is 2^63-1 and os.closerange() 
        
           # occasionally raises out of range error 
        
           max_fd = min(1048576, resource.getrlimit(resource.RLIMIT_NOFILE)[0]) 
        
           spass_fds = sorted(set(pass_fds) | {exec_err_pipe_write}) 
        
           for pair in zip([2] + spass_fds, spass_fds + [max_fd]): 
        
               os.closerange(pair[0]+1, pair[1])

is looping through all possible file descriptors in order to close those (note that closerange() implemented as a loop at least on Linux). In case the limit of open fds (aka ulimit -n, aka RLIMIT_NOFILE, aka SC_OPEN_MAX) is set too high (for example, with recent docker it is 1024*1024), this loop takes considerable time (as it results in about a million close() syscalls).

The solution (at least for Linux and Darwin) is to obtain the list of actually opened fds, and only close those. This is implemented in subprocess module in Python3, and there is a backport of it to Python2 called subprocess32.

This issue was originally reported to docker: docker/for-linux#502

Other good reason for using subprocess (being multithread-safe) is described in #43

isalive(): check process status for non-child process

The original implementation of isalive uses os.waitpid to check if the ptyprocess is alive
os.waitpid can only check the process status if that process is a child of the current process

If the pexpect object is passed from process A to process B
process B can access the pexpect object but it cannot use it because os.waitpid can't find the ptyprocess under the child list of process B
it will throw the no such process error
while the process actually exists and is still running

To solve this
It can use os.kill to send signal 0 to check if process is alive
which achieve the same functionality of os.waitpid but not limit it to child process

Is it possible to add this as an extra check for isalive()?

Potential performance issue with unbuffered IO and the PtyProcess readline method

Calls to self.fileobj.readline() from the PtyProcess readline() method read data one byte at a time (most likely since fileobj is opened with 'buffering=0'.) Thus, this program:

from ptyprocess import PtyProcess, PtyProcessUnicode
p = PtyProcess.spawn(['perl',  '-e', '''use 5.010; foreach my $letter ('a'..'z'){ say $letter x 1000; }'''])
while(1):
    try:
        print p.readline()
    except EOFError:
        break
p.close()

Has pretty poor performance (output from strace):

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 93.48    0.465020          18     26214         1 read
  2.28    0.011353          23       489       381 open
  0.84    0.004197        4197         1           clone
  0.61    0.003037          19       160       113 stat

Is there a compelling reason to specify that the fileobj should have unbuffered IO?

The PtyProcess read() method does not experience this behavior because it uses a default buffer size of 1024.

is it possible to PtyProcess.read() without blocking?

I'm running ptyprocess in read in a greenlet coroutine. so when I .read() and there's no data, it blocks and stops all the other greenlets. is it possible to read without blocking, or at least check if there's any new data?

Sorry it this is a silly question

Thank you

A minor issue found via static analysis

ExceptionPexpect is not defined.

ptyprocess/ptyprocess/_fork_pty.py

Line 66 in 8f1ba73

raise ExceptionPexpect("OSError of errno.ENXIO should be raised.")

Found using cython and
https://gist.github.com/tzickel/ea4541b12c1298a8092c3ec5043ef77e

pip install -I --root temp_ptyprocess ptyprocess
python checkcython.py temp_ptyprocess

The positions of the two arguments in setwinsize function appear to be reversed

Recently I made a Web terminal with ptyprocess, websocket, and XTerm, but eventually found that the Web terminal size could not adapt after the resize event was triggered.

I even thought my computer was actually a big phone until I switched the positions of two parameters and the page appeared normal.

Before:

def _setwinsize(fd, rows, cols):
    # Some very old platforms have a bug that causes the value for
    # termios.TIOCSWINSZ to be truncated. There was a hack here to work
    # around this, but it caused problems with newer platforms so has been
    # removed. For details see https://github.com/pexpect/pexpect/issues/39
    TIOCSWINSZ = getattr(termios, 'TIOCSWINSZ', -2146929561)
    # Note, assume ws_xpixel and ws_ypixel are zero.
    s = struct.pack('HHHH', rows, cols, 0, 0)
    fcntl.ioctl(fd, TIOCSWINSZ, s)

After using the following modified code, the interface display is normal:

def _setwinsize(fd, rows, cols):
    # Some very old platforms have a bug that causes the value for
    # termios.TIOCSWINSZ to be truncated. There was a hack here to work
    # around this, but it caused problems with newer platforms so has been
    # removed. For details see https://github.com/pexpect/pexpect/issues/39
    width, height = cols, rows
    TIOCSWINSZ = getattr(termios, 'TIOCSWINSZ', -2146929561)
    # Note, assume ws_xpixel and ws_ypixel are zero.
    s = struct.pack('HHHH', width, height, 0, 0)
    fcntl.ioctl(fd, TIOCSWINSZ, s)

Help to the xonsh shell

Hello! Thank you for awesome work here!

I just want to let you know about xonsh shell project and that your awesome knowledge can help this project very high.

During my effort around understanding and fixing cases around running interactive tools I diven into PTY I found ptyprocess.

If you have time and chance to take a look into how xonsh works with terminal it will be very helpful to this pure open source crowd development project.

Thanks!

For community

⬇️ Please click the 👍 reaction instead of leaving a +1 or 👍 comment

ptyprocess fails on OpenIndiana 151a9 and Python 2.7.11 when the process lacks a controlling terminal

pexpect is failing in ptyprocess.spawn on OpenIndiana 151a9 (a distro of Illumos/OpenSolaris). pexpect is 4.1.0, ptyprocess is 0.5.1:

Traceback (most recent call last):
  File "/home/majid/bin/upload_file.py", line 26, in <module>
    upload(fn)
  File "/home/majid/bin/upload_file.py", line 7, in upload
    timeout=14400)
  File "/usr/local/lib/python2.7/site-packages/pexpect/pty_spawn.py", line 198, in __init__
    self._spawn(command, args, preexec_fn, dimensions)
  File "/usr/local/lib/python2.7/site-packages/pexpect/pty_spawn.py", line 298, in _spawn
    cwd=self.cwd, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/pexpect/pty_spawn.py", line 309, in _spawnpty
    return ptyprocess.PtyProcess.spawn(args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/ptyprocess/ptyprocess.py", line 223, in spawn
    pid, fd = _fork_pty.fork_pty()
  File "/usr/local/lib/python2.7/site-packages/ptyprocess/_fork_pty.py", line 30, in fork_pty
    pty_make_controlling_tty(child_fd)
  File "/usr/local/lib/python2.7/site-packages/ptyprocess/_fork_pty.py", line 76, in pty_make_controlling_tty
    fd = os.open("/dev/tty", os.O_WRONLY)
OSError: [Errno 6] No such device or address: '/dev/tty'

The script is run under the control of at(1), and the job itself is queued from a crontab. It looks like what is happening is that when the parent process, which has no controlling terminal, calls os.openpty(), it acquires the pty as a controlling terminal. The problem seems to occur when posix_openpty pushes the ptem STREAMS module onto the slave pty fd, the simple call ioctl(slave_fd, I_PUSH, "ptem") causes the parent process to acquire the pty as its controlling terminal, and thus the child, which is in a different session, is unable to acquire it and the failure at line 76 in _fork_pty().

I diffed posixmodule.c on Python 3.5 vs 2.7.11, the implementation of posix_openpty is not much different.

Older versions of pexpect had a different implementation _svr4_openpty(), but those look very similar to the Python posixmodule.c code.

pty cannot read unicode color escapde

My company has a ssh server, which would return QR code (by sending unicode and color escape), show in screenshot.

ssh [email protected]

the intact output of ssh session is in the attachment ssh_output.log

However if i tried to use ptyprocess to read it , it just read the first 3 bytes and then hangs forever.

#!/usr/bin/env python
import ptyprocess

p = ptyprocess.PtyProcess.spawn(["/usr/bin/ssh", "[email protected]"])
s = p.read(1000)
print(s)  # this line return b'\x1b[c'
s = p.read(1000) # this line blocks forever

Why would this happen and how can i solve it?

FreeBSD fails fork_pty: OSError: [Errno 6] Device not configured: '/dev/tty'

Got a FreeBSD (digital ocean droplet, freebsd.pexpect.org) build agent prepared. It raises exception very early in critical codepath causing test runner to fork and eventually the build agent is killed by the kernel due to an OOM condition.

Error is in method pty_make_controlling_tty at:

        # Verify we now have a controlling tty.
        fd = os.open("/dev/tty", os.O_WRONLY)

[freebsd@freebsd ~]$ sudo -u teamcity -s
$ cd /opt/TeamCity/work/210ae16cc3f30c30/ptyprocess
$ . `which virtualenvwrapper.sh`
$ mkvirtualenv pexpect27 --python=`which python2.7`
$ pip install -e .
$ cd ../pexpect
$ python
Python 2.7.9 (default, Jan  8 2015, 21:47:19)
[GCC 4.2.1 Compatible FreeBSD Clang 3.3 (tags/RELEASE_33/final 183502)] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
>>> import pexpect
>>> bash = pexpect.spawn('/bin/bash')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pexpect/pty_spawn.py", line 189, in __init__
    self._spawn(command, args, preexec_fn)
  File "pexpect/pty_spawn.py", line 281, in _spawn
    cwd=self.cwd, **kwargs)
  File "/opt/TeamCity/work/210ae16cc3f30c30/ptyprocess/ptyprocess/ptyprocess.py", line 220, in spawn
    pid, fd = _fork_pty.fork_pty()
  File "/opt/TeamCity/work/210ae16cc3f30c30/ptyprocess/ptyprocess/_fork_pty.py", line 30, in fork_pty
    pty_make_controlling_tty(child_fd)
  File "/opt/TeamCity/work/210ae16cc3f30c30/ptyprocess/ptyprocess/_fork_pty.py", line 76, in pty_make_controlling_tty
    fd = os.open("/dev/tty", os.O_WRONLY)
OSError: [Errno 6] Device not configured: '/dev/tty'

/dev/tty may be opened under normal conditions.

Rather than closing all the file descriptors, set close on exec on them

The code that closes all the file descriptors in the child process before exec breaks practically any and all efforts to debug exec errors; especially if they're not OSErrors.

In my case, I was debugging under pycharm. pydevd monkeypatched all the exec routines and because of a behavioural change in pexpect, where it re-encoded all the arguments as utf-8 encoded binary, rather than strings, when pydevd tried to detect the presence of python in it's code, it crashed out because there was encoding of the strings and bstring.endswith() needs to be passed a b'' string, or else it bombs out.

Incorrect distributive in WHEEL format

I had tried to install the ptyprocess package from "whl" file and got an error:

$ ./bin/wheel install -d ~/wheels/ ptyprocess
Traceback (most recent call last):
  File "./bin/wheel", line 11, in <module>
    sys.exit(main())
  File "/home/cykooz/testvenv/local/lib/python2.7/site-packages/wheel/tool/__init__.py", line 356, in main
    args.func(args)
  File "/home/cykooz/testvenv/local/lib/python2.7/site-packages/wheel/tool/__init__.py", line 299, in install_f
    args.wheel_dirs, args.force, args.list_files)
  File "/home/cykooz/testvenv/local/lib/python2.7/site-packages/wheel/tool/__init__.py", line 221, in install
    wf.install(force=force)
  File "/home/cykooz/testvenv/local/lib/python2.7/site-packages/wheel/install.py", line 268, in install
    if self.parsed_wheel_info['Root-Is-Purelib'] == 'true':
  File "/home/cykooz/testvenv/local/lib/python2.7/site-packages/wheel/decorator.py", line 17, in __get__
    val = self.wrapped(inst)
  File "/home/cykooz/testvenv/local/lib/python2.7/site-packages/wheel/install.py", line 238, in parsed_wheel_info
    return read_pkg_info_bytes(self.zipfile.read(self.wheelinfo_name))
  File "/usr/lib/python2.7/zipfile.py", line 935, in read
    return self.open(name, "r", pwd).read()
  File "/usr/lib/python2.7/zipfile.py", line 630, in read
    data = self.read1(n)
  File "/usr/lib/python2.7/zipfile.py", line 692, in read1
    self._update_crc(data, eof=eof)
  File "/home/cykooz/testvenv/local/lib/python2.7/site-packages/wheel/install.py", line 463, in _update_crc
    raise BadWheelFile("Bad hash for file %r" % ef.name)
wheel.install.BadWheelFile: Bad hash for file 'ptyprocess-0.5.1.dist-info/WHEEL'

This is because of hashes of files (saved in file ptyprocess-0.5.1-py2.py3-none-any.whl/ptyprocess-0.5.1.dist-info/RECORD) stored in invalid format. Instead of using BASE64-encoded SHA256-digest your hashes stored as HEX-string from SHA256 digest.

Please, rebuild WHL distributive of your package and put it into pypi.python.org.

PtyProcess.spawn (and thus pexpect) is not safe for use in multithreaded applications

Just the same as the old pure Python subprocess module from Python 2, the PtyProcess.spawn method is not async signal safe. The entire fork() to exec code path must be written in async signal safe C code. Both pty.fork and your own PtyProcess.spawn method violate this constraint.

See the Python 3 subprocess module's _posixsubprocess extension module (and the https://pypi.python.org/pypi/subprocess32/ backport for Python 2 users).

Likely symptoms when this bites you: random seeming deadlocks and hangs.

Workaround for users today: Do not use pexpect.spawn or pexpect.pty_spawn in a multithreaded application.
A pexpect.popen_spawn module appears to exist and will use the subprocess module. If you are on 2.7 make sure you've replaced sys.modules['subprocess'] with a reference to the subprocess32 module first (if you haven't already done so within your Python 2.7 interpreter's stdlib)

0.5.2 not really released?

The 0.5.2 release doesn't appear in github's list of releases: perhaps you forgot to tag it?

Race condition in tests?

There seems to be some kind of race condition in the testsuite on both python-3.4 and python-2.7 here on Fedora 21 (it seems to occour more often in python-3.4 though).

This is the log of the tests:
self = <tests.test_spawn.PtyTestCase testMethod=test_spawn_sh>

    def test_spawn_sh(self):
        env = os.environ.copy()
        env['FOO'] = 'rebar'
        p = PtyProcess.spawn(['sh'], env=env)
        p.read()
        p.write(b'echo $FOO\n')
        time.sleep(0.1)
        response = p.read()
        assert b'rebar' in response

        p.sendeof()
        p.read()

        with self.assertRaises(EOFError):
>           p.read()
E           AssertionError: EOFError not raised

tests/test_spawn.py:21: AssertionError

What could cause these random tests? How can I help debugging it?

Use stdin in child process

Is it possible?

I'd like to spawn a process that reads stdin.

cat requirements.txt | ./script_that_spawns.py safety --check --stdin

When doing so and trying to read safety output, it blocks, and I have to interrupt it with control-c:

Traceback (most recent call last):
  File "/home/pawamoy/.cache/pypoetry/virtualenvs/mkdocstrings-ytlBmpdO-py3.8/bin/failprint", line 8, in <module>
    sys.exit(main())
  File "/home/pawamoy/.cache/pypoetry/virtualenvs/mkdocstrings-ytlBmpdO-py3.8/lib/python3.8/site-packages/failprint/cli.py", line 125, in main
    return run(
  File "/home/pawamoy/.cache/pypoetry/virtualenvs/mkdocstrings-ytlBmpdO-py3.8/lib/python3.8/site-packages/failprint/cli.py", line 54, in run
    output.append(process.read())
  File "/home/pawamoy/.cache/pypoetry/virtualenvs/mkdocstrings-ytlBmpdO-py3.8/lib/python3.8/site-packages/ptyprocess/ptyprocess.py", line 818, in read
    b = super(PtyProcessUnicode, self).read(size)
  File "/home/pawamoy/.cache/pypoetry/virtualenvs/mkdocstrings-ytlBmpdO-py3.8/lib/python3.8/site-packages/ptyprocess/ptyprocess.py", line 516, in read
    s = self.fileobj.read1(size)
KeyboardInterrupt

Here is the actual Python code I'm using:

process = PtyProcessUnicode.spawn(cmd)

output = []

while True:
    try:
        output.append(process.read())
    except EOFError:
        break

process.close()

use os.posix_spawn when available (i.e, 3.8)

The attached patch is an experiment in using Python 3.8's os.posix_spawn() in ptyprocess. Since it doesn't use fork() it eliminates all those problems. A simple test using pexpect.interact() seemed to work.

I know it has a race with inheritable when called in parallel; and I'm sure there's more.

ptyprocess-posix-spawn.patch.gz

The change looks bigger than it is because I indented all the old code. Below is what matters, which I've included so it is easier to pick it apart....

    if hasattr(os, 'posix_spawn'):
        print("using posix_spawn")
        # Issue 36603: Use os.openpty() (and try to avoid the
        # whole pty module) as that guarentees inheritable (if it
        # ever fails then just file a bug against os.openpty()
        fd, tty = os.openpty()
        # Try to set window size on TTY per below; but is this
        # needed?
        try:
            _setwinsize(tty, *dimensions)
        except IOError as err:
            if err.args[0] not in (errno.EINVAL, errno.ENOTTY):
                raise
        # Try to disable echo if spawn argument echo was unset per
        # below; but does this work?
        if not echo:
            try:
                _setecho(tty, False)
            except (IOError, termios.error) as err:
                if err.args[0] not in (errno.EINVAL, errno.ENOTTY):
                    raise
        # Create the child: convert the tty into STDIO; use the
        # default ENV if needed; and try to make the child the
        # session head using SETSID.  Assume that all files have
        # inheritable (close-on-exec) correctly set.
        file_actions=[
            (os.POSIX_SPAWN_DUP2, tty, STDIN_FILENO),
            (os.POSIX_SPAWN_DUP2, tty, STDOUT_FILENO),
            (os.POSIX_SPAWN_DUP2, tty, STDERR_FILENO),
            (os.POSIX_SPAWN_CLOSE, tty),
            (os.POSIX_SPAWN_CLOSE, fd),
        ]
        spawn_env = env or os.environ
        pid = os.posix_spawn(command, argv, spawn_env,
                             file_actions=file_actions,
                             setsid=True)
        # Child started.  Now close tty and stop PTY(FD) being
        # inherited. Note that there's a race here: a parallel
        # fork/exec would unwittingly inherit this PTY(FD)/TTY
        # pair.  Probably need to wrap all this in a lock?
        os.close(tty)
        os.set_inheritable(fd, False)

Remove all sleep calls?

There are still a number of time.sleep() calls, particularly in close() and terminate(). This is bad for async code, because this will hog control when the event loop could be doing other things. I'd like to work out how to remove these. If that isn't possible, it should at least be marked very clearly which methods do this.

The preexec_fn should be executed before closing the file descriptors.

Currently, the preexec_fn is executed after the file descriptors are closed. This has some unwanted effects:

if preexec_fn opens a file descriptor, it will be inherit by the child process
if preexec_fn relays on having some file descriptor open, it will crash (see pexpect/pexpect#368)

The proposal is to move the "close the fds" section below the "execute the preexec_fn" code: https://github.com/pexpect/ptyprocess/blob/master/ptyprocess/ptyprocess.py#L266-L285

For reference, this is what it does subprocess.Popen: https://github.com/python/cpython/blob/master/Modules/_posixsubprocess.c#L528-L549

If it is okay, I can do a PR with the fix but I would like to hear your opinions about this.

test failure on FreeBSD

Running the tests on FreeBSD 11 I see:

% py.test  
============================= test session starts ==============================
platform freebsd11 -- Python 2.7.9 -- py-1.4.26 -- pytest-2.6.4
collected 5 items 

tests/test_invalid_binary.py .
tests/test_preexec_fn.py ..
tests/test_spawn.py FF


=================================== FAILURES ===================================
__________________________ PtyTestCase.test_spawn_sh ___________________________

self = <tests.test_spawn.PtyTestCase testMethod=test_spawn_sh>

    def test_spawn_sh(self):
        env = os.environ.copy()
        env['FOO'] = 'rebar'
        p = PtyProcess.spawn(['sh'], env=env)
        p.read()
        p.write(b'echo $FOO\n')
        time.sleep(0.1)
        response = p.read()
>       assert b'rebar' in response
E       AssertionError: assert 'rebar' in 'echo $FOO\r\n'

tests/test_spawn.py:15: AssertionError
______________________ PtyTestCase.test_spawn_unicode_sh _______________________

self = <tests.test_spawn.PtyTestCase testMethod=test_spawn_unicode_sh>

    def test_spawn_unicode_sh(self):
        env = os.environ.copy()
        env['FOO'] = 'rebar'
        p = PtyProcessUnicode.spawn(['sh'], env=env)
        p.read()
        p.write(u'echo $FOO\n')
        time.sleep(0.1)
        response = p.read()
>       assert u'rebar' in response
\n'     AssertionError: assert 'rebar' in 'echo $FOO

tests/test_spawn.py:31: AssertionError
====================== 2 failed, 3 passed in 0.80 seconds ======================

The expected output is produced, it is just not returned by the second p.read() in the test. Inserting a dummy p.read() before response = p.read() gets the tests passing.

PtyProcess.spawn just got slower and can trigger problems

This is somewhat related to #43.

Python 3.7.1 contains the change https://bugs.python.org/issue6721 that is trying to avoid a deadlock in the logging code. It works by having os.fork() grab all the logging locks before executing fork(). It's expensive - if the forked process is just going to exec then os.spawn() is preferred. Its also creating, lets say, interesting problems (deadlocks) with code that was working.

Since PtyProcess calls pty.fork() (which calls os.fork() ...) it triggering this code path.

Using some equivalent of os.spawn() would eliminate this.

Wait() should return normally if the child process has already terminated

Calling wait() on a PtyProcess object whose process has already terminated will raise an exception. This behavior is unexpected[1].
I have made a small test to show case the behavior. Using p.isalive() && p.wait() will only make the race condition worse in my tests.

[1]: subprocess.Popen.wait() returns normally even when the process has already terminated, although it doesn't say so explicitly in the documentation. In java the waitFor() method "returns immediately if the subprocess has already terminated."

Logging of stdout and stderr

Hi :)

I would like to subprocess any given command and log its stdout and stderr separately. This would seem like an easy thing to do, but im having no luck because:

Processes which detect that their stdout/stderr are not ttys will modify their output
Processes which detect that their stdout/stderr are not the same file path will assume theres stdout redirection and modify their output.

So in jumps ptys and ptyprocess to the resuce. Tie the stdout and stderr to a pty, you get 1 file path like /dev/tty0021, and isatty() returns true for both. Problem is - now we can't distinguish stdout from stderr.
Ok, no problem, just make two ptys - one for stdout and one for stderr. But now although both pass isatty(), their file paths will now look like /dev/tty0021 and /dev/tty0022 (for example).
The subprocess reacts as if you weren't using a pty in the first place, and you log nothing.

I have been trying for four months to figure a way around this, and because you are the expert in ptys I thought i might ask you directly - could you think of a way to log stdout and stderr separatly in your program, and still fool a script like this: https://bpaste.net/show/000d6f70ef41

THANK YOU :D

PtyProcess.read() returns a different value every call

This is a very severe bug. When calling Ptyprocess.read() the value returned is different almost every time:

ptyprocess.PtyProcess.spawn(['openssl', "ec", '-noout', '-text', '-in', '/opt/key/s128r1.key']).read()

The output:

And again with the same params:

And again:

I don't know what is causing this but this is very weird.

Release?

I just ran into #38, and can confirm installing from master fixes the issue. It would be nice to have a new release of ptyprocess that includes that fix.

ptyprocess.read() does not read last two lines

I can see what's happening in the real terminal with "OK" in the end of the response.
But the read method does not return the last two lines.

    @staticmethod
    def communicate(write_uuid, read_uuid, data):
        
        # Requires a writable and a readable GATT uuid and data to be written
        # The method returns the data retrieved from the readable GATT uuid
        
        hex_string = ''
        for char in data:
            hex_string += hex(ord(char)) + ' '

        timeout = .1
        read_process = ptyprocess.PtyProcessUnicode.spawn(['/bin/bash'])
        commands = ['bluetoothctl', 'menu gatt', f'select-attribute {read_uuid}', 'notify on']
        for command in commands:
            read_process.write(command + '\n')
            time.sleep(timeout)
            read_process.read(1)

        write_process = ptyprocess.PtyProcessUnicode.spawn(['/bin/bash'])
        commands = f'bluetoothctl <<EOF\nmenu gatt\nselect-attribute {write_uuid}\nwrite "{hex_string}"\nEOF\n'
        write_process.write(commands)
        time.sleep(timeout)
        write_process.read(1)
        time.sleep(timeout)

        hex_values = []
        pattern = re.compile(r'\b[0-9a-fA-F]{2}\b')
        lines = read_process.read(size=10000).splitlines()
        for line in lines:
            print(line)

This is the response of the method:

[CHG] Attribute /org/bluez/hci0/dev_CE_EE_A8_9B_17_AC/service000b/char000e Value:
[Device:/service000b/char000e]#                                              
  30 2e 30 2c 20 30 2e 30 2c 20 30 2e 30 2c 20 30  0.0, 0.0, 0.0, 0
[Device:/service000b/char000e]#                                              
  2e 30 2c 20 30 2e 30 2c 20 30 2e 30 2c 20 30 2e  .0, 0.0, 0.0, 0.
[Device:/service000b/char000e]#                                              
  30 2c 20 30 2e 30 2c 20 30 2e 30 2c 20 30 2e 30  0, 0.0, 0.0, 0.0
[Device:/service000b/char000e]#                                              
  2c 20 30 2e 30 2c 20 30 2e 30 2c 20 30 2e 30 2c  , 0.0, 0.0, 0.0,
[Device:/service000b/char000e]�[

This should be the correct response:

[Device:/service000b/char000e]# [CHG] Attribute /org/bluez/hci0/dev_CE_EE_A8_9B_17_AC/service000b/char000e Value:
[Device:/service000b/char000e]#   30 2e 30 2c 20 30 2e 30 2c 20 30 2e 30 2c 20 30  0.0, 0.0, 0.0, 0
[Device:/service000b/char000e]#   2e 30 2c 20 30 2e 30 2c 20 30 2e 30 2c 20 30 2e  .0, 0.0, 0.0, 0.
[Device:/service000b/char000e]#   30 2c 20 30 2e 30 2c 20 30 2e 30 2c 20 30 2e 30  0, 0.0, 0.0, 0.0
[Device:/service000b/char000e]#   2c 20 30 2e 30 2c 20 30 2e 30 2c 20 30 2e 30 2c  , 0.0, 0.0, 0.0,
[Device:/service000b/char000e]#   20 30 2e 30 2c 20 30 2e 30 2c 20 30 2e 30 2c 20   0.0, 0.0, 0.0, 
[Device:/service000b/char000e]#   30 2e 30 2c 20 30 2e 30 20 4f 4b 0a              0.0, 0.0 OK.

As you can see we have 18 values and an OK at the end of the correct response.
While we only have 13 values without OK in the read() method's response.
Also: The read method returns an ESC character for some reason.

ptyprocess 0.5.1

Hello,

I was wondering if we could make ptyprocess 0.5.1 available for [legacy] easy-install installations

Thanks

Carlos

Race condition (fail) in tests.test_spawn.PtyTestCase.test_spawn_sh_unicode

The test is likely at fault. It is attempting to see its own keyboard input in output, but the prompt is displaying late, causing the familiar "input echo intermittently spanned in output" condition:

AssertionError: assert 'echo $ENV_KEY; exit 0' in 'echo $EN(teamcity-pexpect)(teamcity-pexpect)V_KEY; exit 0\r\nenv_value\r\n'

Full output and link

self = <tests.test_spawn.PtyTestCase testMethod=test_spawn_sh_unicode>
ptyp = <class 'ptyprocess.ptyprocess.PtyProcessUnicode'>
cmd = 'echo $ENV_KEY; exit 0\n'
outp = 'echo $EN(teamcity-pexpect)(teamcity-pexpect)V_KEY; exit 0\r\nenv_value\r\n'
env_value = 'env_value'

    def _spawn_sh(self, ptyp, cmd, outp, env_value):
        # given,
        p = ptyp.spawn(['sh'], env=self.env)
        p.write(cmd)

        # exercise,
        while True:
            try:
                outp += p.read()
            except EOFError:
                break

        # verify, input is echo to output
>       assert cmd.strip() in outp
E       AssertionError: assert 'echo $ENV_KEY; exit 0' in 'echo $EN(teamcity-pexpect)(teamcity-pexpect)V_KEY; exit 0\r\nenv_value\r\n'
E        +  where 'echo $ENV_KEY; exit 0' = <built-in method strip of str object at 0x7f2231b5f540>()
E        +    where <built-in method strip of str object at 0x7f2231b5f540> = 'echo $ENV_KEY; exit 0\n'.strip

https://teamcity-master.pexpect.org/viewLog.html?buildId=13605&buildTypeId=Pexpect_LinuxBuild&tab=buildLog

	# Do not allow child to inherit open file descriptors from parent,
	# with the exception of the exec_err_pipe_write of the pipe
	# and pass_fds.
	# Impose ceiling on max_fd: AIX bugfix for users with unlimited
	# nofiles where resource.RLIMIT_NOFILE is 2^63-1 and os.closerange()
	# occasionally raises out of range error
	max_fd = min(1048576, resource.getrlimit(resource.RLIMIT_NOFILE)[0])
	spass_fds = sorted(set(pass_fds) \| {exec_err_pipe_write})
	for pair in zip([2] + spass_fds, spass_fds + [max_fd]):
	os.closerange(pair[0]+1, pair[1])