Potential performance issue with unbuffered IO and the PtyProcess readline method about ptyprocess HOT 7 CLOSED

Is there a compelling reason to specify that the fileobj should have unbuffered IO?

Yes, for terminals, stdout is line-buffered, and stderr is unbuffered: We need to be able to read bytes waiting for input that have not yet sent a line-terminator, such as progress bars or screen-oriented programs using ncurses that never emit a line terminator.

I could be wrong, though. This will require writing tests (and a sub-program that emits 1-byte at a time without flushing, to both stdout and stderr), and various assertions about reading single bytes, blocks of bytes, and lines.

Then, we could simply change the argument in io.open(fd, 'r+b', buffering=0) to -1 or so, and see what happens. If it fails the tests, then the only thing we could do for you is allow you to specify your own buffering option, documenting the differences of behavior, and furthermore writing tests for such behavior.

At least with buffering=0 i know it works without so much testing :)

from ptyprocess.

I don't think I fully understand what you mean, so I apologize if this doesn't make sense.

For a script using PtyProcess, would it make sense for it to call 'readline' if it expected to work with data that did not return a line terminator? Or for it to call 'readline' if it didn't know what to expect from the output (probably in the case of implementing a full terminal.) I would think in applications like those readline should not be used, and instead, assuming unbuffered I/O operations are desired, the 'read' call should be used with size set to 1.

My concern is that in the common case where PtyProcess is used in a similar way to pexpect (or used by pexpect behind the scenes) the readline call will incur huge performance penalties as is demonstrated by the example above.

from ptyprocess.

See #30 - I think it is possible to use buffered reading from the pty, though I want to make sure that it doesn't break anything else.

@anwilli5 if you're still interested, it would be good if you could see whether that PR speeds things up for your use case.

from ptyprocess.

@takluyver Yep, that PR significant speeds up execution of the test script above by avoiding thousands of read calls:

From the trunk:

strace -c python test.py > output_1000_orig.txt
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 85.67    0.044864           2     26215         1 read
  5.24    0.002745        2745         1           clone
  2.58    0.001350           2       543       434 open
...

With the PR:

strace -c python test.py > output_1000_new.txt
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 29.36    0.002999        2999         1           clone
 15.67    0.001601           3       544       434 open
 14.96    0.001528           7       221         1 read
...

For a more extreme case, consider the speed up when you change '1000' to '100000' in the test script above:

From the trunk:

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 99.75    4.616448           2   2600221         1 read
  0.08    0.003859          64        60           write
  0.07    0.003198        3198         1           clone
...

With the PR:

% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 26.99    0.003002        3002         1           clone
 19.92    0.002215           3       824         1 read
 15.84    0.001762           3       543       434 open
...

Nice work! 👍

from ptyprocess.

Great, thanks. @jquast , when you've got a moment, it would be good to get a second pair of eyes looking over the changes in #30.

from ptyprocess.

from ptyprocess.

Super, thanks! I hope there's still some time for relaxing in your holiday weekend too. :)

from ptyprocess.