False OS X Unit Test Failures - e.g. ut_seash_help.py about seash HOT 23 OPEN

Oh, fun! I replaced utf.py's output-on-error (i.e. the "prodced/expected" thing) to show reprs of what the unit test case returns, rather than strs.

480,481c480,481
<       print "."*30 + "Produced" + "."*30 + "\n" + str(produced_val)
<       print "."*30 + "Expected" + "."*30 + "\n" + str(expected_val)
---
>       print "."*30 + "Produced" + "."*30 + "\n" + repr(produced_val)
>       print "."*30 + "Expected" + "."*30 + "\n" + repr(expected_val)

    Running: ut_seash_help.py                                   [ FAIL ]
--------------------------------------------------------------------------------
Standard out :
..............................Produced..............................
'\x1b[?1034h'
..............................Expected..............................
None
--------------------------------------------------------------------------------

/: Yeah, you're right: that's the same output I get when it's set to repr instead of str. So there's an ASCII escape code popping out in the output that shouldn't be there.
On my macbook, I have $TERM set to "xterm-256color". If I change it to "vt100" or "linux" first, the test passes.

In particular, if the term setting specifies an smm value, that value gets spit out at some point into std out. (Possibly at import of the readline library? Ref)

If a term setting is selected that lacks smm, this bug will not arise. The issue is a combination of term settings plus readline library issues, I suppose. I'm not sure how unreasonable it is for readline to do this.

I'm running Python 2.7.11 on my laptop and having this problem.
The Linux VMs on Travis are running 2.6.9 and 2.7.9 and not having this problem.
The OS X Travis-CI VMs having this problem are running Python 2.6.9 and 2.7.5.

So it's not something that varies in recent Python (2) history.

I'm going to toy with TERM values on the Travis-CI VMs and see what happens.

from seash.

Values of TERM on Travis-CI are all "xterm" (OS X and Linux both); however, setting TERM to "linux" averts the issue....

This feels a bit silly. Do we need to tell people to set their terminals to a particular value, or should we adjust utf to disregard escape characters like this? (Maybe just this one?)

from seash.

Can you remove readline.so.mac from your test dir and retry? (It was added as an ugly patch for command-line editing in seash on Mac, which was never a good idea in the first place.)

On a side note, I had problems sshing into machines as they would not work with my xterm-256color setting.

from seash.

Hah. I wish I'd noticed that. Yes: removing that readline resolves the issue for me under Python 2.7.11 - which is to say that the readline that comes with 2.7.11 doesn't have this issue or python 2.7.11 paired with the readline included with it (some patch?) doesn't have this issue.

Here's a solid discussion of the issue in readline from python's perspective .

It looks like readline 6.3 may fix this (item 6.3 1.d in the changelog?) given a comment in that thread.

from seash.

I now wish this were an issue in the utf project instead. /:
Shame issue movement hasn't gone anywhere and github-issue-mover doesn't provide much.

from seash.

So: what impact does it have to remove that exactly? Do people use command-line editing in seash on a Mac? Is there some alternative? (What exactly is command-line editing in seash?)

from seash.

1. I think it is a seash issue really.
2. https://seattle.poly.edu/changeset/6046 --- this is used for tab completion in interactive seash sessions. Remove it, and tab completion vanishes as well. Try with e.g. loadkeys /dir/to/a/user/key/...
3. I use it a lot :-(

from seash.

1- Hmm. Fair point, but that readline version is a valid one that exists out there.... Even if we take it out of seash, someone still has it on some python version. It may just fall to utf to be able to handle a known not-really-output piece of output. (Thus utf issue?)

2&3. Yeah, ok. Possibly clunky, but not meriting removal if it has that impact.

So utf patch like the above? (but more complete :P)

from seash.

Consider this little experiment:

albert$ python -c "import seash" | hexdump
0000000 1b 5b 3f 31 30 33 34 68                        
0000008
albert$ python -c "import readline" | hexdump
albert$ mv readline.so.mac readline.so.mac.OUT_OF_MY_WAY
albert$ python -c "import seash" | hexdump
albert$ 

Maybe you can try and remove the patching-around-existing readline from seash for a test. I don't remember the exact details why the default installed version didn't suffice for our purposes.

from seash.

If I remember correctly, it didn't let you tab complete or use the arrow
keys to view previous commands. It may even be that backspace didn't work.

On Thu, Feb 4, 2016 at 11:21 AM, aaaaalbert [email protected]
wrote:

Maybe you can try and remove the patching-around-existing readline from
seash for a test. I don't remember the exact details why the default
installed version didn't suffice for our purposes.

—
Reply to this email directly or view it on GitHub
#99 (comment)
.

from seash.

Tab is broken. The other things work.

from seash.

Further reading and fixing: http://stackoverflow.com/questions/7116038/python-tab-completion-mac-osx-10-7-lion

from seash.

I use homebrew to install python on OS X, not native Xcode, so I installed using Xcode just now (which is probably still the most common method) and checked to see....

The winner is of course installing python properly, but, basically, we cannot exclude the readline.so in seash, no, if we want people with Xcode-installed python to be able to use tab completion.

We can still upgrade it to readline 6.3...?

from seash.

Great, thanks for checking this out in detail! Now building on the StackOverflow bit mentioned earlier, it looks like half of the problem (viz. tab completion of seash command names) can be fixed like this:

# Via Ned Deily in Python issue 10666, http://bugs.python.org/issue10666
import readline
import rlcompleter

# This is for Python installs whose `readline` uses BSD libedit
readline.parse_and_bind("bind ^I rl_complete")

# For `readline`s using GNU readline
readline.parse_and_bind("tab: complete")

The logic in seash.py around the readline import is strange to me. Why do we go to the trouble of renaming a library back and forth if we can just import as...?

(I ask in case there's something I'm not understanding before I try making changes. (: )

EDIT:
Ah. Nevermind. (:
The library would fail to import if we just renamed it permanently to readline_mac.so and tried to import readline_mac as readline if sys.platform == 'darwin'.

Moving on (:

from seash.

Submitted PR #100.

All tests that should pass pass locally (3 failures: ut_seash_getrelease.py, ut_seash_getrelease_insecure.py, ut_seash_moduleconflicterror.py - just as with vanilla). I no longer get the blank false failures on my OS X laptop.

If PR #98 were merged, the tests via AppVeyor and Travis would occur and show up here.

from seash.

PR #100 also breaks tab completion of local files / folders (only preserving tab completion of command names), which I hadn't previously understood. /:

> loadkeys ~/<TAB>

should provide completion but does not there. Will have to try fixing the code to work with it and issue another PR. (Confusion stemmed from fact that filename tab completion is only available following a command, not on bare line - e.g. > sea<TAB> doesn't display seash.py.)

from seash.

When you type loadkeys ~/ and hit TAB in seash, here's how each readline calls tab_completer.Completer::complete():

• editline (shipped with Xcode python 2.7):
  • completer('~/')
• GNU readline:
  • completer('loadkeys ~/')

The prefixes provided by the libraries aren't the same - editline is just providing the current word, while readline provides the whole line. Therein lies the problem, since tab_completer is only interested in completing local filenames/directories if they follow a command that takes local filenames/directories. In the absence of the command, it does not look.

Off the top of my head, there are a few ways we could work around this (other than just shipping a modern GNU readline, which might be a licensing issue (GPL) aside from being a somewhat distasteful procedure...?):

• Abandon the feature of only looking at local filenames after a command that would sensibly take them (and always look). This would clutter up tab completion and maybe make things more confusing for users, but it would work. We could also elect to do this only if editline is being used, so it would only affect those users.
• Modern editlines might not have this issue, so I could check that out if there's a sense that that would be a palatable solution. (not GPL license, which is presumably why OS X went that direction anyway).
• Possible clever hackery plugging into something else editline and readline provide in order to have access to the full line regardless? Have to look.

from seash.

Yeah, editline seems to be ignoring readline.set_completer_delims("") in the tab completion setup in seash.py.

It's not just that it's not happy with the empty string, either - it ignores readline.set_completer_delims("s") just as happily, while you can see readline clearly making use of it.
/:

from seash.

While I didn't see some better alternative binding for editline, I do almost have it working correctly through the use of readline.get_line_buffer() to replace the shortened prefix. File tab completion now works, along with tildes, using that, on editline, but...

... I'm getting some odd behavior I haven't quite figured out yet - prefixes are repeating in a way that readline doesn't do when I apply exactly the same code to it. There seems to be some other difference in what the libraries are doing.

Still working on it. I don't see any other impact yet, even though it feels a little weird to be doing.

Code diff here: master...awwad:readline_clean_take_2

from seash.

On a tangent: Have you seen this? I got the idea from https://www.gnu.org/software/bash/manual/html_node/Readline-Init-File-Syntax.html#Readline-Init-File-Syntax , and indeed if I parse_and_bind tab to something else, it shows in this list.

~ albert$ python
Python 2.7.10 (default, Oct 23 2015, 18:05:06) 
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import readline
>>> readline.parse_and_bind("bind")
Standard key bindings
"^@"           ->  em-set-mark
"^A"           ->  ed-move-to-beg
"^B"           ->  ed-prev-char
"^C"           ->  ed-tty-sigint
"^D"           ->  em-delete-or-list
"^E"           ->  ed-move-to-end
"^F"           ->  ed-next-char
"^G"           ->  is undefined
"^H"           ->  em-delete-prev-char
"^I"           ->  ed-insert
"^J"           ->  ed-newline
"^K"           ->  ed-kill-line
"^L"           ->  ed-clear-screen
"^M"           ->  ed-newline
"^N"           ->  ed-next-history
"^O"           ->  ed-tty-flush-output
"^P"           ->  ed-prev-history
"^Q"           ->  ed-tty-start-output
"^R"           ->  ed-redisplay
"^S"           ->  ed-tty-stop-output
"^T"           ->  ed-transpose-chars
"^U"           ->  em-kill-line
"^V"           ->  ed-quoted-insert
"^W"           ->  ed-delete-prev-word
"^X"           ->  ed-sequence-lead-in
"^Y"           ->  em-yank
"^Z"           ->  rl_tstp
"^["           ->  ed-sequence-lead-in
"^\"           ->  ed-tty-sigquit
"^]"           ->  ed-tty-dsusp
" "  to "/"    ->  ed-insert
"0"  to "9"    ->  ed-digit
":"  to "~"    ->  ed-insert
"^?"           ->  em-delete-prev-char
"\U+0088"      ->  ed-delete-prev-word
"\U+008C"      ->  ed-clear-screen
"\U+009F"      ->  em-copy-prev-word
"\U+00A0" to "\U+00FE"->  ed-insert
"\U+00FF"      ->  ed-quoted-insert
Alternative key bindings
Multi-character bindings
"^[^H"         ->  ed-delete-prev-word
"^[^L"         ->  ed-clear-screen
"^[^_"         ->  em-copy-prev-word
"^[0"          ->  ed-argument-digit
"^[1"          ->  ed-argument-digit
"^[2"          ->  ed-argument-digit
"^[3"          ->  ed-argument-digit
"^[4"          ->  ed-argument-digit
"^[5"          ->  ed-argument-digit
"^[6"          ->  ed-argument-digit
"^[7"          ->  ed-argument-digit
"^[8"          ->  ed-argument-digit
"^[9"          ->  ed-argument-digit
"^[B"          ->  ed-prev-word
"^[C"          ->  em-capitol-case
"^[D"          ->  em-delete-next-word
"^[F"          ->  em-next-word
"^[L"          ->  em-lower-case
"^[N"          ->  ed-search-next-history
"^[P"          ->  ed-search-prev-history
"^[U"          ->  em-upper-case
"^[W"          ->  em-copy-region
"^[X"          ->  ed-command
"^[b"          ->  ed-prev-word
"^[c"          ->  em-capitol-case
"^[d"          ->  em-delete-next-word
"^[f"          ->  em-next-word
"^[l"          ->  em-lower-case
"^[n"          ->  ed-search-next-history
"^[p"          ->  ed-search-prev-history
"^[u"          ->  em-upper-case
"^[w"          ->  em-copy-region
"^[x"          ->  ed-command
"^[^?"         ->  ed-delete-prev-word
"^[[A"         ->  ed-prev-history
"^[[B"         ->  ed-next-history
"^[[C"         ->  ed-next-char
"^[[D"         ->  ed-prev-char
"^[[H"         ->  ed-move-to-beg
"^[[F"         ->  ed-move-to-end
"^[[3~"        ->  ed-delete-next-char
"^[OA"         ->  ed-prev-history
"^[OB"         ->  ed-next-history
"^[OC"         ->  ed-next-char
"^[OD"         ->  ed-prev-char
"^[OH"         ->  ed-move-to-beg
"^[OF"         ->  ed-move-to-end
"^X^X"         ->  em-exchange-mark
Arrow key bindings
down           ->  ed-next-history
up             ->  ed-prev-history
left           ->  ed-prev-char
right          ->  ed-next-char
home           ->  ed-move-to-beg
end            ->  ed-move-to-end
delete         ->  ed-delete-next-char