ksh93 / ksh Goto Github PK

View Code? Open in Web Editor NEW

This project forked from att/ast

165.0 165.0 29.0 23.87 MB

ksh 93u+m: KornShell lives! | Latest release: https://github.com/ksh93/ksh/releases

License: Eclipse Public License 2.0

Shell 16.20% C 63.74% Scilab 12.61% Roff 7.45%

bourne-shell c kornshell ksh ksh93 shell unix-shell

ksh's People

Contributors

Stargazers

Watchers

ksh's Issues

pty.c: SIGTSTP handling is broken

See prior discussion:
att#375
#45 (comment)

TODO: after fixing this bug, re-enable regression test disabled in 1ca9286.

Versioning?

It's not "broken", but what is the standard for ksh93/ksh?

[10:04 AM][ttys000][~/src/ksh]
[245] mbp13 $ /usr/local/bin/ksh --version
  version         sh (AT&T Research) 93u+m 2020-07-16
[10:05 AM][ttys000][~/src/ksh]
[246] mbp13 $ echo $KSH_VERSION           
Version AJM 93u+m 2020-07-16
[10:05 AM][ttys000][~/src/ksh]
[247] mbp13 $ git log                     

commit 36f55f1f857f84c3ca88664b8050853e15350ade (HEAD -> master, origin/master, origin/HEAD)
Author: Martijn Dekker <[email protected]>
Date:   Sun Jul 19 06:42:53 2020 +0100

Huh?

Regression with command substitution when compiled with SHOPT_SPAWN

After fixing #79 there is another regression that shows up when ksh is compiled with SHOPT_SPAWN to use sh_ntfork() which uses AST spawnveg(3) which uses posix_spawn(3). A minimal reproducer for the regression involves a combination of command substitution, a here-document, and an external command:

output=$(
cat <<EOF
$(ls /dev/null)
EOF
)
echo "$output"

Actual output: (empty)
Expected output: /dev/null

I can reproduce this regression on a 93u+ 2012-08-01 compiled to use SHOPT_SPAWN, so 93u+m did not introduce this bug.

If ksh is compiled with -DSHOPT_SPAWN=0 the bug does not occur.

If the command substitution is forked, e.g. with ulimit -t unlimited, the bug does not occur:

output=$(
ulimit -t unlimited
cat <<EOF
$(ls /dev/null)
EOF
)
echo "$output"

Output as expected: /dev/null

So this is a bug that is, in part, related to non-forking subshells. There must be a bug somewhere in sh_ntfork(), or some function called directly or indirectly by it, that causes I/O not to be set up correctly.

Faulty logic for disabling job control in virtual subshells

~~Job control should be disabled in subshells as it makes no sense there; subshells are never interactive and job control only makes sense on interactive shells. However,~~ for non-forking subshells, ksh currently only disables job control for command substitutions:

ksh/src/cmd/ksh93/sh/subshell.c

Lines 570 to 577 in f207cd5

    
           if(comsub) 
        
           { 
        
           	/* disable job control */ 
        
           	shp->spid = 0; 
        
           	sp->jobcontrol = job.jobcontrol; 
        
           	sp->monitor = (sh_isstate(SH_MONITOR)!=0); 
        
           	job.jobcontrol=0; 
        
           	sh_offstate(SH_MONITOR);

ksh/src/cmd/ksh93/sh/subshell.c

Lines 629 to 636 in f207cd5

    
           if(comsub) 
        
           { 
        
           	/* re-enable job control */ 
        
           	if(!sp->nofork) 
        
           		sh_offstate(SH_NOFORK); 
        
           	job.jobcontrol = sp->jobcontrol; 
        
           	if(sp->monitor) 
        
           		sh_onstate(SH_MONITOR);

This causes "interesting" behaviour, e.g.:

$ (sleep 1 & :)
[1]	25185

This wrongly prints a job number.

$ sleep 10 &
$ (fg)

This waits for sleep to end, but should complain about no job control instead. However:

$ (sleep 10 & fg)
[1]	25144
arch/darwin.i386-64/bin/ksh: fg: No job control

So that's inconsisistent.

Related: #79, #86

Also:

$ (sleep 1 &)
$ [1]	25187

This wrongly prints a job number after the prompt, i.e. the job number is printed from a background process! So the main shell forks as well as the background process. That's a separate bug, probably a parser bug causing a double fork, akin to the one I fixed in 1026006.

FreeBSD & Debian: pty.sh fail: process/terminal group exercise

On this FreeBSD version, the following regression test failure occurs:

$ uname -vm
FreeBSD 12.1-RELEASE-p6 GENERIC  amd64
$ bin/shtests -p pty
[...]
        pty.sh[445]: process/terminal group exercise: line 451: read timeout

On Debian 10 (Buster) as hosted by shell.xs4all.nl, this test also fails, slightly differently:

$ cat /etc/issue; uname -srm
Debian GNU/Linux 10 \n \l

Linux 4.9.0-9-amd64 x86_64
$ bin/shtests -p pty
[...]
        pty.sh[445]: process/terminal group exercise: line 451: expected "Stopped", got ""

The test in question:

ksh/src/cmd/ksh93/tests/pty.sh

Lines 445 to 454 in f811482

    
           TERM=vt100 tst $LINENO <<"!" 
        
           L process/terminal group exercise 
        
           w m=yes; while true; do echo $m-$m; done | less 
        
           u :$|:\E|lines 
        
           c \cZ 
        
           r Stopped 
        
           w fg 
        
           u yes-yes 
        
           !

Add '-o posix' shell option

Recent email correspondence with @kdudka has made me think that maybe, in making POSIX compliance front and centre to development here, I've been giving insufficient consideration to the fact that there is an installed base of set-and-forget ksh93 scripts still running, which may somehow depend on old non-compliant behaviour.

In principle, I'm still thinking that anyone who needs 100% bug compatibility should stick with exactly the version they are running. However, we can restore the documented or otherwise clearly intended non-compliant behaviour as part of an option that can be turned on and off.

Other shells such as bash, mksh and yash can do set -o posix to enable POSIX compliance mode. It is worth considering adding a similar option to ksh93. ksh should probably default to POSIX mode when invoked as sh, and to "legacy" otherwise. Having the posix option off would restore behaviour such as what has been fixed in 6a49720 (edit: nah, that's just pure breakage, and completely undocumented), eeee77e (edit: done in c607c48), and maybe 36da314 (edit: nah, that was just a bug).

It would also keep the Bourne shell compatibility hack for a literal [ -t ] or test -t being interpreted as [ -t 1 ]/test -t 1. Korn hacked the parser so that only a literal [ -t ] is changed and not something like x=-t; [ "$x" ]. Still, that hack violates POSIX and should be deactivated in any new POSIX mode. (edit: done in 55f0f8c)

Let's think about this for a while before doing anything. We can talk about it in this issue's thread, and figure out a list of things that need to be switched by such an option.

Crash in command line editing

This has happened repeatedly today:

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   ksh                           	0x000000010aa7a1f6 job_unpost + 49
1   ksh                           	0x000000010aa7a9d1 job_reap + 1639
2   libsystem_platform.dylib      	0x00007fff7dbf1b5d _sigtramp + 29
3   ???                           	0x000000010ab6cb08 stkdisc + 40
4   ksh                           	0x000000010aa6c0a9 ed_read + 313
5   ksh                           	0x000000010aa6c478 ed_getchar + 102
6   ksh                           	0x000000010aaab15f ed_emacsread + 659
7   ksh                           	0x000000010aa76045 slowread + 491
8   ksh                           	0x000000010aaf9314 sfrd + 1045
9   ksh                           	0x000000010aaf558a _sffilbuf + 589
10  ksh                           	0x000000010aaf9ad6 sfreserve + 655
11  ksh                           	0x000000010aa5b3af exfile + 1939
12  ksh                           	0x000000010aa5bb08 sh_main + 1110
13  libdyld.dylib                 	0x00007fff7da063d5 start + 1

While it ends up at job_unpost again, this wasn't trying to replicate either of the crashers. This was just typing bin/pscal (I'm probably the only person who still uses pscal) at a prompt, and the shell crashed. No command substitution, no timeout reads, no backgrounding.

I've now backed up to 49ae483 to see if I can replicate it without any of the job control related patches.

>(Process substitution) b0rken when combined with redirection

Process substitution of the form >(list) is utterly broken:

echo ok > >(sed s/ok/good/)

Expected output: good (as on bash and zsh)

Actual output:

either none, with a file with a bizarre binary file name created
in the current working directory that contains ok plus a newline
(which is what happens on Linux and Solaris);
or (depending on the file system and/or OS and/or locale) an
error message indicating the refusal to create that file:
ksh: ?: cannot create [Illegal byte sequence]
(which is what happens on my Mac).

Before I spend hours searching for the cause, does anyone have any idea where to begin debugging this?

`readonly` on special vars in subshell affects main shell

When declaring certain special variables readonly in a (virtual/nonforked) subshell, ksh fails to restore them when leaving that subshell, and errors out as if a shell assignment to these variables had failed.

$ ksh -c '(readonly RANDOM;echo one);echo two'
one
ksh: RANDOM: is read only
$ ksh -c '(readonly LINENO;echo one);echo two'
one
ksh: LINENO: is read only

As usual, a workaround is to fork the subshell using ulimit -t unlimited. This must be done before doing the readonly for the bug to go away, which suggests the readonly command is erroneously affecting the main shell environment.

Fix the bash compatibility mode

Does anyone care about ksh's bash compatibility mode?

It's not compiled in my default. The RELEASE file says it's incomplete and experimental.

bash is pretty ubiquitous. I think if people want to run bash scripts, they'll run bash.

Would anyone object to getting rid?

Main shell forks after making PATH readonly in subshell

$ cat > bug.sh
(
	sh -c 'echo DEBUG1 $PPID'
	readonly PATH
	sh -c 'echo DEBUG2 $PPID'
)
sh -c 'echo DEBUG3 $PPID'
$ ksh bug.sh
DEBUG1 7546
DEBUG2 7546
$ DEBUG3 7549

The DEBUG3 output is done in an erroneous background job, as evidenced by the fact that the process ID changed, and it was printed after returning to the shell prompt.

This means that, when the state of PATH is restored while exiting the virtual subshell, it erroneously forks the main shell environment.

A little go-back-and-recompile trip through the git history reveals that this bug was introduced by commit 102868f - very probably by the change it made in name.c.

Sure enough, commenting out the following lines makes the bug go away (while of course reintroducing the corresponding hash table regression test failure in subshell.c):

ksh/src/cmd/ksh93/sh/name.c

Lines 1609 to 1614 in bd3e2a8

    
           	/* 
        
           	 * Resetting the PATH in a non-forking subshell will reset the parent shell's 
        
           	 * hash table, so work around the problem by forking before sh_assignok 
        
           	 */ 
        
           	if(shp->subshell && !shp->subshare && np == PATHNOD) 
        
           		sh_subfork();

I would conclude that nv_putval() is not a safe place from which to call sh_subfork() as it is also used for restoring the state after leaving a virtual subshell. The forking should probably happen somewhere we can be sure that an assignment is being processed – possibly in xec.c.

With SHOPT_DEVFD, process substitution leaks file descriptor when passed to function as argument

edit: In ab5dedd I've applied a workaround that disables the use of /dev/fd/NN pseudofiles and falls back to the older FIFO method instead. (That commit also improves the robustness of the FIFO method.) This effectively works around this bug, but is not a real fix. The /dev/fd method is the most robust as it doesn't involve the file system. So I'm leaving this issue open. To debug this, we now first need to comment out/remove the redefinition of SHOPT_DEVFD to 0 in src/cmd/ksh93/include/defs.h.

This was first reported on the old mailing list.

Reproducer based on the one in that message:

fdUser() {
	cat "$1"
}

for ((i=1; i<10; i++))
do	fdUser <(echo '========')
	if	[ -d /proc ]
	then	ls /proc/$$/fd
	else	lsof -p $$ | wc -l
	fi
done

The output shows one more open file for each function call. This bug does not occur if you replace the function call with a built-in or external command.

Unsetting a running function is silently ignored

The segfault has been fixed. Now the problem is that both types of functions, POSIX and ksh, silently fail to unset themselves. See b7932e8 and 9751174 and the issues referenced in them for more information.

[Original report] It should not be this easy to make ksh segfault:

$ ksh -c 'fn() { unset -f fn; echo end; }; fn'
Segmentation fault: 11

Tab completion menu causes inconsistent editor state in emacs

Line editing is set thus:

[44] mbp13 $ set -o
emacs                    on
vi                       off
viraw                    on

Start typing, get to bin/p, hit TAB. Nothing happens. Hit TAB again. (why do I have to hit TAB twice?) Get:

[04:59 PM][ttys001][~]
[45] mbp13 $ bin/p
1) pkg-config
2) pscal
3) pscal.saf
[45] mbp13 $ bin/pscal

Hit '2', hit TAB again, shell fills out bin/pscal.

[45] mbp13 $ bin/pscal

Now, backspace over the command line...

[04:59 PM][ttys001][~]
[45] mbp13 $ bin/p
1) pkg-config
2) pscal
3) pscal.saf
[45] mbp13 $ b

I'm left with the 'b', which isn't actually there, I can hit ENTER and not get an error message, and it's not stored in history.

10.15.5, ksh93u+m, ksh93u+ does it too.

What to do about typeset -L/-R combined with -i/-F/-E types?

This is not currently supported. See #47.

After applying #47, typeset -R10 -i foo=1; echo "[$foo]" gives a usage error and something like typeset -R10 -E3 foo=12345; echo "[$foo]" or typeset -R20 -F foo=12345; echo "[$foo]" simply ignores the -R flag.

Which precedent should we follow? Maybe they should all error out? I think giving a usage error is more logical since this combination of flags is not actually possible. But it should be preceded by an informative error message such as "incompatible options combination".

Or maybe we should just fix it so they can be combined? I see no real reason why this should be impossible. On mksh (which only supports integer arithmetic) the following works fine:

$ mksh -c 'typeset -R10 -i foo=123 bar=4567; printf "%s\n" "[$foo]" "[$bar]"'
[       123]
[      4567]

The other ksh-inspired shell that supports floating point arithmetic is zsh, and it has no problem combining any of these:

$ zsh -c 'typeset -R20 -F foo=12345; echo "[$foo]"'
[    12345.0000000000]
$ zsh -c 'typeset -R20 -E3 foo=12345; echo "[$foo]"'
[            1.23e+04]
$ zsh -c 'typeset -R20 -i foo=12345; echo "[$foo]"'
[               12345]

Community opinions invited. Should this be supported, or should it be an error, or should the -L/-R be ignored for these?

Arrays leak out of non-forked subshells

Previous discussion and reproducer: att#416

A straightforward workaround/fix would be to fork when an associative array is declared in a subshell:

diff --git a/src/cmd/ksh93/bltins/typeset.c b/src/cmd/ksh93/bltins/typeset.c
index 906d0a7..0ae7d8c 100644
--- a/src/cmd/ksh93/bltins/typeset.c
+++ b/src/cmd/ksh93/bltins/typeset.c
@@ -259,6 +259,8 @@ int    b_typeset(int argc,register char *argv[],Shbltin_t *context)
 				tdata.tname = opt_info.arg;
 				break;
 			case 'A':
+				if(tdata.sh->subshell && !tdata.sh->subshare)
+					sh_subfork();
 				flag |= NV_ARRAY;
 				break;
 			case 'C':

Before applying that fix/workaround I'd like to try to figure out if it's possible to implement a fix for the non-forking subshell mechanism. I don't really have time to dive deeply into this right now, so… any ideas, anyone?

Memory leak

[32] iMac $ git pull
Updating f9d2893..c4236cc
Fast-forward
 NEWS                            | 5 +++++
 src/cmd/ksh93/include/version.h | 2 +-
 src/cmd/ksh93/sh/parse.c        | 2 +-
 src/cmd/ksh93/tests/types.sh    | 3 +++
 4 files changed, 10 insertions(+), 2 deletions(-)
[ 9:59 AM][ttys000][~/src/ksh]

but...

[10:07 AM][ttys000][~/src/ksh]
[36] iMac $ bin/shtests leaks  
#### Regression-testing /Users/mwilson/src/ksh/arch/darwin.i386-64/bin/ksh ####
test leaks begins at 2020-07-10+10:08:09
test leaks passed at 2020-07-10+10:08:09 [ 6 tests 0 errors ]
test leaks(C.UTF-8) begins at 2020-07-10+10:08:09
test leaks(C.UTF-8) passed at 2020-07-10+10:08:10 [ 6 tests 0 errors ]
test leaks(shcomp) begins at 2020-07-10+10:08:10
	shcomp-leaks.ksh[102]: memory leak with read -C when using <<< (leaked 80 bytes)
test leaks(shcomp) failed at 2020-07-10+10:08:10 with exit code 1 [ 6 tests 1 error ]
Total errors: 1
CPU time       user:      system:
main:       0m00.00s     0m00.01s
tests:      0m00.27s     0m00.55s
[10:08 AM][ttys000][~/src/ksh]
[37] iMac $ git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean

FreeBSD: pty.sh fail: POSIX sh 101(C)

On this system:

$ uname -vm
FreeBSD 12.1-RELEASE-p6 GENERIC  amd64

The following regression test failure occurs:

        pty.sh[291]: POSIX sh 101(C): line 323: expected "^backslash-erase=:\r?\n$", got EOF

The test in question:

ksh/src/cmd/ksh93/tests/pty.sh

Lines 291 to 333 in f811482

    
           tst $LINENO <<"!" 
        
           L POSIX sh 101(C) 
        
           # If the User Portability Utilities Option is supported and shell 
        
           # command line editing is supported:  When in insert mode a control-V 
        
           # causes the next character to be inserted even in the case that the 
        
           # character is a special insert mode character. 
        
           # Testing Requirements: The assertion must be tested with at least the 
        
           # following set of characters: <newline>, erase, interrupt, kill, 
        
           # control-V, control-W, end-of-file, backslash \ (followed by erase or 
        
           # kill) and <ESC>. 
        
           d 10 
        
           p :test-1: 
        
           w stty erase ^H intr ^C kill ^X 
        
           p :test-2: 
        
           w echo erase=:\cV\cH: 
        
           u ^erase=:\r?\n$ 
        
           p :test-3: 
        
           w echo kill=:\cV\cX: 
        
           u ^kill=:\cX:\r?\n$ 
        
           p :test-4: 
        
           w echo control-V=:\cV\cV: 
        
           u ^control-V=:\cV:\r?\n$ 
        
           p :test-5: 
        
           w echo control-W:\cV\cW: 
        
           u ^control-W:\cW:\r?\n$ 
        
           p :test-6: 
        
           w echo EOF=:\cV\cD: 
        
           u ^EOF=:\004:\r?\n$ 
        
           p :test-7: 
        
           w echo backslash-erase=:\\\cH: 
        
           u ^backslash-erase=:\r?\n$ 
        
           p :test-8: 
        
           w echo backslash-kill=:\\\cX: 
        
           u ^backslash-kill=:\cX:\r?\n$ 
        
           p :test-9: 
        
           w echo ESC=:\cV\E: 
        
           u ^ESC=:\E:\r?\n$ 
        
           p :test-10: 
        
           w echo interrupt=:\cV\cC: 
        
           u ^interrupt=:\cC:\r?\n$ 
        
           !

Versioning?

Too much memory is allocated when reading big lines

There is a severe memory allocation bug that was reported in att#1401. Too much memory is being allocated when ksh reads lines of increasing size from a file. In the following example ksh uses 12GB of RAM to read a 41MB file:

$ dd if=/dev/zero of=./zeroes count=4096 bs=10000
$ ksh -c 'read line < ./zeroes; grep VmPeak /proc/$$/status ; echo "."'
VmPeak:  12548348 kB
.

The cause is located in the vmalloc's memory allocation. This bug disappears when ksh is compiled with -D_std_malloc added to CCFLAGS because that flag causes vmalloc to become a wrapper for the operating system's malloc implementation (i.e., the native malloc and free functions are used instead of vmalloc's replacements):

# Run after deleting ./arch and recompiling ksh with `-D_std_malloc`
$ dd if=/dev/zero of=./zeroes-std count=4096 bs=10000
$ arch/*/bin/ksh -c 'read line < ./zeroes-std; grep VmPeak /proc/$$/status ; echo "."'
VmPeak:  88492 kB
.

For reference, ksh93v- and ksh2020 use the following amounts of memory with the reproducer:

# ksh93v- uses a more reasonable amount of memory, indicating vmalloc was (edit: partially) fixed after ksh93u+
$ ksh93v -c 'read line < ./zeroes; grep VmPeak /proc/$$/status ; echo "."'
VmPeak:  131572 kB
.

# ksh2020 uses standard malloc instead of vmalloc
$ ksh2020 -c 'read line < ./zeroes; grep VmPeak /proc/$$/status ; echo "."'
VmPeak:  88480 kB
.

'=' file created during build

There appears to be a spurious '=' file created at the top of the source tree after a standard build of 'bin/package make'.

$ bin/package make
...
$ ls
'='   LICENSE.md   NEWS   README.md   TODO   arch   bin   docs	 lib   src

Tab completion wrongly activated (and broken) when neither emacs or vi mode is active

According to sh.1, tab completion is available in "each of the edit modes" (see under "In-line Editing Options").

However, there is often a broken form of tab completion when no edit mode is active. ~~It is unpredictable whether this happens, which suggests a failure to initialise a variable somewhere.~~

A fix should make sure tab completion is properly deactivated when neither emacs nor vi mode is active.

Shell-quoting garbled after parsing UTF-8 characters

#! /bin/ksh
LANG=C.UTF-8
printf '%q\n' 'Is shell-quoting garbled?'
print -nr $'\303\274' | read -n1 g
printf '%q\n' 'Is shell-quoting garbled?'

When running this, the second line is output without the final quote.

When you run this with set -x, the issue happens to quoted strings in the xtrace as well, suggesting that the global shell-quoting algorithm is somehow affected.

LINENO corrupted upon autoloading function

This bug is also in 93u+ 2012-08-01, as well as ksh2020.

Reproducer:

tmp=/tmp/LINENO_bug.$$
trap 'rm -rf "$tmp"' EXIT
mkdir "$tmp" || exit
cd "$tmp" || exit
print $'foobar() {\n: two\n: three\n: four\n} #five' > $tmp/foobar
FPATH=$tmp
saveLINENO=$LINENO
foobar
diff=$((LINENO - saveLINENO))
if      ((diff != 2))
then    echo "bug: diff == $diff"
        exit 1
else    echo "ok"
fi

Output:

bug: diff == 7

It appears that the five lines from the function definition file are wrongly added to the main script's $LINENO. So the fix probably involves saving and restoring $LINENO upon autoloading a function.

First item missed in compound assignment of associative enum arrays

Scalar arrays (-a) and associative arrays (-A) of a type created by enum allow values not specified by the enum type, yielding corrupted results.

Reproducer:

$ enum Color_t=(red green blue orange yellow)
$ Color_t -a clr=(red blue WRONG)
$ printf '%s\n' "${clr[@]}"
red
blue
red
$ Color_t -A clr=([foo]=red [bar]=blue [bad]=WRONG)
$ printf '%s\n' "${clr[@]}"
1440
1728
2016

(the numbers are random, which is evidence of memory corruption)

Expected in both cases: an error message refusing to create the array due to an invalid value, as in this scalar asssignment:

$ Color_t clr=BAD
ksh: clr: invalid value BAD

This can also be reproduced in 93u+ and in ksh2020, so it doesn't look like there's an existing fix to backport.

'return' combined with '||' statement has wrong exit status

Running return || true or return || false in a function causes the function to return with exit status zero instead of one:

# This bug is also in ksh93n-, ksh93u+, ksh93v- and ksh2020
$ cat ./reproducer.sh
foo() {
	false
	return || true
}
foo
if [ $? = 0 ]; then
	echo 'Failure'
	exit 1
fi
exit 0
$ ksh ./reproducer.sh
Failure
$ dash ./reproducer.sh # No output when using other shells
$

I found this problem while debugging BUG_LOOPRET2, which is similar (although that bug only applies to loops, while the above reproducer doesn't use a loop):

ksh/TODO

Lines 57 to 61 in d03e948

    
           - BUG_LOOPRET2: If a 'return' command is given without a status argument 
        
             within the set of conditional commands in a 'while' or 'until' loop (i.e., 
        
             between 'while'/'until' and 'do'), the exit status passed down from the 
        
             previous command is ignored and the function returns with status 0 
        
             instead.

Filename expansion of * includes '.' and '..' if FIGNORE is set

Filename expansion using bare asterisk expands . and .. as well.

$ print *
. .. bar foo

The behavior here should be that it excludes the files with leading periods unless the period is specified in the pattern as well.

Ref: 2.13.3.2 Patterns Used for Filename Expansion

Reproduced on: 93u+m 2020-07-02

Locale-based 'printf %T' output is broken

E.g. on macOS, as well as FreeBSD:

$ LANG=nl_NL.UTF-8 TZ=Europe/Helsinki arch/*/bin/ksh -c 'date; printf %T\\n now'
di 30 jun 2020 03:49:12 EEST
di 30 jun 03:49:12 2020

(the time zone is missing from printf %T output and the year is printed in the wrong order)

And on Linux (Debian):

$ LANG=nl_NL.UTF-8 TZ=Europe/Helsinki arch/*/bin/ksh -c 'date; printf %T\\n now'
di 30 jun 2020  3:50:40 EEST
di 30 jun 2020 di jun 30  EEST 2020:50:40 EEST

That different systems produce differently corrupted output is interesting.

Related: #6, #38, #62

Update time-related data in src/lib/libast/tm

On FreeBSD I get the following regression test failure (among others):

test builtins begins at 2020-06-12+22:04:08
        builtins.sh[315]: printf "%T" now

This test compares the output of printf "%T" now with that of the external command date and checks if they are identical. On my cloud server and in my (Dutch) locale and time zone, they produce:

$ date
vr 12 jun. 2020 22:12:37 CEST
$ LC_ALL=C date
Fri Jun 12 22:13:20 CEST 2020
$ arch/freebsd12.amd64-64/bin/ksh -c 'printf %T\\n now'
vr 12 jun. 22:13:44 2020
$ LC_ALL=C arch/freebsd12.amd64-64/bin/ksh -c 'printf %T\\n now'
Fri Jun 12 22:13:54 NST 2020

The last one is funny: it puts me in Newfoundland Standard Time.

Clearly, the time zone data compiled into ksh is broken. I also suspect it's roughly 27 years out of date.

Shells really have no business dealing with time zones. Providing and maintaining time zone data is the job of the operating system. Maybe it was justifiable in 1993 when not all OSs supported those facilities, but that time is long behind us.

So, a proper fix for this bug, with backward compatibility in mind, will be to:

Make printf %T ~~use the operating system's standard date utility. It should search for it in the default utility path as in command -p so that it works regardless of $PATH~~ somehow use the time zone data provided by the OS.
Figure out what else might depend on the broken time zone data, and how to get that to function in an alternative way.
Remove the internal time zone data.

`whence -v` canonicalises paths improperly

Commands:

(cd /; whence -v usr/bin/env)

Actual output:

usr/bin/env is //usr/bin/env

Expected output:

usr/bin/env is /usr/bin/env

Multiple slashes are usually ignored, but a double initial slash is permitted by POSIX to have a special meaning, and in fact has a special meaning on Cygwin (indicating a Windows UNC network path). So it should not be adding one for that reason alone. It's also ugly.

shell crashes when read times out

[ 6:38 PM][ttys001][~/test/bz573936]
[211] mbp13 $ type ksh
ksh is a tracked alias for /usr/local/bin/ksh
[ 6:38 PM][ttys001][~/test/bz573936]
[212] mbp13 $ ksh
[ 6:38 PM][ttys001][~/test/bz573936]
[213] mbp13 $ TMOUT=5
[ 6:38 PM][ttys001][~/test/bz573936]
[214] mbp13 $ read
[ 6:39 PM][ttys001][~/test/bz573936]
[215] mbp13 $                           
shell will timeout in 60 seconds due to inactivity
Memory fault
[ 6:39 PM][ttys001][~/test/bz573936]
[215] mbp13 $

I read https://bugzilla.redhat.com/show_bug.cgi?id=573936 and tested it, didn't see the behavior the bug talked about, figured it must no longer be a problem.

Then the window I was running ksh in closed.

The read timed out after 5 seconds, and the prompt came back. Then ksh popped the timeout message and the memory fault, and the shell crashed.

I can't make it happen if I put it in a script and call the script, but it happens repeatably if I do it interactively. If I don't set a timeout and just let the read sit there, it doesn't crash.

(macOS 10.15.6, Version AJM 93u+m 2020-07-29)

Restore nmake, make it build

I get it, ksh93/ksh doesn't have nmake. Thus:

[ 7:59 AM][ttys000][../src/ksh]
iMac $ bin/package write tgz source ksh   
package: must have nmake to generate archives
[ 8:00 AM][ttys000][../src/ksh]
iMac $ bin/package write tgz binary ksh
package: must have nmake to generate archives

So... how is it done now? If there's something obvious I should have seen in 'package --man', feel free to point it out.

(There was a posting on the Korn Shell list that I have to find again, that made it sound like it was terribly easy to bring nmake back... have to find that again)

Memory leak and eventual crash when defining and running function in virtual subshell

This bug was discovered and identified as separate while fixing #113.

When defining and running function in a (virtual/non-forked) subshell many times, a crash occurs.

Reproducer (minimal except for the counter):

for ((n=0; n < 50000; n++))
do	print -n $'\r'$n  # optional counter
	(function foo { :; }; foo)
done

When compiled with vmalloc, the reproducer consistently crashes at iteration #32766. This is 2^15-2, suggesting an overflow of a short somewhere (max value: 2^15-1, 32767).

When compiled without vmalloc and using the native malloc on macOS, it crashes slightly later and at unpredictable numbers of iterations, but all quite close to a short overflow: 32907, 33191, 32986, 33038, 32832, 32783, etc.

Backtraces of the native macOS malloc crash are here.

The consistent crash with vmalloc always has the same backtrace (edit: updated with backtrace with source line numbers after recompiling with -O0 -g):

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   ksh                                 0x000000010f5fd8d0 _sfpmove + 80 (sfpool.c:235)
1   ksh                                 0x000000010f5f16c5 sfclose + 837 (sfclose.c:96)
2   ksh                                 0x000000010f5b6469 stkclose + 121 (stk.c:317)
3   ksh                                 0x000000010f562bb2 sh_exec + 10674 (xec.c:1516)
4   ksh                                 0x000000010f564a99 sh_exec + 18585 (xec.c:2012)
5   ksh                                 0x000000010f55c7ed sh_subshell + 2605 (subshell.c:605)
6   ksh                                 0x000000010f564402 sh_exec + 16898 (xec.c:1886)
7   ksh                                 0x000000010f564a99 sh_exec + 18585 (xec.c:2012)
8   ksh                                 0x000000010f56571c sh_exec + 21788 (xec.c:2201)
9   ksh                                 0x000000010f564a99 sh_exec + 18585 (xec.c:2012)
10  ksh                                 0x000000010f4e7dbb exfile + 3243 (main.c:582)
11  ksh                                 0x000000010f4e8ce7 sh_main + 3367 (main.c:353)
12  ksh                                 0x000000010f4ce796 main + 38 (pmain.c:45)
13  libdyld.dylib                       0x00007fff670df3d5 start + 1

external command causes SIGINT to be ignored within non-forking subshell

On a ksh compiled with -DSHOPT_SPAWN=0 to disable AST spawnveg(3) (see #79), running an external command in a non-forking subshell blocks SIGINT until that subshell is exited. If the subshell never exits, ksh must be killed from another shell using another signal.

This applies to both interactive shells and scripts.

Reproducer:

$ arch/*/bin/ksh -c '(/usr/bin/true; while :; do :; done); exit'
^C^C^C^C^C

(the extra 'exit' is to avoid the subshell being the last command, which would optimise it out when using -c)

Memory fault in tests/variables.sh

Attn: @JohnoKing
I may have been too quick in merging #22. On my system, that commit introduced a memory fault in one of the regression tests in variables.c:

$ bin/shtests -x -p variables
#### Regression-testing /usr/local/src/ksh93/ksh/arch/darwin.i386-64/bin/ksh ####
test variables begins at 2020-06-17+19:37:26
+ [0s|S0,tests/variables.sh,L26,e0] alias err_exit='err_exit $LINENO'

[...long-ish trace trimmed...]

+ [0s|S0,tests/variables.sh,L172,e1] COUNT=0
+ [0s|S0,tests/variables.sh,L173,e1] (( COUNT++ ))
+ [0s|S0,tests/variables.sh,L174,e1] (( COUNT != 1 || ACCESS!=2 ))
+ [0s|S0,tests/variables.sh,L177,e0] LANG=C
+ [0s|S0,tests/variables.sh,L177,e0] 1> /dev/null 2>& 1
shtests[343]: eval: line 1: 61715: Memory fault
test variables failed at 2020-06-17+19:37:26 with exit code 267 [ 125 tests (killed by SIGSEGV) ]

Here is the backtrace from lldb:

* thread #1, stop reason = signal SIGSTOP
  * frame #0: 0x0000000105f605a3 ksh`lookup + 734
    frame #1: 0x0000000105f8808d ksh`varsub + 6329
    frame #2: 0x0000000105f84a1e ksh`copyto + 2580
    frame #3: 0x0000000105f85660 ksh`sh_macexpand + 468
    frame #4: 0x0000000105f8a91b ksh`sh_macpat + 135
    frame #5: 0x0000000105fa595d ksh`sh_exec + 4194
    frame #6: 0x0000000105fa56ea ksh`sh_exec + 3567
    frame #7: 0x0000000105f5cb2b ksh`exfile + 3243
    frame #8: 0x0000000105f5da27 ksh`sh_main + 3367
    frame #9: 0x00007fff5832b3d5 libdyld.dylib`start + 1
    frame #10: 0x00007fff5832b3d5 libdyld.dylib`start + 1

It makes little sense to me. Why SIGSTOP when it was killed by SIGSEGV?

'[ -t 1 ]' and 'exec >...' breakage in ${ shared; } command substitutions

Commit cafe33f (see there for links/documentation) fixed test -t 1 in command substitutions by implementing a workaround: fork the subshell.

A real fix would be preferable, because test -t 1 and [ -t 1 ] and [[ -t 1 ]] continue to be wrong for "shared" command substitutions of the form var=${ list; }:

$ v=${ [ -t 1 ] && echo whoops; }; echo $v
whoops

The forking workaround is not possible there, as forking would defeat the "shared" aspect of the shared command substitution.

Implement hash tables for virtual subshells

The forking fix implemented in 102868f and 9d428f8, which stops the main shell's hash table from being cleared if PATH is changed in a subshell, can cause a significant performance penalty for certain scripts that do something like

( PATH=... command foo )

in a subshell, especially if done repeatedly. This is because the hash table is cleared (and hence a subshell forks) even for temporary PATH assignments preceding commands.

The modernish regression tests take about half a second longer to run on 93u+m than they do on 93u+ as PATH=$DEFPATH command ... is a common idiom in that library, and there's also this report of a 5.5% slowdown running a configure script which I suspect might be due to this change.

Virtual subshells would not need to fork when changing PATH if they get their own hash tables. Perhaps the code for alias subshell trees (which was removed in ec88886 because they were broken and unneeded) can provide the beginning of a template for their implementation.

Update zlib in src/cmd/INIT/ratz.c

src/cmd/INIT/ratz.c is a self-contained .tar.gz extractor. It still incorporates zlib 1.2.3 from 2005, whereas zlib 1.2.11 is out since 2017.

Related: #28

Does not compile with musl library under alpine

When doing a ./bin/package make the compilation is broken of when compiling under alpine linux with musl library

+ cc -D_BLD_DLL -fPIC -D_BLD_ast -O -I. -I/root/git_remote/ksh/src/lib/libast -Icomp -I/root/git_remote/ksh/src/lib/libast/comp -Iinclude -I/root/git_remote/ksh/src/lib/libast/include -Istd -I/root/git_remote/ksh/src/lib/libast/std -D_PACKAGE_ast -c /root/git_remote/ksh/src/lib/libast/misc/state.c
In file included from /root/git_remote/ksh/src/lib/libast/include/ast_std.h:66,
                 from /root/git_remote/ksh/src/lib/libast/include/ast.h:34,
                 from /root/git_remote/ksh/src/lib/libast/misc/state.c:26:
./ast_fs.h:135:18: warning: 'struct statvfs64' declared inside parameter list will not be visible outside of this definition or declaration
 #define statvfs  statvfs64
                  ^~~~~~~~~
./ast_sys.h:14:26: note: in definition of macro '__PROTO__'
 #    define __PROTO__(x) x
                          ^
/usr/include/sys/statvfs.h:48:19: note: in expansion of macro 'statvfs'
 #define statvfs64 statvfs
                   ^~~~~~~
./ast_fs.h:135:18: warning: 'struct statvfs64' declared inside parameter list will not be visible outside of this definition or declaration
 #define statvfs  statvfs64
                  ^~~~~~~~~
./ast_sys.h:14:26: note: in definition of macro '__PROTO__'
 #    define __PROTO__(x) x
                          ^
/usr/include/sys/statvfs.h:48:19: note: in expansion of macro 'statvfs'
 #define statvfs64 statvfs
                   ^~~~~~~
In file included from /root/git_remote/ksh/src/lib/libast/include/ast.h:34,
                 from /root/git_remote/ksh/src/lib/libast/misc/state.c:26:
/root/git_remote/ksh/src/lib/libast/include/ast_std.h:300:16: error: unknown type name 'off64_t'; did you mean 'off_t'?  #define off_t  off64_t
                ^~~~~~~
/usr/include/sys/mman.h:142:17: note: in expansion of macro 'off_t'
 #define off64_t off_t
                 ^~~~~
/root/git_remote/ksh/src/lib/libast/include/ast_std.h:300:16: error: unknown type name 'off64_t'
 #define off_t  off64_t
                ^~~~~~~
/usr/include/sys/mman.h:142:17: note: in expansion of macro 'off_t'
 #define off64_t off_t
                 ^~~~~
/root/git_remote/ksh/src/lib/libast/include/ast_std.h:300:16: error: unknown type name 'off64_t'; did you mean 'off_t'?  #define off_t  off64_t
                ^~~~~~~
/usr/include/sys/mman.h:142:17: note: in expansion of macro 'off_t'
 #define off64_t off_t
                 ^~~~~
/root/git_remote/ksh/src/lib/libast/include/ast_std.h:300:16: error: unknown type name 'off64_t'; did you mean 'off_t'?  #define off_t  off64_t
                ^~~~~~~
/usr/include/sys/mman.h:142:17: note: in expansion of macro 'off_t'
 #define off64_t off_t
                 ^~~~~
mamake [lib/libast]: *** exit code 1 making state.o
mamake: *** exit code 1 making lib/libast
mamake: *** exit code 1 making all
package: make done  at Wed Jun 10 14:47:48 UTC 2020 in /root/git_remote/ksh/arch/linux.i386-64

Memory leak when initialising associative array in subshell

When doing this in a virtual/non-forked subshell:

(typeset -A foo=([a]=1 [b]=2 [c]=3))

a memory leak occurs. The memory occupied by the array is not freed when exiting the subshell.

A reproducer with ps can confirm this bug exists on 93u+ 2012-08-01:

for ((i=1; i<=10; i++)); do
        for ((n=1; n<=100000; n++)); do
                (typeset -A foo=([a]=1 [b]=2 [c]=3))
        done
        ps -p "$$" -o rss=,vsz=
done

Output on macOS:

 15900  4285776
 29996  4299856
 44092  4313968
 58188  4328048
 72288  4342160
 86380  4356240
100480  4370352
114572  4384432
128672  4398544
142772  4412656

Reproducer with vmstate, ~~which is compiled into 93u+m by default~~ (edit: as of f9364b1, you have to pass -D_AST_vmalloc to enable it):

builtin vmstate || exit
for ((i=1; i<=10; i++)); do
        (typeset -A foo=([a]=1 [b]=2 [c]=3))
        vmstate
done

Output (note increasing busy and decreasing free):

region=0x10ffe4928 method=best flags=0 size=393216 segments=4 busy=(282256,170,65552) free=(107776,9,34288)
region=0x10ffe4928 method=best flags=0 size=393216 segments=4 busy=(283280,185,65552) free=(106560,6,34288)
region=0x10ffe4928 method=best flags=0 size=393216 segments=4 busy=(283408,188,65552) free=(106368,7,34288)
region=0x10ffe4928 method=best flags=0 size=393216 segments=4 busy=(283440,191,65552) free=(106128,17,34288)
region=0x10ffe4928 method=best flags=0 size=393216 segments=4 busy=(283664,194,65552) free=(105952,11,34288)
region=0x10ffe4928 method=best flags=0 size=393216 segments=4 busy=(283712,197,65552) free=(105728,19,34288)
region=0x10ffe4928 method=best flags=0 size=393216 segments=4 busy=(283888,200,65552) free=(105600,13,34288)
region=0x10ffe4928 method=best flags=0 size=393216 segments=4 busy=(283920,203,65552) free=(105392,21,34288)
region=0x10ffe4928 method=best flags=0 size=393216 segments=4 busy=(284096,206,65552) free=(105392,7,34288)
region=0x10ffe4928 method=best flags=0 size=393216 segments=4 busy=(284208,209,65552) free=(105120,14,34288)

'kill %%' and 'kill %+' segfault

Any reference to the current job (%% or %+) before there is one, passed to the kill command, currently causes a segfault. This is also the case in the 93u+ 2012-08-01 release. For example:

$ ksh -c 'kill -s STOP %%; echo exit status $?'
Segmentation fault: 11

Backtrace from lldb:

* thread #1, stop reason = signal SIGSTOP
  * frame #0: 0x000000010ff80944 ksh`job_kill + 20
    frame #1: 0x000000010ff80809 ksh`job_walk + 309
    frame #2: 0x000000010ff5856d ksh`b_kill + 806
    frame #3: 0x000000010ffab18b ksh`sh_exec + 16128
    frame #4: 0x000000010ff5f48b ksh`exfile + 3243
    frame #5: 0x000000010ff60387 ksh`sh_main + 3367
    frame #6: 0x00007fff5832b3d5 libdyld.dylib`start + 1

edit: Same happens when using %- to refer to the previous job before there is one.

printf %T formats do not follow strftime(3)

It's simple:

[6] iMac $ date +'%l:%M %p'
 8:26 PM

but:

[8] iMac $ printf '%(%l:%M %p)T\n'
Jul  4 20:27:27 PM

The man page for strftime(3) says %l is supposed to be the 12-hour hour without leading zeroes:

     %l    is replaced by the hour (12-hour clock) as a decimal number (1-12); single digits are preceded by
           a blank.

And I didn't ask for the seconds at all, but ksh is displaying them.

93u+ and 93u+m both do the same thing. I can't find %()T anywhere in the new Korn book, but Learning the Korn Shell 2nd ed says it should reformat it "according to the date(1) specification." (p225)

(macOS 10.14.6)

A crash report for readonly.exe?

This showed up in ~/Library/Logs/DiagnosticReports...

readonly.exe_2020-08-13-195404_mbp13.crash

I have no idea what it is and the shell I was running didn't crash or anything else. Putting it out there in case someone else knows what it is. Not running HEAD, if that matters, my current KSH is at commit 61437b2.

Process:               readonly.exe [10458]
Path:                  /private/var/folders/*/readonly.exe
Identifier:            readonly.exe
Version:               0
Code Type:             X86-64 (Native)
Parent Process:        ksh [9009]
Responsible:           Terminal [1301]
User ID:               501

Date/Time:             2020-08-13 19:54:02.532 -0700
OS Version:            Mac OS X 10.15.6 (19G2021)
Report Version:        12
Bridge OS Version:     4.6 (17P6610)
Anonymous UUID:        FDAFEF58-17EF-0FD0-8BE4-E164085D539D


Time Awake Since Boot: 4400 seconds

System Integrity Protection: enabled

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BAD_ACCESS (SIGBUS)
Exception Codes:       KERN_PROTECTION_FAILURE at 0x000000010c5a8fab
Exception Note:        EXC_CORPSE_NOTIFY

Termination Signal:    Bus error: 10
Termination Reason:    Namespace SIGNAL, Code 0xa
Terminating Process:   exc handler [10458]

VM Regions Near 0x10c5a8fab:
--> __TEXT                 000000010c5a8000-000000010c5a9000 [    4K] r-x/r-x SM=COW  /private/var/folders/*/*.exe
    __LINKEDIT             000000010c5a9000-000000010c5aa000 [    4K] r--/r-- SM=COW  /private/var/folders/*/*.exe

Application Specific Information:
dyld2 mode

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   readonly.exe                  	0x000000010c5a8f9b main + 11
1   libdyld.dylib                 	0x00007fff6cf43cc9 start + 1

Thread 0 crashed with X86 Thread State (64-bit):
  rax: 0x000000010c5a8f90  rbx: 0x0000000000000000  rcx: 0x00007ffee36578d8  rdx: 0x00007ffee3657798
  rdi: 0x0000000000000001  rsi: 0x00007ffee3657788  rbp: 0x00007ffee3657760  rsp: 0x00007ffee3657760
   r8: 0x0000000000000000   r9: 0x0000000000000000  r10: 0x0000000000000000  r11: 0x0000000000000000
  r12: 0x0000000000000000  r13: 0x0000000000000000  r14: 0x0000000000000000  r15: 0x0000000000000000
  rip: 0x000000010c5a8f9b  rfl: 0x0000000000010246  cr2: 0x000000010c5a8fab
  
Logical CPU:     2
Error Code:      0x00000007 (invalid protections for user data write)
Trap Number:     14


Binary Images:
       0x10c5a8000 -        0x10c5a8fff +readonly.exe (0) <BA175183-1A29-388A-9FDF-CD35D0641EA8> /private/var/folders/*/readonly.exe
       0x11b650000 -        0x11b6e1f47  dyld (750.6) <F9D4DEDC-8296-3E3F-B517-9C8B89A4C094> /usr/lib/dyld
    0x7fff69f28000 -     0x7fff69f29fff  libSystem.B.dylib (1281.100.1) <C0C9872A-E730-37EA-954A-3CE087C15535> /usr/lib/libSystem.B.dylib
    0x7fff6a20e000 -     0x7fff6a260fff  libc++.1.dylib (902.1) <59A8239F-C28A-3B59-B8FA-11340DC85EDC> /usr/lib/libc++.1.dylib
    0x7fff6a261000 -     0x7fff6a276ffb  libc++abi.dylib (902) <E692F14F-C65E-303B-9921-BB7E97D77855> /usr/lib/libc++abi.dylib
    0x7fff6bd88000 -     0x7fff6bdbbfde  libobjc.A.dylib (787.1) <6DF81160-5E7F-3E31-AA1E-C875E3B98AF6> /usr/lib/libobjc.A.dylib
    0x7fff6cd25000 -     0x7fff6cd2aff3  libcache.dylib (83) <AF488D13-9E89-35E0-B078-BE37CC5B8586> /usr/lib/system/libcache.dylib
    0x7fff6cd2b000 -     0x7fff6cd36fff  libcommonCrypto.dylib (60165.120.1) <C7912BE5-993E-3581-B2A0-6AABDC8C5562> /usr/lib/system/libcommonCrypto.dylib
    0x7fff6cd37000 -     0x7fff6cd3efff  libcompiler_rt.dylib (101.2) <49B8F644-5705-3F16-BBE0-6FFF9B17C36E> /usr/lib/system/libcompiler_rt.dylib
    0x7fff6cd3f000 -     0x7fff6cd48ff7  libcopyfile.dylib (166.40.1) <3C481225-21E7-370A-A30E-0CCFDD64A92C> /usr/lib/system/libcopyfile.dylib
    0x7fff6cd49000 -     0x7fff6cddbfdb  libcorecrypto.dylib (866.140.1) <60567BF8-80FA-359A-B2F3-A3BAEFB288FD> /usr/lib/system/libcorecrypto.dylib
    0x7fff6cee8000 -     0x7fff6cf28ff0  libdispatch.dylib (1173.100.2) <CD9C059C-91D9-30E8-8926-5B9CD0D5D4F5> /usr/lib/system/libdispatch.dylib
    0x7fff6cf29000 -     0x7fff6cf5ffff  libdyld.dylib (750.6) <789A18C2-8AC7-3C88-813D-CD674376585D> /usr/lib/system/libdyld.dylib
    0x7fff6cf60000 -     0x7fff6cf60ffb  libkeymgr.dylib (30) <DB3337BE-01CA-3425-BD0C-87774FC0CDC0> /usr/lib/system/libkeymgr.dylib
    0x7fff6cf6e000 -     0x7fff6cf6eff7  liblaunch.dylib (1738.140.1) <AFBCBDD3-0B55-3ECD-8E04-A73A3A57356B> /usr/lib/system/liblaunch.dylib
    0x7fff6cf6f000 -     0x7fff6cf74ff7  libmacho.dylib (959.0.1) <AA613A9C-961A-3B67-B696-4622FA59FC4E> /usr/lib/system/libmacho.dylib
    0x7fff6cf75000 -     0x7fff6cf77ff3  libquarantine.dylib (110.40.3) <F234E51D-FD0B-3EE4-B679-AE3EE9C536C3> /usr/lib/system/libquarantine.dylib
    0x7fff6cf78000 -     0x7fff6cf79ff7  libremovefile.dylib (48) <7C7EFC79-BD24-33EF-B073-06AED234593E> /usr/lib/system/libremovefile.dylib
    0x7fff6cf7a000 -     0x7fff6cf91ff3  libsystem_asl.dylib (377.60.2) <1563EE02-0657-3B78-99BE-A947C24122EF> /usr/lib/system/libsystem_asl.dylib
    0x7fff6cf92000 -     0x7fff6cf92ff7  libsystem_blocks.dylib (74) <0D53847E-AF5F-3ACF-B51F-A15DEA4DEC58> /usr/lib/system/libsystem_blocks.dylib
    0x7fff6cf93000 -     0x7fff6d01afff  libsystem_c.dylib (1353.100.2) <BBDED5E6-A646-3EED-B33A-91E4331EA063> /usr/lib/system/libsystem_c.dylib
    0x7fff6d01b000 -     0x7fff6d01effb  libsystem_configuration.dylib (1061.141.1) <0EE84C33-64FD-372B-974A-AF7A136F2068> /usr/lib/system/libsystem_configuration.dylib
    0x7fff6d01f000 -     0x7fff6d022fff  libsystem_coreservices.dylib (114) <A199156E-058D-3ABB-BCE9-4B9F20DCED0F> /usr/lib/system/libsystem_coreservices.dylib
    0x7fff6d023000 -     0x7fff6d02bfff  libsystem_darwin.dylib (1353.100.2) <5B12B5DB-3F30-37C1-8ECC-49A66B1F2864> /usr/lib/system/libsystem_darwin.dylib
    0x7fff6d02c000 -     0x7fff6d033fff  libsystem_dnssd.dylib (1096.100.3) <EBB4C2C2-E031-3094-B40A-E67BF261D295> /usr/lib/system/libsystem_dnssd.dylib
    0x7fff6d034000 -     0x7fff6d035ffb  libsystem_featureflags.dylib (17) <29FD922A-EC2C-3F25-BCCC-B58D716E60EC> /usr/lib/system/libsystem_featureflags.dylib
    0x7fff6d036000 -     0x7fff6d083ff7  libsystem_info.dylib (538) <8A321605-5480-330B-AF9E-64E65DE61747> /usr/lib/system/libsystem_info.dylib
    0x7fff6d084000 -     0x7fff6d0b0ff7  libsystem_kernel.dylib (6153.141.1) <2B6311E6-6240-3EF7-8C87-475B66F7452C> /usr/lib/system/libsystem_kernel.dylib
    0x7fff6d0b1000 -     0x7fff6d0f8fff  libsystem_m.dylib (3178) <00F331F1-0D09-39B3-8736-1FE90E64E903> /usr/lib/system/libsystem_m.dylib
    0x7fff6d0f9000 -     0x7fff6d120fff  libsystem_malloc.dylib (283.100.6) <8549294E-4C53-36EB-99F3-584A7393D8D5> /usr/lib/system/libsystem_malloc.dylib
    0x7fff6d121000 -     0x7fff6d12effb  libsystem_networkextension.dylib (1095.140.2) <F06C65C5-2CBE-313C-96E1-A09240F9FE57> /usr/lib/system/libsystem_networkextension.dylib
    0x7fff6d12f000 -     0x7fff6d138ff7  libsystem_notify.dylib (241.100.2) <FA22F928-D91B-3AA5-96BB-3186AC0FB264> /usr/lib/system/libsystem_notify.dylib
    0x7fff6d139000 -     0x7fff6d141fef  libsystem_platform.dylib (220.100.1) <009A7C1F-313A-318E-B9F2-30F4C06FEA5C> /usr/lib/system/libsystem_platform.dylib
    0x7fff6d142000 -     0x7fff6d14cfff  libsystem_pthread.dylib (416.100.3) <62CB1A98-0B8F-31E7-A02B-A1139927F61D> /usr/lib/system/libsystem_pthread.dylib
    0x7fff6d14d000 -     0x7fff6d151ff3  libsystem_sandbox.dylib (1217.141.1) <6DE13684-5A67-3009-A53A-C9086CF241C3> /usr/lib/system/libsystem_sandbox.dylib
    0x7fff6d152000 -     0x7fff6d154fff  libsystem_secinit.dylib (62.100.2) <F80872AA-E1FD-3D7E-8729-467656EC6561> /usr/lib/system/libsystem_secinit.dylib
    0x7fff6d155000 -     0x7fff6d15cffb  libsystem_symptoms.dylib (1238.120.1) <5820A2AF-CE72-3AB3-ABCC-273A3419FB55> /usr/lib/system/libsystem_symptoms.dylib
    0x7fff6d15d000 -     0x7fff6d173ff2  libsystem_trace.dylib (1147.120) <04B47629-847B-3D74-8ABE-C05EF9DEEFE4> /usr/lib/system/libsystem_trace.dylib
    0x7fff6d175000 -     0x7fff6d17aff7  libunwind.dylib (35.4) <42B7B509-BAFE-365B-893A-72414C92F5BF> /usr/lib/system/libunwind.dylib
    0x7fff6d17b000 -     0x7fff6d1b0ffe  libxpc.dylib (1738.140.1) <3E243A41-030F-38E3-9FD2-7B38C66C35B1> /usr/lib/system/libxpc.dylib

External Modification Summary:
  Calls made by other processes targeting this process:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0
  Calls made by this process:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0
  Calls made by all processes on this machine:
    task_for_pid: 3624
    thread_create: 0
    thread_set_state: 0

VM Region Summary:
ReadOnly portion of Libraries: Total=393.4M resident=0K(0%) swapped_out_or_unallocated=393.4M(100%)
Writable regions: Total=17.3M written=0K(0%) resident=0K(0%) swapped_out=0K(0%) unallocated=17.3M(100%)
 
                                VIRTUAL   REGION 
REGION TYPE                        SIZE    COUNT (non-coalesced) 
===========                     =======  ======= 
Kernel Alloc Once                    8K        1 
MALLOC                            9300K        8 
MALLOC guard page                   24K        5 
STACK GUARD                       56.0M        1 
Stack                             8192K        1 
__DATA                             596K       39 
__DATA_CONST                        20K        1 
__LINKEDIT                       388.8M        3 
__OBJC_RO                         32.3M        1 
__OBJC_RW                         1904K        1 
__TEXT                            4752K       40 
shared memory                        8K        2 
===========                     =======  ======= 
TOTAL                            501.2M      103

Backspacing past backslash in vi mode is broken in two ways

This bug was signalled and videoed by @saper in ksh-community/ksh#8 (comment) and the subsequent comment. Video 1; video 2 (see also instructions there).

The entire ksh-community/ksh#8 thread contains background information worth reading.

Note that the first variant of this bug also applies to raw POSIX (neither vi nor emacs) editing mode if a UTF-8 locale is active, as this raw editing mode is handled by edit/vi.c in that case.

Failure to launch external commands due to race condition in 'posix_spawn' usage

There has been a bug report of an exec format error when executing "cat TODO | while read line; do ls; done".

Ref:

I was able to reproduce this issue on 93u+m 2020-07-14 and it appears to be an issue at least since 93u+ 2012-08-01.

$ cat TODO
1
2
$ cat TODO | while read line; do ls; done
ls: ls: cannot execute [Exec format error]
ls: ls: cannot execute [Exec format error]

Tagging @chrisbertin and @mirabilos - reporters of the issue.

Memory Leak (spurious?) is back, built from HEAD on 10.15.6.

The problem is back, built from HEAD on 10.15.6 today. It's not the same test that fails as the one in #70.

[ 9:03 AM][ttys000][~/src/ksh][master]
[359] mbp13 $ bin/shtests leaks
#### Regression-testing /Users/mwilson/src/ksh/arch/darwin.i386-64/bin/ksh ####
test leaks begins at 2020-07-29+09:04:11
test leaks passed at 2020-07-29+09:04:11 [ 6 tests 0 errors ]
test leaks(C.UTF-8) begins at 2020-07-29+09:04:11
	leaks.sh[72]: variable value reset memory leak -- 48 bytes after 1000 iterations
test leaks(C.UTF-8) failed at 2020-07-29+09:04:11 with exit code 1 [ 6 tests 1 error ]
test leaks(shcomp) begins at 2020-07-29+09:04:11
test leaks(shcomp) passed at 2020-07-29+09:04:11 [ 6 tests 0 errors ]
Total errors: 1
CPU time       user:      system:
main:      0m00.007s    0m00.014s
tests:     0m00.153s    0m00.220s
[ 9:04 AM][ttys000][~/src/ksh][master]
[360] mbp13 $ git log | tee
commit 05081dfc1c6ef18eee06d2a31c3f1cb4bb68d9d7
Author: Johnothan King <[email protected]>
Date:   Mon Jul 27 05:27:20 2020 -0700

Originally posted by @posguy99 in #70 (comment)

Intermittent hang, KSH with backgrounded job

@McDutchie points out that this hang pre-dates #103, see last comment in that issue.

[10:37 AM][ttys000 +1][~/src/ksh-segfault][master]
[1049] iMac $ arch/darwin.i386-64/bin/ksh
[10:37 AM][ttys000 +2][~/src/ksh-segfault][master]
[1050] iMac $ top &
[1]	72570
[10:37 AM][ttys000 +2][~/src/ksh-segfault][master]
[1051] iMac $ 
[1] + Stopped (SIGTTOU)        top &

[10:37 AM][ttys000 +2][~/src/ksh-segfault][master]
[1051] iMac $ fg
top 
[10:38 AM][ttys000 +2][~/src/ksh-segfault][master]
[1052] iMac $

And now the shell is unresponsive.

This is related to #89 , right?

I've managed to get it to do that four times now just doing the above, but it's not reliably repeatable. If it does not display the 'Stopped' line without hitting Return again, it seems to never do it. Only if it does that on its own, and not always.

I have to kill it from somewhere else or close Terminal.app.

74575 s000 S+ 0:00.02 arch/darwin.i386-64/bin/ksh

^ from the latest one.

Originally posted by @posguy99 in #103 (comment)

'whence -a'/'type -a' fails to list builtin if function exists

Commands:

(printf() { :; }; whence -a printf)
(printf() { :; }; type -a printf)

Actual output:

printf is a function
printf is /usr/bin/printf

Expected output:

printf is a function
printf is a shell builtin
printf is /usr/bin/printf

Commands:

(autoload printf; whence -a printf)
(autoload printf; type -a printf)

Actual output:

printf is an undefined function
printf is /usr/bin/printf

Expected output:

printf is an undefined function
printf is a shell builtin
printf is /usr/bin/printf

Spurious 'undefined function' msg in 'whence -a' output after loading builtin

$ ksh -c 'builtin cat; whence -a cat'
cat is a shell builtin
cat is /bin/cat
cat is an undefined function

The third line should not appear.

This bug was found by @JohnoKing while working on #25.

Variable assignment in subshell affects parent.

This is a SSCCE:

$ cat bug.sh
#!/bin/ksh93

unset a b
c=0

function set_ac { a=1; c=1; }
function set_abc { ( set_ac ; b=1 ) }

set_abc
echo "a=$a b=$b c=$c"
$ ./bug.sh
a=1 b= c=0

Hence, although set_abc calls set_ac in a subshell, the assignment a=1 propagates to the parent shell. There are a few known workarounds [1]. Replace the definition of set_abc with either

function set_abc { ( set_ac ; b=1 ) | echo -n; }

function set_abc { ( ulimit -t unlimited ; set_ac ; b=1 ) }

Here is a third workaround that shows how bad subshells step on each other's toes. By setting a in a second subshell, it prevents the first subshell to set a in the parent shell.

function set_abc { ( set_ac ; b=1 ) }
( a=1 )

With either of the three above we get

$ ./bug.sh
a= b= c=0

References:
[1] att#73 (comment)
[2] https://stackoverflow.com/q/62385690/1137388

	if(comsub)
	{
	/* disable job control */
	shp->spid = 0;
	sp->jobcontrol = job.jobcontrol;
	sp->monitor = (sh_isstate(SH_MONITOR)!=0);
	job.jobcontrol=0;
	sh_offstate(SH_MONITOR);

	if(comsub)
	{
	/* re-enable job control */
	if(!sp->nofork)
	sh_offstate(SH_NOFORK);
	job.jobcontrol = sp->jobcontrol;
	if(sp->monitor)
	sh_onstate(SH_MONITOR);

	TERM=vt100 tst $LINENO <<"!"
	L process/terminal group exercise

	w m=yes; while true; do echo $m-$m; done \| less
	u :$\|:\E\|lines
	c \cZ
	r Stopped
	w fg
	u yes-yes
	!

	/*
	* Resetting the PATH in a non-forking subshell will reset the parent shell's
	* hash table, so work around the problem by forking before sh_assignok
	*/
	if(shp->subshell && !shp->subshare && np == PATHNOD)
	sh_subfork();

	tst $LINENO <<"!"
	L POSIX sh 101(C)

	# If the User Portability Utilities Option is supported and shell
	# command line editing is supported: When in insert mode a control-V
	# causes the next character to be inserted even in the case that the
	# character is a special insert mode character.
	# Testing Requirements: The assertion must be tested with at least the
	# following set of characters: <newline>, erase, interrupt, kill,
	# control-V, control-W, end-of-file, backslash \ (followed by erase or
	# kill) and <ESC>.

	d 10
	p :test-1:
	w stty erase ^H intr ^C kill ^X
	p :test-2:
	w echo erase=:\cV\cH:
	u ^erase=:\r?\n$
	p :test-3:
	w echo kill=:\cV\cX:
	u ^kill=:\cX:\r?\n$
	p :test-4:
	w echo control-V=:\cV\cV:
	u ^control-V=:\cV:\r?\n$
	p :test-5:
	w echo control-W:\cV\cW:
	u ^control-W:\cW:\r?\n$
	p :test-6:
	w echo EOF=:\cV\cD:
	u ^EOF=:\004:\r?\n$
	p :test-7:
	w echo backslash-erase=:\\\cH:
	u ^backslash-erase=:\r?\n$
	p :test-8:
	w echo backslash-kill=:\\\cX:
	u ^backslash-kill=:\cX:\r?\n$
	p :test-9:
	w echo ESC=:\cV\E:
	u ^ESC=:\E:\r?\n$
	p :test-10:
	w echo interrupt=:\cV\cC:
	u ^interrupt=:\cC:\r?\n$
	!

	- BUG_LOOPRET2: If a 'return' command is given without a status argument
	within the set of conditional commands in a 'while' or 'until' loop (i.e.,
	between 'while'/'until' and 'do'), the exit status passed down from the
	previous command is ignored and the function returns with status 0
	instead.

ksh93 / ksh Goto Github PK

ksh's People

Contributors

Stargazers

Watchers

Forkers

ksh's Issues

Recommend Projects

Recommend Topics

Recommend Org