hosh-shell / hosh Goto Github PK

Problem:
Thread name should be improved, version (v0.0.27) reports:

[command='Sink'] instead [command='sink']
[command='ListFiles /'] instead of [command='ls /']
[command='ExternalCommand pom.xml'] instead of [command='cat pom.xml']
[command='DefaultCommandWrapper 1'] instead of [command='benchmark 1']

Solution:
provide a simple API to describe commands, something like Junit Description. As bonus would be nice to provide line number and original arguments (before expansion).

File git.properties contains several other useful properties, such as:

• git.commit.id.describe=v0.0.23-19-ge13c22b-dirty
• git.commit.time=2019-02-13T21\:43\:30+0100

These properties would be very useful in bug reports and for general usage.

User should be able to interrupt current command by using CTRL-C.

https://github.com/jline/jline3/tree/master/style

Hosh has different value types buried in the modules:

• None
• Duration (withTime)
• Numbers
• Instant
• Regex/Pattern
• Error, often we capture an exception and create a simple Text

Right now Hosh on Windows tries just ".exe". This could be improved by using PATHEXT:

PathExt environment variable returns a list of the file extensions that the operating system considers to
be executable. When executing a command line that does not contain an extension, the command
interpreter (cmd.exe) uses the value of this environment variable to determine which extensions to look
for and in what order.[1]
http://environmentvariables.org/PathExt

Looks like in certain cases it is not possible to capture the output produced by the after method in CommandWrapper

This is working as expected:

hosh:1> benchmark 1 { ls } | schema
path size

Instead this is not working (no output):

hosh:2> benchmark 1 { ls | sink } | schema

whereas this is working as expected:

hosh:4> benchmark 1 { sink } | schema
count best worst average

Try to optimize cpu/memory usage by creating specialized versions of Record with 0, 1, more keys. By now, most of Record instances contain only 1 key.

Since Record don't allows removal and it uses a fluent interface should be simple to change implementation:

Empty -> SIngleton -> Generic

Problem:
hitting ctrl-C when a single, not pipeline, external or a native command is running, terminates the whole shell

Solution:
let the Interpreter handle the execution + handling of ctrl-C. Probably a ExecutorAware interface is needed to let the pipeline to submit jobs

Benefit:
have a single point where futures are created, thread name assigned.

to simplify debug with jstack

problem:

"ls /tmp" in macosx yields no results since /tmp is a symlink

solution:
hosh should follow symlink by default

move all exception handling in the Supervisor, keeping the Interpreter simpler

Actually this is accepted:
withTime { ls } }}}}}}

It looks like that JDK11 ships a module named jdk.internal.jline, used by jdk.jshell. Could be worth to reuse?

Problem:
unit tests have these asserts:

assertThat(exitStatus.value()).isEqualTo(21);
assertThat(exitStatus.value()).isEqualTo(0);
assertThat(exitStatus.isSuccess()).isTrue();

Solution:
custom assert for ExitStatus to be more readable

less is not working properly when used in a pipeline:

less pom.xml works fine (i.e. pagination)
cat pom.xml | less does not work (i.e. no pagination)

Tested on macOS Mojave:

less 487 (POSIX regular expressions)
Copyright (C) 1984-2016  Mark Nudelman

Introduce a Key type for Records. Add also a some constant definitions to avoid errors such as:

• Keys.MESSAGE human formatted message
• Keys.ERROR to store an error message
• Keys.EXCEPTION to store an exception

In bash:
export HISTCONTROL=ignoredups

Hosh should have this configuration by default

This is how sort should be used:

hosh:1> help | sort command
bell
cat
cd
clear
count
...

Bugs:

1. this uses external command 'sort', causing the shell to stuck

hosh:2> help | sort 

2. when key is not found nothing is signaled to the user

hosh:1> help | sort asdasdasdasdasdasd
... normal output

Latest version is: 1.4.5

           <plugin>
                <groupId>org.pitest</groupId>
                <artifactId>pitest-maven</artifactId>
                <version>${version.plugin.pitest}</version>
                <configuration>
                    <targetClasses>
                        <param>${pitest.target}.*</param>
                    </targetClasses>
                    <targetTests>
                        <param>${pitest.target}.*</param>
                    </targetTests>
                </configuration>
            </plugin>

• cp
• mv
• rm
• symlink (a good use case for showing portability in windows)
• hardlink
• realpath (see https://stackoverflow.com/questions/5265702/how-to-get-full-path-of-a-file)
• probe https://docs.oracle.com/javase/7/docs/api/java/nio/file/Files.html#probeContentType(java.nio.file.Path)
• partitions (df -h)

Problem:

hosh:69> ls
/tmp/KSDownloadAction.mMjQifWXyB

there is no clue that this is an error. Only with debug it is possible to see it:

2019-03-23T09:26:43.142 [WARNING] [command='ListFiles'] - caught I/O error
java.nio.file.AccessDeniedException: /tmp/KSDownloadAction.mMjQifWXyB
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)

Picocli is quite interesting, looks like there is some integration with JLine3:

jline/jline3#349
https://github.com/remkop/picocli/tree/master/picocli-shell-jline3

Problem:
/usr/ TAB yields /usr without proposing further matches
additionally at start of line should pick only executables

Solution:
TDB

Benefit:
to have a useful filesystem autocomplete

problem:
lines | wc -l does not work!

solution:
read from Channel in and write to the native process OutputStream

benefit:
interoperability between native and non-native commands

Just discovered this old blog post:

https://puredanger.github.io/tech.puredanger.com/2009/02/28/java-7-transferqueue/

Some unscientific benchmarks (benchmarks/rand_take.hosh):

transfer() (commit 77b9700):
50 PT0.036367452S PT0.131597582S PT0.047171256S

put() (commit c845043):
50 PT0.009320667S PT0.082922534S PT0.019954243S

ArrayBlockingQueue + fair=false (commit e1dd261):
50 PT0.010402944S PT0.094943864S PT0.022610688S

dogfooding : watch for changes in a directory sending events downstream.
Events could be records with two keys: type (added, removed, modified) and path, downstream consumer could start a job (e.g. mvn clean verify).

Ideally command completion should be programmable. But let's try to provide some obvious
defaults:

• cd should propose only directories
• kill should propose only local process ids, along with a brief description

Problem:

Windows 7 (running OpenJDK 11.0.2)

hosh:1> benchmark 100 { rand | take 10000 | count }
100 1,144 1,246 1,159

Same Hosh version on Macosx (OpenJDK 11.0.2)

hosh:1> benchmark 100 { rand | take 10000 | count }
100 8 97 13

Problem:
have a more efficient way to create records with more than 2 keys (instead of discarding 2 or 3 hashmaps)

Solution:
have a builder patter for Record, something like:

new Record.Builder().entry(k1, v1).entry(k2, v2)....build()

right now hosh supports only pipelines composed of 2 commands

See:
https://junit.org/junit5/docs/snapshot/user-guide/#migrating-from-junit4

JDK11 has a new module for HTTP (https://openjdk.java.net/jeps/321).

It would be convenient to provide a very simple curl/wget command that depends on only standard JDK classes.

Nice intro:
https://openjdk.java.net/groups/net/httpclient/intro.html
https://openjdk.java.net/groups/net/httpclient/recipes.html

For testing:
https://httpbin.org/

This should not be accepted:

ls | drop 1 |

Problem:
it would be useful to capture output coming from wrappers, especially benchmark:

hosh:5> benchmark 10  { ls | drop 1000 } | table
10 1 3 2

Solution: ???

problem:
cannot resolve variables in certain cases:

echo ${OS_ENV_VARIABLE} | take 1
withTime { echo ${OS_ENV_VARIABLE} } 

solution:
using another (!) marker interface to inject an ArgumentResolver

Problem:
since it is hard to test all possible errors it would be nice to have a way to inject failures
without writing java code

Solution:
create a command to raise exceptions according to some rules:

• interrupted
• null pointers

Benefit:
it would be easy to test error scenarios for pipelines and wrappers

this is key to show the some usability progress

Would be nice to have a human readable syntax for simple text processing tasks. Something like:

https://github.com/andrewbihl/bsed/blob/master/README.md

SIGINT handling is quite easy to break and very boring to be manually tested.
This is just a note about some scenarios to test:

• single external command like cat
• single built-in command like rand
• pipeline with built-in and external commands
• pipeline with wrappers
• benchmark

replace with JUL for several reason:

• simplicity: just use JDK classes
• nicer lazy logging API:
  info​(Supplier<String> msgSupplier) -- Log a INFO message, which is only to be constructed if the logging level is such that the message will actually be logged.
• JMX support of out the box and can be changed at runtime https://docs.oracle.com/en/java/javase/11/docs/api/java.management/java/lang/management/PlatformLoggingMXBean.html
• it is possible to use just String.format (no special syntax like {})
• saving around 100KB in the final jar

problem:
measure progress in performance and optmization

solution:
write a command that repeat a given block for several times and report the best/worst/average

benefit:
explore possible optimization faster (e.g. #30 and #3)

Compiler should create and assemble pipelines as Command to keep the interpreter ignorant as much as possible.

Same technique could be used for compiling a CommandWrapper.

Output captured by an external command should be converted into a Record and sent to the out channel.

The goal is to have mixed pipelines working e.g. ifconfig | take 1.

Problem:
Commands need to access system services and user data. Code should be cluttered by manual DI injection e.g.PipelineCommand' must be StateAwarejust to forward State to children. Additionally builtin commands are singleton (i.e. sole instance registered) and yet they have a mutablesetState()orsetTerminal()`.

Solution:
https://github.com/andrus/di-comparison

DI, but spring is a big dependency (spring-core 1.2MB + spring-beans 600KB). Dagger 2 could be a good trade off:
https://github.com/google/dagger

Benefit:
Remove some boilerplate code and tests.

Problem:
set and unset variables

Solution:
set VARNAME VALUE to create/update a variable in State
untset VARNAME to remove a variable from State

NB: fishshell has a nice syntax for set/unset variables:

https://fishshell.com/docs/current/faq.html