Code Monkey home page Code Monkey logo

shcl's Introduction

SHCL: Shell Meets Common Lisp

SHCL is

  1. a very customizable shell made with secret alien technology, and
  2. an unholy union of POSIX Shell and Common Lisp.

SHCL is more than just a shell. It is a mutual embedding of POSIX Shell and Common Lisp. Behold Common Lisp embedded in POSIX shell embedded in Common Lisp! Notice that the Common Lisp form embedded in the shell expression can access the lexical environment.

(let ((rld "rld"))
  (capture (:stdout)
    #$ echo Hello ,(concatenate 'string "Wo" rld) #$))
; => "Hello World"

Now lay your eyes on a lisp function participating in a pipeline!

shcl> : ,(shcl/core/debug:graph-dependencies) | dot -Tpng > graph.png

The #$ reader macro isn’t just some hack that constructs a string to be evaluated by a “real” shell. The #$ reader macro fully parses the shell expression and constructs an equivalent Common Lisp form. SHCL IS the “real” shell!

SHCL/CORE/LISP-INTERPOLATION> (macroexpand-1 '#$ if true; then echo woo; fi #$)
(SHCL/CORE/SHELL-FORM:SHELL-IF
 (SHCL/CORE/SHELL-FORM:SHELL-RUN
  (WITH-FD-STREAMS NIL
    (EXPANSION-FOR-WORDS (LIST #<NAME "true">) :EXPAND-ALIASES T
                         :EXPAND-PATHNAME-WORDS T :SPLIT-FIELDS NIL))
  :ENVIRONMENT-CHANGES NIL :FD-CHANGES NIL)
 (SHCL/CORE/SHELL-FORM:SHELL-RUN
  (WITH-FD-STREAMS NIL
    (EXPANSION-FOR-WORDS (LIST #<NAME "echo"> #<NAME "woo">) :EXPAND-ALIASES T
                         :EXPAND-PATHNAME-WORDS T :SPLIT-FIELDS NIL))
  :ENVIRONMENT-CHANGES NIL :FD-CHANGES NIL))
T

Building SHCL

SHCL is only really tested against SBCL and CCL, but it should be portable to other lisp compilers. Be aware that ECL is known to be problematic because it tries to reap child processes automatically.

First, you’ll need to install some dependencies. To start with, you’ll need Clang and libedit. There’s also some Common Lisp dependencies that need to be taken care of: SBCL, Quicklisp, and cffi-grovel. If you’re new to building Common Lisp projects, you might want to let Roswell set up your lisp environment for you.

# Set up Clang, libedit, and Roswell
make LISP='ros -s cffi-grovel run --'

You can skip Roswell if you want. Just make sure that you set LISP to a command that runs SBCL with Quicklisp and cffi-grovel loaded. For example,

# Set up Clang, libedit, SBCL, and Quicklisp
QUICKLISP_SETUP=~/quicklisp/setup.lisp # or wherever you installed quicklisp
make LISP="sbcl --no-userinit --load \"$QUICKLISP_SETUP\" --eval '(ql:quickload :cffi-grovel)'"

If you use the Nix package manager, building SHCL is super easy! SHCL has a default.nix file, so you just need to run nix-build.

nix-build

Congratulations! You built SHCL! If you try to run shcl you’ll probably find that it doesn’t work because it can’t find libshcl-support. As part of the build, SHCL produces a shared library named (you guessed it!) libshcl-support. That library needs to be installed somewhere that the dynamic linker can find it. So, go ahead and use sudo make install to install SHCL and its support library! Don’t forget to set the PREFIX to something you’re happy with. Alternatively, you can just use the run-shcl script included in the repository. run-shcl just adds $(pwd) to the dynamic linker’s search path before invoking ./shcl.

Note: if you build SHCL using nix-build, then you don’t have to worry about libshcl-support. SHCL will know how to find it!

Example Usage

I don’t know what you’re expecting to see here. Its a POSIX-like shell. You can do (almost) all your normal POSIX shell stuff in it.

shcl> echo foobar
foobar
shcl> { echo foobar ; echo baz ; echo blip ; } | tail -n 1
blip
shcl> shcl-enable-lisp-syntax
shcl> if [ ,(+ 1 2 3) = ,(* 2 3) ]; then
> echo woah wait what
> fi
woah wait what
shcl> shcl-repl
shcl (lisp)> (define-builtin upcase ()
> (loop :for line = (read-line *standard-input* nil :eof)
>       :until (eq line :eof) :do
>       (format "~A~%" (string-upcase line)))
> 0)
UPCASE
shcl (lisp)> ^D
shcl> { echo ahhh ; echo what is going on ; } | upcase
AHHH
WHAT IS GOING ON

Okay, actually, that kind of went off the rails.

shcl's People

Contributors

ahefner avatar diasbruno avatar phmarek avatar squirclespace avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

shcl's Issues

Failed to install with `sudo make install`

make LISP='ros -s cffi-grovel run --'

worked fine.

sudo make install

right after that - copied from the README - failed:

debugger invoked on a MISSING-DEPENDENCY in thread #<THREAD "main thread" RUNNING {1001890373}>: Component "cffi-grovel" not found, required by NIL

This worked:

sudo make LISP='ros -s cffi-grovel run --' install

Should the README be changed?
(I am a complete LISP newb)

Linux instructions, er, are sparse.

The instructions for

Building without nix:

Acquire dependencies (don't forget the lisp dependencies!)
make

"Acquire dependencies…"? That's kinda sparse for instructions. So, to spare others the pleasure of running and re-running sbcl, here's what I did on my Arch Linux system:

  1. Installed extra/sbcl and aur/quicklisp in addition to cloning shcl.
  2. Append to ~/.sbclrc:
    (load "/usr/lib/quicklisp/setup") (ql:quickload "trivial-gray-streams") (ql:quickload "cffi") (ql:quickload "misc-extensions") (ql:quickload "mt19937") (ql:quickload "fset") (ql:quickload "closer-mop") (ql:quickload "trivial-garbage") (ql:quickload "cl-ppcre") (ql:quickload "cl-fad") (ql:quickload "lisp-namespace") (ql:quickload "swank")
  3. Run make and then execute sbcl --load 'make.lisp' from within the cloned shcl directory.
    and then executed sbcl --load 'make.lisp' from within the cloned shcl directory.

Of course, I'm not a lisper, so there may be a Better Way(tm).

Attempted build, got "Fatal error: The variable +WHITESPACE-CHARACTERS+ is unbound."

I'm on OSX. I installed Roswell from Brew, and from there installed SBCL (1.5.4), Quicklisp, and cffi-grovel. Then I ran make LISP='ros -s cffi-grovel run --' and got, after a few things had loaded (apparently successfully; there were naming-convention warnings but no errors), the following:

; Loading "shcl"

Fatal error: The variable +WHITESPACE-CHARACTERS+ is unbound.

I don't see +WHITESPACE-CHARACTERS+ anywhere in the repo, from a grep. right, case-insensitive.

histedit.h "no such file" error

when I run make LISP='ros -s cffi-grovel run --' I get:

......

; compilation finished in 0:00:00.001
; cc -o /home/slade/.cache/common-lisp/sbcl-1.4.14-linux-x64/home/slade/Documents/sourcecode/shcl/shell/prompt-types-grovel__grovel-tmpONTOHUJ2.o -c -g -Wall -Wundef -Wsign-compare -Wpointer-arith -O3 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Wunused-parameter -fno-omit-frame-pointer -momit-leaf-frame-pointer -fPIC -I/home/slade/.roswell/lisp/quicklisp/dists/quicklisp/software/cffi_0.20.1/ /home/slade/.cache/common-lisp/sbcl-1.4.14-linux-x64/home/slade/Documents/sourcecode/shcl/shell/prompt-types-grovel__grovel.c
/home/slade/.cache/common-lisp/sbcl-1.4.14-linux-x64/home/slade/Documents/sourcecode/shcl/shell/prompt-types-grovel__grovel.c:6:10: fatal error: histedit.h: No such file or directory
    6 | #include <histedit.h>
      |          ^~~~~~~~~~~~
compilation terminated.

Fatal error: Subprocess #<UIOP/LAUNCH-PROGRAM::PROCESS-INFO {100748F7C3}>
 with command ("cc" "-o" "/home/slade/.cache/common-lisp/sbcl-1.4.14-linux-x64/home/slade/Documents/sourcecode/shcl/shell/prompt-types-grovel__grovel-tmpONTOHUJ2.o" "-c" "-g" "-Wall" "-Wundef" "-Wsign-compare" "-Wpointer-arith" "-O3" "-D_LARGEFILE_SOURCE" "-D_LARGEFILE64_SOURCE" "-D_FILE_OFFSET_BITS=64" "-Wunused-parameter" "-fno-omit-frame-pointer" "-momit-leaf-frame-pointer" "-fPIC" "-I/home/slade/.roswell/lisp/quicklisp/dists/quicklisp/software/cffi_0.20.1/" "/home/slade/.cache/common-lisp/sbcl-1.4.14-linux-x64/home/slade/Documents/sourcecode/shcl/shell/prompt-types-grovel__grovel.c")
 exited with error code 1
; 
; compilation unit aborted
;   caught 1 fatal ERROR condition
;   caught 16 STYLE-WARNING conditions
;   printed 868 notes
make: *** [Makefile:67: shcl] Error 1

Static linking of libshclsupport

SHCL depends on a small C library named libshclsupport. SHCL loads it with cffi. That's annoying. It would be way cooler if SHCL didn't depend on a special purpose shared library. Distributing SHCL is going to be awkward as long as SHCL needs a support library.

Supposedly, static linking is a thing, now. Maybe that's a viable option?

SHCL hangs forever

To reproduce:

  1. Build shcl like so:
    sbcl --load make.lisp
  2. Run ./shcl
  3. Enter the following commands
    echo
    FOO=123

Export builtins for use with tools like find

This is a completely crazy idea that almost no one actually wants, but...

Tools like find and xargs take in a command to run. Normally, you have to name a command backed by a binary. You can’t really tell xargs to run a shell function you defined earlier in your script.

Or can you?

Imagine there was an executable on disk that represented the builtin command, and running that executable magically got SHCL to run the appropriate builtin in the original shell process. You could just point xargs at that binary and suddenly it would be effectively calling a shell builtin. SHCL could create these files on demand and clean them up when xargs exits.

With FUSE, you could do some even whackier stuff along these lines.

Can't find sbcl.core

I managed to build shcl on guix, but get this error when trying to run.

$ shcl
fatal error encountered in SBCL pid 10164 tid 10164:
Can't find sbcl.core

this is what the installed directory looks like:

├── bin
│   └── shcl
├── lib
│   └── libshcl-support.so
└── share
    └── doc
        └── sbcl-shcl-0.0.1
            └── LICENSE.txt

am I missing something?

"SHCL doesn't fork" - discussion

Hi,

I'm not sure whether the "SHCL doesn't fork" will work in practice.
How would you get $$ and $PPID correct? A shell script that does

(echo $$ > logfile ; exec long-running-process ) &
...
kill $(cat logfile)

would kill the whole SHCL process, and not just the child.
Even if $$ and $PPID got faked in a subshell, forking would be inevitable as soon as external commands are being called; and investigating the sub-shell command line to find out whether to fork or not feels quite unclean, too.

So I ask you to discuss for and against forking.

Thank you!

Empty input lines are no good

$ true | ./shcl 
$ echo | ./shcl 

debugger invoked on a SHCL/CORE/SHELL-FORM::UNBOUND-SHELL-FORM-TRANSLATOR in thread
#<THREAD "main thread" RUNNING {10006905B3}>:
  Symbol TRUTHY-EXIT-INFO is unbound in namespace SHELL-FORM-TRANSLATOR

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [USE-VALUE] USE-VALUE
  1: [IGNORE   ] IGNORE
  2: [DIE      ] SHCL/SHELL/MAIN::DIE
  3: [ABORT    ] Exit from the current thread.

((:METHOD SHCL/CORE/SHELL-FORM::TRANSLATE-SHELL-FORM (CONS T)) (SHCL/CORE/EXIT-INFO:TRUTHY-EXIT-INFO) #<NULL-LEXENV>) [fast-method]
   source: (SYMBOL-SHELL-FORM-TRANSLATOR (CAR FORM))
0] 3

Ie. an empty line, followed by EOF (Ctrl-D interactively), gives this error.

[packaging] reference to $srcdir; fstat is undefined.

I'm trying to package for archlinux. (https://github.com/WillForan/shcl/blob/archlinux/archlinux/PKGBUILD)
I added options=(!strip) to keep makepkg from totally destroying the shcl binary, but as a result:

==> WARNING: Package contains reference to $srcdir
usr/bin/shcl

shcl launches and there is prompt. but I pushed tab and see

debugger invoked on a SB-KERNEL::UNDEFINED-ALIEN-FUNCTION-ERROR in thread
#<THREAD "main thread" RUNNING {1002DF8103}>:
The alien function "fstat" is undefined.

make fails on macOS

I tried to compile on macOS and looks like it is calling a Linux only function clearenv

 $ make
clang -fPIC -o core/support/macros.o -c core/support/macros.c
clang -fPIC -o core/support/spawn.o -c core/support/spawn.c
core/support/spawn.c:137:14: warning: implicit declaration of function 'clearenv' is invalid in C99
      [-Wimplicit-function-declaration]
    if (0 != clearenv()) {
             ^
1 warning generated.
clang -shared -o libshcl-support.so core/support/macros.o core/support/spawn.o
Undefined symbols for architecture x86_64:
  "_clearenv", referenced from:
      _shcl_spawn in spawn.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [libshcl-support.so] Error 1

Simple custom builtin failing with stack exhausted error.

I wanted to write a simple alias like alias ls=ls --color, so I wrote a simple builtin to start off:

(define-builtin ls (&rest args)
  (shcl/core/lisp-interpolation:evaluate-constant-shell-string "ls --color"))

This fails quite spectularly:

ralt@zap:~/common-lisp/shcl$ ls
INFO: Control stack guard page unprotected
Control stack guard page temporarily disabled: proceed with caution

debugger invoked on a SB-KERNEL::CONTROL-STACK-EXHAUSTED in thread
#<THREAD "main thread" RUNNING {1002DD8083}>:
  Control stack exhausted (no more space for function call frames).
This is probably due to heavily nested or infinitely recursive function
calls, or a tail call that SBCL cannot or has not optimized away.

PROCEED WITH CAUTION.

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [IGNORE] IGNORE
  1: [DIE   ] SHCL/SHELL/MAIN::DIE

(SB-KERNEL::CONTROL-STACK-EXHAUSTED-ERROR)
0]

Is that something you're familiar with?

Signal handling

SHCL doesn't support any sort of signal handling. It should.

This will be trickier than in your typical shell. SHCL doesn't fork for subshells. Since signals are delivered to a process, it might be impossible to be 100% POSIX compliant for signal handling. Still, being as-compliant as possible would be good.

Import commands as lisp functions

We’ve got good support for exposing lisp code to shell. Let’s bridge the other way, too! Shell commands should be exposable as lisp functions!

Integration with stumpwm

Stumpwm's exec command builds a string for /bin/sh to eval. Stumpwm could just load SHCL and have SHCL do the eval.

Prompt customization and others

Hello,

Instead of shcl>, it'd be nice to have something else, like current directory or w/e. Looking at the source, it seems to be hardcoded right now: https://github.com/bradleyjensen/shcl/blob/master/shell/main.lisp#L65

Relatedly, I haven't seen anywhere where it loads a user-provided file to execute it (e.g. LOAD it).

I guess the 2 birds could be killed with one stone; make the prompt funcall a function that can be provided by the user in an RC file, and that should be good. (Related: do you have an easy way to call a shell command from Lisp? That could be used in the RC file.)

What do you think?

Elisp Splices

When running a shell command from Emacs (e.g. M-!), it would be handy if the lisp splice syntax (i.e. ,) didn't evaluate the lisp form as Common Lisp in SHCL, but instead evaluated it as Emacs Lisp within Emacs.

There are a couple of ways this can be accomplished.

  1. SHCL is just running inside Emacs. If SHCL is linked by Emacs then SHCL should be able to call out to Emacs Lisp somehow. Practically speaking this means SHCL will need to build as an ECL library or something. Alternatively, we could implement Common Lisp in Emacs Lisp...

  2. SHCL uses emacsclient to get Emacs to evaluate the expression. This requires Emacs to be running a server, but that shouldn't be too hard. It seems like Emacs Lisp's syntax is a strict subset of Common Lisp's syntax, so SHCL could use READ like it does today. Instead of evaluating the form it would hand emacsclient a string containing the input that READ consumed.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.