knrafto / language-bash Goto Github PK

View Code? Open in Web Editor NEW

35.0 35.0 9.0 163 KB

Parse and pretty-print Bash shell scripts

License: BSD 3-Clause "New" or "Revised" License

Haskell 98.11% Shell 1.89%

language-bash's People

Contributors

Stargazers

Watchers

Forkers

jfoutz rhendric jb55 mgsloan hardentoo fosskers mmhat forist2034

language-bash's Issues

Why not treat escaped oct as a whole?

For example, "x\012y" will be parsed as [Char 'x',Escape '0',Char '1',Char '2',Char 'y'].

For other situation, I could deal with the span individually (like from [Char 'l', Char 's'] to ExtCmd ls).

But for the above example, I'd have to make a special case for Escape '0'. Why not make it Escape '\n' (I am not sure if \012 is \n), since actually "\012" is a char in system.

On the other hand, one might say "parser should return what it is, not what it means". But then I should not get Escape '0', either. I should get Char '\', Char '0'.

Here documents are rendered poorly

Consider the following bash snippet:

cat <<EOF
here document
EOF

If one parses it with Language.Bash.Parse.parse and pretty prints it with Language.Bash.Pretty.prettyText it looks like that:

cat <<EOF
here document
EOF
;

Note the semicolon at the end: Bash will refuse the script with "line 4: syntax error near unexpected token `;'".

AndOr is not parsed right.

parse "" "ls && ls"
Right (List [Statement (Last (Pipeline {timed = False, timedPosix = False, inverted = False, commands = [Command (SimpleCommand [] [[Char 'l',Char 's'],[Char '&',Char '&'],[Char 'l',Char 's']]) []]})) Sequential])

Rename ppCond ppCondList

Minor leftover point from #33.

IMO I don't think "conditional command" is accurate, maybe "block command" would be better terminology.

Annotate AST with comments and source positions

It would be useful for me if comments were put in the AST in a "Comment" construct (I plan to translate them into Comments in my new language).

Redirection not supported.

Prelude> :m + Language.Bash.Parse
Prelude Language.Bash.Parse> parse "Test" "ls > log"
Loading package array-0.5.0.0 ... linking ... done.
Loading package deepseq-1.3.0.2 ... linking ... done.
Loading package bytestring-0.10.4.0 ... linking ... done.
Loading package transformers-0.3.0.0 ... linking ... done.
Loading package mtl-2.1.2 ... linking ... done.
Loading package text-1.1.0.0 ... linking ... done.
Loading package parsec-3.1.5 ... linking ... done.
Loading package pretty-1.1.1.1 ... linking ... done.
Loading package language-bash-0.3.0 ... linking ... done.
Left "Test" (line 1, column 6):
unexpected ">"
expecting redirection, |, |&, &&, ||, list terminator or pipeline

Add Monoid/Semigroup instance for Lists

It would be nice if we had Monoid and Semigroup instances for the List
type.

Triple-parens "(((" appears to cause an infinite loop

Test case:

module Main (main) where
import Language.Bash.Parse (parse)
main = print (parse "<input>" "(((")

This program uses up lots of memory and does not terminate for me. (I'm using language-bash version 0.6.1 from Hackage.)

[ is not treated as a builtin

[ is not parsed into Cond, Binary, etc, as [[ is. Based on [1], [2], and the code for test and [[, I'm guessing this is an oversight rather than a design decision.

Happy to code up a fix, which would reuse the test code, but I wanted to check that it would be accepted before I did.

[1] https://www.gnu.org/software/bash/manual/html_node/Bourne-Shell-Builtins.html#Bourne-Shell-Builtins
[2] https://www.gnu.org/software/bash/manual/html_node/Bash-Conditional-Expressions.html#Bash-Conditional-Expressions,

Expose ByteString version of 'parse'

https://hackage.haskell.org/package/language-bash-0.9.0/docs/Language-Bash-Parse.html#v:parse is only String. That is really unfortunate. String is both a weird (not even UTF-8) format and is unefficient. I generally don't use it at all.

Nothing else from the module is exposed, so it's hard to build my own version without forking the library.

Arithmetic is not parsed

Looks like arithmetic becomes an ArithSubst String, and it isn't parsed any further. What's the plan there? Should I instead try to parse the arithmetic strings as C?

Pretty-printed while command indents too much

> let Right e = parse "<stdin>" "while true; do :; done;" in putStrLn $ prettyText e
while true; do
                :;
            done;

For reference, the output from bash's declare -f is:

$ f() { while true; do :; done }
$ declare -f f
f () 
{ 
    while true; do
        :;
    done
}

It seems like doDone l binds too tightly that the whole block is now beside the while p; text.

Parse pipelines that span multiple lines

e.g.

echo 'foo' |
  cat

Right now we can't handle a newline after |.

Escape octal issue.

Sorry, I do not know how to open issue 4 again. I found it worse.

$ cat /tmp/test.sh
echo -e 'x\064y'
$ echo -e 'x\064y'
x4y

λ>bash <- readFile "/tmp/test.sh"
λ>parse "" bash
Right (List [Statement (Last (Pipeline {timed = False, timedPosix = False, inverted = False, commands = [Command (SimpleCommand [] [[Char 'e',Char 'c',Char 'h',Char 'o'],[Char '-',Char 'e'],[Single [Char 'x',Char '\\',Char '0',Char '6',Char '4',Char 'y']]]) []]})) Sequential])

Test suite

One option: round-trip testing by generating syntax trees, pretty-printing them, and parsing them. But this doesn't test one element at a time or every possible variation of Bash syntax.

Pipeline is not prettied correct.

> let Right s = parse "" "ls | tail"
> prettyText s
"ls tail;"

Brace expression property tests fails occasionally

QuickCheck complains with "Use --quickcheck-replay=735772 to reproduce."

Double-quoted strings are not parsed by parseTestExpr

Again, wondering if this is a design decision?

When I call parseTestExpr on a list of words (the '[' and ']' have already been stripped off), the words are not parsed and double-quoted strings with command substitutions in them are not processed.

How were you thinking this should be accomplished?

convertTest :: [W.Word] -> Expr
convertTest ws = case condExpr of
    Left  err -> Debug $ "doesn't parse" ++ (show err) ++ (show strs)
    Right e -> convertCommand e
    where condExpr = C.parseTestExpr strs
          strs = (map W.unquote ws)

Provide optics?

I tried to get all top-level assignments and ended up with:

extractAssignments :: List -> [Assign]
extractAssignments (List stms) = join $ fmap getAssign $ getCommands stms
 where
  getCommands :: [Statement] -> [Command]
  getCommands = join . fmap commands . catMaybes . fmap findPipes
   where
    findPipes (Statement (Last p@(Pipeline{})) Sequential) = Just p
    findPipes _ = Nothing

  getAssign :: Command -> [Assign]
  getAssign (Command (SimpleCommand ass _) _) = ass
  getAssign _ = []

I believe this would be easier with optics. I would propose the optics package, because it has better documentation, an opaque interface with useful type errors and safer behavior in some corner cases (e.g. less implicit monoidal behavior) than lens.

Test pretty printing

While working on #24 it turned out that the current test suite performed poorly.
Proposal:

Drop a bash snippet somewhere in the tests/ directory
Discover those snippets
Parse them
Pretty print them
Compare the output to a golden file using tasty-golden
Run bash on the original file and the pretty printed one and compare the output

This is of course only for those tests involving pretty printing; we can keep the existing unit tests as they are.
Additionally we could fuzzy test the implementation with QuickCheck.

Performance

I use language-bash in ghcup to parse variable assignments from files like /etc/os-release: https://gitlab.haskell.org/haskell/ghcup-hs/-/blob/master/lib/GHCup/Utils/Bash.hs

This works relatively well, but when I profile ghcup on the simplest command ghcup list, I get this:

	Thu Jun 25 21:56 2020 Time and Allocation Profiling Report  (Final)

	   ghcup +RTS -p -RTS list

	total time  =        0.05 secs   (52 ticks @ 1000 us, 1 processor)
	total alloc =  56,666,312 bytes  (excludes profiling overheads)

COST CENTRE       MODULE                            SRC                                                      %time %alloc

word              Language.Bash.Parse.Word          src/Language/Bash/Parse/Word.hs:300:1-43                  15.4   40.2
jstring           Data.Aeson.Parser.Internal        Data/Aeson/Parser/Internal.hs:316:1-32                    11.5    5.5
try               Control.Monad.Catch               src/Control/Monad/Catch.hs:789:1-47                        9.6    9.4
skipSpace         Language.Bash.Parse.Word          src/Language/Bash/Parse/Word.hs:(81,1)-(86,50)             5.8    1.5
<?>               Data.Aeson.Types.Internal         Data/Aeson/Types/Internal.hs:532:1-74                      3.8    0.4
unfoldDirContents Streamly.External.Posix.DirStream src/Streamly/External/Posix/DirStream.hs:(50,1)-(59,69)    3.8    1.1
dirContentsStream Streamly.External.Posix.DirStream src/Streamly/External/Posix/DirStream.hs:(70,1)-(72,83)    3.8    2.0
lexeme            Text.Megaparsec.Lexer             Text/Megaparsec/Lexer.hs:78:1-23                           3.8    3.2
pathParser'       URI.ByteString.Internal           src/URI/ByteString/Internal.hs:(567,1)-(569,82)            3.8    1.6
decimal           Data.Attoparsec.ByteString.Char8  Data/Attoparsec/ByteString/Char8.hs:(447,1)-(448,49)       1.9    0.1
copy              Data.HashMap.Array                Data/HashMap/Array.hs:(328,1)-(333,30)                     1.9    0.0
object_'          Data.Aeson.Parser.Internal        Data/Aeson/Parser/Internal.hs:(135,49)-(137,22)            1.9    0.5
catchAll          Control.Monad.Catch               src/Control/Monad/Catch.hs:744:1-16                        1.9    1.5
mapExcepts        Haskus.Utils.Variant.Excepts      src/lib/Haskus/Utils/Variant/Excepts.hs:102:1-33           1.9    0.0
liftE             Haskus.Utils.Variant.Excepts      src/lib/Haskus/Utils/Variant/Excepts.hs:110:1-38           1.9    0.0
mapVariantAt      Haskus.Utils.Variant              src/lib/Haskus/Utils/Variant.hs:(413,1)-(416,47)           1.9    0.0
uncons            Data.ByteString.Lazy.UTF8         Data/ByteString/Lazy/UTF8.hs:(228,1)-(229,38)              1.9    0.6
parseAbs          HPath                             src/HPath.hs:(142,1)-(147,38)                              1.9    0.0
word              Language.Bash.Parse.Internal      src/Language/Bash/Parse/Internal.hs:175:1-66               1.9    0.1
ioDesc            Language.Bash.Parse.Internal      src/Language/Bash/Parse/Internal.hs:188:1-46               1.9    0.3
assign            Language.Bash.Parse.Internal      src/Language/Bash/Parse/Internal.hs:217:1-43               1.9    0.2
anyOperator       Language.Bash.Parse.Internal      src/Language/Bash/Parse/Internal.hs:209:1-51               1.9    3.0
select            Language.Bash.Operator            src/Language/Bash/Operator.hs:19:1-43                      1.9    1.8
runParsecT        Text.Megaparsec.Internal          Text/Megaparsec/Internal.hs:(591,1)-(596,56)               1.9    0.1
getOptic          Optics.Internal.Optic             src/Optics/Internal/Optic.hs:70:5-12                       1.9    0.0
wrapCompile       Text.Regex.Posix.Wrap             src/Text/Regex/Posix/Wrap.hsc:(372,1)-(384,42)             1.9    0.0
regNameParser     URI.ByteString.Internal           src/URI/ByteString/Internal.hs:(533,1)-(535,59)            1.9    0.3
getDownloadsF     GHCup.Download                    lib/GHCup/Download.hs:(105,1)-(127,42)                     1.9    0.0
printListResult   Main                              app/ghcup/Main.hs:(1332,1)-(1377,18)                       1.9    0.2
satisfying        Language.Bash.Parse.Internal      src/Language/Bash/Parse/Internal.hs:(130,1)-(132,49)       0.0    1.3
pack              Language.Bash.Parse.Internal      src/Language/Bash/Parse/Internal.hs:(99,1)-(122,14)        0.0    2.2
anyWord           Language.Bash.Parse.Internal      src/Language/Bash/Parse/Internal.hs:171:1-39               0.0    2.4
name              Language.Bash.Parse.Word          src/Language/Bash/Parse/Word.hs:(313,1)-(316,38)           0.0    1.8
functionName      Language.Bash.Parse.Word          src/Language/Bash/Parse/Word.hs:(320,1)-(324,31)           0.0    1.5
getDownloads      GHCup.Download                    lib/GHCup/Download.hs:(143,1)-(261,42)                     0.0    2.2

This is on a 7 LOC bash file:

NAME="Exherbo"
PRETTY_NAME="Exherbo Linux"
ID="exherbo"
ANSI_COLOR="0;32"
HOME_URL="https://www.exherbo.org/"
SUPPORT_URL="irc://irc.freenode.net/#exherbo"
BUG_REPORT_URL="https://bugs.exherbo.org/"

I'm wondering if there's a way to reduce the allocations.

Brace expansion is not correct

bash$ echo {{a,b}
{a {b

ghci> map (\c -> case c of Char x -> x) (head (braceExpand (map Char "{{a,b}")))
"{{a,b}"

Since the bash expansion algorithm knows about quoting and escaping, this would need to be made aware of those too. (It's really ugly...)

bash$ echo {q\{b{zp,\}c}{kmy\}}{tjl}
{q{bzp{kmy}}{tjl} {q{b}c{kmy}}{tjl}

Brace expansion edge case

QuickCheck found this in this job

Tests
  Properties
    brace expansion:     FAIL (0.08s)
      *** Failed! Assertion failed (after 56 tests): 
      "\\}\\,,{y}{\\{\\ }\\}{}\\}s\\ p\\{\\},\\,lhhiq\\,qv}{}\\},}\\}\\ \\,{\\}\\ \\{a\\ \\,\\,z\\,t"
      Use --quickcheck-replay '55 TFGenR 0000000309618D3500000000000F4240000000000000E1390000000056F692C0 0 72057594037927935 56 0' to reproduce.

Pretty printing of here documents without trailing newline

Consider the following here document:

Heredoc Here "EOF" False (stringToWord "here document")

If pretty printed it looks like that:

<<EOF
here documentEOF

This is clearly not the intended result. Maybe we should add a trailing newline if it is not present in the Word being rendered.

Switch from Parsec to Megaparsec

Megaparsec is more actively maintained, and will give us better error messages

Switch to a more advanced pretty printing library

It would be nice if we support Text pretty printing in addition to simple Strings.
We could use grouping features to improve the output as well.
Personal choice: prettyprinter