blynn / nex Goto Github PK

Are multiline comments possible with nex? I've been playing with nex quite a bit and I'm very impressed. However, I'm having trouble figuring out how to handle multiline comments. Can this been done with the nesting feature? I've tried but could not figure it out. Below is one thing I tired but I regex is greedy the nesting grabs the rest of the document.

/[0-9]+/            { println("INTEGER:", txt()) }
/[0-9]+\.[0-9]*/    { println("FLOAT:", txt()) }
/if|then|begin|end|procedure|function/
                    { println( "KEYWORD:", txt()) }
/[a-z][a-z0-9]*/    { println( "ID:", txt()) }
/\+|-|\*|\//        { println("OP:", txt()) }
/[ \t]+/            { /* eat up whitespace */ }
/\/\*.*|\n*\*\// <  { println("BEG_MULTI:", txt()) }
  /.*|\n*\*\//      { println("BEG_COMMENT:", txt()) }
  /\n/              { println("NEWLINE") }
>                   { println("END_COMMENT", txt()) }
/\n/                { println("NEWLINE") }
/./                 { println("UNRECOGNIZED CHAR:", txt()) }
//
package main
import (
  "os"
)

func main() {
  lex := NewLexer(os.Stdin)
  txt := func() string { return lex.Text() }
  NN_FUN(lex)
}

Should be as simple as renaming nex.go to main.go.

Description

In the source there is func (yylex *Lexer) Line() int func (yylex *Lexer) Column() int. I think it would also be useful to have func (yylex *Lexer) Size() int which returns position in bytes relative to the start of the files.

Current approach

I'd like to lex and parse a number of different files together. The approach I'm attempting is the concatenate them together with a https://golang.org/pkg/io/#MultiWriter. I also store a list of cumulative file.Size() offsets for each file. I get this via Stat when opening each file for the MultiWriter.

At the end of lexing/parsing if I get an error, I work backwards from position of error to line number and file.

Problem

The problem is that I'm doing the "math" with line numbers instead of Size offsets. This means that I need to loop through each file (before lexing/parsing) and count all the newlines. If Size was available in addition to Column and Row, then this would be much more direct.

I haven't yet worked out how I'd get from the correct size offset to Column and Line, but I figure that's doable somehow.

Help

Help is appreciated if anyone can contribute. This currently works with line number alone, but as I mentioned a Size offset might be preferable. If there's an alternative technique for lexing/parsing multiple files together as one, please let me know!

Thanks!

Is this possible? Currently I have to do something like:

in parser.y:

top:	prog
	{
		langGlobal = $1.prog
	}
;

in main.go:

	lexer := NewLexer(os.Stdin)
	yyParse(lexer) // writes the result to langGlobal

	log.Printf("behold, the AST: %+v\n", langGlobal)

I know it's not directly nex related, but I was hoping someone here might know! Thanks :)

As far as I can tell, there doesn't seem to be a way to change the func (yylex Lexer) Error(e string) function. This function gets called automatically from a go tool yacc generated parser, so it would be nice to be able to customize it.

I was having trouble using nex with go yacc so I copied the example to test it out. I compiled, but it didn't work, as soon as I enter some text it breaks

Using:
Mac OS 10.10.1
Go version go1.3.3 darwin/amd64

panic: syntax error

goroutine 16 [running]:
runtime.panic(0xa6300, 0x2081aa300)
    /usr/local/Cellar/go/1.3.3/libexec/src/pkg/runtime/panic.c:279 +0xf5
main.Lexer.Error(0x2081ae0c0, 0x2081cc240, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0xe67f0, ...)
    /Users/donnanicolas/go/src/tu-carrito.com/backend/extractor/rp/rp.nn.go:239 +0x67
main.(*Lexer).Error(0x2081c21e0, 0xe67f0, 0xc)
    <autogenerated>:3 +0xa5
main.yyParse(0x22081bc340, 0x2081c21e0, 0x170bc8)
    /Users/donnanicolas/go/src/tu-carrito.com/backend/extractor/rp/yaccpar:155 +0x563
main.main()
    /Users/donnanicolas/go/src/tu-carrito.com/backend/extractor/rp/rp.nn.go:263 +0x84

goroutine 19 [finalizer wait]:
runtime.park(0x17f90, 0x170b90, 0x16ff09)
    /usr/local/Cellar/go/1.3.3/libexec/src/pkg/runtime/proc.c:1369 +0x89
runtime.parkunlock(0x170b90, 0x16ff09)
    /usr/local/Cellar/go/1.3.3/libexec/src/pkg/runtime/proc.c:1385 +0x3b
runfinq()
    /usr/local/Cellar/go/1.3.3/libexec/src/pkg/runtime/mgc0.c:2644 +0xcf
runtime.goexit()
    /usr/local/Cellar/go/1.3.3/libexec/src/pkg/runtime/proc.c:1445

goroutine 20 [chan send]:
main.func·003(0x2081ae120, 0x2081ae0c0, 0x2081ea000, 0x3, 0x3, 0x0, 0x1)
    /Users/donnanicolas/go/src/tu-carrito.com/backend/extractor/rp/rp.nn.go:136 +0x9c8
created by main.NewLexerWithInit
    /Users/donnanicolas/go/src/tu-carrito.com/backend/extractor/rp/rp.nn.go:193 +0x677

I have the following regex:

/[a-z]([a-z0-9:]*[a-z0-9]+)?/

This is intended to match:
g
or
hey42
or
hey:there
but NOT:
blah:
trailing colons should not be allowed. Yet it does.

I need more extensive testing, but AFAIK this is pretty serious matching bug!

I also tested

/[a-z]+([a-z0-9:]*[a-z0-9]+)?/

Which has the same problem :/

Digging deeper... Help welcome =D

It seems that running nex multiple times on the same file generates different output sometimes. Example:

/a|A/ { return A }
//
package main

I get two different versions (filenames are their shas):

→ diff 55b29ae5b5679cab92281c8e9d3d493cb0f5be6c.go 73edf909a04d60e4f253d456fc4bf8b3ca6eee9c.go
155,156c155,156
<       case 65: return 1
<       case 97: return 2

---
>       case 65: return 2
>       case 97: return 1

I haven't looked at your code yet, but a possible source of randomness could be go's map iteration order (c.f. http://stackoverflow.com/a/9621526/220918).

I'm having a hard time understanding the behavior of the lexer in the following case:

/\(/   { fmt.Printf("-> %q\n", yylex.Text()) }
/\)/   { fmt.Printf("-> %q\n", yylex.Text()) }
/[^( ][^ ]*[^ )]/ { fmt.Printf("-> %q\n", yylex.Text()) }
//

package main
import ("fmt")
func main() {
  fmt.Printf("lexing %q:\n", "(rule)")
    NN_FUN(NewLexer(strings.NewReader("(rule)")))
  fmt.Printf("lexing %q:\n", "( rule  )")
    NN_FUN(NewLexer(strings.NewReader("( rule )")))
}

Output of nex -r -s huh.nex:

lexing "(rule)":
-> "("
-> "rule"
lexing "( rule  )":
-> "("
-> "rule"
-> ")"

Why is the lexer swallowing the trailing bracket when there is no space between the content and the surrounding brackets?. This looks like a bug to me. It has something to do with second character class in the content regex ([^ ]*), when I change that to also not match ) it works.

Is there a way to define case insensitive regular expressions? I am making a SQL parser and I need to be able to parse correctly SELECT, select or any of its variants.

I do know that Flex has %option caseless, is there something similar for Nex?

It would be great if nex had an argument just like yacc's -p to change yy to other things. As it is, I seem to have to write a bunch of wrapper code to make things line up.

I wrote a complex regex and run nex but found an 'out-of-memory' error.
My vps has 4G memory so I doubt somewhere may went wrong.
I tried to find a simpler regex which can cause the same problem, finally, I found one like below:

/(([ \t]*)*)*/ { /* eat up whitespace */ }

It is sensed that nex may hang into inf-loop in cases which has nested klein-closures.
the crash is sth like below:

fatal error: runtime: out of memory

runtime stack:
runtime.throw(0x546c99, 0x16)
/usr/local/go/src/runtime/panic.go:566 +0x95
runtime.sysMap(0xc440200000, 0x20000000, 0xc4401a3e00, 0x5ff718)
/usr/local/go/src/runtime/mem_linux.go:219 +0x1d0
runtime.(*mheap).sysAlloc(0x5e6ec0, 0x20000000, 0x0)
/usr/local/go/src/runtime/malloc.go:407 +0x37a
runtime.(*mheap).grow(0x5e6ec0, 0x10000, 0x0)
/usr/local/go/src/runtime/mheap.go:726 +0x62
runtime.(*mheap).allocSpanLocked(0x5e6ec0, 0x10000, 0x0)
/usr/local/go/src/runtime/mheap.go:630 +0x4f2
runtime.(*mheap).allocStack(0x5e6ec0, 0x10000, 0x0)
/usr/local/go/src/runtime/mheap.go:597 +0x62
runtime.stackalloc(0xc420000000, 0xc4301a3ee0, 0x8000000, 0xc4200001a0, 0x0, 0x0)
/usr/local/go/src/runtime/stack.go:395 +0x2ed
runtime.copystack(0xc4200001a0, 0x20000000, 0x5e3d01)
/usr/local/go/src/runtime/stack.go:839 +0x83
runtime.newstack()
/usr/local/go/src/runtime/stack.go:1070 +0x370
runtime.morestack()
/usr/local/go/src/runtime/asm_amd64.s:366 +0x7f

goroutine 1 [copystack]:
main.gen.func17.1(0x2)
/home/kongdeyu/goprojs/src/github.com/blynn/nex.go:522 fp=0xc4301a42b8 sp=0xc4301a42b0
main.gen.func17.1(0x1)
/home/kongdeyu/goprojs/src/github.com/blynn/nex.go:527 +0xf8 fp=0xc4301a4308 sp=0xc4301a42b8
main.gen.func17.1(0x2)
/home/kongdeyu/goprojs/src/github.com/blynn/nex.go:527 +0xf8 fp=0xc4301a4358 sp=0xc4301a4308
main.gen.func17.1(0x1)
/home/kongdeyu/goprojs/src/github.com/blynn/nex.go:527 +0xf8 fp=0xc4301a43a8 sp=0xc4301a4358
main.gen.func17.1(0x2)
/home/kongdeyu/goprojs/src/github.com/blynn/nex.go:527 +0xf8 fp=0xc4301a43f8 sp=0xc4301a43a8
main.gen.func17.1(0x1)

Currently, if you try to put dash (-) inside a character group (at the beginning or at the end of it) nex will panic: bad range in character class:

/[-a-z]/ {
}
//
package main
import ("fmt";"os")
func main() {
  NN_FUN(NewLexer(os.Stdin))
}

I wish to be able to use dashes like that, because else regexp becomes cumbersome if your identifiers can contain a dash.

Hi,

After typing in line count example in readme, I redirected it's output to a file and issued "go build" on it. And I got "undefine os" error for NewLexer(os.Stdin).

Readme says generated code already imports os, but mine (1eadfa3) didn't put it. I am on windows 7.

By using nex to parse SQL, I encounter the following problem:
Say two statements "BETWEEN expr AND expr" and "IF( expr AND expr)", there are two "AND" token that are different. In flex we can write rule like this:

%s BTWMODE
AND { BEGIN INITIAL; return AND; }
AND { return ANDOP; }
BETWEEN { BEGIN BTWMODE; return BETWEEN; }

%s represents an inclusive-mode lexing.(%x exclusive-mode correspondingly)

What is the equivalent counterpart of it?
In addition, flex has an option "case-insensitive" to ignore case, seems nex has no such switch, so I have to write rules like /[Ss][Ee][Ll][Ee][Cc][Tt]/ to represent the "select" keyword. Is there a recommend way to achieve this?

hey there,

playing with nex, and using the below... The problem is I had to add an ERROR identifier to match anything not previously caught since otherwise random typos in the lexed code are skipped silently. Any advice on how to improve this?

Thanks!

/[ \t\n]/	{ /* Skip blanks and tabs. */ }
/{/		{ return OPEN_CURLY }
/}/		{ return CLOSE_CURLY }
/if/		{ return IF }
/else/		{ return ELSE }
/=/		{ return EQUALS }
/\$[a-z]+/	{
			s := yylex.Text()
			lval.str = s[1:len(s)] // remove the leading $
			return VAR_IDENTIFIER
		}
/[a-z]+/	{
			lval.str = yylex.Text()
			return IDENTIFIER
		}
/./		{ return ERROR }
//
package main
import ()

I previously couldn't figure out how to match single line comments. Now solved, so I'm opening and closing this issue so it will get seen as reference for future users.

Match the single line comment:

/#[^\n]*/
	{	// this matches a (#) pound char followed by any
		// number of chars that aren't the (\n) newline!
		s := yylex.Text()
		lval.str = s[1:len(s)] // remove the leading #
		log.Printf("lang: lexer: comment: `%s`", lval.str)
		//return COMMENT // skip return to avoid parsing
	}

The trick was the ^ negates the newline char, which matches all the junk you want, up until the newline. In the source this is the negate property.

Sadly it seems as if all the regexp code was copy+pasted incompletely from the golang lib, and as a result, a lot of standard regexp things are missing such as :alpha: for example.

Hope this hidden tutorial helped you!

There is currently no possibility I could find to get the shortest possible match (non-greedy behaviour).

There should be a possibility to split the following snippet:

<?php
  b
?>
text
<?php
  a
?>

to these 2 matches:

<?php
  b
?>

and

<?php
  a
?>

Currently the regex /<\?php.*\?>/ matches the whole text.

Or did I simply miss something?
Thanks

Writing a little programming language, and trying to lex a "string":

Eg:

$foo = "hello, world"

This is kind of tricky, in particular in the docs:

**Matching Nuances

Among rules in the same scope, the longest matching pattern takes precedence. In event of a tie, the first pattern wins.**

This means that if I have other similar patterns in the code, this will match when I don't expect it. Additionally, is there some way to specify a string? I don't want to write in every unicode char, and if you do a \*\ match it matches too much!!

I haven't done super serious profiling yet, but in a rather large application we run, our lexer keeps coming up in goroutine dumps at what seems to be a disproportionately high rate, especially given that we lex very small strings as a tiny part of everything our app does. I found a bunch of routines awaiting chan receive in the generated lexer, which kind of immediately raised my eyebrow. Given what the lexer does, does it really need to use channels and goroutines to accomplish its job? I'm not familiar with the design constraints or exactly what kind of guarantees we want to make for the user code that gets embedded in the generator, but it seems like this could all be done more efficiently single-threaded.

Thoughts?

Quite hilariously, an unpaired curly brace inside nex golang code causes lex/parse errors with nex itself!

Example:

/\$[a-z][a-z0-9]*{[0-9]+}/
		{
			yylex.pos(lval) // our pos
			s := yylex.Text()
			a := strings.Split(s, "{") // XXX: close match here: }
			lval.str = a[0]
			return IDENTIFIER
		}

Note the comment I added with a close brace. I added that as a workaround so that this works. Remove it and you'll see nex errors:

panic: unmatched '{'

goroutine 16 [running]:
runtime.gopanic
	../../../libgo/go/runtime/panic.go:493
main.$nested34
	/builddir/build/BUILD/nex-5344f151fd3251726650dffd30a531d3f1bddc17/nex.go:1027
main.$nested35
	/builddir/build/BUILD/nex-5344f151fd3251726650dffd30a531d3f1bddc17/nex.go:1094
main.process
	/builddir/build/BUILD/nex-5344f151fd3251726650dffd30a531d3f1bddc17/nex.go:1099
main.main
	/builddir/build/BUILD/nex-5344f151fd3251726650dffd30a531d3f1bddc17/main.go:81
runtime_main
	../../../libgo/runtime/proc.c:606

HTH

10 type yySymType struct {
11 yys int n int } <---- won't compile

Currently ^ and $ symbols convey no special meaning and match literally. I want to be able to use these special symbols inside regular expressions, including nested ones. In the latter case I want them to match at the beginning and at the end of the substring, matched by an outer regular expression.

The following code

/[a-z]+: [a-z]+/ <
  { fmt.Println("BEGIN"); }
  /[a-z]+:/ {
    fmt.Println(1, yylex.Text())
  }
  /[a-z]+/ {
    fmt.Println(2, yylex.Text())
  }
> { fmt.Println("END"); }
//
package main
import ("fmt";"os")
func main() {
  NN_FUN(NewLexer(os.Stdin))
}

(when processed with nex and executed as echo name: value | ./testcase.nn) prints

BEGIN
1 name:
2 value
END

...as you would expect. However, if you'll change the second nested expression to /.+/ --

/[a-z]+: [a-z]+/ <
  { fmt.Println("BEGIN"); }
  /[a-z]+:/ {
    fmt.Println(1, yylex.Text())
  }
  /.+/ {
    fmt.Println(2, yylex.Text())
  }
> { fmt.Println("END"); }
//
package main
import ("fmt";"os")
func main() {
  NN_FUN(NewLexer(os.Stdin))
}

it will print only

BEGIN
2 name: value
END

That is, in the last case the first nested expression is never matched.

what if I don't want to panic(err) when there is a parsing error?

for example flex supports something like this:

<<EOF>>

I need a way to return an EOF token to the parser.

So I was trying to migrate a BISON/FLEX basic parser written in "C" to GO. Settled on goyacc and nex. But was seeing bizarre behavior, where lines were not being processed until I entered a newline again. To eliminate questions about my grammar, I tried building the rp test program, and saw the same behavior. e.g. I enter '111 111 * 3 /', and nothing is printed until I enter another newline (or ctrl-d), at which point it prints the answer. I patched yyDebug to 3 to observe the grammar actions, and saw that it wasn't completing the parse, because the terminal newline was not being returned to the parser.

Output with yyDebug==3:

reduce 1 in:
state-0
111 111 * 3 /
lex NUM(57346)
reduce 5 in:
state-5
lex NUM(57346)
reduce 5 in:
state-5
lex '*'(42)
reduce 8 in:
state-11
lex NUM(57346)
reduce 5 in:
state-5
lex '/'(47)
reduce 9 in:
state-12

(notice how it hasn't printed anything yet?) Now I enter another newline:

lex '\n'(10)
reduce 4 in:
state-6
4107
reduce 2 in:
state-2

and it finishes up.

p.s. I cloned this straight out of github...

I'm guessing that Nex is not thread safe if it uses globals, am I wrong? My program needs to be able to parse in multiple threads and I'm wondering if I will need to lock a mutex for every time Lex will be called.

hi,

how can i generate a syntax error using nex and add more information the the syntax error?

i tried to create a costum error function for nex:
nex -e=true lexer.nex

here my nex file:
/while/ { return WHILE }
/print/ { return PRINT }
/;/ { return END_LINE }
/+|-/ { lval.s = yylex.Text(); return ADD_OP }
/|// { lval.s = yylex.Text(); return MUL_OP }
/=/ { return ASSIGN }
/(/ { return BEGIN_EXPRESSION }
/)/ { return END_EXPRESSION }
/{/ { return BEGIN_BLOCK }
/}/ { return END_BLOCK }
/[0-9]+/ { lval.s = yylex.Text(); return NUMBER }
/[a-z][a-z0-9]/ { lval.s = yylex.Text(); return IDENTIFIER }
/\n+/ { lineno += len(yylex.Text()) }
//
package dsl
import "fmt"

var lineno int // idk if thats good to put it global?

func (yylex Lexer) Error(e string) {
yylex.p.err = fmt.Sprintf("Syntax error in line %d", lineno+1)
}

i can print the line number where the syntax error happend.
EDIT: there are those 2 variables in the type Lexer struct:
l, c int // line number and character position
are they incremented automatically and can i remove my lineno variable?

how can you add more information to the error message? i want to add the token/s that was not recognized in the parsing...

e.g.
how can i keep track of the last token that i sent to the parser, then on error just say Printf("error after %s on line %d" , humanReadable(lastToken,lastLine) ??

hi,

could the following hack be added by nex?
printf '/NEX_END_OF_LEXER_STRUCT/i\np *Tacky\n.\nw\nq\n' | ed -s tacky.nn.go

also the nex file expects a newline at the end of the file... why?

Given the nex file:

%{
package main

%}

%union {
    n int
}


%token NUMBER
%token ADD SUB MUL DIV ABS
%token EOL

%%

start:           term
        ;
term:           NUMBER
        |       ABS term { $$ = $2 >= 0? $2 : - $2; }
        ;
%%

Got:

# command-line-arguments
desk.y:33[/Users/drew/src/snazzle/desk/y.go:468]: syntax error: unexpected ?

Expected:
$2 is returned as positive number