chriswailes / rltk Goto Github PK

View Code? Open in Web Editor NEW

452.0 452.0 36.0 1.69 MB

The Ruby Language Toolkit

Home Page: http://chriswailes.github.io/RLTK/

License: University of Illinois/NCSA Open Source License

Ruby 100.00%

rltk's People

Contributors

Stargazers

Watchers

rltk's Issues

Calling #type on RLTK::CG::Value throws an error

Here's a snippet:

require "rltk"
require "rltk/cg"
require "rltk/cg/module"
require "rltk/cg/contractor"

@module = RLTK::CG::Module.new("Hello")

f = @module.functions.add("hello", RLTK::CG::VoidType, [RLTK::CG::NativeIntType])
f.params[0].type
# ArgumentError: wrong number of arguments (0 for 1..2)
#0: /Users/gunca/.rvm/gems/ruby-2.1.5/gems/rltk-3.0.0/lib/rltk/cg/type.rb:158:in `initialize'
#1: /Users/gunca/.rvm/gems/ruby-2.1.5/gems/rltk-3.0.0/lib/rltk/cg/type.rb:50:in `new'
#2: /Users/gunca/.rvm/gems/ruby-2.1.5/gems/rltk-3.0.0/lib/rltk/cg/type.rb:50:in `from_ptr'
#3: /Users/gunca/.rvm/gems/ruby-2.1.5/gems/rltk-3.0.0/lib/rltk/cg/value.rb:125:in `type'

The exact version of LLVM from brew I'm using is this:
llvm34: stable 3.4.2 (bottled), HEAD

A more complete example for error production

I am trying to use the error production feature for handling typos in a really simple context-free grammar. Could you please give a more advanced example?

Requires Ruby >= 2.4.0 due to Filigree

With the new dependency on filigree, gem will now choke trying to install on Ruby versions lower than 2.4.0:

steven@macbook:~/workspace/apps/ios $ gem install rltk
Fetching: ffi-1.9.25.gem (100%)
Building native extensions.  This could take a while...
Successfully installed ffi-1.9.25
Fetching: filigree-0.4.1.gem (100%)
ERROR:  Error installing rltk:
	filigree requires Ruby version >= 2.4.0.```

JITCompiler segfaults on OSX

It seems like it doesn't matter what I do, the result's always the same. Every time I try to call run_function, it blows up.

require "rltk"
require "rltk/cg"
require "rltk/cg/module"
require "rltk/cg/contractor"
require "rltk/cg/execution_engine"

# ENV["DYLD_FALLBACK_LIBRARY_PATH"] = "/usr/local/lib/llvm-3.4/lib/"
RLTK::CG::LLVM.init(:X86)
@module = RLTK::CG::Module.new("Hello")
@engine = RLTK::CG::JITCompiler.new(@module)

f = @module.functions.add("hello", RLTK::CG::VoidType, [])
f.blocks.append('entry') do
  ret_void
end

f.verify!

@engine.run_function(f, [])

/Users/gunca/.rvm/gems/ruby-2.1.5/gems/rltk-3.0.0/lib/rltk/cg/execution_engine.rb:104: [BUG] Segmentation fault at 0x00000000000000
ruby 2.1.5p273 (2014-11-13 revision 48405) [x86_64-darwin14.0]

-- Crash Report log information --------------------------------------------
   See Crash Report log file under the one of following:
     * ~/Library/Logs/CrashReporter
     * /Library/Logs/CrashReporter
     * ~/Library/Logs/DiagnosticReports
     * /Library/Logs/DiagnosticReports
   for more details.

-- Control frame information -----------------------------------------------
c:0004 p:---- s:0020 e:000019 CFUNC  :run_function
c:0003 p:0075 s:0013 e:000011 METHOD /Users/gunca/.rvm/gems/ruby-2.1.5/gems/rltk-3.0.0/lib/rltk/cg/execution_engine.rb:104
c:0002 p:0156 s:0005 E:000b60 EVAL   binding_test.rb:26 [FINISH]
c:0001 p:0000 s:0002 E:000868 TOP    [FINISH]

binding_test.rb:26:in `<main>'
/Users/gunca/.rvm/gems/ruby-2.1.5/gems/rltk-3.0.0/lib/rltk/cg/execution_engine.rb:104:in `run_function'
/Users/gunca/.rvm/gems/ruby-2.1.5/gems/rltk-3.0.0/lib/rltk/cg/execution_engine.rb:104:in `run_function'

-- C level backtrace information -------------------------------------------
0   ruby                                0x00000001036a6e50 rb_vm_bugreport + 144
1   ruby                                0x00000001035589b1 report_bug + 305
2   ruby                                0x0000000103558874 rb_bug + 180
3   ruby                                0x00000001036291bc sigsegv + 156
4   libsystem_platform.dylib            0x00007fff8f1f3f1a _sigtramp + 26
5   libLLVM-3.4.dylib                   0x000000010429534a LLVMRunFunction + 96
6   ???                                 0x00007fc92b226560 0x0 + 140501988828512

-- Other runtime information -----------------------------------------------

* Loaded script: binding_test.rb

* Loaded features:

    0 enumerator.so
    1 enc/encdb.so
    2 enc/trans/transdb.so
    3 /Users/gunca/.rvm/rubies/ruby-2.1.5/lib/ruby/2.1.0/x86_64-darwin14.0/rbconfig.rb
    4 thread.rb
    5 thread.so
    6 /Users/gunca/.rvm/rubies/ruby-2.1.5/lib/ruby/site_ruby/2.1.0/rubygems/compatibility.rb
    7 /Users/gunca/.rvm/rubies/ruby-2.1.5/lib/ruby/site_ruby/2.1.0/rubygems/defaults.rb
    8 /Users/gunca/.rvm/rubies/ruby-2.1.5/lib/ruby/site_ruby/2.1.0/rubygems/deprecate.rb
    9 /Users/gunca/.rvm/rubies/ruby-2.1.5/lib/ruby/site_ruby/2.1.0/rubygems/errors.rb
   10 /Users/gunca/.rvm/rubies/ruby-2.1.5/lib/ruby/site_ruby/2.1.0/rubygems/version.rb
   11 /Users/gunca/.rvm/rubies/ruby-2.1.5/lib/ruby/site_ruby/2.1.0/rubygems/requirement.rb
   12 /Users/gunca/.rvm/rubies/ruby-2.1.5/lib/ruby/site_ruby/2.1.0/rubygems/platform.rb
   13 /Users/gunca/.rvm/rubies/ruby-2.1.5/lib/ruby/site_ruby/2.1.0/rubygems/basic_specification.rb
   14 /Users/gunca/.rvm/rubies/ruby-2.1.5/lib/ruby/site_ruby/2.1.0/rubygems/stub_specification.rb
   15 /Users/gunca/.rvm/rubies/ruby-2.1.5/lib/ruby/site_ruby/2.1.0/rubygems/util/stringio.rb
   16 /Users/gunca/.rvm/rubies/ruby-2.1.5/lib/ruby/site_ruby/2.1.0/rubygems/specification.rb
   17 /Users/gunca/.rvm/rubies/ruby-2.1.5/lib/ruby/site_ruby/2.1.0/rubygems/exceptions.rb
   18 /Users/gunca/.rvm/rubies/ruby-2.1.5/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_gem.rb
   19 /Users/gunca/.rvm/rubies/ruby-2.1.5/lib/ruby/2.1.0/monitor.rb
   20 /Users/gunca/.rvm/rubies/ruby-2.1.5/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb
   21 /Users/gunca/.rvm/rubies/ruby-2.1.5/lib/ruby/site_ruby/2.1.0/rubygems.rb
   22 /Users/gunca/.rvm/rubies/ruby-2.1.5/lib/ruby/site_ruby/2.1.0/rubygems/path_support.rb
   23 /Users/gunca/.rvm/rubies/ruby-2.1.5/lib/ruby/site_ruby/2.1.0/rubygems/dependency.rb
   24 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/rltk-3.0.0/lib/rltk.rb
   25 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/rltk-3.0.0/lib/rltk/cg.rb
   26 /Users/gunca/.rvm/gems/ruby-2.1.5/extensions/x86_64-darwin-14/2.1.0-static/ffi-1.9.8/ffi_c.bundle
   27 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/ffi-1.9.8/lib/ffi/platform.rb
   28 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/ffi-1.9.8/lib/ffi/types.rb
   29 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/ffi-1.9.8/lib/ffi/library.rb
   30 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/ffi-1.9.8/lib/ffi/errno.rb
   31 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/ffi-1.9.8/lib/ffi/pointer.rb
   32 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/ffi-1.9.8/lib/ffi/memorypointer.rb
   33 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/ffi-1.9.8/lib/ffi/struct_layout_builder.rb
   34 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/ffi-1.9.8/lib/ffi/struct.rb
   35 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/ffi-1.9.8/lib/ffi/union.rb
   36 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/ffi-1.9.8/lib/ffi/managedstruct.rb
   37 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/ffi-1.9.8/lib/ffi/callback.rb
   38 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/ffi-1.9.8/lib/ffi/io.rb
   39 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/ffi-1.9.8/lib/ffi/autopointer.rb
   40 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/ffi-1.9.8/lib/ffi/variadic.rb
   41 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/ffi-1.9.8/lib/ffi/enum.rb
   42 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/ffi-1.9.8/lib/ffi/ffi.rb
   43 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/ffi-1.9.8/lib/ffi.rb
   44 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/filigree-0.3.2/lib/filigree/boolean.rb
   45 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/rltk-3.0.0/lib/rltk/version.rb
   46 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/rltk-3.0.0/lib/rltk/cg/generated_bindings.rb
   47 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/rltk-3.0.0/lib/rltk/cg/bindings.rb
   48 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/rltk-3.0.0/lib/rltk/cg/context.rb
   49 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/rltk-3.0.0/lib/rltk/cg/module.rb
   50 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/filigree-0.3.2/lib/filigree/class_methods_module.rb
   51 /Users/gunca/.rvm/rubies/ruby-2.1.5/lib/ruby/2.1.0/ostruct.rb
   52 /Users/gunca/.rvm/rubies/ruby-2.1.5/lib/ruby/2.1.0/singleton.rb
   53 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/filigree-0.3.2/lib/filigree/abstract_class.rb
   54 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/filigree-0.3.2/lib/filigree/types.rb
   55 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/filigree-0.3.2/lib/filigree/class.rb
   56 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/filigree-0.3.2/lib/filigree/match.rb
   57 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/filigree-0.3.2/lib/filigree/visitor.rb
   58 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/rltk-3.0.0/lib/rltk/cg/type.rb
   59 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/rltk-3.0.0/lib/rltk/cg/value.rb
   60 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/rltk-3.0.0/lib/rltk/cg/instruction.rb
   61 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/rltk-3.0.0/lib/rltk/cg/builder.rb
   62 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/rltk-3.0.0/lib/rltk/cg/contractor.rb
   63 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/rltk-3.0.0/lib/rltk/cg/pass_manager.rb
   64 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/rltk-3.0.0/lib/rltk/cg/triple.rb
   65 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/rltk-3.0.0/lib/rltk/cg/target.rb
   66 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/rltk-3.0.0/lib/rltk/cg/execution_engine.rb
   67 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/rltk-3.0.0/lib/rltk/cg/llvm.rb
   68 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/rltk-3.0.0/lib/rltk/cg/basic_block.rb
   69 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/rltk-3.0.0/lib/rltk/cg/function.rb
   70 /Users/gunca/.rvm/gems/ruby-2.1.5/gems/rltk-3.0.0/lib/rltk/cg/generic_value.rb

[NOTE]
You may have encountered a bug in the Ruby interpreter or extension libraries.
Bug reports are welcome.
Don't forget to include the above Crash Report log file.
For details: http://www.ruby-lang.org/bugreport.html

Abort trap: 6

Cannot Generate YARD documentation

Hi, the generation of the yardoc produce this error:

[debug]: Parsing ["{lib,app}/**/*.rb", "ext/**/*.{c,cc,cxx,cpp,rb}"] with `ruby` parser
[debug]: Parsing lib/rltk.rb
[debug]: Parsing lib/rltk/cg.rb
[debug]: Parsing lib/rltk/ast.rb
[debug]: Parsing lib/rltk/cfg.rb
[debug]: Parsing lib/rltk/lexer.rb
[debug]: Parsing lib/rltk/token.rb
[debug]: Parsing lib/rltk/parser.rb
Traceback (most recent call last):
        25: from /home/didacus/.gem/ruby/2.7.0/bin/yardoc:23:in `<main>'
        24: from /home/didacus/.gem/ruby/2.7.0/bin/yardoc:23:in `load'
        23: from /home/didacus/.gem/ruby/2.7.0/gems/yard-0.9.24/bin/yardoc:13:in `<top (required)>'
        22: from /home/didacus/.gem/ruby/2.7.0/gems/yard-0.9.24/lib/yard/cli/command.rb:14:in `run'
        21: from /home/didacus/.gem/ruby/2.7.0/gems/yard-0.9.24/lib/yard/cli/yardoc.rb:258:in `run'
        20: from /home/didacus/.gem/ruby/2.7.0/gems/yard-0.9.24/lib/yard/registry.rb:210:in `lock_for_writing'
        19: from /home/didacus/.gem/ruby/2.7.0/gems/yard-0.9.24/lib/yard/registry_store.rb:202:in `lock_for_writing'
        18: from /home/didacus/.gem/ruby/2.7.0/gems/yard-0.9.24/lib/yard/serializers/yardoc_serializer.rb:56:in `lock_for_writing'
        17: from /home/didacus/.gem/ruby/2.7.0/gems/yard-0.9.24/lib/yard/cli/yardoc.rb:259:in `block in run'
        16: from /home/didacus/.gem/ruby/2.7.0/gems/yard-0.9.24/lib/yard.rb:20:in `parse'
        15: from /home/didacus/.gem/ruby/2.7.0/gems/yard-0.9.24/lib/yard/parser/source_parser.rb:113:in `parse'
        14: from /home/didacus/.gem/ruby/2.7.0/gems/yard-0.9.24/lib/yard/logging.rb:182:in `enter_level'
        13: from /home/didacus/.gem/ruby/2.7.0/gems/yard-0.9.24/lib/yard/parser/source_parser.rb:114:in `block in parse'
        12: from /home/didacus/.gem/ruby/2.7.0/gems/yard-0.9.24/lib/yard/parser/source_parser.rb:371:in `parse_in_order'
        11: from /home/didacus/.gem/ruby/2.7.0/gems/yard-0.9.24/lib/yard/parser/source_parser.rb:45:in `parse'
        10: from /home/didacus/.gem/ruby/2.7.0/gems/yard-0.9.24/lib/yard/logging.rb:82:in `capture'
         9: from /home/didacus/.gem/ruby/2.7.0/gems/yard-0.9.24/lib/yard/parser/source_parser.rb:46:in `block in parse'
         8: from /home/didacus/.gem/ruby/2.7.0/gems/yard-0.9.24/lib/yard/parser/source_parser.rb:442:in `parse'
         7: from /home/didacus/.gem/ruby/2.7.0/gems/yard-0.9.24/lib/yard/parser/ruby/ruby_parser.rb:17:in `parse'
         6: from /home/didacus/.gem/ruby/2.7.0/gems/yard-0.9.24/lib/yard/parser/ruby/ruby_parser.rb:56:in `parse'
         5: from /home/didacus/.gem/ruby/2.7.0/gems/yard-0.9.24/lib/yard/parser/ruby/ruby_parser.rb:56:in `parse'
         4: from /home/didacus/.gem/ruby/2.7.0/gems/yard-0.9.24/lib/yard/parser/ruby/ruby_parser.rb:165:in `on_def'
         3: from /home/didacus/.gem/ruby/2.7.0/gems/yard-0.9.24/lib/yard/parser/ruby/ruby_parser.rb:238:in `visit_event'
         2: from /home/didacus/.gem/ruby/2.7.0/gems/yard-0.9.24/lib/yard/parser/ruby/ast_node.rb:64:in `source_range'
         1: from /home/didacus/.gem/ruby/2.7.0/gems/yard-0.9.24/lib/yard/parser/ruby/ast_node.rb:348:in `reset_line_info'
/home/didacus/.gem/ruby/2.7.0/gems/yard-0.9.24/lib/yard/parser/ruby/ast_node.rb:348:in `first': cannot get the first element of beginless range (RangeError)

Is it a problem of rltk or yard ?
By the way, this is an amazing project. It's fun creating programming languages with this library. Thank you!

Can't generate parser using Rubinius

When attempting to run the test suite against Rubinius I get the following backtrace:

An exception occurred running /Users/jnh/.rvm/rubies/rbx-head-rbx19/bin/testrb
    undefined method `id' on nil:NilClass. (NoMethodError)

Backtrace:
                                         Kernel(NilClass)#id (method_missing) at kernel/delta/kernel.rb:81
                                            { } in RLTK::Parser.grammar_prime at lib/rltk/parser.rb:689
                                                                   Array#each at kernel/bootstrap/array.rb:68
                                            { } in RLTK::Parser.grammar_prime at lib/rltk/parser.rb:686
                                                                   Array#each at kernel/bootstrap/array.rb:68
                                            { } in RLTK::Parser.grammar_prime at lib/rltk/parser.rb:681
                                              { } in RLTK::Parser::State#each at lib/rltk/parser.rb:1505
                                                                   Array#each at kernel/bootstrap/array.rb:68
                                                     RLTK::Parser::State#each at lib/rltk/parser.rb:1505
                                            { } in RLTK::Parser.grammar_prime at lib/rltk/parser.rb:676
                                                                   Array#each at kernel/bootstrap/array.rb:68
                                                   RLTK::Parser.grammar_prime at lib/rltk/parser.rb:675
                                                    { } in RLTK::Parser.prune at lib/rltk/parser.rb:1076
                                                                   Array#each at kernel/bootstrap/array.rb:68
                                                    { } in RLTK::Parser.prune at lib/rltk/parser.rb:1074
                                                                   Array#each at kernel/bootstrap/array.rb:68
                                                    { } in RLTK::Parser.prune at lib/rltk/parser.rb:1068
                                                                   Array#each at kernel/bootstrap/array.rb:68
                                                           RLTK::Parser.prune at lib/rltk/parser.rb:1058
                                                        RLTK::Parser.finalize at lib/rltk/parser.rb:615
             ParserTester::EmptyListParser0.__class_init__ (EmptyListParser0) at test/tc_parser.rb:111
                                   ParserTester.__class_init__ (ParserTester) at test/tc_parser.rb:108
                                                            Object#__script__ at test/tc_parser.rb:26
                                                 Rubinius::CodeLoader.require at kernel/common/codeloader.rb:212
                                                       Kernel(Object)#require at kernel/common/kernel.rb:649
                                                            Object#__script__ at test/ts_rltk.rb:34
                                                 Rubinius::CodeLoader.require at kernel/common/codeloader.rb:212
                               Kernel(Test::Unit::AutoRunner::Runner)#require at kernel/common/kernel.rb:649
  { } in Test::Unit::RequireFiles(Test::Unit::AutoRunner::Runner)#non_options at /Users/jnh/.rvm/rubies/rbx-head-rbx19/lib/19/test/unit.rb:221
                                                                   Array#each at kernel/bootstrap/array.rb:68
         Test::Unit::RequireFiles(Test::Unit::AutoRunner::Runner)#non_options at /Users/jnh/.rvm/rubies/rbx-head-rbx19/lib/19/test/unit.rb:215
             Test::Unit::Options(Test::Unit::AutoRunner::Runner)#process_args at /Users/jnh/.rvm/rubies/rbx-head-rbx19/lib/19/test/unit.rb:52
                                          Test::Unit::AutoRunner#process_args at /Users/jnh/.rvm/rubies/rbx-head-rbx19/lib/19/test/unit.rb:625
                                                            Object#__script__ at /Users/jnh/.rvm/rubies/rbx-head-rbx19/bin/testrb:5
                                             Rubinius::CodeLoader#load_script at kernel/delta/codeloader.rb:68
                                             Rubinius::CodeLoader.load_script at kernel/delta/codeloader.rb:118
                                                      Rubinius::Loader#script at kernel/loader.rb:615
                                                        Rubinius::Loader#main at kernel/loader.rb:816
Coverage report generated for Unit Tests to /Users/jnh/Dev/forks/RLTK/coverage. 699 / 763 LOC (91.61%) covered.
rake aborted!
Command failed with status (1): [/Users/jnh/.rvm/rubies/rbx-head-rbx19/bin/...]

Tasks: TOP => test
(See full trace by running task with --trace)

The backtrace points at lib/rltk/parser.rb:689 as being the culprit:

      def grammar_prime
        if not @grammar_prime
          @grammar_prime = CFG.new

          @states.each do |state|
            state.each do |item|
              lhs = "#{state.id}_#{item.next_symbol}".to_sym

              next unless CFG::is_nonterminal?(item.next_symbol) and not @grammar_prime.productions.keys.include?(lhs)

              @grammar.productions[item.next_symbol].each do |production|
                rhs = ""

                cstate = state

                production.rhs.each do |symbol|
                  rhs += "#{cstate.id}_#{symbol} "

                  cstate = @states[cstate.on?(symbol).first.id]  # <-- here
                end

                @grammar_prime.production(lhs, rhs)
              end
            end
          end
        end

        @grammar_prime
      end

The parser code is too complex for me to figure out at first blush, so I thought I'd raise an issue in the hopes that you had any ideas/pointers on where I should look?

bug in evaluating right hand sides

There seems to be a problem with regard to nondeterminism in the productions results:
parsing AB under S-> (A B? B?)* results in duplicated output. Never mind the global variable, its just to test that the output is indeed generated twice, and not just duplicated.

https://gist.github.com/ranmrdrakono/dfbbe4dbfae599f5b8c7

The output should be (in my opinion) [:body, [[:a, 1]]] but acctually is [:body, [[:a, 1], [:a, 2]]]

"String not in language" error

We have a JRuby-on-Rails application that interprets TWS definitions using RLTK that then inserts them into database tables. The application previously was using this version and has now been updated to 2.2.1.

We initially saw an error saying NameError exception occurred: undefined local variable or method 'scanner' for #<RLTK::Lexer::Environment:0x2838abe5>, which seemed to be related to these lines in lib/tws_parser/parser.rb:

    # Punctuation
    rule(/#/)                         {
      # A hash indicates a comment-line iff it is the first character on the line
      scanner.pos -= 1
      token = scanner.bol? ? nil : :HASH
      if token
        scanner.pos += 1
      else
        scanner.scan_until(/\n/)
      end
      token
    }

After replacing these lines we now receive a new error:

18-Aug-2020 21:02:17.190 INFO [http-nio-7100-exec-10] org.apache.catalina.core.ApplicationContext.log Parsing schedules...
18-Aug-2020 21:02:17.198 INFO [http-nio-7100-exec-10] org.apache.catalina.core.ApplicationContext.log RLTK::NotInLanguage exception occurred: String not in language.
18-Aug-2020 21:02:17.200 INFO [http-nio-7100-exec-10] org.apache.catalina.core.ApplicationContext.log lib/tws_parser/builder.rb:72:in `process'
    app/controllers/tivoli_uploads_controller.rb:37:in `create'
   app/controllers/tivoli_uploads_controller.rb:19:in `create'

I believe the issue is in the same file as before, here is the contents of lib/tws_parser/parser.rb, lines 103 to 129:

      # A hash and then a space indicates a comment-line
      rule(/^#/)           { push_state :comment }
      rule(/\n/, :comment) { pop_state }
      rule(/./, :comment)
     
      # Punctuation
    rule(/,/)                         { :COMMA }
    rule(/:/)                         { :COLON }
    rule(/\./)                        { :DOT }
    rule(/@/)                         { :AT_SYM }
    rule(/\+/)                        { :PLUS }
    rule(/\-/)                        { :MINUS }
 
    # Text
    rule(/"(\\"|[^"])*"/) do |t|
      str = t[1..-2].gsub('\"', '"')
      str.gsub!(/%([^%]+)%/) do |m|
        (@mappings && @mappings[$1]) || m
      end
      [:STRING, str]
    end
    rule(/\([^\)]*\)/)                          { |t| [:QUALIFIER, t] }
    rule(/[A-Za-z%][A-Za-z0-9_\-\\%]*/) do |t|
      t =~ /^%(.+)%$/
      mapped = $1 && @mappings && @mappings[$1]
      [:NAME, mapped || t]
    end

I guess the issue is related to the block that assigns [:STRING, str] but I don't know what I am supposed to change it to - can you please help?

RLTK::Parser.finalize use: fails to load cache on Windows

Problem: Windows paths contain a colon

Solution: rltk-3.0.1\lib\rltk\parser.rb:564

Current:

def_file = caller()[2].split(':')[0] if opts[:use]

Should be (choose one):

def_file = caller()[2].split(':')[0..-3].join(':') if opts[:use]
def_file = caller()[2][/^(.*):[^:]+:[^:]+$/, 1] if opts[:use]

Fix EBNF/selector interplay bug

A new testcase has been added that checks the interplay between EBNF elemination and the production value selectors. It currently doesn't work and should be fixed.

Remove support for LLVM::JIT in anticipation of LLVM 3.6 release

LLVM 3.6 is removing support for the legacy JIT. The bindings need to be updated to use only the MCJIT.

Check gemspec into repository

If you could please check a gemspec into the repository then I will be able to use RLTK head from Bundler.

bug in empty_list?

I believe the documentation for RLTK::Parser.empty_list '(as well as nonempty_list) is either a little unspecific or the functions do not behave as expected. If one uses an expressions such as "COMMA NL?" for the seperator only the first list value will be returned. This behaviour is fixed when the expression is outsourced into an own rule.
See this example for clarification:

https://gist.github.com/ranmrdrakono/28307c9d5d4cf9b60f2d

Rejecting a clause in a production

Hi,
I am currently working on a natural language parser and I wonder whether it is possible to reject a clause during the parsing.
What I mean is that I defined a grammar but, as in the end all tokens are words, I need to be able to reject some constructions if the word is, say, a verb and the parser is currently in a clause needing a noun. Without that, I don't know how to specify the differences between each class of words.
Cheers,

Emmanuel

Update LLVM Bindings to LLVM 3.2

LLVM 3.2 was released a couple of days ago. RLTK's LLVM bindings should be updated.

Project Activity?

I'm curious about using RLTK for some experiments regarding converting an existing DSL into something which can be executed natively, as well as some general experimenting with compiler-writing, because compilers are awesome.

However, it appears that the last commit is close to six months old, and there are no apparent branches that have more recent work. Is RLTK abandoned/moved, or is it just reached a point of stability.

Thanks, RLTK looks like a great project, and I'm really hoping to use it, but I don't want to hang on something that isn't being maintained.

Underscores in tokens break parser (but not lexer).

I'm working on a ruby runtime for the Handlebars.js templating language. I ran into an issue where the parser erroneously raises a string not in language exception when tokens contain underscores.

Changing the following rule:

rule /{{/, :default do                                                                       
  push_state :expression
  [ :EXPRESSION_START ]
end

rule /{{/, :default do                                                                       
  push_state :expression
  [ :EXPRESSIONSTART ]
end

Allows the parser to succeed.

I have prepared examples of working and non-working lexers, parsers and specs for your enjoyment at jamesotron/FlavourSaver on the broken_parser and working_parser branches. In both branches if you run the specs you will find that the lexer output passes but the parser will change behaviour.

Thanks for RLTK, it's an awesome library!

require 'rltk/ast' pollute main Object with #match method, leaking it in a global scope

Hi!

First of all thanks for this tool!

I have noticed that the following line in the ASTNode

require 'filigree/match'

https://github.com/chriswailes/RLTK/blob/master/lib/rltk/ast.rb

Actually will include this in the main Object

def match(*objects, &block)
  me = Filigree::MatchEnvironment.new
  me.instance_exec &block
  me.find_match(objects)
end

https://github.com/chriswailes/filigree/blob/master/lib/filigree/match.rb#L157

It looks like RLTK doesn't use #match but only the module Filigree::Destructurable as described in this commit that introduced the inclusion: 3cf628c

I was wondering if we could avoid leaking the match method in a global scope.
I have noticed since including rltk/ast started to conflict with rspec

Thanks!

LLVM

RLTK targets an older version of the LLVM (3.4). Is there any future plans of upgrading to support newer versions of the LLVM?

capture groups in lexers

Hey there.

This is awesome. I'm wondering if there's a way to get access to the groups captured by a particular rule's regex, as in String#scan. For example:

class MyLexer < RLTK::Lexer
  rule(/\.(\w+)\[/) { [:THING, $1] }
end

Currently it looks like the block is being passed only the whole match. I'm happy to patch it to be more like the core ruby matching methods, but I wanted to check first :)

Split the LLVM bindings into their own project: RCGTK

The code generation library, used to interact with LLVM, will be moved to its own project. This will allow developers to upgrade RLTK and the LLVM bindings independently, easing the upgrade path for both RLTK and LLVM installations.

If this will cause any major issues for anyone please let me know.

Make a release?

Happy birthday to v3.0.1 - its been 5 years 🍰

On the serious side: Can we get a release?

Today I fought a bit with the BadToken error and was confused as the code in the repo didn't match the error received. #50 improved this error a bit. Although not a lot has happened, there are a few things in master that might be worth releasing.

Cheers

Ruby 1.9 Style

As Ruby 1.9 is starting to find its way into mainline distributions of Linux it doesn't seem to unreasonable to drop support for Ruby 1.8 syntax and code. Unless anyone who is currently using RLTK has any major objections I will be making some changes to the codebase that will prevent RLTK from running with Ruby versions < 1.9.

rltk (3.0.1) does not work with filigree (0.4.1) due to check_type signature

That gem version combination raises:

#<ArgumentError: wrong number of arguments (given 4, expected 2)>
	.../gems/ruby-2.5.1/gems/filigree-0.4.1/lib/filigree/types.rb:30:in `check_type'
	.../gems/ruby-2.5.1/gems/rltk-3.0.1/lib/rltk/ast.rb:169:in `block in define_accessor'
	.../gems/ruby-2.5.1/gems/rltk-3.0.1/lib/rltk/ast.rb:458:in `block in initialize'
	.../gems/ruby-2.5.1/gems/rltk-3.0.1/lib/rltk/ast.rb:457:in `each'
	.../gems/ruby-2.5.1/gems/rltk-3.0.1/lib/rltk/ast.rb:457:in `initialize'
	.../gems/ruby-2.5.1/gems/filigree-0.4.1/lib/filigree/abstract_class.rb:75:in `new'
	.../gems/ruby-2.5.1/gems/filigree-0.4.1/lib/filigree/abstract_class.rb:75:in `new'

while instantiating an RLTK::ASTNode derived class like this:

class Participant < RLTK::ASTNode
    value :id, String
    value :text, String
end

due to updated method signature of check_type from filigree gem.

check_type(obj, type, blame: nil, nillable: false, strict: false)

It is called from RLTK like so:

self.instance_variable_set(ivar_name, check_type(value, type, nil, true))

List of items

To parse list of items is so usual that I'd like some syntax sugar about it.

Kazoo example:

production(:arg_defs) do
  clause('')    { || [] }
  clause('arg_def_list')    { |args| args }
end

production(:arg_def_list) do
  clause('IDENT')   { |i| [i] }
  clause('IDENT COMMA arg_def_list')    { |i, _, defs| [i] + defs }
end

I know this is the LALR(1) way, but something like the following is shorter and easier to read:

production(:arg_defs, separator: 'COMMA', required: false) do
  clause    { |items| defs << items }
end

This way it would be optional, of course. Don't be afraid, LALR(1) purists!

Thank you again for RLTK!

Recommended LLVM installation/configuration

I've been working on a new programming language, Rip, for a while now. I am able to construct the AST. I didn't know about RLTK before I got started, so I'm just trying to use RLTK to generated LLVM IR. I've compiled LLVM 3.5 from source based on the official instructions. The binaries and libraries are in a non-standard location (/Users/thomas/Code/llvm/build/Debug+Asserts/{bin,lib}) though.

Running RLTK::CG::Module.new('foo_module') results in the following error. What is the preferred way to install LLVM so I can use RLTK?

/usr/local/opt/rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/ffi-1.9.3/lib/ffi/library.rb:133:in `block in ffi_lib': Could not open library 'LLVM-3.0': dlopen(LLVM-3.0, 5): image not found. (LoadError)
Could not open library 'libLLVM-3.0.dylib': dlopen(libLLVM-3.0.dylib, 5): image not found
    from /usr/local/opt/rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/ffi-1.9.3/lib/ffi/library.rb:100:in `map'
    from /usr/local/opt/rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/ffi-1.9.3/lib/ffi/library.rb:100:in `ffi_lib'
    from /usr/local/opt/rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/rltk-2.2.1/lib/rltk/cg/bindings.rb:46:in `<module:Bindings>'
    from /usr/local/opt/rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/rltk-2.2.1/lib/rltk/cg/bindings.rb:44:in `<module:CG>'
    from /usr/local/opt/rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/rltk-2.2.1/lib/rltk/cg/bindings.rb:23:in `<top (required)>'
    from /usr/local/opt/rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/rltk-2.2.1/lib/rltk/cg/module.rb:11:in `require'
    from /usr/local/opt/rbenv/versions/2.1.1/lib/ruby/gems/2.1.0/gems/rltk-2.2.1/lib/rltk/cg/module.rb:11:in `<top (required)>'
    from /Users/thomas/Code/rip-rip/lib/rip/nodes/module.rb:1:in `require'
    from /Users/thomas/Code/rip-rip/lib/rip/nodes/module.rb:1:in `<top (required)>'
    from /Users/thomas/Code/rip-rip/lib/rip/nodes.rb:28:in `require'
    from /Users/thomas/Code/rip-rip/lib/rip/nodes.rb:28:in `<top (required)>'
    from /Users/thomas/Code/rip-rip/lib/rip.rb:47:in `require'
    from /Users/thomas/Code/rip-rip/lib/rip.rb:47:in `<top (required)>'
    from ./bin/rip:3:in `require_relative'
    from ./bin/rip:3:in `<main>'

Wrong associativity of MUL and DIV in the example

In the example, both MUL and DIV are using right associativity, which should be left. It is misleading to some extent.

To illustrate right associativity, better to use exponentiation in the example.

Move to LLVM 3.6

Change the required version of RLTK to LLVM 3.6.

example of ASTNode child elements as arrays

Do you have any examples of child elements of an ASTNode that is an array?

the first problem I ran into was the value being initialized to nil which caused the initialization to error out, I started initializing the array children with [] and this gets past initialization, but the AST cannot be traversed (since during traversal it encounters an Array as the value and calls .each(:pre) which violates the arity of a normal Array)

Was hoping that I am just not understanding something here? (also happy to submit a patch with default initialization and an Array subclass with an appropriate arity on each to handle traversal)

undefined method 'subclass_of?', no such file or directory

I've been following the tutorial to build Kazoo, but have come across 2 problems:

When finalizing, if the .tbl file exists, it gives me this stack trace:

S:/Ruby31-x64/lib/ruby/gems/3.1.0/gems/rltk-3.0.1/lib/rltk/parser.rb:568:in `mtime': No such file or directory @ rb_file_s_mtime - J (Errno::ENOENT)
        from S:/Ruby31-x64/lib/ruby/gems/3.1.0/gems/rltk-3.0.1/lib/rltk/parser.rb:568:in `finalize'
        from J:/Programs/ruby/beetbasicrb/parser.rb:58:in `<class:Parser>'
        from J:/Programs/ruby/beetbasicrb/parser.rb:4:in `<module:BeetBasic>'
        from J:/Programs/ruby/beetbasicrb/parser.rb:3:in `<top (required)>'
        from <internal:S:/Ruby31-x64/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from <internal:S:/Ruby31-x64/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from main.rb:4:in `<main>'

My parser.rb calls

finalize({:use => 'bbparser.tbl'})

and my directory structure looks like:

When I delete the file, it generates as planned.

When it gets past the tbl issue, it gives me the following trace:

S:/Ruby31-x64/lib/ruby/gems/3.1.0/gems/rltk-3.0.1/lib/rltk/ast.rb:120:in `child': undefined method `subclass_of?' for BeetBasic::Expression:Class (NoMethodError)

                                if not t.subclass_of?(ASTNode)
                                        ^^^^^^^^^^^^^
Did you mean?  subclasses
        from J:/Programs/ruby/beetbasicrb/ast.rb:17:in `<class:Binary>'
        from J:/Programs/ruby/beetbasicrb/ast.rb:16:in `<module:BeetBasic>'
        from J:/Programs/ruby/beetbasicrb/ast.rb:3:in `<top (required)>'
        from <internal:S:/Ruby31-x64/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from <internal:S:/Ruby31-x64/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
        from main.rb:5:in `<main>'

I guess they must have removed this method? I'm defining Expression and Binary like this:

class Expression < RLTK::ASTNode; end
#...
class Binary < Expression
        child :left, Expression #* child is a method that takes a symbol and a type. The lack of parenthesis is kiling me.
        child :right, Expression
    end

I don't think I can fix this one on my end.

I'm new to Ruby (although not new to programming) so if I made a rookie mistake let me know. I'm using the Gem version of RLTK and the latest release of Ruby, if that wasn't clear.
If you need a closer look, this project is a public repo.

left/right associativity maintains separate precedence stacks

I have the following associativity map:
left :MOD
left :GT, :GTE, :LT, :LTE
left :SUB
right :ADD
left :DIV
right :MUL
right :EXP
right :DOLLAR
right :VAR

I want DOLLAR and VAR to have higher precedence than anything else. However, since RLTK sets operator precedence for left-assoc and right-assoc operators separately, if I add another left-associative operators up top (e.g. left :QUESTION), DOLLAR, which right now has precedence 4, will suddenly become lower-precedence than DIV, which currently also has precedence 4.

This seems wrong to me - I don't see why precedence should be tied to left- right- or non-associativity.

There is also another issue that setting precedence based on the token type does not always work out, since the same symbol may have multiple meanings - for example I have COLON showing up in a ternary expression production clause and also in a dictionary item label clause - it doesn't make sense to set one precedence for COLON since these are two very different operations.

undefined method `subclass_of?'

Affected Versions
Affects version 3.0.0 and version 3.0.1. Does not affect version 2.2.1 or prior.

Behavior
In the more recent versions of the application when attempting to use a trivial parser we get a NoMethodError.

ast = TaskGrammar::TaskParser::parse(TaskGrammar::TaskLexer::lex(" "))
#=> NoMethodError (undefined method `subclass_of?' for TaskGrammar::TaskCompletion:Class)

When running this with 2.2.1 of the gem, this completes successfully.

ast = TaskGrammar::TaskParser::parse(TaskGrammar::TaskLexer::lex(" "))
=> #<TaskGrammar::TaskAst:0x000055d585cb2e00 @notes={}, @parent=nil, @c=#<TaskGrammar::TaskCompletion:0x000055d585cb34e0 @notes={}, @parent=#<TaskGrammar::TaskAst:0x000055d585cb2e00 ...>, @prefix="a", @infix="b">>

Investigation
It appears that the ast.rb file moved the subclass_of? definition to the rltk/util/monkeys file in commit 4565d5c (4565d5c).

In version 2.2.1 the util folder exists and contains the monkeys.rb file that defines subclass_of? (https://github.com/chriswailes/RLTK/tree/v2.2.1/lib/rltk). In version 3.0.0 the utils folder has been deleted (https://github.com/chriswailes/RLTK/tree/v3.0.0/lib/rltk), but no replacement method of subclass_of? appears to be defined.

RLTK::LexingError carries a wrong line_offset.

Let's say that I have the following rule in the lexer:

rule(/a/)

and the following string (with 3 leading spaces):

   aaab

Lexing it, what I get is a RLTK::LexingError whose line_offset is supposed to be 6, but I actually get a way lower number (I guess in this case it would be 3)

Let me know if you want me to write a test that fails for this bug, in case my description is not clear enough :^)

cc: @libo

:pointer argument is not a valid pointer

This very simple snippet doesn't seem to work for me and I don't know why:

mod = RLTK::CG::Module.new("Hello")
mod.globals.add(RLTK::CG::Int32Type, "hello")

Stacktrace:

ArgumentError: :pointer argument is not a valid pointer
from /Users/gunca/.rvm/gems/ruby-2.1.5/gems/rltk-3.0.0/lib/rltk/cg/module.rb:391:in `add_global'

Running against LLVM-3.4 from Homebrew.

bug in ASTNode#each

Traversing the AST fails if some children have Array types:

https://gist.github.com/ranmrdrakono/82832db8708af1c47495

chriswailes / rltk Goto Github PK

rltk's People

Contributors

Stargazers

Watchers

Forkers

rltk's Issues

Recommend Projects

Recommend Topics

Recommend Org