julianthome / inmemantlr Goto Github PK

For some reason, the lexer isn't being generated in V1.3.1. Any ideas? Also, the Eclipse compiler isn't being brought in for some reason via your POM reference. So I had to add it to the build.
(See https://github.com/cyberthinkers/typescript-to-scala-js-facade - it's a work in progress.)

Exception in thread "main" java.lang.IllegalArgumentException: bytecode of class GenScalaFacadesFromTypescriptLexer is empty
at org.snt.inmemantlr.comp.SpecialJavaFileManager.getByteCodeFromClass(SpecialJavaFileManager.java:99)
at org.snt.inmemantlr.comp.StringCompiler.compile(StringCompiler.java:135)
at org.snt.inmemantlr.GenericParser.compile(GenericParser.java:293)
at com.cyberthinkers.tools.GenScala$.parseSource(GenScalaFacadesFromTypescriptBaseListenerScala.scala:35)
at com.cyberthinkers.tools.GenScala$.main(GenScalaFacadesFromTypescriptBaseListenerScala.scala:27)
at com.cyberthinkers.tools.GenScala.main(GenScalaFacadesFromTypescriptBaseListenerScala.scala)

I suppose inmemantlr doesn't support the C# grammar provided at: https://github.com/antlr/grammars-v4

Is there a possibility to get an AST using inmemantlr from parsers generated by ANTLR4?

Thanks, and sorry for opening an issue for this.

I tried to do this:

val file = Source.fromFile(src).mkString
val read = file //.getLines
val dtListener = new DefaultTreeListener()

val stream = new ANTLRInputStream(read)

val lexer = new CSharpLexer(stream)

val tokens = new CommonTokenStream(lexer)

val parser = new CSharpParser(tokens)

parser.addParseListener(dtListener)

val ast: Ast = dtListener.getAst

But doesn't seem to work.

Hello.

InmemantlrTool here, while setting lexer/parser, gets a package name using this logic which includes only a terminal-based package name, as noted multiple times in antlr4 lib.

But if we set those packages using .g4 actions like in this example:

...
@header {
    package foo.bar;
}
...

GenericParser may throw an exception, for example, here, because SpecialClassLoader can't find a class with a name which doesn't includes packages. m may contain full class names here, but the method parameter name is only a class name without packages, because, as I said earlier, InmemantlrTool uses only terminal-defined packades.

We can workaround this by setting lexer or parser names (with packages) manually after a parser was compiled, but it doesn't seem like a good soltuion for me. But if it's intended and a good way to set up these names, it would be nice to have some notes in documentation or exceptions. Currently, this may lead to this exception: java.lang.NullPointerException: Cannot invoke "org.antlr.v4.runtime.Lexer.addErrorListener(org.antlr.v4.runtime.ANTLRErrorListener)" because "lex" is null which doesn't contain any hints about lexerName or parserName.

My ENV:
IDEA 2016.2 (default junit4)
JDK 1.8

Question:
why use junit5 ? (org.junit.jupiter.api)
how to run these test case ? give me your develop env info.

Dear Julian
Recently I applied filtering for CPP code to get only function body.. Can you please guide me how to generate all the descending nodes of function body.. I mean I m interested in getting all the nodes of function body till last level.. I tried calling getNodes() method from DefaultTreelistner class. But it is creating error as java.util.List cannot be converted to ParseTreeNode.. Kindly help

Is this supported? How can I do that?

Additional compiler options to have more impact on grammar compilation

Remove the output scheme where textfragments are displayed with inside the node. This is too error prone because the text may contain invalid characters depending on what input language you are parsing.

Make sure the parameter order is the same for all GenericParser functions.

Hi Julian,
when I use this tool, it displays as follows.
Error: (4, 26) java: Unable to access org.snt.inmemantlr.GenericParser
Wrong class file: /E:/maven/maven_rep1/com/github/julianthome/inmemantlr-api/1.7.0/inmemantlr-api-1.7.0.jar!/org/snt/inmemantlr/GenericParser.class
The class file has the wrong version 54.0, it should be 52.0
Please delete the file or make sure that the file is in the correct classpath subdirectory.

I am working a project to general runtime grammars for parse email and I am getting this when trying to parse my files:

153. ERROR in /PisosGrammarParser.java (at line 1724)
	_decisionToDFA[i] = new DFA(_ATN.getDecisionState(i), i);
	                                                     ^^^^
ATN cannot be resolved to a type
----------
----------
154. ERROR in /PisosGrammarListener.java (at line 2)
	import org.antlr.v4.runtime.tree.ParseTreeListener;
        ^^^^^^^^^
The import org.antlr cannot be resolved

I generate a spring boot jar with spring boot maven plugin.

Do you have any idea about this issue?

Thanks!

Add utility operations for manipulating the parse tree: parse tree injection

Would be great to have an option to serialize the ANTLR Java classes to the project classpath. This is useful for cases where the grammar is ready-to-use for production and no code-generation/compilation is required anymore.

It would be nice to have a feature which allows one to save a GenericParser on disk after using its compile method so one does not have to compile a specific grammar every time the software starts. I tried ObjectOutputStream which was expected to fail at this task and it did. Do you have any suggestions?

Btw I am glad that you created this antlr wrapper putting a parse result into a real tree and provided a working graphviz output (unlike antlr4).

Hi Julian,

everything works fine when creating a fresh Parser.
However, when I serialize the parser and de-serialize it (successfully without any error), the parse will fail:

Caused by: java.lang.NullPointerException
	at org.snt.inmemantlr.memobjects.MemoryByteCode.getBytes(MemoryByteCode.java:92)
	at org.snt.inmemantlr.comp.SpecialClassLoader.findClass(SpecialClassLoader.java:65)
	at org.snt.inmemantlr.comp.StringCompiler.findClass(StringCompiler.java:152)
	at org.snt.inmemantlr.comp.StringCompiler.instanciateLexer(StringCompiler.java:181)
	at org.snt.inmemantlr.GenericParser.parse(GenericParser.java:526)
	at org.snt.inmemantlr.GenericParser.parse(GenericParser.java:420)
	at com.manticore.jsqlformatter.JavaTools.formatJava(JavaTools.java:153)
	... 1 more

Shortened Java Code

    GenericParser gp =null;
    File cacheFolder = new File(System.getProperty( "java.io.tmpdir"));
    if (cacheFolder.exists() && cacheFolder.isDirectory()) {
      File cacheFile = new File(cacheFolder, "java_grammar.out");
      if (cacheFile.exists() && cacheFile.canRead()) {
        LOGGER.info("Found cache file " + cacheFile.getAbsolutePath());
        try {
          gp = GenericParser.load(cacheFile.getAbsolutePath());
        } catch (Exception ex) {
          LOGGER.log(Level.WARNING, "Failed to read the cached grammer from " + cacheFile.getAbsolutePath(), ex);
          cacheFile.delete();
        }
      }
    }

    gp.setListener(t);
    try {
      ParseTree parseTree;
      gp.parse(source.toString(), GenericParser.CaseSensitiveType.NONE);
      parseTree = t.getParseTree();
    } catch (Exception ex) {
      throw new Exception("Could not parse Java Code:\n" + javaCode, ex);
    }

Before I have serialized the Parser like shown below:

    if (gp==null) {
      LOGGER.info("Compile new Generic Parser.");
       
      gp = new GenericParser(lexerContent, parserContent);
      gp.compile();
      if (cacheFolder.exists() && cacheFolder.isDirectory()) {
        File cacheFile = new File(cacheFolder, "java_grammar.out");
        gp.store(cacheFile.getAbsolutePath(), true);
        
        LOGGER.info("Saved cache file " + cacheFile.getAbsolutePath());
      }
    }

The Parser is correct and works fine when I create it afresh.
But it fails with the NPE when loading it from the file.

I have the File java_grammar.out attached:
java_grammar.zip

Sticking to your sample code in the readme, I can't get any of the grammars-v4 to work. When initializing GenericParser with java8.g4, for instance, it says:

error(50): :1448:3: syntax error: '1-9' came as a complete surprise to me while matching a lexer rule
error(50): :1469:7: syntax error: 'xX' came as a complete surprise to me while looking for lexer rule element
error(50): :1479:3: syntax error: '0-9a-fA-F' came as a complete surprise to me while matching a lexer rule
error(50): :1505:3: syntax error: '0-7' came as a complete surprise to me while matching a lexer rule
error(50): :1521:7: syntax error: 'bB' came as a complete surprise to me while looking for lexer rule element
error(50): :1531:3: syntax error: '01' came as a complete surprise to me while matching a lexer rule
error(50): :1567:3: syntax error: 'eE' came as a complete surprise to me while matching a lexer rule
error(50): :1577:3: syntax error: '+-' came as a complete surprise to me while matching a lexer rule
error(50): :1582:3: syntax error: 'fFdD' came as a complete surprise to me while matching a lexer rule
error(50): :1593:7: syntax error: 'xX' came as a complete surprise to me while looking for lexer rule element
error(50): :1603:3: syntax error: 'pP' came as a complete surprise to me while matching a lexer rule
error(50): :1622:4: syntax error: ''\\\r\n]\n\t;\n\n// §3.10.5 String Literals\n\nStringLiteral\n\t:\t'"' StringCharacters? '"'\n\t;\n\nfragment\nStringCharacters\n\t:\tStringCharacter+\n\t;\n\nfragment\nStringCharacter\n\t:\t~["\\\r\n]\n\t|\tEscapeSequence\n\t;\n\n// §3.10.6 Escape Sequences for Character and String Literals\n\nfragment\nEscapeSequence\n\t:\t'\\' [btnfr"'\\' came as a complete surprise to me while looking for lexer rule element
error(50): :1660:3: syntax error: '0-3' came as a complete surprise to me while matching a lexer rule
error(50): :1735:3: syntax error: 'a-zA-Z$_' came as a complete surprise to me while matching a lexer rule
error(50): :1737:3: syntax error: '\u0000-\u007F\uD800-\uDBFF' came as a complete surprise to me while looking for lexer rule element
error(50): :1740:2: syntax error: '\uD800-\uDBFF' came as a complete surprise to me while matching a lexer rule
error(50): :1746:3: syntax error: 'a-zA-Z0-9$_' came as a complete surprise to me while matching a lexer rule
error(50): :1748:3: syntax error: '\u0000-\u007F\uD800-\uDBFF' came as a complete surprise to me while looking for lexer rule element
error(50): :1751:2: syntax error: '\uD800-\uDBFF' came as a complete surprise to me while matching a lexer rule
error(50): :1766:7: syntax error: ' \t\r\n\u000C' came as a complete surprise to me while matching a lexer rule
error(50): :1774:14: syntax error: '\r\n' came as a complete surprise to me while looking for lexer rule element

The line producing the output above is simply:

this.parser = new GenericParser(grammarFile);

where grammarFile is the java8.g4 file.

Ignoring the messages above and going on with the following:

this.parser.setListener(new ParserListener());
this.parser.compile();

fails with this exception:

 org.snt.inmemantlr.exceptions.CompilationException: No string code pipeline availabe
	at org.snt.inmemantlr.GenericParser.compile(GenericParser.java:313)

Using version 1.4 of inmemantlr (for compatibility with Java 8). Any pointers are appreciated.

I'm getting this dump:

Exception in thread "main" java.lang.IllegalArgumentException: bytecode of class GenScalaFacadesFromTypescriptLexer is empty
at org.snt.inmemantlr.comp.SpecialJavaFileManager.getByteCodeFromClass(SpecialJavaFileManager.java:99)
at org.snt.inmemantlr.comp.StringCompiler.compile(StringCompiler.java:135)
at org.snt.inmemantlr.GenericParser.compile(GenericParser.java:293)
at com.cyberthinkers.tools.GenScala$.parseSource(GenScalaFacadesFromTypescriptBaseListenerScala.scala:36)
at com.cyberthinkers.tools.GenScala$.main(GenScalaFacadesFromTypescriptBaseListenerScala.scala:27)
at com.cyberthinkers.tools.GenScala.main(GenScalaFacadesFromTypescriptBaseListenerScala.scala)

I have a TypeScript ambient grammar that I wrote, and your library is crashing when your librar tries to compile it. It works correctly when I use the standard library.

Class ParseTreeNode has methods getSidx() and getEidx(), but these methods are not sufficient.

There should also be methods getBeginLine(), getBeginCharPositionInLine(), genEndLine(), and getEndCharPositionInLine().

This way, we can quickly locate where the node is in the source parsed. Useful for output detailed error messages.

Dear Julian and Team,

first of all, thanks a lot for providing this project.
I have figured out, how to make use of the Utility files. See the example below.

Furthermore I would like to suggest to implement the TreeNode interface in order to make it easier to use the UI.
Please see the attached patch.

Cheers!

import java.awt.Dimension;
import java.io.File;
import javax.swing.JFrame;
import javax.swing.JScrollPane;
import javax.swing.JTree;
import org.apache.commons.io.FileUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.snt.inmemantlr.GenericParser;
import org.snt.inmemantlr.exceptions.CompilationException;
import org.snt.inmemantlr.exceptions.IllegalWorkflowException;
import org.snt.inmemantlr.exceptions.ParsingException;
import org.snt.inmemantlr.listener.DefaultTreeListener;
import org.snt.inmemantlr.tree.ParseTree;

/**
 *
 * @author are
 */
public class TestSimple {

  private static final Logger LOGGER = LoggerFactory.getLogger(TestSimple.class);

  public static void main(String[] args) throws Exception {
    ClassLoader cl = TestSimple.class.getClassLoader();

    File lexerFile = new File(cl.getResource("com/manticore/g4/PlSqlLexer.g4").toURI());
    File parserFile = new File(cl.getResource("com/manticore/g4/PlSqlParser.g4").toURI());
    
    File abstractLexerClassFile = new File(FileUtils.getTempDirectory(), "PlSqlLexerBase.java");
    abstractLexerClassFile.deleteOnExit();
    
    File abstractParserClassFile = new File(FileUtils.getTempDirectory(), "PlSqlParserBase.java");
    abstractLexerClassFile.deleteOnExit();
    
    FileUtils.copyInputStreamToFile(cl.getResourceAsStream("com/manticore/g4/PlSqlLexerBase.java.txt"), abstractLexerClassFile);
    FileUtils.copyInputStreamToFile(cl.getResourceAsStream("com/manticore/g4/PlSqlParserBase.java.txt"), abstractParserClassFile);

    GenericParser gp = new GenericParser(lexerFile, parserFile);

    gp.addUtilityJavaFiles(abstractLexerClassFile, abstractParserClassFile);

    DefaultTreeListener t = new DefaultTreeListener();
    gp.setListener(t);

    boolean compile;
    try {
      gp.compile();
      compile = true;
      // gp.writeAntlrAritfactsTo("/tmp/grammar");
      // gp.store("/tmp/grammar/gp.out", true);
    } catch (CompilationException e) {
      LOGGER.error(e.getMessage(), e);
      compile = false;
    }

    // this example shows you how one could use inmemantlr for incremental parsing
    try {
      ParseTree parseTree;
      gp.parse("SELECT a, b, c from t1 inner join t2 on t1.a=t2.b;", GenericParser.CaseSensitiveType.UPPER);
      parseTree = t.getParseTree();

      JTree tree = new JTree(parseTree.getRoot());
      JScrollPane scrollPane = new JScrollPane(tree);
      JFrame frame = new JFrame("Simple SQL parser test");
      frame.add(scrollPane);
      frame.setPreferredSize(new Dimension(480, 640));
      frame.pack();
      frame.setDefaultCloseOperation(JFrame.DISPOSE_ON_CLOSE);

      frame.setVisible(true);

    } catch (IllegalWorkflowException | ParsingException e) {
      LOGGER.error(e.getMessage(), e);
    }
  }
}

ParseTreeNode.java.zip

inmemantlr cannot process ANTLR grammars when they are not provided in one single file. We have to add support for multi-file grammars (Thanks to mrn-aglic for the feature request).

Add an extra flag to the default tree listener for omitting/adding terminal nodes. At the moment they are omitted by default.

Is there any reason for it to not to be released on maven central? This plugin is very relevant and useful.

Replace the Eclipse compiler, which adds too much complexity implementation-wise, with the standard java compiler.