oracle / graal Goto Github PK

Some languages, like R, have immutable data types, but allow modifications by using value semantics for updates.
An example:

x <- list(a=1) # creates a new list with element "a" == 1 and assigns it to variable x
x$b <- 2 # adds an element "b" == 2

While the second statement seems to modify the list, it doesn't: the parser desugars this into the following statement:

x <- `$<-`(x, "b", 2)

The call to the update function $<- returns a new list with the original "a" field and the new "b" field.
In cases like this one, where it does not influence language semantics, the runtime can modify the list, but this is only an internal optimization and not something that can be exposed to language users.

What should happen when other languages try to change an immutable data structure?
I see three options:

1. Prevent any modifications from other languages; immutable data structures simply do not understand the write message.
2. Allow modifications from other languages. This will lead to unexpected and undeterministic behavior in the original language.
3. Implement some clever copy-on-write scheme that keeps a copy of the original data, but allows for modifications (which are then only visible in other languages).

I think that 1) is the correct solution: if other languages want to modify the data, they need to copy it into their own, modifiable, data structures.

/cc @jtulach @smarr @chumer

The debugger needs to decide which stack frame slots to display during normal debugging. What's needed is a language-specific recommendation on whether the name and value of any given slot would be sensible (useful) to a guest language programmer.

The debugger might display such slots anyway, for example in a special mode for language implementors.

@mickjordan reports that this would be useful for current work

I'd like to propose an extension to Truffle DSL that help Truffle language implementations avoid unnecessary code duplication.

Given a simple Node like this:

class IsFoo extends Node {
    abstract boolean execute(Object value);
    @Specialization
    static boolean fooA(A value) {
        return ...;
    }
    @Specialization
    static boolean fooB(B value) {
        return ...;
    }
    @Fallback
    static boolean fooFallback(Object value) {
        return ...;
    }
}

Truffle language developers often hit situations where they would like to use the same functionality in a place where they have no node in which to place the IsFoo node.
It is not feasible to enforce all code that is executed to be within compilable nodes: apart from the fact that it's sometimes just infeasible maintain nodes for deeply nested functionality, this would lead to too much (compiled) code duplication, and if the relevant code is behind a TruffleBoundary, the compilation and inlining heuristics are disturbed.
One possible solution would be to provide a "fake" node that can contain other nodes even though it does not have a parent itself, but this disturbs many assumptions that the Truffle system takes.

In reality, the simplest solution is often to accept the code duplication and create the following helper method:

static boolean isFoo(Object value) {
    if (value instanceof A) {
        return ...;
    } else if (value instanceof B) {
        return ...;
    } else {
        return ...;
    }
}

All information needed to create this method is available to Truffle DSL, so that adding @NodeInfo(staticEval=true) could automatically generate the following code in IsFooNodeGen:

static boolean eval(Object value) {
    CompilerAsserts.neverPartOfCompilation(); // or @TruffleBoundary
    if (value instanceof A) {
        return fooA((A) value);
    } else if (value instanceof B) {
        return fooB((B) value);
    } else {
        return fooFallback(value);
    }
}

There are some limitations on the specializations for generating these static eval methods:

• they need to be static methods, so they cannot use instance fields
• they cannot have Frame parameters (at least no VirtualFrame parameters)
• @Cached(...) will behave differently: they might be disallowed, implemented as static fields, or re-created every time

What is the future of Node.assignSourceSection()?

Currently, the reassigning of a source section is protected by an IllegalStateException.
However, the API also provides a public clearSourceSection(), which enables reassignment.

The main reason for this issue is that the API is marked for revision, so, I hope we can settle on an API considering for instance the needs for the new instrumentation API.

@chumer suggested as solution a protected setSourceSection(.), which could be used depending on a language's needs.

Other interested parties are most likely @woess and @jtulach.

A little bit of background: the new instrumentation API currently manages tags as part of SourceSections. For me this creates the need to reassign the source sections on a node.
The situation in SOMns is, that I got a recursive descent parser, and thus, for instance when parsing expression trees recursively, I might only know after creating a full node with source section and state etc that I needs a tag for some tool. So, I got the need to update the tag and reassign the source section.
Using clearSourceSection() the use case is supported.

Restore the ability for debugging and other tools to enumerate tags applied at particular code locations. It can be slow, and should be clearly identified as such.

We should start classify errors that can occur in guest languages and instruments and report them in a checkable form. We can and should at least differentiate between ParseErrors, RuntimeErrors and InternalErrors. All of these errors that are reported to the outside should be equipped with structured information so they can relate to the given Source in PolyglotEngine#eval. They should have a base class which guest languages should be able to subclass. If guest languages cannot use these error classes for whatever reason they have to translate the errors in TruffleLanguage.

Discussion item: I think we should avoid checked exceptions (IOException) for generic errors in PolyglotEngine (I think checked exceptions only make sense if we need to ensure that they are always checked immediatly after using a method ie. UnexpectedResultException). Instead we should provide a runtime exception base class for Parse and Runtime errors.

Are methods withName, withShortName, and withPath supposed to be public rather than package-private?

Javadoc of withPath should probably say "..with the specified path" rather than "...with the specified short name".

The following node definition causes the DSL to generated invalid code:

public abstract class BreakpointNode extends Node {

  public abstract boolean executeCheckIsSetAndEnabled();

  protected BreakpointInfo getBreakpointStatus() {
    return null;
  }

  @Specialization(guards = "!info.hasBreakpoint")
  public boolean noBreakpoint(@Cached("getBreakpointStatus()") final BreakpointInfo info) {
    return false;
  }

  @Specialization(guards = {"info.hasBreakpoint", "!info.breakpoint.isEnabled()"})
  public boolean breakpointDisabled(@Cached("getBreakpointStatus()") final BreakpointInfo info) {
    return false;
  }
}

Specifically, this generates the following createNext(.) method:

        @Override
        protected final SpecializationNode createNext(Frame frameValue) {
            BreakpointInfo info1 = (root.getBreakpointStatus());
            if ((!(info1.hasBreakpoint))) {
                return NoBreakpointNode_.create(root, info1);
            }
            BreakpointInfo info2 = (root.getBreakpointStatus()); else if ((!(info2.breakpoint.isEnabled()))) {
                return BreakpointDisabledNode_.create(root, info2);
            }
            return null;
        }

Would be nice and simple to use a standard build system instead of MX

Right now the graal-core uses Truffle class from an implementation package: https://github.com/graalvm/graal-core/blob/6a3900d09cc4aa94196c58a9b136e4d422b6055b/graal/com.oracle.graal.truffle/src/com/oracle/graal/truffle/OptimizedCallTarget.java#L630

Such tight coupling between Truffle and graal-core will force us to release and use Truffle and graal+core in the same time. That is far from ideal. There shall be explicit, more stable API (probably more an SPI) for communication between Truffle and various runtimes.

I would expect that Java interop message handling to dispatch messages in accordance with Java overload resolution rules, but the current implementation just chooses the first candidate with matching name and matching number of arguments.

The current implementation doesn't even work well for single-argument overloads, where (the equivalent of) System.out.println(0) could dispatch either to the version with double and print 0.0, or dispatch to the Object version and (by chance) do the right thing, or the String version that would cause a dispatch error.

Handling overloads with multiple arguments correctly would be even trickier, as the method selection might be ambiguous.

From @chrisseaton and 7d1f405, iterateFrames() now iterates the topmost frame instance, that is the frame instance returned by getCurrentFrame().
If that is the case, it seems very important to fix the javadoc which states the opposite.

For the same reason that name, uri and mimeType can be configured explicitly, path should be explicitly configurable as well.

If the type used in a @Cached entry requires qualification, the DSL truncates after the . and ends up generating the wrong type.

E.g., I have the following:

@Specialization(guards = { "!isEmpty(string)", "isSingleByteOptimizable(string)" })
public int ordAsciiOnly(DynamicObject string,
      @Cached("create(getContext(), getSourceSection())") RopeNodes.GetByteNode getByteNode) {
...
}

The generated class for this specialization ends up looking like:

@GeneratedBy(methodName = "ordAsciiOnly(DynamicObject, GetByteNode)", value = OrdNode.class)
private static final class OrdAsciiOnlyNode_ extends BaseNode_ {

  @Child private org.jruby.truffle.core.rope.RopeNodes getByteNode;

   ....
}

Note that the @child field is of type RopeNodes rather than RopeNodes.GetByteNode.

On a clean Truffle repo:

dsimon@kurz ~/g/truffle> mx --strict build -n --no-javac-crosscompile --dependencies com.oracle.truffle.api.interop.java
Compiling com.oracle.truffle.api with javac... [/Users/dsimon/graal/truffle/mxbuild/truffle/com.oracle.truffle.api/bin/com/oracle/truffle/api/Assumption.class does not exist]
/Users/dsimon/graal/truffle/truffle/com.oracle.truffle.api/src/com/oracle/truffle/api/instrument/impl/DefaultVisualizer.java:91: warning: [deprecation] displayIdentifier(FrameSlot) in Visualizer has been deprecated
    public String displayIdentifier(FrameSlot slot) {
                  ^
/Users/dsimon/graal/truffle/truffle/com.oracle.truffle.api/src/com/oracle/truffle/api/instrument/impl/DefaultVisualizer.java:83: warning: [deprecation] displayValue(Object,int) in Visualizer has been deprecated
    public String displayValue(Object value, int trim) {
                  ^
/Users/dsimon/graal/truffle/truffle/com.oracle.truffle.api/src/com/oracle/truffle/api/instrument/impl/DefaultVisualizer.java:78: warning: [deprecation] displayCallTargetName(CallTarget) in Visualizer has been deprecated
    public String displayCallTargetName(CallTarget callTarget) {
                  ^
/Users/dsimon/graal/truffle/truffle/com.oracle.truffle.api/src/com/oracle/truffle/api/instrument/impl/DefaultVisualizer.java:66: warning: [deprecation] displayMethodName(Node) in Visualizer has been deprecated
    public String displayMethodName(Node node) {
                  ^
/Users/dsimon/graal/truffle/truffle/com.oracle.truffle.api/src/com/oracle/truffle/api/instrument/impl/DefaultVisualizer.java:49: warning: [deprecation] displaySourceLocation(Node) in Visualizer has been deprecated
    public String displaySourceLocation(Node node) {
                  ^
/Users/dsimon/graal/truffle/truffle/com.oracle.truffle.api/src/com/oracle/truffle/api/instrument/impl/DefaultVisualizer.java:44: warning: [deprecation] getASTPrinter() in Visualizer has been deprecated
    public ASTPrinter getASTPrinter() {
                      ^
6 warnings
Compiling com.oracle.truffle.api.dsl with javac... [dependency com.oracle.truffle.api updated]
Compiling com.oracle.truffle.api.interop with javac... [dependency com.oracle.truffle.api updated]
Compiling com.oracle.truffle.api.instrumentation with javac... [dependency com.oracle.truffle.api updated]
Compiling com.oracle.truffle.dsl.processor with javac... [dependency com.oracle.truffle.api.dsl updated]
Compiling com.oracle.truffle.api.object with javac... [dependency com.oracle.truffle.api.interop updated]
Compiling com.oracle.truffle.api.object.dsl with javac... [dependency com.oracle.truffle.api.object updated]
Compiling com.oracle.truffle.object.dsl.processor with javac... [dependency com.oracle.truffle.api.object.dsl updated]
Archiving TRUFFLE_DSL_PROCESSOR_INTERNAL... [dependency com.oracle.truffle.dsl.processor updated]
Compiling com.oracle.truffle.api.interop.java with javac... [dependency TRUFFLE_DSL_PROCESSOR_INTERNAL updated]
/Users/dsimon/graal/truffle/truffle/com.oracle.truffle.api.interop.java/src/com/oracle/truffle/api/interop/java/ArrayGetSizeNode.java:33: error: cannot find symbol
final class ArrayGetSizeNode extends ArrayGetSizeBaseNode {
                                     ^
  symbol: class ArrayGetSizeBaseNode
/Users/dsimon/graal/truffle/truffle/com.oracle.truffle.api.interop.java/src/com/oracle/truffle/api/interop/java/ArrayHasSizeNode.java:33: error: cannot find symbol
final class ArrayHasSizeNode extends ArrayHasSizeBaseNode {
                                     ^
  symbol: class ArrayHasSizeBaseNode
/Users/dsimon/graal/truffle/truffle/com.oracle.truffle.api.interop.java/src/com/oracle/truffle/api/interop/java/ExecuteNode.java:39: error: cannot find symbol
final class ExecuteNode extends JavaFunctionBaseNode {
                                ^
  symbol: class JavaFunctionBaseNode
/Users/dsimon/graal/truffle/truffle/com.oracle.truffle.api.interop.java/src/com/oracle/truffle/api/interop/java/InvokeNode.java:36: error: cannot find symbol
final class InvokeNode extends InvokeBaseNode {
                               ^
  symbol: class InvokeBaseNode
/Users/dsimon/graal/truffle/truffle/com.oracle.truffle.api.interop.java/src/com/oracle/truffle/api/interop/java/IsExecutableNode.java:31: error: cannot find symbol
final class IsExecutableNode extends IsExecutableBaseNode {
                                     ^
  symbol: class IsExecutableBaseNode
/Users/dsimon/graal/truffle/truffle/com.oracle.truffle.api.interop.java/src/com/oracle/truffle/api/interop/java/NewNode.java:34: error: cannot find symbol
final class NewNode extends JavaNewBaseNode {
                            ^
  symbol: class JavaNewBaseNode
/Users/dsimon/graal/truffle/truffle/com.oracle.truffle.api.interop.java/src/com/oracle/truffle/api/interop/java/NullCheckNode.java:31: error: cannot find symbol
final class NullCheckNode extends NullCheckBaseNode {
                                  ^
  symbol: class NullCheckBaseNode
/Users/dsimon/graal/truffle/truffle/com.oracle.truffle.api.interop.java/src/com/oracle/truffle/api/interop/java/ReadFieldNode.java:40: error: cannot find symbol
final class ReadFieldNode extends ReadFieldBaseNode {
                                  ^
  symbol: class ReadFieldBaseNode
/Users/dsimon/graal/truffle/truffle/com.oracle.truffle.api.interop.java/src/com/oracle/truffle/api/interop/java/WriteFieldNode.java:33: error: cannot find symbol
final class WriteFieldNode extends WriteFieldBaseNode {
                                   ^
  symbol: class WriteFieldBaseNode
/Users/dsimon/graal/truffle/truffle/com.oracle.truffle.api.interop.java/src/com/oracle/truffle/api/interop/java/ExecuteNode.java:41: error: cannot find symbol
    @Child private DoExecuteNode doExecute;
     ^
  symbol:   class Child
  location: class ExecuteNode
/Users/dsimon/graal/truffle/truffle/com.oracle.truffle.api.interop.java/src/com/oracle/truffle/api/interop/java/InvokeNode.java:38: error: cannot find symbol
    @Child private DoExecuteNode doExecute;
     ^
  symbol:   class Child
  location: class InvokeNode
/Users/dsimon/graal/truffle/truffle/com.oracle.truffle.api.interop.java/src/com/oracle/truffle/api/interop/java/WriteFieldNode.java:35: error: cannot find symbol
    @Child private ToJavaNode toJava = new ToJavaNode();
     ^
  symbol:   class Child
  location: class WriteFieldNode
/Users/dsimon/graal/truffle/truffle/com.oracle.truffle.api.interop.java/src/com/oracle/truffle/api/interop/java/ExecuteNode.java:41: error: Exception thrown during processing: java.lang.ClassCastException: com.sun.tools.javac.code.Symbol$VarSymbol cannot be cast to javax.lang.model.element.ExecutableElement
    @Child private DoExecuteNode doExecute;
                                 ^
    at com.oracle.truffle.dsl.processor.verify.VerifyTruffleProcessor.process(VerifyTruffleProcessor.java:109)
    at com.sun.tools.javac.processing.JavacProcessingEnvironment.callProcessor(JavacProcessingEnvironment.java:793)
    at com.sun.tools.javac.processing.JavacProcessingEnvironment.discoverAndRunProcs(JavacProcessingEnvironment.java:722)
    at com.sun.tools.javac.processing.JavacProcessingEnvironment.access$1700(JavacProcessingEnvironment.java:97)
    at com.sun.tools.javac.processing.JavacProcessingEnvironment$Round.run(JavacProcessingEnvironment.java:1029)
    at com.sun.tools.javac.processing.JavacProcessingEnvironment.doProcessing(JavacProcessingEnvironment.java:1163)
    at com.sun.tools.javac.main.JavaCompiler.processAnnotations(JavaCompiler.java:1108)
    at com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:824)
    at com.sun.tools.javac.main.Main.compile(Main.java:439)
    at com.sun.tools.javac.main.Main.compile(Main.java:353)
    at com.sun.tools.javac.main.Main.compile(Main.java:342)
    at com.sun.tools.javac.main.Main.compile(Main.java:333)
    at com.sun.tools.javac.Main.compile(Main.java:76)
    at com.sun.tools.javac.Main.main(Main.java:61)
/Users/dsimon/graal/truffle/truffle/com.oracle.truffle.api.interop.java/src/com/oracle/truffle/api/interop/java/InvokeNode.java:38: error: Exception thrown during processing: java.lang.ClassCastException: com.sun.tools.javac.code.Symbol$VarSymbol cannot be cast to javax.lang.model.element.ExecutableElement
    @Child private DoExecuteNode doExecute;
                                 ^
    at com.oracle.truffle.dsl.processor.verify.VerifyTruffleProcessor.process(VerifyTruffleProcessor.java:109)
    at com.sun.tools.javac.processing.JavacProcessingEnvironment.callProcessor(JavacProcessingEnvironment.java:793)
    at com.sun.tools.javac.processing.JavacProcessingEnvironment.discoverAndRunProcs(JavacProcessingEnvironment.java:722)
    at com.sun.tools.javac.processing.JavacProcessingEnvironment.access$1700(JavacProcessingEnvironment.java:97)
    at com.sun.tools.javac.processing.JavacProcessingEnvironment$Round.run(JavacProcessingEnvironment.java:1029)
    at com.sun.tools.javac.processing.JavacProcessingEnvironment.doProcessing(JavacProcessingEnvironment.java:1163)
    at com.sun.tools.javac.main.JavaCompiler.processAnnotations(JavaCompiler.java:1108)
    at com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:824)
    at com.sun.tools.javac.main.Main.compile(Main.java:439)
    at com.sun.tools.javac.main.Main.compile(Main.java:353)
    at com.sun.tools.javac.main.Main.compile(Main.java:342)
    at com.sun.tools.javac.main.Main.compile(Main.java:333)
    at com.sun.tools.javac.Main.compile(Main.java:76)
    at com.sun.tools.javac.Main.main(Main.java:61)
/Users/dsimon/graal/truffle/truffle/com.oracle.truffle.api.interop.java/src/com/oracle/truffle/api/interop/java/WriteFieldNode.java:35: error: Exception thrown during processing: java.lang.ClassCastException: com.sun.tools.javac.code.Symbol$VarSymbol cannot be cast to javax.lang.model.element.ExecutableElement
    @Child private ToJavaNode toJava = new ToJavaNode();
                              ^
    at com.oracle.truffle.dsl.processor.verify.VerifyTruffleProcessor.process(VerifyTruffleProcessor.java:109)
    at com.sun.tools.javac.processing.JavacProcessingEnvironment.callProcessor(JavacProcessingEnvironment.java:793)
    at com.sun.tools.javac.processing.JavacProcessingEnvironment.discoverAndRunProcs(JavacProcessingEnvironment.java:722)
    at com.sun.tools.javac.processing.JavacProcessingEnvironment.access$1700(JavacProcessingEnvironment.java:97)
    at com.sun.tools.javac.processing.JavacProcessingEnvironment$Round.run(JavacProcessingEnvironment.java:1029)
    at com.sun.tools.javac.processing.JavacProcessingEnvironment.doProcessing(JavacProcessingEnvironment.java:1163)
    at com.sun.tools.javac.main.JavaCompiler.processAnnotations(JavaCompiler.java:1108)
    at com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:824)
    at com.sun.tools.javac.main.Main.compile(Main.java:439)
    at com.sun.tools.javac.main.Main.compile(Main.java:353)
    at com.sun.tools.javac.main.Main.compile(Main.java:342)
    at com.sun.tools.javac.main.Main.compile(Main.java:333)
    at com.sun.tools.javac.Main.compile(Main.java:76)
    at com.sun.tools.javac.Main.main(Main.java:61)
/Users/dsimon/graal/truffle/truffle/com.oracle.truffle.api.interop.java/src/com/oracle/truffle/api/interop/java/ArrayGetSizeNode.java:33: error: cannot find symbol
final class ArrayGetSizeNode extends ArrayGetSizeBaseNode {
                                     ^
  symbol: class ArrayGetSizeBaseNode
/Users/dsimon/graal/truffle/truffle/com.oracle.truffle.api.interop.java/src/com/oracle/truffle/api/interop/java/ArrayHasSizeNode.java:33: error: cannot find symbol
final class ArrayHasSizeNode extends ArrayHasSizeBaseNode {
                                     ^
  symbol: class ArrayHasSizeBaseNode
/Users/dsimon/graal/truffle/truffle/com.oracle.truffle.api.interop.java/src/com/oracle/truffle/api/interop/java/ExecuteNode.java:39: error: cannot find symbol
final class ExecuteNode extends JavaFunctionBaseNode {
                                ^
  symbol: class JavaFunctionBaseNode
/Users/dsimon/graal/truffle/truffle/com.oracle.truffle.api.interop.java/src/com/oracle/truffle/api/interop/java/InvokeNode.java:36: error: cannot find symbol
final class InvokeNode extends InvokeBaseNode {
                               ^
  symbol: class InvokeBaseNode
/Users/dsimon/graal/truffle/truffle/com.oracle.truffle.api.interop.java/src/com/oracle/truffle/api/interop/java/IsExecutableNode.java:31: error: cannot find symbol
final class IsExecutableNode extends IsExecutableBaseNode {
                                     ^
  symbol: class IsExecutableBaseNode
/Users/dsimon/graal/truffle/truffle/com.oracle.truffle.api.interop.java/src/com/oracle/truffle/api/interop/java/NewNode.java:34: error: cannot find symbol
final class NewNode extends JavaNewBaseNode {
                            ^
  symbol: class JavaNewBaseNode
/Users/dsimon/graal/truffle/truffle/com.oracle.truffle.api.interop.java/src/com/oracle/truffle/api/interop/java/NullCheckNode.java:31: error: cannot find symbol
final class NullCheckNode extends NullCheckBaseNode {
                                  ^
  symbol: class NullCheckBaseNode
/Users/dsimon/graal/truffle/truffle/com.oracle.truffle.api.interop.java/src/com/oracle/truffle/api/interop/java/ReadFieldNode.java:40: error: cannot find symbol
final class ReadFieldNode extends ReadFieldBaseNode {
                                  ^
  symbol: class ReadFieldBaseNode
/Users/dsimon/graal/truffle/truffle/com.oracle.truffle.api.interop.java/src/com/oracle/truffle/api/interop/java/WriteFieldNode.java:33: error: cannot find symbol
final class WriteFieldNode extends WriteFieldBaseNode {
                                   ^
  symbol: class WriteFieldBaseNode
/Users/dsimon/graal/truffle/truffle/com.oracle.truffle.api.interop.java/src/com/oracle/truffle/api/interop/java/ExecuteNode.java:41: error: cannot find symbol
    @Child private DoExecuteNode doExecute;
     ^
  symbol:   class Child
  location: class ExecuteNode
/Users/dsimon/graal/truffle/truffle/com.oracle.truffle.api.interop.java/src/com/oracle/truffle/api/interop/java/InvokeNode.java:38: error: cannot find symbol
    @Child private DoExecuteNode doExecute;
     ^
  symbol:   class Child
  location: class InvokeNode
/Users/dsimon/graal/truffle/truffle/com.oracle.truffle.api.interop.java/src/com/oracle/truffle/api/interop/java/WriteFieldNode.java:35: error: cannot find symbol
    @Child private ToJavaNode toJava = new ToJavaNode();
     ^
  symbol:   class Child
  location: class WriteFieldNode
15 errors

We should not expose the collection of all call targets in the TruffleRuntime. If we still need a way to access all call targets, consider moving it to implementation classes.

The documentation for the Factory10 interface at the moment says that the accessXXX messages can return null if the message is not supported. However, example in SL shows that this is not done that way. Instead, the SL implementation always raise an UnsupportedMessageException (i.e., doing UnsupportedMessageException.raise(Message.XXX); )

Can you clarify in the documentation why is there two ways to do this and which one is the recommended one ?

Use of PolyglotEngine is required to make certain parts of Truffle work (e.g. retrieval of ExecutionContext). From what I've heard, PolyglotEngine may even become mandatory in the future.

However, I found that when writing tools tailored to a specific language, PolyglotEngine isn't a good fit: Values have to be unwrapped, exceptions have to be unwrapped, only Sources (but not ASTs) can be evaluated, etc. After a lot of fighting, I gave up and decided to bypass PolyglotEngine for the time being.

To solve this problem, I propose that PolyglotEngine should provide an additional lower-level execution mode that allows to run any callback within an engine context, without the baggage of Value, Source, etc. This would greatly simplify writing of language-specific tools.

Thanks to Chris Seaton for bringing up the idea of a lower-level execution mode.

Isn't it the case that the list of types used in the DSL type system is largely redundant now? So at least should it be given a default value of {} so that we can write:

@TypeSystem

instead of

@TypeSystem({})

The Source class contains support for describing text segments in several ways, including both 0-based character offsets and 1-based {row,line} coordinates. Although correctly documented, this support has a design flaw that makes it impossible to create a SourceSection that represents the contents of an empty Source instance. JRuby has needed a workaround for this, as likely have others. The "text map", its method documentation, and detailed tests must be reimplemented.

Certain constraints on AST structure and "source attribution" (attached SourceSection instances) make the ASTs well behaved for a variety of purposes, especially for tools. Examples include fast searching for AST nodes corresponding to particular text selections (e.g. point and click in a debugger), and some kinds of source mapping. The basic idea is that the AST naturally represents a subset (projection) of a parse tree. Another way to view it is that the SourceSection on a node represent exactly the text that was parsed to produce the node.

Create an auditor to check for this property. More specifically;

1. The SourceSection of a parent node includes the text of every child node'sSourceSection.
2. The SourceSection text of a node contains text that strictly precedes (in the original Source) the SourceSection of every sibling node to the right.
3. The SourceSection text of a node contains text that strictly follows (in the original Source) the SourceSection of every sibling node to the left.

Truffle DSL has an optimization for creating Classes from the @specialization annotation: In cases where there is only one Specialization, it avoids generating the Base_, Uninitialized_ and [SpecializationName]* classes, but gets along with just the [NodeName]Gen class.

This optimization seems to fail on two occasions:

1. when an @cached annotation is used in this specialization's argument list
2. when a specific datatype is used as argument, i.e. when there might be a need to throw an UnsupportedSpecializationException.

While the second one should rarely occur due to the imminent exception (but is in fact occurring in internal helper nodes), I would expect the first one to occur a lot in all language implementations, generating lots of unnecessary extra classes. Fixing this would reduce code size and potentially improve startup/warmup time.

In Travis we execute

• mx checkoverlap
• mx canonicalizeprojects
• mx checkstyle
• sh .travis.sigtest.sh

as separate tasks but these are also part of mx gate --strict-mode:

• Distribution Overlap Check
• Canonicalization Check
• Checkstyle
• Truffle Signature Tests

In graal-core we use mx gate --tags to implement non-redundant parallel gates (see graalvm/graal-core#17).

The debugger needs to decide which stack frames to display, for example in a backtrace, during normal debugging. What's needed is a language-specific recommendation on whether any given frame would be sensible (useful) to a guest language programmer.

The debugger might display such frames anyway, for example in a special mode for language implementors.

To discuss: should the recommendation that a particular frame be hidden also be treated as a suggestion that the call-site that created the frame also be hidden? I'm thinking of Ruby's implementation of certain (but not all) looping constructs as calls. I specifically hid those calls in an early debugger prototype.

@mickjordan reports that this would be useful for current work

We want to avoid people running JRuby+Truffle on a standard JVM, or even a JVM with Graal but the wrong flags, and being disappointed about performance. When JRuby+Truffle starts we check that the TruffleRuntime is a Graal runtime, and if it isn't we present a warning. Doing this requires looking at the undocumented runtime name.

Truffle.getRuntime().getName().toLowerCase(Locale.ENGLISH).contains("graal");

I wonder if it would be useful to add a method boolean isOptimizing() to TruffleRuntime. This hides the implementation detail that it is Graal being used, and leaves the door open for any other hypothetical optimising Truffle runtimes, while still allowing us to show our warning.

Another option would be the inverse - boolean isInterpreter().

SourceSection Javadoc does not say how equality/hash are defined, especially so with respect to associated tags.

Ruby has a built-in coverage tool. It returns an array of values for each file, one value per line. If that line had code on it but wasn't executed the value is 0. If it didn't have any code on it anyway then the value is nil. I'm not sure how to recreate this distinction between no-code and never-executed in the new instrumentation API.

I'm thinking that one solution is to be able to run a query something like this:

boolean hasCode = instrumenter.doesExist(SourceSectionFilter.newBuilder()
    .sourceIs(source).lineIs(line).tagIs(LINE_TAG).build())

But I am of course open to any other idea that achieves the same result. I've got tests that fail until I can solve this.

I know that laziness makes it a tricky problem.

We should revisit and document the Truffle stack walking API.
The stack trace API consists of the following elements:

• class FrameInstance
• class FrameInstance.FrameAccess
• class FrameInstanceVisitor
• method TruffleRuntime#getCurrentFrame()
• method TruffleRuntime#getCallerFrame()
• method TruffleRuntime#iterateFrames(FrameInstanceVisitor)

The following things should be considered:

A) The naming in FrameInstance and FrameInstanceVisitor might be misleading, as they are not just containing a frame but also the CallTarget and the CallNode which is usually called a stack frame in other APIs. So the more established StackFrameElement could be used instead.
A new API could look like:

• T TruffleRuntime#iterateStackFrameElements(int skipElements, StackElementVisitor visitor)
• StackElement TruffleRuntime#peekStackFrameElement(int skipElements)
  Please feel free to comment on that.

B) Add javadoc to the API. ASCII art similar to below could be useful.

C) FrameInstance#isVirtualFrame is currently never true in the default runtime. Its true for the topmost frame in the Graal runtime. @lukasstadler Maybe you can comment what the use-case is for this method?

D) In FrameInstance#getFrame(FrameAccess, boolean slowPath) whats the purpose of slowpath? @lukasstadler Do you want to comment on that? Maybe we can solve this in a different way by checking a CompilerDirective in graal?

E) We should revisit the question of what FrameInstance#getCallNode() should return.
First grouping option (current state):

                    ===============
getCurrentFrame(): |  CallTarget   | FrameInstance
                    ===============
getCallerFrame():  |  CallNode     | FrameInstance
                   |  CallTarget   |
                    ===============
                   |  CallNode     | FrameInstance
                   |  CallTarget   |
                    ===============
                         ...
                    ===============
                   |  CallNode     | FrameInstance
Initial call:      |  CallTarget   |
                    ===============

Second grouping option:

                    ===============
getCurrentFrame(): |  CallTarget   | FrameInstance
                   |  CallNode     | 
                    ===============
getCallerFrame():  |  CallTarget   | FrameInstance
                   |  CallNode     | 
                    ===============
                   |  CallTarget   | FrameInstance
                   |  CallNode     | 
                    ===============
                         ...
                    ===============
Initial call:      |  CallTarget   |
                    ===============

As far as I understand the first has the advantage that you can easily find the current line number where execution is currently at for the frame. And the second option would have the advantage that you find out how you were called (some languages tend to print stack traces with a called-as note).

The changelog hasn't been updated since Oct 2015 and doesn't include the latest releases.

It is a common error in guest languages to use nodes with source sections and that are @Instrumentable with the wrong field type so that the WrapperNode cannot get inserted.

For example:

@Instrumentable(InstrumentableNodeWrapper.class)
class InstrumentableNode extends Node {}

class InstrumentableNodeSubclass extends InstrumentableNode {}

class SomeOtherNode {
@Child InstrumentableNodeSubclass child;
}

Replaces on SomeOtherNode#child will fail if InstrumentableNodeSubclass is wrapped by the instrumentation framework because the wrapper just extends InstrumentableNode but not as required InstrumentableNodeSubclass. We should check for this error in an annotation processor that gets triggered on @child annotations.

Source is inflexible with respect to source encodings.

I think that @NodeInfo should be @Inherited, to make it easier to retrieve from node subclasses (e.g. those generated by Truffle codegen).

SourceSection is old prototype code and is overdue for redesign and reimplementation. One goal is to reduce memory footprint. There are also open issues around requirements for variants that carry useful information when actual source is either unavailable or nonexistent; this is especially important for tools. Among others, a representative of each language team should be included in a discussion of requirements for "unavailable" as well as any other requirements that experience has revealed.

The Debugger currently halts execution by throwing a KillException, which unwinds immediately all instrumentation and guest language execution code through the original call, e.g. PolyglotEngine.eval(). Language implementations must now support this by ensuring that the exception is correctly passed through any catch.

@chumer suggests replacing this implementation with something more general that does not depend on the language implementations for correctness.

Note: The Debugger also uses the QuitException, which requires exactly the same support. Only one kind of exception is actually required, so support for QuitException is not a separate requirement.

A very common way to read Javadoc is through the sources (Jars) attached to the IDE project. (It's much more convenient to search/browse sources in an IDE than Javadoc on a web page, and seeing both documentation and source code at the same time is very useful.) However, seeing Javadoc such as Usage example: {@codesnippet BranchProfileSample} isn't helpful - instead I'd like to see the code snippet itself. Therefore, please embed code snippets in sources before creating sources Jars. (And please make sure line numbers in class files remain intact.)

Currently, the Truffle DSL has the "rewriteOn" annotation that removes the current instance of a specialization. However, according to the javadoc, it also removes all the other instances of that specialization and prevents instances of this specialization to be created again.

This behavior does not allow replacement of an individual instance in the cached specialization chain. Having a cache with no replacement is a serious limitation in general.

Single node replacement could be done similarly to the rewriteOn attribute, for example, by adding a new attribute with a list of exceptions. Throwing one such exception would remove only the current instance.

This means that most Truffle users won't be able to navigate to these snippets in their IDE.

Cases I encountered:

• com.oracle.truffle.api.dsl.test.interop.Snippets.ExampleTruffleObject#getForeignAccessMethod
• com.oracle.truffle.api.dsl.test.interop.Snippets.ExampleTruffleObjectMR

For languages that allow a function to contain lexically nested functions, e.g. R, there needs to be an easy way to restrict a SourceSectionFilter to the body of the enclosing function but excluding the bodies of the nested functions. The indexIn method for the outer function (which must be used to uniquely identify the function in the face if duplicates in the "same" source") necessarily includes the bodies of the nested functions. The consequence is that, .e.g. a debugger will step into the bodies of nested functions when it should step over because the filter will match statements in the nested functions. Generating a list of indexNotIn filters would be possible but tedious.

http://lafo.ssw.uni-linz.ac.at/nexus/content/repositories/repos-public/com/oracle/truffle/truffle-dsl-processor/ only has 0.12-SNAPSHOT, whereas other modules (e.g. http://lafo.ssw.uni-linz.ac.at/nexus/content/repositories/repos-public/com/oracle/truffle/truffle-api/) have 0.13-SNAPSHOT.

mx maven-install does produce a 0.13-SNAPSHOT:

[INFO] Installing /xxx/truffle/mxbuild/dists/truffle-dsl-processor.jar to /xxx/.m2/repository/com/oracle/truffle/truffle-dsl-processor/0.13-SNAPSHOT/truffle-dsl-processor-0.13-SNAPSHOT.jar

Introduction of Debugger.pause() (see the pull request #153), revealed insufficient synchronization of code operating on roots map and bindings list.
roots is a synchronized map, but there are several unsynchronized iterations over it's key set. This leads to ConcurrentModificationException.

Bindings can be added asynchronously at any time while a language is executed. This is the case for e.g. debugger breakpoints and debugger pause. That means that bindings list can be changed at any time. This makes the code in AbstractBindingsVisitor problematic, shouldVisit() might not use the same list of elements as visit(Node) and there is a risk of IndexOutOfBoundsException when e.g. breakpoint is removed - one binding is removed.

The similar lack of synchronization may apply to instrumenterMap map.

Something I realized when reviewing https://github.com/graalvm/truffle/pull/87/files.
Source reloading does not seem to be thread-safe.
In particular Source#getTextMap().

Truffle 0.14

From TypeCheck's Javadoc:

Is:

This is useful for types where the guest language specific type check can be implemented more efficiently than a direct cast.

Should be:

This is useful for types where the guest language specific type check can be implemented more efficiently than an instanceof check.

From TypeCast's Javadoc:

Is:

This is useful for types where the guest language specific type cast can be implemented more efficiently than an instanceof check.

Should be:

This is useful for types where the guest language specific type cast can be implemented more efficiently than a direct cast.

Currently, Source has at least three distinct responsibilities:

1. Load source code from different origins
2. Cache source code once loaded
3. Work with source text (SourceSection, etc.)

I'm happy with how Source handles 3, but have different requirements for 1 and 2 (i.e. I need full control over them). Can these responsibilities be extracted into separate classes (e.g. SourceLoader) that are not mandatory for using the PolyglotEngine/TruffleLanguage/Source APIs? Also, can you add a way to set the URI for a(ny) Source, so that its origin can be cleanly attached?

The InstrumentationHandler.visitRoot() method sets fields of an AbstractNodeVisitor (root and providedTags), at the end, it sets the fields to null.

This seems thread-unsafe. Considering InstrumentationHandler.installRootNode() calls visitRoot() with the addAllBindingsVisitor, this seems to be unsafe, because multiple threads can end up using the same instance of addAllBindingsVisitor object, and mutating it unsynchronized.

This seems to be introduced with 2aa2eff

/cc @mlvdv

The debugger needs access to language-specific textual display of language values. This was formerly supported by the (now deprecated):
String TruffleLanugage.getVisualizer().displayValue(Object value, int trim)

The REPL debugger now uses Object.toString().

@mickjordan reports that this is important for current work.

TVMCI.findLanguageClass assumes that Accessor.SPI is not null. This might be the case when PolyglotEngine is not initialized.

Javadoc missing on FrameInstance.getFrame()

O.17 has deprecated SourceSection.getIdentifier(). Presumably this means that the second parameter to SourceSection.createUnavailable() should also be deprecated, as all it does it to set identifier.