$ starlark-go -c '0in()'

(exit 0)

$ starlark-rust -c '0in()'
error[CL00]: Parse error
 --> [command flag]:1:2
  |
1 | 0in()
  |  ^ Character not valid at present location

(exit 2)

Related: #44

I'm writing my own file loader right now and finding it awkward to differentiate between EvalException and Diagnostic, moreso because EvalException is #[doc(hidden)]. Looking at the implementations for NoFileLoader and SimpleFileLoader, they both wrap a Diagnostic in EvalException::DiagnosedError already, so there shouldn't be any difference in behavior if the return type were changed to just Diagnostic (IIUC).

It's a breaking change to a public trait, so it'd need a new minor version I think. Thoughts?

$ cat x.star 
print("hi")
$ ./target/release/starlark-repl x.star
error[CM01]: Variable 'print' not found
 --> x.star:1:1
  |
1 | print("hi")
  | ^^^^^ Variable was not found

The spec is here: https://github.com/google/starlark-go/blob/master/doc/spec.md#print

The context, if you were curious, is that I was kicking the tires to see whether I could do some fuzzing of the Go vs the Rust implementations, to find bugs in each of them. It seems like maybe this project isn't ready for that. Does that sound right?

@adonovan said "For loops at top level give a parse error. It's kinder to parse them and reject them as a later pass (and allow them in the REPL)."

$ starlark-go -c '0/(1/2)'
cmdline:1:2: this Starlark dialect does not support floating point (use //)
(exit 1)

$ starlark-go -float -c '0/(1/2)'

(exit 0)

$ starlark-rust -c '0/(1/2)'
error[CV08]: Cannot divide by zero
 --> [command flag]:1:1
  |
1 | 0/(1/2)
  | ^^^^^^^ Division by zero

(exit 2)

starlark-rust should either reject this program because floating point support is not enabled or return 0 without error.

See https://docs.rs/crate/starlark/0.1.0/builds/118604

It seems like it doesn't like the single quotes. Let just replace with double quotes

I was thinking of implementing some starlark-exposed functions in Rust (outside of the starlark crate itself), but I noticed that FunctionSignature::new is pub(crate), and much of the machinery surrounding NativeFunction is doc(hidden). Is there another way to implement functions and exposing them to an Environment? If not, how would folks feel about opening up FunctionSignature::new? I think I'm fine with leaving them undoc'd (they're slightly weird APIs, and I'm not sure we think they're stable extension points at the moment), but I'm curious as to why we ended up with them restricted.

/cc @stepancheg @damienmg

a = []
b = a
a += [1]
(a, b)

This should return

([1], [1])

according to Starlark spec, but returns

([1], [])

in current impl.

This is just to test the command line itself not starlark where we have dedicated tests.

There is a println!("{:?}", self) at 57aa687#diff-3aaa36185590979cba661351eaa38456R277 that doesn't belong.

I'm blocked on releasing https://github.com/indygreg/PyOxidizer 0.8 to crates.io (due to published crate dependency requirement) due to this bug because I must call this function :(

See discussion in #32 for some additional context.

reported by @adonovan

$ starlark-repl -c "'a'.index('', 1, 0)"
thread 'main' panicked at 'called `Option::unwrap()` on a `None` value', src/libcore/option.rs:355:21
note: Run with `RUST_BACKTRACE=1` for a backtrace.
(exit 101)

Found while investigating google/starlark-go#134.

Hello! Thank you for creating this project: it's a wonderful contribution to both the rust and starlark communities.

I've started work on a series of 4-5 (relatively small) PRs that would add initial support for Python's yield syntax for coroutines/generators. Our (in http://github.com/pantsbuild/pants/) usecase only requires consuming the coroutines from the host language, so those PRs would not add builtins/stdlib-fns for resuming generators from starlark*, and would instead assume that the host language would be directly aware of a Generator TypedValue, and would resume evaluation that way.

Would the committer(s) be willing to accept these patches? The end-user visible differences could be very limited: it should be straightforward to disable the yield statement via an option if need be.

* which would require a bit of spec work, since starlark does-not/should-not support exceptions, and Python uses StopIteration in its interface.

Starlark has at least 3 implementations: a Java one for Bazel, a go one and this one.

"Go", not "go".

This interpreter was made using the specification from the go version and the Python 3 documentation when things were unclear.

What was unclear? Please file a bug report against doc/spec.md for each issue and I will fix them by clarifying the spec.

It does not include the set() type either (the Java implementation use a custom type, depset, instead).

The depset data type is a feature of the Bazel build language, not core Starlark. (I have a Go implementation but it does not belong in the Starlark-in-Go repository.)

Is there any way to access local variables declared outside the starlark_module! macro, or to allow for closures to be provided as a NativeFunction?

Otherwise, how could I import external functions or variables into a Starlark module?

Hi, thanks a ton for this project! I'm relatively new to Rust and found integrating starlark-rust into a program a bit intimidating. Would you be open to adding some examples to the documentation? They could go in the README.md, an examples directory, or the rustdoc comments themselves. I ended up reading code in the starlark-repl directory to get familiar with how to use it, but it's a bit more complicated to parse than I had hoped. The specific information I was hoping to see was how codemap and starlark::stdlib::global_environment were meant to be used with eval, and how to use a Value returned from eval.

I made some sample code for myself, but it might be a bit too long for the README.md (it's also not directly compilable being outside a function, but I think that would make it even longer):

extern crate codemap;
extern crate codemap_diagnostic;
extern crate starlark;

use std::sync::{Arc, Mutex};

use codemap_diagnostic::{ColorConfig, Emitter};
use starlark::eval::simple::eval;
use starlark::stdlib::global_environment;

let starlark_input = "str([2 * x for x in range(5)])";

let global_env = global_environment();
global_env.freeze();
let mut env = global_env.child("shell_loop");
let map = Arc::new(Mutex::new(codemap::CodeMap::new()));

let res = match eval(&map, "string input", &starlark_input, true, &mut env) {
    Err(diag) => {
        let cloned_map = Arc::clone(&map);
        let unlocked_map = cloned_map.lock().unwrap();
        Emitter::stderr(ColorConfig::Always, Some(&unlocked_map)).emit(&[diag]);
        panic!("Error interpreting code '{}'", starlark_input);
    },
    Ok(res) => match res.get_type() {
        "string" => res.to_str(),
        _ => panic!(
            "Error interpreting code '{}': result must be string! (was {})",
            starlark_input,
            res.get_type()
        )
    }
};

I've signed the CLA so feel free to use or modify it if it seems useful. Thanks again, and looking forward to hearing your response!

The error Not a base 8 integer is returned when I try the follwing:

int("0", 8)
int("-0", 8)
int("+0", 8)

Although these examples work fine and return 0:

int("0", 16)
int("-0")
int("+00", 8)
int("00", 8)

I have pushed a master into a new branch to trigger CI job:

https://travis-ci.org/google/starlark-rust/builds/573286283

Beta and Nightly builds timed out. Any idea why and how to fix it?

The Go implementation accepts a -c flag to provide a program on the command line. This is helpful for quick and dirty "what does this do?" investigation, and for fuzzing. -c was chosen to match python. It'd be nice if the rust repl also accepted -c, instead of requiring a temp file to provide input.

spec: https://github.com/google/starlark-go/blob/master/doc/spec.md#string·elem_ords

$ starlark-go -c "''.elem_ords()"

(exit 0)

$ starlark-rust -c "''.elem_ords()"
error[CV00]: Cannot .elem_ords on type string
 --> [command flag]:1:1
  |
1 | ''.elem_ords()
  | ^^^^^^^^^^^^ .elem_ords not supported for type string

(exit 2)

I assume I'm holding it wrong, but the documentation doesn't tell me how to hold it right.

$ uname -a 
Linux xxx 5.2.17-1rodete3-amd64 #1 SMP Debian 5.2.17-1rodete3 (2019-10-21 > 2018) x86_64 GNU/Linux
$ date
Sun 29 Mar 2020 01:03:20 PM EDT
$ mkdir  /tmp/foo
$ cd /tmp/foo
$ git clone https://github.com/google/starlark-rust
...
Unpacking cargo (0.37.0-3) ...
...

$ cd starlark-rust/
$ cargo run  --example starlark-simple-cli
    Updating crates.io index
   Compiling libc v0.2.68
   Compiling autocfg v0.1.7
   Compiling rand_core v0.4.2
   Compiling proc-macro2 v1.0.9
   Compiling unicode-xid v0.2.0
   Compiling siphasher v0.2.3
   Compiling typenum v1.11.2
   Compiling syn v1.0.17
   Compiling memchr v2.3.3
   Compiling string_cache_shared v0.3.0
   Compiling byteorder v1.3.4
   Compiling log v0.4.8
   Compiling lazy_static v1.4.0
   Compiling serde v1.0.105
   Compiling byte-tools v0.3.1
   Compiling regex-syntax v0.6.17
   Compiling cfg-if v0.1.10
   Compiling ordermap v0.3.5
   Compiling either v1.5.3
   Compiling fixedbitset v0.1.9
   Compiling precomputed-hash v0.1.1
   Compiling bit-vec v0.5.1
   Compiling new_debug_unreachable v1.0.4
   Compiling fake-simd v0.1.2
   Compiling opaque-debug v0.2.3
   Compiling term v0.4.6
   Compiling strsim v0.9.3
   Compiling diff v0.1.12
   Compiling lalrpop-util v0.16.3
   Compiling unicode-xid v0.1.0
   Compiling codemap v0.1.2
   Compiling termcolor v1.1.0
   Compiling linked-hash-map v0.5.2
   Compiling thread_local v1.0.1
   Compiling block-padding v0.1.5
   Compiling phf_shared v0.7.24
   Compiling rand_core v0.3.1
   Compiling rand_jitter v0.1.4
   Compiling itertools v0.8.2
   Compiling rand_pcg v0.1.2
   Compiling rand_chacha v0.1.1
   Compiling rand v0.6.5
   Compiling bit-set v0.5.1
   Compiling petgraph v0.4.13
   Compiling rand_xorshift v0.1.1
   Compiling rand_isaac v0.1.1
   Compiling rand_hc v0.1.0
   Compiling ena v0.11.0
   Compiling aho-corasick v0.7.10
   Compiling rand_os v0.1.3
   Compiling atty v0.2.14
   Compiling codemap-diagnostic v0.1.1
   Compiling generic-array v0.12.3
   Compiling ascii-canvas v1.0.0
   Compiling quote v1.0.3
   Compiling digest v0.8.1
   Compiling block-buffer v0.7.3
   Compiling sha2 v0.8.1
   Compiling phf_generator v0.7.24
   Compiling string_cache_codegen v0.4.4
   Compiling string_cache v0.7.5
   Compiling regex v1.3.6
   Compiling serde_derive v1.0.105
   Compiling docopt v1.1.0
   Compiling lalrpop v0.16.3
   Compiling starlark v0.3.1-pre (/tmp/foo/starlark-rust/starlark)
error[E0432]: unresolved import `std::any::type_name`
  --> starlark/src/values/mod.rs:94:5
   |
94 | use std::any::type_name;
   |     ^^^^^^^^^^^^^^^^^^^ no `type_name` in `any`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0432`.
error: Could not compile `starlark`.

To learn more, run the command again with --verbose.

$

I don't know why, I have those test running without issue on my linux machine but today I used my OS X laptop and they froze. It is not so important I think, as long as those test works on Linux, maybe we should just have a configuration to skip them on macOS?

/cc @illicitonion fyi

From @adonovan:

Strings are UTF-8. (Good.) Not sure why they they print funny:

>>> "Hello, 世界"
'Hello, \u{4e16}\u{754c}'

observed at commit 2682e5a

$ cat a.star
def f():
  x = None
  for i in range(100000):
    x = [x]

f()

$ target/release/starlark-rust a.star 

thread 'main' has overflowed its stack
fatal runtime error: stack overflow
Aborted

There is no obvious recursive function call in this program.

Functions use context of caller function instead of context where they were defined.

Example 1

bb.star:

x = 1

def f():
    print(x)

aa.star

load("bb.star", "f")

f()

% starlark ./aa.star
1

% starlark-repl ./aa.star
error[CM01]: Variable 'x' not found
 --> bb.star:4:11
  |
4 |     print(x)
  |           ^ Variable was not found

Example 2

bb.star

def f():
    print(x)

aa.star

load("bb.star", "f")

x = 1

f()

% starlark ./aa.star
Traceback (most recent call last):
  ./aa.star:1: in <toplevel>
Error: cannot load bb.star: bb.star:2:11: undefined: x

% starlark-repl ./aa.star
1

Example 3

cc.star

def f():
    print(x)

def g():
    x = 1
    f()

g()

% starlark cc.star
cc.star:2:11: undefined: x

% starlark-repl cc.star
1

This is nitpicking, but it'd be nice for users if the output of dir was alphabetized. It is currently not just unalphabetized, but also unstable from run to run:

$ starlark-repl -c "dir([])"
["remove", "extend", "index", "append", "pop", "insert", "clear"]
$ starlark-repl -c "dir([])"
["extend", "insert", "clear", "index", "pop", "remove", "append"]
$ starlark-repl -c "dir([])"
["index", "extend", "insert", "append", "pop", "remove", "clear"]
$ starlark-repl -c "dir([])"
["append", "clear", "extend", "remove", "insert", "index", "pop"]
$ starlark-repl -c "dir([])"
["index", "extend", "pop", "append", "insert", "remove", "clear"]
$ starlark-repl -c "dir([])"
["insert", "clear", "remove", "append", "pop", "extend", "index"]

Python 2/3 and starlark-go all alphabetize.

I was hoping to get plain text output from starlark-repl, for when it is being invoked by another program (fuzz). This is what happens currently:

$ TERM=dumb starlark-repl -c "x" 
thread 'main' panicked at 'failed to emit error: operation not supported by the terminal', /Users/josh/.cargo/registry/src/github.com-1ecc6299db9ec823/codemap-diagnostic-0.1.0/src/emitter.rs:881:27
note: Run with `RUST_BACKTRACE=1` for a backtrace.

$ starlark-rust -c "''.splitlines('0')"
error[CV02]: string.splitlines() expect a bool as first parameter whilegot a value of type string.
 --> [command flag]:1:1
  |
1 | ''.splitlines('0')
  | ^^^^^^^^^^^^^^^^^^ type string while expected bool

(exit 2)

Typo: "whilegot". Perhaps the "while" should just be deleted and replaced with a comma? (And if so, in the inline error message as well.)

In the docs for the eval functions, the build argument is documented as:
"build: set to true if you want to evaluate a BUILD file or false to evaluate a .bzl file"

That doesn't seem particularly meaningful to me. I think it would be helpful to have a more descriptive explanation of the difference somewhere in the docs.

I just tried using starlark-rust for a rewrite/extension of an old side-project of mine, and I've got an experience report in progress: Using the starlark_module! macro requires somewhat unergonomic boilerplate code.

Instead of the following 2015 edition rust (for 2018 edition rust, you have to pull in the macros explicitly, which is a bit more unwieldy since you also have to import all the macros that the starlark_module block expands into):

#[macro_use]
extern crate starlark;
use starlark::values::Value;

// all of the below things are required to expand the macro correctly 
// (I kept adding them until rustc stopped screaming at me):
use starlark::environment::Environment;
use starlark::values::function;
use starlark::values::ValueResult;

starlark_module!{
    ohai =>
        hi(_a) {
            Ok(Value::from(None))
        }
}

I believe this could be avoided by making the macro expansion use the $crate variable (weirdly enough not documented in newer editions of the rust book) to canonize names exported by starlark-rust. I'm happy to draft up a PR, it would be a straightforward search&replace job.

The starlark crate contains not only the core Starlark parser and evaluator, but also the interactive REPL with its huge dependency tree. This sharply limits the utility of starlark-rust as an embedded library.

Following the example of https://github.com/rust-lang-nursery/futures-rs, can the current monolithic crate be split up? Keeping it in the same repo is fine, I just want separate compilation units.

Using latest master (commit 4fc595a) I think I've uncovered a bug in the parsing of triple quoted strings:

Welcome to Starlark REPL, press Ctrl+D to exit.
>>> """foo"bar"""
"foo\"ar"
>>> """foo'bar"""
"foo\'bar"
>>> '''foo'bar'''
"foo\'ar"
>>> '''foo"bar'''
"foo\"bar"

It looks like a single quote character matching the outer triple quoted character somehow results in the character after it getting swallowed. I'm not sure if this a quirk of Starlark or a bug in this implementation. But I figured I should report it.

The starlark-repl binary uses custom getopts parsing to parse its command-line arguments. This requires a lot of hand-written boilerplate, such as manually printing out help. I'd like to convert it to use structopt, where command-line arguments are defined in a structured way to reduce boilerplate. Before I do that, I wanted to see what the maintainers think. Would this change be well-received? Thanks!

I'm currently working with defining my own starlark types, and wanting to write some unit tests for the starlark code. This is effectively what I'm trying to do:

#[test]
pub fn simple_test() {
  {
    let simple = run_starlark_and_get_type::<PortDefinition>("pyra_port('8080')");
    assert_eq!(simple.port, 8080);
    assert_eq!(simple.tcp, true);
  }
}

fn run_starlark_and_get_type<T: starlark::values::TypedValue>(starlark_code: &str) -> starlark::values::cell::ObjectRef<T> {
  let res = run_starlark(starlark_code);
  if let Err(err) = res {
    panic!(
      "Running Starlark Code Error'd:\n-----CODE-------\n{}\n------------\n\n------ERROR-----\n{:?}\n------------\n",
      starlark_code,
      err,
    );
  }
  let value = res.unwrap();

  let downcast_attempt = value.downcast_ref::<T>();
  if downcast_attempt.is_none() {
    panic!(
      "Running Starlark Code Returned Incorrect type. Actual type is: {:?}\n-----CODE-------\n{}\n------------\n",
      value.get_type(),
      starlark_code,
    );
  }

  downcast_attempt.unwrap()
}

fn run_starlark(starlark_code: &str) -> Result<Value> {
  let (mut global_env, mut type_values) = global_environment();
  pyra_module(&mut global_env, &mut type_values);
  global_env.freeze();
  let mut env = global_env.child("simple-unit-test");
  let map = Arc::new(Mutex::new(codemap::CodeMap::new()));

  let result = eval(
      &map,
      "stdin",
      &starlark_code,
      Dialect::Bzl,
      &mut env,
      &type_values,
      global_env.clone(),
  );

  if let Err(err) = result {
      Err(color_eyre::eyre::eyre!(format!("{:?}", err)))
  } else {
      Ok(result.unwrap())
  }
}

Unfortunately this does not work because starlark::values::cell is pub(crate) rather than being pub. Is there any particular reason for this cell to be public crate instead of public in general? Is there an easier way to do this "run this string, and get me this TypedValue"?

It seems the FileLoader implementation accepts a path as a &str. While this is simplistic allowing for someone to simply time load("my_constant_file"), this actually presents a theoretical problem as Linux Path's aren't guaranteed to be UTF8, but rust string types are. Rust offers both Path, and PathBuf for dealing with paths (which do work for non-utf8 paths, and they also handle path separators for you).

I just noticed the string*bool operator:

I wonder: what set of features were you working toward?

PyOxidizer is using this crate to power its configuration files. See https://pyoxidizer.readthedocs.io/en/stable/config_api.html for the current configuration API.

As part of porting PyOxidizer from crate version 0.2 to 0.3, I ran into a roadblock with commit
048cd83 - the one removing access to Environment from call sites and replacing it with TypeValues. And the reason I hit a roadblock is because PyOxidizer is using Environment.get() in a number of places to retrieve global variables in the Environment. e.g. PyOxidizer defines an opaque CONTEXT global variable that wraps a Rust struct holding things like a logger instance to use. Various Starlark functions and methods implemented in Rust do an env.get("CONTEXT") and downcast so they can get access to this internal state.

With the introduction of TypeValues, Rust-implemented Starlark functions can no longer access global variables on the Environment, just type values.

The commit message for 048cd83 calls out specific examples of why things were changed and how the old behavior can get you in trouble. I understand why this was done. However, I'd like to gently push back and question that change.

My reading of the commit message is the new behavior better aligns with Bazel's use of Starlark. That's all fine of course. But Starlark is a generic language and has use cases outside of Bazel and its execution semantics. And my use of Starlark in PyOxidizer purposefully uses Starlark more like a traditional Python script.

Reading the Starlark specifications at https://github.com/google/starlark-go/blob/master/doc/spec.md and https://github.com/bazelbuild/starlark/blob/master/spec.md, I'm not seeing anything that would prohibit a Starlark function from accessing a variable in an outer scope. Actually, my read of the specification is that an inner scope should be able to access symbols from the outer scope. Using the language of this crate, a Rust function implementing a Starlark function should be able to access variables from outer scopes.

Is my reading of the Starlark specification correct? If so, is this crate purposefully diverging from the Starlark specification or was the change to TypeValues which blocked access to outer scope variables an inadvertent departure?

If this crate will only allow access to typed values going forward, may I ask how custom Starlark dialects (like PyOxidizer's config files) should provide access to global state within a Starlark execution context? i.e. from my Rust code implementing a Starlark function, how should that code access special "context" needed to implement that function? One (hacky) solution is with a Rust static variable guarded by a mutex. But I would very much prefer not to use global variables because they are an anti-pattern. Should I instead do something like define a "Context" type and hang off methods like Context.global() that returns a reference to a global variable? But even with this option, I'm not so sure how to implement that any more, as you seemingly need the Rust code behind the function to access an Environment's variable or a Rust static.

Keep up the great work on this crate! I'm hoping to continue porting PyOxidizer to version 0.3 once I figure out how to make it work in the new Environment-less world.

This could come up being handy to be able to be able to build this project with Bazel and/or Pants.

@pcj tried once: google/cargo-raze#64

According to the spec, strings are supposed to support the bytes() method, but trying it in the repl fails:

$ cargo run -- -r
    Finished dev [unoptimized + debuginfo] target(s) in 0.07s                                                                                                           
     Running `target/debug/starlark -r`
Welcome to Starlark REPL, press Ctrl+D to exit.
>>> "foo".bytes()
error[CV00]: Cannot .bytes on type string
 --> <1>:1:1
  |
1 | "foo".bytes()
  | ^^^^^^^^^^^ .bytes not supported for type string

>>> 

As a follow-up of #23, I'll make the codemap dependency optional also because it might not necessarly make sense to use it depending on your application.

(I've been poking around the interpreter to understand Rust.)

The Starlark spec permits recursive data structures (and they do occur in practice), but the Rust implementation causes cycle-forming operations to fail dynamically. For example:

$ cat a.star 

x = [0]
d = dict(a=x)
y = struct(f=d)

x[0] = y

$ cargo run a.star
error[CV14]: This operation create a recursive data structure. Recursive datastructure are disallowed because infinite loops are disallowed in Starlark.
 --> a.star:6:1
  |
6 | x[0] = y
  | ^^^^ Unsupported recursive data structure

The implementation of this assignment check requires traversing the complete object graph of x to see if it contains a references to x itself. This is asymptotically expensive, as it requires pulling the complete object graph of x in from main memory. There's no efficient general way to implement this check, so it doesn't seem like a good language feature, even though it's a neat idea and solves other problems, such as recursive functions (print, hash, equals) getting stuck.

I suspect the real reason for this check is that Rust has no garbage collector. Rather than implement one, you used Rc reference counts to manage the lifetimes of Starlark values, but a known limitation of plain reference counting is that it fails to collect cycles, so you outlawed cycles, contra the spec. (Am I right?)

To support cycles, Starlark-Rust will need a partial tracing garbage collector; take a look at Retracer ("Concurrent Cycle Collection in Reference Counted System", Bacon & Rajan, ECOOP 2001; Googlers: GCL contains a C++ implementation) or you could do what CPython does. You already implemented tracing (is_descendant). In effect, the Rust owner of all values would be the GC.

https://github.com/bazelbuild/starlark/blob/master/spec.md#repr

Originally posted by @LegNeato in #5 (comment)

Hi and thanks for the awesome crate!

It's been more than a year since the last release on crates.io (2018). Would you like to cut a 3.0 release with all the new changes?