missing print built-in about starlark-rust HOT 9 CLOSED

I'll keep filing bugs bit by bit. In the meantime, here's a stab at instructions. Please ask if you have any questions or if it doesn't work out of the box. It's all too easy to miss steps when giving instructions, and these are only partly tested.

• Install a recent version of Go. Make sure it is on your PATH.
• Fetch Go starlark code and go-fuzz: go get -u go.starlark.net/... github.com/dvyukov/go-fuzz; go install github.com/dvyukov/go-fuzz/go-fuzz; go install github.com/dvyukov/go-fuzz/go-fuzz-build
• Find out your GOPATH by running go env GOPATH. I'll refer to it as just GOPATH from here on.
• Put GOPATH/bin on your PATH, so that you can easily execute go-fuzz-build and go-fuzz.
• Put starlark-repl somewhere on your PATH.
• cd GOPATH/src/go.starlark.net
• git remote add josharian [email protected]:josharian/starlark-go.git
• git fetch josharian
• git checkout josharian/fuzz
• cd fuzz
• Instrument code for fuzzing: go-fuzz-build -o r.zip -tags fuzzrust -func FuzzRust go.starlark.net/fuzz
• Fuzz: go-fuzz -bin=r.zip -dup -workdir=.

This will start printing output as it makes progress. If you experience is like mine, there'll be a bunch of "crashes" reported pretty quickly. You'll want to ctrl-c before they become overwhelming. :) Crashes are in the crashers subdirectory. You'll see there are three files per hash: One with the raw input, one with the input escaped, and one explaining what the Go and Rust output was. Note that you should delete/ignore the first character of the input.

There'll be a bunch of duplicates, so I recommend finding a bug, fixing/filing it, and then deleting the crashers and suppressions directories and restarting.

If you find false positives, you can suppress them in the fuzzing script (fuzz-rust.go). See examples in fuzz.go for how to do that. If you change any of the Go code, you need to re-run go-fuzz-build.

Ask with questions! :)

from starlark-rust.

The reason the print function was not made is it was not needed so far and the spec specify it should be left to the application to define its exact behavior so the function cannot be implemented in the library crate. It however make sense to add it to the REPL binary. Sending a PR for that.

I don't know what you mean by this project isn't ready for fuzzing but I would love to see result of your fuzzing on it :)

from starlark-rust.

I don't know what you mean by this project isn't ready for fuzzing but I would love to see result of your fuzzing on it :)

Apologies if that sounded harsh. I was just surprised that the first thing I tried didn't work.

I just set up a quick and dirty fuzz test. I'm using go-fuzz. I'm using the Go implementation to guide the fuzzing, and then comparing the Go repl vs the Rust repl. As a first pass, I'm just checking whether they disagree about whether an input is valid or not.

I hit another immediate speed bump: starlark-repl always returns a 0 error code, so I have to parse the output to see whether the input was valid. I'll file a new issue for that. You may assume that other issues I file in the near term are all around getting fuzzing up and running.

For the record, I have written zero lines of rust in my life. I've been meaning to learn, but I hope you'll forgive me if I don't send any PRs soon to fix the issues I'm filing. (And if I do send PRs, I'll need some patience from the reviewers!)

from starlark-rust.

Don't worry, I didn't find that harsh, I was curious about the cause of it. I am perfectly fine with you filling all those bugs don't worry. I would love to increase test coverage for it.

from starlark-rust.

Thanks for the -c / exit code fixes. I got a couple dozen mismatches within 15 seconds of starting fuzzing. I've filed a few of them.

If you're interested, I can write up tidy instructions for running this yourself. Otherwise, I'll just keep finding, minimizing, and then filing them incrementally--this helps not overwhelm you or bore me. :)

from starlark-rust.

Instructions are welcomed and precise bugs are even more welcome so whichever you prefer to do is fine with me, this is great feedback overall.

from starlark-rust.

from starlark-rust.

:)

You might also find this simple tool useful:

go get -u github.com/josharian/starbug

It just executes multiple starlark-y implementations and prints the output. Use like this: starbug 'print(1+1)'. Makes writing bug reports easier. :)

from starlark-rust.

You might also find this useful:

go get -u github.com/josharian/rmcrashers

It makes it easy to remove sets of crashers found by go-fuzz, to make triage easier. go-fuzz can turn up a lot of minor variations on a theme very quickly.

We might also want to get set up with fuzzbuzz.io. Lots of TODOs, not enough time. :)

from starlark-rust.