Comments (14)
It would be good to coordinate with @seantalts about the amount of unit testing this will require and for which components. The old system never really unit tested the generator in any significant way and the AST tests were huge but trivial. Mostly it was testing end-to-end results on parsing grammar fragments to make sure they either parsed (at all, usually not with generated code checks) or generated the correct error messages.
from stanc3.
Yes. If you guys have thoughts on what precisely should be unit tested, I would love to know.
I am currently planning to add some expect tests to make sure that:
- if you strip the decorations of the AST after semantic checking, you end up with the original AST (very minimal test for semantic_check)
- if you parse, pretty print and parse again, that's the same as parsing once (pretty decent test for correctness of the pretty printer, provided that the parser is correct).
What I really don't know is what unit tests to write for the parser. I feel like the semantic check is tested pretty well by the end-to-end tests. The parser, I am less sure about, until we start generating code and testing that. The AST does not need any tests, I would think.
from stanc3.
I'm not sure that's a useful test for the type checker - is that an invariant we care about?
I think the way to think about these more granular tests is that anywhere where a function is doing something pretty tricksy or unobvious, you can add a test that demonstrates its behavior in that circumstance. For me, this often occurs while writing the functions because I actually need to test what I'm writing as I'm writing it. Then, you can just leave the test in.
from stanc3.
I'm not sure that's a useful test for the type checker - is that an invariant we care about?
I think it is an invariant we care about. The semantic check should not change the AST. (Otherwise something is quite clearly wrong.) It should only decorate it or maybe throw. I agree it's a pretty low bar and by no means shows correctness, but it's a good sanity check.
I think the way to think about these more granular tests is that anywhere where a function is doing something pretty tricksy or unobvious, you can add a test that demonstrates its behavior in that circumstance. For me, this often occurs while writing the functions because I actually need to test what I'm writing as I'm writing it. Then, you can just leave the test in.
Thought about that way, I think the only part of the parser that was tricky to get right was the precedence of the operators and the dangling else clauses. So should I write expect tests for those?
For semantic checking, I think the integrations tests have good coverage. Whenever there was something tricky, I've added test models if there weren't any yet.
Does anything else come to mind?
from stanc3.
I think it is an invariant we care about. The semantic check should not change the AST. (Otherwise something is quite clearly wrong.) It should only decorate it or maybe throw. I agree it's a pretty low bar and by no means shows correctness, but it's a good sanity check.
I think this would be an interesting use case to try out Jane Street's quickcheck testing library, since you basically just need to fuzz input and check a simple invariant.
Thought about that way, I think the only part of the parser that was tricky to get right was the precedence of the operators and the dangling else clauses. So should I write expect tests for those?
That sounds good. I can then add some more for anything else I find confusing once the testable hooks are there (e.g. additional Parser.Interpreted.<start symbol>
functions are available).
I think you're generally right about our end-to-end test coverage being good. I am just thinking about these more granular tests as being documentation that is automatically tested to be up-to-date. So in addition to testing anything that was very tricky, you could add tests to illustrate examples and help explain the code.
from stanc3.
Thanks, @seantalts ! Will do.
from stanc3.
I tested that the pretty printer is a one-sided inverse to the parser. Moreover, the integration tests are tracking changes to the expected pretty printed code. This expected output is now probably correct.
Also, I just added unit tests for the parser to check that it does operator precedence correctly.
from stanc3.
@seantalts , I've added a bunch of unit tests for the parser. Could you have a look if they suffice?
On a different note, both @bob-carpenter and I felt like it might be cleaner to move the unit tests to their own file.
from stanc3.
Sure thing! I think you forgot to open a PR though. Could you do that and I'll comment on it?
from stanc3.
Also I think that sounds reasonable to have most unit tests in their own file. I also think it's worth having very short expect (and maybe quickcheck) inline to serve as documentation.
from stanc3.
Will open a PR moving the unit tests to their own files once my current PR is moved. (Would like to avoid merge conflicts.)
from stanc3.
I'm not sure if this solves the merge conflicts thing, but with git you can also base a branch off an existing branch: git checkout existing-branch
and then just create a new branch there: git checkout -b new-branch-name
. If you end up making changes to existing-branch
, you can merge those in with new-branch-name
normally (ie while you're on the new branch, git merge existing-branch
). Conflicts should be minimal.
from stanc3.
Thanks!
from stanc3.
This seems like an interesting test tool: https://github.com/mirage/alcotest
from stanc3.
Related Issues (20)
- [BUG] (Nightly) mac-stanc --version prints %%NAME%%3 ... HOT 1
- [BUG] Build error: This expression has type [ `Use_Sys_unix ]. This is not a function; it cannot be applied. HOT 18
- Improving external C++ integration HOT 4
- SoA optimization can over-promote objects used inside loops to produce slower code
- Expose `hypergeometric_2F1` and `hypergeometric_3F2` HOT 2
- Creating empty arrays with {} HOT 13
- [BUG] Inline matrix construction fails to promote with mixed types
- [BUG] eigenvalues_sym fails to compile in expression context HOT 3
- [BUG] Compiler optimizations can prevent bounds checks
- [BUG] `--standalone-functions` ignores undefined functions (external c++) HOT 4
- [BUG] `--info` does not print function information HOT 9
- [BUG] `transform_inits` code for complex numbers is incorrect.
- [FR] `--standalone-functions` change auto return to plain type? HOT 1
- [BUG] `transform_inits` does not check validity of var_context
- Debug flag `--debug-generate-inits` fails if sizes of parameters are based on data HOT 2
- [FR] UDFs taking another UDF as argument HOT 1
- [FR] Bool data type? HOT 1
- `--filename-in-msg` should also update error strings in generated C++
- [BUG] Array literals in `return` statements may generate wrong type in C++
- [BUG] Tilde syntax does not properly generate promote_scalar for array arguments HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from stanc3.