Extracting feature request from #845

As a user or dev I would like to easily

1. create and submit new spec tests for bugs
2. activate passing spec tests
3. push these to personal github repo
4. automatically create a pull request

I already took a first shot at this with gen_libsass_todo. There are quite a few problematic steps involved, mainly due to the interaction with git. The workflow I implemented in that batch file is:

• take issue number from input
• read content for test from %issue%.scss
• fetch latest version from upstream git (i.e. origin)
• create branch todo/issue_%ISSUE% from master
• create folder in todo-tests and copy test case
• run sass-spec to generate files for expected output
• add files to git and commit with standard message
• optionally push to remote repository (2nd argument)

I think this per se is pretty much the minimal workflow I would expect to support devs (and would fulfill point 1). The process is error prone, as it is vital to how we interact with git. One question is how fault-tolerant it must be. IMO it will never be 100% bullet proof, just too many variables in between local and remote. I see most problems on the local side, since we need to create new branches (I guess that is a given), since that is only possible with a clean tree. There are a lot of abort conditions we need to check and the question remains how far we want to try to help the user to resolve any git problems.

I guess in a first shot we need to rely on the help of potential users (probably just "us") to progress from there. If you mess up you may need to remove remote branches that were created so you can push again and so on. My only hope here is that we probably can use a ruby git binding (I suspect such a thing exists?) to make our live easier with interacting with git (therefore giving end-users a better experience).

Ultimately we may want to directly create github issues through automated bug reports (automatically attaching the offending code, if you want to got that far). But in the end this always has to do with authentification at some point. And for me that's were it definitely becomes infeasible.

Initial request/goal for me would be to get something up and running in official spec-runner similar to gen_libsass_todo batch script. For me it has proven to work pretty well, so I personaly am not even really in a hurry to get it baked into official sass-spec runner. But the long term use-case for such a bug reporting system seems appealing, even though I'm not yet sure how this could play out (just think of how firefox etc. include bug reporting tools out of the box nowaday).

I was thinking something like the following

Not sure if it makes sense on a "per suite"/folder, and also include a visualization of some of the options

It's easy to split a spec into a version that targets an older version of Sass and one that targets a newer version, but if this split ends up being incorrect, it's difficult to merge them again. It would be nice to have a way of doing this automatically. Bonus points if --interactive mode detects that the actual output of a failing test is the same as the expected output of a different version and presents an option for merging.

A lot of the extends tests seem to be totally empty and also be repetitively named.

209_test_pseudo_element_superselector
210_test_pseudo_element_superselector
211_test_pseudo_element_superselector
212_test_pseudo_element_superselector
213_test_pseudo_element_superselector
214_test_pseudo_element_superselector

What are these for? Can they be deleted?

/ping @michaek

https://github.com/sass

IMO it would be nice to also check the todo tests on CI. We should not alter the overall error state for these, but it should report the spec tests that are unexpectedly passing. We sometimes add todo tests for segfaults, which should not be included here, since it would abort the whole test runner. Therefore I propose to add tests for segfaults to another folder, that would be excluded from this proposed test run.

From sass/libsass#1865 as it fits better here.
We should already do the right thing but could use these specs:

Each of these produces a parsing error:

@warn;
@error;
@debug;
@warn("foo";
@error("foo";
@debug("foo";

As I suspected, there are some errors which differ for varying output styles. Today the check I've added in the perl-libsass runner (I mentioned that in a slack chat AFAIR) catched a first sample:

foo {
  bar: (rgba(16,16,16,0.5) + #AAA);
}

results

Error: Alpha channels must be equal: rgba(16,16,16,0.5) + #AAA // compressed
Error: Alpha channels must be equal: rgba(16, 16, 16, 0.5) + #AAA // all others

//CC @saper

According to current specification, it says the expected output should be:

Error: Invalid CSS after "  --prop: [{": expected "}", was "];"

But that is incorrect since what we are expecting is an expression. Something more like:

Error: Invalid CSS after "  --prop:": expected expression (e.g. 1px, bold), was "];"

Discovered in #529.

If a spec is nuke'd multiple times, and one run produces an error, the error file is never cleaned up.
I expect the same applies in the inverse. If nuking an error spec multiple times and one mistaken produces output, this output is not cleaned up (untested).

Directories which do not have valid input files (or output files before #494) are silently skipped.
This is how not-really-magic disabled keyword works (renaming input.scss to input.disabled.scss).

> ruby sass-spec.rb -c ~/sw/sass/bin/sass -s spec/libsass-todo-tests/css_error_with_windows_newlines
Recursively searching under directory 'spec/libsass-todo-tests/css_error_with_windows_newlines' for test files to test '/home/saper/sw/sass/bin/sass' with.
Sass 3.4.14.2e0f33b (Selective Steve)
Run options: --seed 19577

# Running:



Finished in 0.001196s, 0.0000 runs/s, 0.0000 assertions/s.

0 runs, 0 assertions, 0 failures, 0 errors, 0 skips

This is confusing especially given there is always a chance for a typo. See also #501 and #505 for funny cases where it confuses people.

I think we should add those directories to "skips" to indicate this. Especially useful when running against a single or a small group of directories.

When I run into an error with --interactive, it would be nice to see the filename being tested so that I have a sense of the feature whose spec isn't passing.

This issue: sass/libsass#442

libsass inserts whitespace when interpolating. Our current iteration of sass-spec ignores whitespace so I can't make a failing test for this issue yet.

Can we have an option when running the tests that allows whitespace to be counted?

My environment:

• sass/node-sass@2fce303 (master as of now)
• libsass 3.2.1 (sass/libsass@e716caa) (master as of now)
• 3dafe99 (master as of now)

(libsass has been compiled separately and linked to the node-sass binding as a shared library)

When running npm test I get the following warning:

      ✓ bool
Warning: `compact` has been removed from libsass because it's not part of the Sass spec
Warning: `compact` has been removed from libsass because it's not part of the Sass spec
Warning: `compact` has been removed from libsass because it's not part of the Sass spec
Warning: `compact` has been removed from libsass because it's not part of the Sass spec
      ✓ bourbon (43ms)
      ✓ calc

but the tests pass. Trying however to run mocha in the coverage testing mode fails immediately:

>  npm run coverage

> [email protected] coverage /home/saper/node_modules/node-sass
> node scripts/coverage.js

Warning: `compact` has been removed from libsass because it's not part of the Sass spec


npm ERR! FreeBSD 10.1-STABLE
npm ERR! argv "node" "/usr/local/bin/npm" "run" "coverage"
npm ERR! node v0.12.2
npm ERR! npm  v2.8.4
npm ERR! code ELIFECYCLE
npm ERR! [email protected] coverage: `node scripts/coverage.js`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] coverage script 'node scripts/coverage.js'.
npm ERR! This is most likely a problem with the node-sass package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node scripts/coverage.js
npm ERR! You can get their info via:
npm ERR!     npm owner ls node-sass
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /home/saper/node_modules/node-sass/npm-debug.log

Maybe the warning should be relaxed or compact test files removed from the spec?

Ping sass/node-sass#904 sass/libsass#1055 #359

It looks like there used to be diff output during failures https://github.com/hcatlin/sass-spec/blob/master/lib/sass_spec/runner.rb#L74

There is also a reference to minitest in the gemfile which also supports a diff comparison

I'd like to track down some of the @extend failures, but it's difficult without seeing the diff.

PS: you can see an example with a Mocha/Node version of the spec testing https://travis-ci.org/nschonni/node-sass/builds/16844931

The following and probably related cases have AFAICS no spec test yet:

$map: ( foo: bar );
test { baz: $map; }

I know we keep discussing various meta-comment-style-shit things to help track issues. But, one of the most annoying things right now is that the "todo" flag only applies to libsass and isn't setup for multiple projects.

How about we drop a file in todo-tests called libsass.todo. So, your test folder /extends/basic might include a new file called rubysass.todo and that would indicate to the ruby people that it's still a known breaking test for them.

Please see #713 and https://travis-ci.org/sass/sass-spec/builds/103013873

They should pass, but fail due to different encoding (spec runner seems not to recognise it correctly)

The tests were originally imported from 3.2's test suite and the tests haven't been updated.

The biggest change required is the variable scoping changes. I have started a branch to work on this called scoped-variables

@xzyfer mentioned the need for this on sass/libsass#521 — and I thought it deserved an issue over here. The current organization of sass-spec tests is quite difficult to sort through, based partially on what feature is being tested and partially on where support already exists.

I think the only organization that makes sense long-term is feature-based. Is there something stopping us from moving completely in that direction? Is there anything we can use as a "full list of features" to work against? Any inherent structure in the code, or somewhere in the RubySass project?

Looks like Libsass has issues with indented syntax that aren't in the scss syntax sass/libsass#845. As is stands we follow the normal procedure of writing a spec because sass-spec.rb can only handle one format per run.

This is likely related to #82, and #46.

/cc @chriseppstein

When running a subprocess with -c, for example sassc or Ruby sass CLI, the value returned in the status variable is a Process::Status, that contains the PID, an exit code and other information.
Ruby sass CLI returns with an exit code 65 when failed with a SyntaxError (in fact any error ocuring during Sass processing, also coming from an @error directive). sassc fails with an exit code 1 in that case.

When using Ruby Sass as a ruby module, we are getting a numeric output: 0 for OK, 1 for a Sass SyntaxError or 2 for some other exception.

This inconsistency prevents us from using the status code in the test framework consistently. For example, when being written to a file Process::Status will produce pid 123123 exit 1 string which is not a simple numeric exit code one can compare against.

We're getting to the point now where spec are being written for stylistic issues. e.g.

#127
#114
#113
#112

It appears that some cases the test-runner doesn't see whitespace as significant, which it is in stylistic terms.

An example of this broken behaviour is https://github.com/sass/sass-spec/pull/127/files. This spec although clearly broken (http://sassmeister.com/gist/9b125ed4200df25a4dd6) passes.

ruby x 3
libsass x 1

It is only updated once it exists - it should be created if requested

@saper has been making a lot of great updates to Sass spec recently. These should be documented in the read me. There is also information on the read me that is no longer correct that should be removed.

As it stands it's not possible to assert an error condition. i.e error on duplicate keys in maps sass/libsass#628

There are cases where a spec like the following should be possible https://github.com/sass/sass-spec/pull/133/files.

Maybe we can redirect error output in the assertion some how. Thoughts?

/cc @lunelson @hcatlin

If there’s bugs we submit on libsass that are supported in the Ruby version, would it be helpful to add tests here? Any chance of creating a quick CONTRIBUTING.md file to make sure that can be done how you’d like? Thanks!

Regenereated all spec tests with ruby sass 3.4.9.
Got a few incompatibilities and I thought I'd post it here:

#   Failed test 'sass-spec t/sass-spec/spec/basic/12_pseudo_classes_and_elements/input.scss'
#          got: 'a b {
#  color: red; }
# a b :first-child, a b :nth-of-type(-2n+1) {
#  blah: bloo; }
# a b :first-child .foo#bar:nth-child(even), a b :nth-of-type(-2n+1) .foo#bar:nth-child(even) {
#  hoo: goo; }
# a b :first-child ::after, a b :nth-of-type(-2n+1) ::after {
#  content: "glux"; color: green; }
# a b :first-child :not(.foo), a b :nth-of-type(-2n+1) :not(.foo) {
#  hoo: boo; }
# a b :first-child :not(:not(:not(.foo[bleeble="blabble"] > .hello, .gluxbux))), a b :nth-of-type(-2n+1) :not(:not(:not(.foo[bleeble="blabble"] > .hello, .gluxbux))) {
#  hoo: boo; }
# a b :first-child a, a b :nth-of-type(-2n+1) a {
#  b: c; }'
#     expected: 'a b {
#  color: red; }
# a b :first-child, a b :nth-of-type(-2n+1) {
#  blah: bloo; }
# a b :first-child .foo#bar:nth-child(even), a b :nth-of-type(-2n+1) .foo#bar:nth-child(even) {
#  hoo: goo; }
# a b :first-child ::after, a b :nth-of-type(-2n+1) ::after {
#  content: "glux"; color: green; }
# a b :first-child :not(.foo), a b :nth-of-type(-2n+1) :not(.foo) {
#  hoo: boo; }
# a b :first-child , a b :nth-of-type(-2n+1) {
#  hoo: boo; }
# a b :first-child a, a b :nth-of-type(-2n+1) a {
#  b: c; }'

#   Failed test 'sass-spec t/sass-spec/spec/basic/16_hex_arithmetic/input.scss'
#          got: 'div {
#  p01: #abc; p02: #aabbcc; p03: #aabbcchello; p04: #abbccd; p05: #aabbdd; p06: #0101ff; p07: blue; p08: cyan; p09: #000000; p10: black; p11: black; p12: yellow; p13: #020202; p14: black; p15: 10-#222222; p16: black; p17: magenta; p18: 10 #232323; p19: 10/#222222; p20: #0b0a0b; p21: white; }'
#     expected: 'div {
#  p01: #abc; p02: #aabbcc; p03: #abchello; p04: #abbccd; p05: #aabbdd; p06: #0101ff; p07: blue; p08: cyan; p09: #000000; p10: black; p11: black; p12: yellow; p13: #020202; p14: black; p15: 10-#222; p16: black; p17: magenta; p18: 10 #232323; p19: 10/#222; p20: #0b0a0b; p21: white; }'

#   Failed test 'sass-spec t/sass-spec/spec/basic/18_mixin_scope/input.scss'
#          got: 'div {
#  a: global x; b: global y; f-a: arg; f-b: global y; f-a: local x changed by foo; f-b: global y changed by foo; f-c: new local z; a: global x; b: global y changed by foo; }'
#     expected: 'div {
#  a: global x; b: global y; f-a: arg; f-b: global y; f-a: local x changed by foo; f-b: global y changed by foo; f-c: new local z; a: global x; b: global y; }'

#   Failed test 'sass-spec t/sass-spec/spec/basic/19_full_mixin_craziness/input.scss'
#          got: 'div {
#  /* begin foo */ margin: 1 2; /* end foo */ /* begin foo */ margin: 1 3; /* end foo */ margin: 1 2 zee; margin: 1 kwd-y kwd-z; }
# div blip {
#  hey: now; }
# div blip {
#  hey: now; }
# div {
#  /* begin hux */ color: global-y; /* begin foo */ margin: called-from-hux global-y; /* end foo */ /* end hux */ }
# div blip {
#  hey: now; }
# div {
#  /* begin hux */ color: calling-hux-again; /* begin foo */ margin: called-from-hux calling-hux-again; /* end foo */ /* end hux */ }
# div blip {
#  hey: now; }
# div {
#  blah: original-bung; }
# div {
#  blah: redefined-bung; }
# div {
#  /* calls to nullary mixins may omit the empty argument list */ blah: redefined-bung; }
# div {
#  /* begin foo */ margin: kwdarg1 kwdarg2; /* end foo */ }
# div blip {
#  hey: now; }
# hoo {
#  color: boo; }
# div {
#  blah: boogoo some other default; }
# div {
#  value: original; }
# div {
#  value: no longer original; }
# div {
#  arg: changed local x; blarg: changed global y; a: global-x; b: changed global y; }'
#     expected: 'div {
#  /* begin foo */ margin: 1 2; /* end foo */ /* begin foo */ margin: 1 3; /* end foo */ margin: 1 2 zee; margin: 1 kwd-y kwd-z; }
# div blip {
#  hey: now; }
# div blip {
#  hey: now; }
# div {
#  /* begin hux */ color: global-y; /* begin foo */ margin: called-from-hux global-y; /* end foo */ /* end hux */ }
# div blip {
#  hey: now; }
# div {
#  /* begin hux */ color: calling-hux-again; /* begin foo */ margin: called-from-hux calling-hux-again; /* end foo */ /* end hux */ }
# div blip {
#  hey: now; }
# div {
#  blah: original-bung; }
# div {
#  blah: redefined-bung; }
# div {
#  /* calls to nullary mixins may omit the empty argument list */ blah: redefined-bung; }
# div {
#  /* begin foo */ margin: kwdarg1 kwdarg2; /* end foo */ }
# div blip {
#  hey: now; }
# hoo {
#  color: boo; }
# div {
#  blah: boogoo some other default; }
# div {
#  value: original; }
# div {
#  value: no longer original; }
# div {
#  arg: changed local x; blarg: changed global y; a: global-x; b: different-global-y; }'

#   Failed test 'sass-spec t/sass-spec/spec/basic/25_basic_string_interpolation/input.scss'
#          got: 'div {
#  blah: "hello 4 world px bloo\n blah"; }'
#     expected: 'div {
#  blah: "hello 4 world px bloon blah"; }'

#   Failed test 'sass-spec t/sass-spec/spec/basic/30_if_in_function/input.scss'
#          got: 'div {
#  content: foo; content: bar; content: foo; content: bar; content: bar; }'
#     expected: 'div {
#  content: foo; content: foo; content: foo; content: foo; content: foo; }'

#   Failed test 'sass-spec t/sass-spec/spec/basic/31_if_in_mixin/input.scss'
#          got: 'div {
#  content: foo; content: bar; content: foo; content: foo; }'
#     expected: 'div {
#  content: foo; content: foo; content: foo; content: foo; }'

#   Failed test 'sass-spec t/sass-spec/spec/basic/38_expressions_in_at_directives/input.scss'
#          got: '@foo 1 2, hux {
#  bar {
#  whatever: whatever; }
#  }'
#     expected: '@foo $x $y, hux {
#  bar {
#  whatever: whatever; }
#  }'

#   Failed test 'sass-spec t/sass-spec/spec/basic/42_css_imports/input.scss'
#          got: '@import url(hux\ bux.css); @import url(foo.css); @import url(bar.css); div {
#  color: red; }
# span {
#  color: blue; }'
#     expected: '@import url(hux bux.css); @import url(foo.css); @import url(bar.css); div {
#  color: red; }
# span {
#  color: blue; }'

#   Failed test 'sass-spec t/sass-spec/spec/basic/48_case_conversion/input.scss'
#          got: 'div {
#  bar: "BLAH"; bar: "BLAH"; bar: "BLAH"; bar: "1232178942"; bar: "øáéíóúüñ¿éàŤDžǂɊɱʭʬѪ҈ݓ"; bar: BLAH; bar: BLAH; bar: BLAH; bar: ""; bar: "blah"; bar: "blah"; bar: "blah"; bar: "1232178942"; bar: "øáéíóúüñ¿éàŤDžǂɊɱʭʬѪ҈ݓ"; bar: blah; bar: blah; bar: blah; bar: ""; }'
#     expected: '@charset "UTF-8"; div {
#  bar: "BLAH"; bar: "BLAH"; bar: "BLAH"; bar: "1232178942"; bar: "├©├í├®├¡├│├║├╝├▒┬┐├®├á┼ñÃàÃé╔è╔▒╩¡╩¼Ð¬Êê¦ô"; bar: BLAH; bar: BLAH; bar: BLAH; bar: ""; bar: "blah"; bar: "blah"; bar: "blah"; bar: "1232178942"; bar: "├©├í├®├¡├│├║├╝├▒┬┐├®├á┼ñÃàÃé╔è╔▒╩¡╩¼Ð¬Êê¦ô"; bar: blah; bar: blah; bar: blah; bar: ""; }'

#   Failed test 'sass-spec t/sass-spec/spec/extend-tests/180_test_basic_extend_loop/input.scss'
#          got: '.bar, .foo {
#  a: b; }
# .foo, .bar {
#  c: d; }'
#     expected: '.foo, .bar {
#  a: b; }
# .bar, .foo {
#  c: d; }'

#   Failed test 'sass-spec t/sass-spec/spec/extend-tests/181_test_three_level_extend_loop/input.scss'
#          got: '.baz, .bar, .foo {
#  a: b; }
# .foo, .baz, .bar {
#  c: d; }
# .bar, .foo, .baz {
#  e: f; }'
#     expected: '.foo, .baz, .bar {
#  a: b; }
# .bar, .foo, .baz {
#  c: d; }
# .baz, .bar, .foo {
#  e: f; }'

#   Failed test 'sass-spec t/sass-spec/spec/libsass/list-evaluation/input.scss'
#          got: 'div {
#  content: red 2/3 blue; content: 2/3; content: number; content: color; /**** 4 ****/ content: 0.5 3/40.83333 7/8; content: 0.5 3/4, 0.83333 7/8; /**** ****/ foo: 1; bar: 2; foo: 2; bar: 3; foo: 0.75; bar: 1.75; /*** ***/ stuff: 1, 2 3/4 5, 6; stuff: 0.25; }'
#     expected: 'div {
#  content: red 2/3 blue; content: 0.66667; content: number; content: color; /**** 4 ****/ content: 0.5 3/40.83333 7/8; content: 0.5 3/4, 0.83333 7/8; /**** ****/ foo: 1; bar: 2; foo: 2; bar: 3; foo: 0.75; bar: 1.75; /*** ***/ stuff: 1, 2 3/4 5, 6; stuff: 0.25; }'
WARNING on line 7, column 2 of t/sass-spec/spec/libsass-closed-issues/issue_308/input.scss:
You probably don't mean to use the color value `orange' in interpolation here.
It may end up represented as #ffa500, which will likely produce invalid CSS.
Always quote color names when using them as strings (for example, "orange").
If you really want to use the color value here, use `"" + $var'.

Error: fetched
        on line 2 of t/sass-spec/spec/libsass-closed-issues/issue_799/input.scss
  Use --trace for backtrace.

#   Failed test 'sass-spec t/sass-spec/spec/libsass-closed-issues/issue_799/input.scss'
#          got: '.test {
#  content: "fetched"; }'
#     expected: ''

#   Failed test 'sass-spec t/sass-spec/spec/scss/default-args/input.scss'
#          got: 'div {
#  value: 1, 2; value: 2, 3; value: 1, 3; }
# div {
#  value: ho; }'
#     expected: 'div {
#  value: 1, 2; value: 2, 3; value: 1, 3; }
# div {
#  value: hey; }'

#   Failed test 'sass-spec t/sass-spec/spec/scss/each-in-function/input.scss'
#          got: 'div {
#  a: 0; b: global each 50% 50% type1 number type2 number each cover circle type1 string type2 string each red blue type1 color type2 color; c: a, b, color, d; }'
#     expected: 'div {
#  a: 0; b: global; c: a, b, color, d; }'

#   Failed test 'sass-spec t/sass-spec/spec/scss/env/input.scss'
#          got: 'div {
#  /* 0 */ font: 0; /* 1 */ font: 1; /* 2 */ font: 2; }
# div span {
#  /* 2 */ font: 2; }
# div p {
#  /* 2 */ font: 2; }
# div {
#  @foo {
#  font: 2; }
# @bar {
#  font: 3; }
#  }
# div {
#  content: "foo"; font: fudge; width: "block for foo!"; }'
#     expected: 'div {
#  /* 0 */ font: 0; /* 1 */ font: 1; /* 2 */ font: 2; }
# div span {
#  /* 2 */ font: 2; }
# div p {
#  /* 2 */ font: 2; }
# @foo {
#  div {
#  font: 2; }
#  }
# @bar {
#  div {
#  font: 3; }
#  }
# div {
#  content: "foo"; font: fudge; width: "block for foo!"; }'

#   Failed test 'sass-spec t/sass-spec/spec/scss/if-in-function/input.scss'
#          got: 'div {
#  content: foo; content: bar; content: foo; content: bar; content: bar; }'
#     expected: 'div {
#  content: foo; content: foo; content: foo; content: foo; content: foo; }'

#   Failed test 'sass-spec t/sass-spec/spec/scss/if-in-mixin/input.scss'
#          got: 'div {
#  content: foo; content: bar; content: foo; content: foo; }'
#     expected: 'div {
#  content: foo; content: foo; content: foo; content: foo; }'

#   Failed test 'sass-spec t/sass-spec/spec/scss/selectors/input.scss'
#          got: 'div span, div p, div span {
#  color: red; }
# div a.foo.bar.foo {
#  color: green; }
# div:nth(-3) {
#  color: blue; }
# @-webkit-keyframes {
#  from {
#  left: 0px; }
# from 10% {
#  whatever: hoo; }
# to {
#  left: 200px; }
#  }
# div {
#  @whatever {
#  blah: blah; stuff {
#  blah: bloh; }
#  }
#  }
# a, b {
#  color: red; }
# a c, a d, b c, b d {
#  height: 10px; }
# a c e, a c f, a d e, a d f, b c e, b c f, b d e, b d f {
#  width: 12px; }'
#     expected: 'div span, div p, div span {
#  color: red; }
# div a.foo.bar.foo {
#  color: green; }
# div:nth(-3) {
#  color: blue; }
# @-webkit-keyframes {
#  from {
#  left: 0px; 10% {
#  whatever: hoo; }
#  }
# to {
#  left: 200px; }
#  }
# @whatever {
#  div {
#  blah: blah; }
# div stuff {
#  blah: bloh; }
#  }
# a, b {
#  color: red; }
# a c, a d, b c, b d {
#  height: 10px; }
# a c e, a c f, a d e, a d f, b c e, b c f, b d e, b d f {
#  width: 12px; }'

#   Failed test 'sass-spec t/sass-spec/spec/scss/unquote/input.scss'
#          got: 'div {
#  a: foo; b: I'm a "fashion" "expert".; c: \"wha; d: column1\tcolumn2; e: 24; f: 37%; g: null; j: 1; k: 2; l: a b; m: a 1, b 2; n: 1; }'
#     expected: 'div {
#  a: foo; b: I'm a "fashion" "expert".; c: \"wha; d: column1tcolumn2; e: 24; f: 37%; g: null; j: 1; k: 2; l: a b; m: a 1, b 2; n: 1; }'

#   Failed test 'sass-spec t/sass-spec/spec/scss-tests/047_test_unknown_directive_bubbling/input.scss'
#          got: '.foo {
#  @fblthp {
#  .bar {
#  a: b; }
#  }
#  }'
#     expected: '@fblthp {
#  .foo .bar {
#  a: b; }
#  }'

#   Failed test 'sass-spec t/sass-spec/spec/scss-tests/135_test_simple_at_root/input.scss'
#          got: '.foo {
#  @at-root {
#  .bar {
#  a: b; }
#  }
#  }'
#     expected: '.bar {
#  a: b; }'

#   Failed test 'sass-spec t/sass-spec/spec/scss-tests/136_test_at_root_with_selector/input.scss'
#          got: '.foo {
#  @at-root .bar {
#  a: b; }
#  }'
#     expected: '.bar {
#  a: b; }'

#   Failed test 'sass-spec t/sass-spec/spec/scss-tests/137_test_at_root_in_mixin/input.scss'
#          got: '.foo {
#  @at-root .bar {
#  a: b; }
#  }'
#     expected: '.bar {
#  a: b; }'

#   Failed test 'sass-spec t/sass-spec/spec/scss-tests/138_test_at_root_in_media/input.scss'
#          got: '@media screen {
#  .foo {
#  @at-root .bar {
#  a: b; }
#  }
#  }'
#     expected: '@media screen {
#  .bar {
#  a: b; }
#  }'

#   Failed test 'sass-spec t/sass-spec/spec/scss-tests/139_test_at_root_in_bubbled_media/input.scss'
#          got: '@media screen {
#  .foo {
#  @at-root .bar {
#  a: b; }
#  }
#  }'
#     expected: '@media screen {
#  .bar {
#  a: b; }
#  }'

#   Failed test 'sass-spec t/sass-spec/spec/scss-tests/140_test_at_root_in_unknown_directive/input.scss'
#          got: '@fblthp {
#  .foo {
#  @at-root .bar {
#  a: b; }
#  }
#  }'
#     expected: '@fblthp {
#  .bar {
#  a: b; }
#  }'

# Looks like you failed 28 tests of 601.
Failed 28/601 subtests 

Whats the best approach here. This may be kinda an edge case but how do you plan to handle sass functions that may have different output across different sass versions?

Example: http://sass-lang.com/documentation/Sass/Script/Functions.html#feature_exists-instance_method

Sometimes when I select the "Migrate copy of the test to pass current version" option in interactive mode, I get the error "Cannot migrate test. Test does not apply to an earlier version"—which makes sense. But then the test runner just starts testing new tests. I'd expect it to let me try choose a different option for the test I tried to migrate, such as updating the expected output.

Currently, every test requires its own directory, with two specifically named files: input.scss and expected_output.css. Because there are a lot of files, this makes reading through the tests and even grouping contextually related tests difficult.

I'd like to suggest some changes that would allow us to name files after their tests, and express multiple assertions per file. We could take our current extend-tests/001_test_basic and put the following in extend-tests/basic.scss:

/*
.foo, .bar {
  a: b; }
*/

.foo {a: b}
.bar {@extend .foo}

Because block comments are preserved in the output, we can compare the output from our scss to the contents of the comment block after compiling. It would be pretty easy to include multiple scss/comment pairs:

/*
.foo, .bar {
  a: b; }
*/

.foo {a: b;}
.bar {@extend .foo;}

/*
.foo, .bar {
  a: b; }
*/

.bar {@extend .foo;}
.foo {a: b;}

I think this would aid comprehensibility, and keep the parts of sass-spec that were derived from Ruby Sass's test cases closer to the structure of those tests. Thoughts?

https://ci.appveyor.com/project/sass/libsass/branch/master/job/myy1tliqmg5fibbl

  1) Failure:
SassSpec::Test#test__sass-spec/spec/libsass-closed-issues/issue_1803/nested [C:/projects/libsass/sass-spec/lib/sass_spec/test.rb:101]:
No error message produced.
--- expected
+++ actual
@@ -1 +1 @@
-"Error: Illegal nesting: Only properties may be nested beneath properties."
+nil

//CC @chriseppstein @xzyfer

While implementing the 3.4 selector functions, I know how sensitive they can be to bad input.
Currently our test (https://github.com/sass/sass-spec/tree/master/spec/libsass-todo-issues/issue_963/selector-functions), are only being passed ideal input values.

Can we add checks to make sure the passed in types are correct (for example passing in "12" in some functions is considered bad.

The ruby sass test suite has some methods for that, but i'm not sure what the format would be to translate those here.

For example:

    assert_error_message("$selectors: At least one selector must be passed for `selector-append'",
      "selector-append()")
    assert_error_message("Can't append \"> .bar\" to \".foo\" for `selector-append'",
      "selector-append('.foo', '> .bar')")
    assert_error_message("Can't append \"*.bar\" to \".foo\" for `selector-append'",
      "selector-append('.foo', '*.bar')")
    assert_error_message("Can't append \"ns|suffix\" to \".foo\" for `selector-append'",
      "selector-append('.foo', 'ns|suffix')")

I wanted to make a website showing libsass's progress so people can see if they can use it yet.

To do this I need a JSON output of the tests with diffs of expected vs actual output. Using minitest as sass-specs' test runner would allow me to make a custom reporter that showed the info I need.

I have a POC over at https://github.com/blopker/sass-spec
The changes are mostly in https://github.com/blopker/sass-spec/blob/master/lib/sass_spec/runner.rb and https://github.com/blopker/sass-spec/blob/master/lib/sass_spec/test.rb

I still need to port all the options over, but it works. Also, it's faster because minitest gives us parallelism for free.

Should I continue to port the options over?

010_test_multiple_extends_with_multiple_extenders_and_single_target and 217_test_parent_and_sibling_extend both fail under Ruby 2.1.0, but not other Ruby versions. This isn't likely to be a significant problem, but we should have a way of documenting this.

(Strange behavior seen #463)

As a consequence, --nuke does not generate error files after #494

  7) Error:
SassSpec::Test#test__expanded_spec/libsass-todo-issues/issue_1418/static:
RuntimeError: Command `sass` did not complete:


    /home/saper/sw/libsass/sass-spec/lib/sass_spec/test.rb:17:in `run_spec_test'
    /home/saper/sw/libsass/sass-spec/lib/sass_spec/test.rb:50:in `block (2 levels) in create_tests'

How should we manage different Sass versions? A few ideas come to mind:

• maintain several branches, one for each major/minor version of Sass. This means that new test cases would frequently be added to multiple branches.
• Have some yaml/json file in each test's directory that specifies what versions it applies to. Or have this as an optional preamble in the input and output sass file(s). It's likely that output would change across versions for the same input. This means that there would no longer be a single input or output file in a particular test, but there would be only one possible per sass version.

Maybe you have other ideas. We also need a way for selecting what versions of Sass the implementation is implementing.

I was looking at the cleaning up the node-sass test suite, but realized we need to keep our sourcemap tests right now.
What do you think of including sourcemaps for the expected.css? Also maybe some option.yml frontmatter to cover the other sourcemap options (inline, relative, etc..)?

The recently added --nuke flag appears to mangle the output before writing.

Mostly targeted towards the recent work done by @chriseppstein.

1. thanks for the work, I like that someone is taking care of sass-spec
2. don't take this feedback too serious, just wanted to give my 2 cents
3. also in regard that perl-libsass (and node-sass) have their own spec runners

• I like the approach with the options.yml in general
• I do not like the split of different out styles on the first glance
  • it increased size considerabely for perl dist (1.3M to 2.1M)
  • not sure if this is fully because of the split, but I guess so
  • counter arg: it's the most common "multiple" option; in old implementation ...
  • ... each expected output was optional (equivalent to the directories now)
  • still after rethinking it for some time, I think that design made sense.
• --run-todo does not make much sense to me in the current state
  • I don't see a use case where runner should abort if todo test does not pass
  • Only thing that comes closed would be with a very specific filter argument
  • I added the behavior I would expect under option --probe-todo
  • Disclaimer: I am not sure if exit code signals error for --run-todo
  • I propse to consider to replace --run-todo with the new behavior.
• Message for --skip confuses me (maybe option name is too generic?)
• I have no real clue what the migrate option does (just glanced the code)
• I like the interactive mode (altough I just barely used it for adding --probe-todo)
• A bit more documentation would be nice (IMO best with some real walk-throughs)
• parallel mode would be nice (in perl-libsass I opted for distinct output files for each test)
  • I tried with stdin, stderr redirect, but it's doomed with fork; and threads are not so portable
  • so I just assign unqiue files for stdin, stderr and parallelize the commands of sassc
  • this way I just need to know the the exit status, which works also well on windows
  • pretty sure ruby will have the same problems, so this has proven to work in perl

Further ideas to improve here:

• Add some workflow to create (and activate) todo spec tests (incl. commits and pushes). I already added a batch file I used to create the missing spec tests under tools folder. I just added it so others may use it if it works for them (and also for myself), but no support is given. Having this workflow automated could help to encourage users to submit their spec tests on their own. In the long shot the spec runner could therefore act as an official bug report tool. I guess I don't have to mention this also allows to catch a lot of possible human errors.

Of course I do not really like that I probably have to update the spec runner in perl-libsass 😉 For now I just exclude offending directories by hardcoding them. I know it's rather optional to have this included in perl-libsass, but the nice thing is that this gets us automated coverage accross a bigger set of systems than with our CI.

Really looking forward to see sass-spec evolve with ruby sass on board!
I marked three items above which I would consider as DoD for this issue.

I'm getting an error from Sass on spec/basic/44_bem_selectors:

Syntax error: Invalid CSS after "  &": expected "{", was "_foo {"

On that syntax error, the test execution is aborted.

Having ported #324 to #494 error testing platform I got the error message mismatch.

Normal execution from the shell:

m> ~/sw/sass/bin/sass spec/libsass-todo-issues/issue_1093/parameter/function/input.scss
Error: Invalid CSS after "$foo: foo(#{": expected expression (e.g. 1px, bold), was "});"
        on line 5 of spec/libsass-todo-issues/issue_1093/parameter/function/input.scss
  Use --trace for backtrace.
m> ~/sw/sassc/bin/sassc spec/libsass-todo-issues/issue_1093/parameter/function/input.scss 
Error: Invalid CSS after "$foo: foo(#{": expected expression (e.g. 1px, bold), was "});"
        on line 5 of spec/libsass-todo-issues/issue_1093/parameter/function/input.scss
>> $foo: foo(#{});
   ----------^

(the first lines seem to be identical)

But the test fails:

SassSpec::Test#test__compact_spec/libsass-todo-issues/issue_1093/parameter/mixin [/home/saper/sw/libsass/sass-spec/lib/sass_spec/test.rb:63]:
Expected did not match error.
--- expected
+++ actual
@@ -1 +1 @@
-"Error: Invalid CSS after \"  @include foo(\#{\": expected expression (e.g. 1px, bold), was \"});\""
+"Error: Invalid CSS after \"...@include foo(\#{\": expected expression (e.g. 1px, bold), was \"});\""

Something gets omitted (probably expanded?) here.

The message to match has been generated correctly via --nuke:

Error: Invalid CSS after "$foo: foo(#{": expected expression (e.g. 1px, bold), was "});"
        on line 5 of /home/saper/sw/libsass/sass-spec/spec/libsass-todo-issues/issue_1093/parameter/function/input.scss
  Use --trace for backtrace.

The --unexpected-pass should consider a spec as unexpectedly passing, if and only if, all the output styles for that spec also pass.

To demonstrate this, please use https://github.com/saper/sass-spec/tree/repro_497 branch (it fixes #495, #496 and includes #494 as well as sample tests for @error and @warn).

Ruby Sass version 3.4.18.4ef8e31 (Selective Steve).

SassC version is sassc: 3.3.0-beta1, libsass: 3.3.0-beta2-1-g1efd, sass2scss: 1.0.3

SassEngineAdapter:

Command to test:

ruby -I <path-to-ruby-sass> sass-spec.rb -s spec/misc/error-directive/

Results:

Standard error: empty
Effect on standard output: https://gist.github.com/a4432027a9f3545531a8
A full exception with a backtrace is returned formatted as a CSS comment

ExecutableEngineAdapter with Ruby Sass:

Command to test:

ruby sass-spec.rb -c <path-to-ruby-sass>/bin/sass -s spec/misc/error-directive/

Results:

Standard error: https://gist.github.com/3c6133b710852b965cb1

Error: Buckle your seatbelt Dorothy, 'cause Kansas is going bye-bye
        on line 1 of /home/saper/sw/libsass/sass-spec/spec/misc/error-directive/input.scss
  Use --trace for backtrace.

Standard output: empty

ExecutableEngineAdapter with Ruby Sass:

Command to test:

ruby sass-spec.rb -c <path-to-sassc>/bin/sassc -s spec/misc/error-directive/

Results:

Standard error: https://gist.github.com/5ac0b9e18252c4906593

Error: Buckle your seatbelt Dorothy, 'cause Kansas is going bye-bye
        on line 1 of spec/misc/error-directive/input.scss
>> @error "Buckle your seatbelt Dorothy, 'cause Kansas is going bye-bye"
   ^

Standard output: empty

Clearly a backtrace should not be used to test against errors. We need to make sure the exception is dumped in one place (probably standard output) and that the format is the same in all cases.

As of the latest refactor the --unexpected-pass flag produces a lot of errors

534) Error:
SassSpec::Test#test__expanded_spec/libsass-todo-issues/issue_1029:
NoMethodError: undefined method `assert_not_equal' for #<SassSpec::Test:0x0000010e1e2670>
    /Users/michael/Projects/Sass/sass-spec/lib/sass_spec/test.rb:39:in `run_spec_test'
    /Users/michael/Projects/Sass/sass-spec/lib/sass_spec/test.rb:50:in `block (2 levels) in create_tests'

/cc @hcatlin

SassEngineAdapter attempts to redirect standard error output coming from the Sass module:

    begin
      captured_stderr = StringIO.new
      real_stderr, $stderr = $stderr, captured_stderr
      begin
        css_output = Sass.compile_file(sass_filename.to_s, :style => style.to_sym)
        [css_output, captured_stderr.string, 0]
      rescue Sass::SyntaxError => e
        ["", "Error: " + e.message.to_s, 1]
      rescue => e
        ["", e.to_s, 2]
      end
    ensure
      $stderr = real_stderr

This unfortunately does not work reliably, since we ask MiniTest to parallelize testing and $stderr redirection is global.

When running some test very fast one notices that sometimes stderr output is not captured, sometime one test captures stderr of two tests etc. This is because at the time we try to redirect
the stderr is already redirect in some other test.

The solution is to:

• abandon SassEngineAdapter (which was introduced in ba737f5 for speed)
• abandon parallel tests
• find better way to capture errors from Sass.

The contents of these directories in spec/ seem very similar, if not exactly the same. Should we refactor to just one directory?

Apparently we have a lot of broken and builds against Sass 3.4.13

https://travis-ci.org/sass/sass-spec/builds/60883789

Can I nuke and fix these?

@mgreter it seems like your test of using non-utf-8 characters broke Ruby Sass... spec/libsass/length