fulldecent / structured-acceptance-test Goto Github PK

I merged a PR supposedly adding structured-acceptance-test support to my project, but now I get a nil error:

$ cucumber
Feature: Run example tests

  Scenario: Running example tests            # features/run_example_tests.feature:2
/Users/andrew/.gem/ruby/2.3.7/gems/structured-acceptance-test-0.0.6/lib/finding.rb:57:in `block in categories=': undefined method `include?' for nil:NilClass (NoMethodError)
	from /Users/andrew/.gem/ruby/2.3.7/gems/structured-acceptance-test-0.0.6/lib/finding.rb:55:in `each'
	from /Users/andrew/.gem/ruby/2.3.7/gems/structured-acceptance-test-0.0.6/lib/finding.rb:55:in `categories='
	from /Users/andrew/go/src/github.com/mcandre/cowl/lib/cowl.rb:88:in `to_finding'
	from /Users/andrew/go/src/github.com/mcandre/cowl/lib/cowl.rb:152:in `block in check'
	from /Users/andrew/go/src/github.com/mcandre/cowl/lib/cowl.rb:151:in `each'
	from /Users/andrew/go/src/github.com/mcandre/cowl/lib/cowl.rb:151:in `check'
	from /Users/andrew/go/src/github.com/mcandre/cowl/bin/cowl:72:in `block in main'
	from /Users/andrew/go/src/github.com/mcandre/cowl/bin/cowl:58:in `each'
	from /Users/andrew/go/src/github.com/mcandre/cowl/bin/cowl:58:in `main'
	from /Users/andrew/go/src/github.com/mcandre/cowl/bin/cowl:100:in `<top (required)>'
	from /Users/andrew/.gem/ruby/2.3.7/bin/cowl:23:in `load'
	from /Users/andrew/.gem/ruby/2.3.7/bin/cowl:23:in `<main>'
    Given the program has finished           # features/step_definitions/steps.rb:1
    Then the output is correct for each test # features/step_definitions/steps.rb:12
      
      expected: true
           got: false
      
      (compared using ==)
       (RSpec::Expectations::ExpectationNotMetError)
      features/run_example_tests.feature:4:in `Then the output is correct for each test'

Failing Scenarios:
cucumber features/run_example_tests.feature:2 # Scenario: Running example tests

1 scenario (1 failed)
2 steps (1 failed, 1 passed)

We may want to be compatible with them.

• https://github.com/vfile/vfile-message
• https://github.com/Microsoft/vscode-languageserver-node/blob/release/types/3.13.0/types/src/main.ts#L452

https://github.com/exakat/php-static-analysis-tools

This is the test case:

bash example-process/example-process.bash | ruby stat-reader/stat-reader.rb 

We want to get the expected output:

WARNING: Go to your room
WARNING: Go to your room
FAILURE: Go to your room
SUMMARY: 2 warnings, 1 failure

Importantly, the findings should be presented in realtime as they are output by the process.

• Move stat-reader to example-consumer
• Use our ruby gem
  • If necessary, propose changes to our Gem to support this incremental reading. We may have to use a different JSON library.
• Hard code the output formatting to match the above (issue #26 will introduce a built-in pretty formatting we will use later)
• Make it work as expected, including incremental rendering
• Add a README.md

Accept format like this:

<?xml version='1.0' encoding='UTF-8'?>
<testsuites tests='4' failures='0'>
  <testsuite name='Impostor_Tests.ImpostorGameModel_Tests' tests='3' failures='0'>
    <testcase classname='Impostor_Tests.ImpostorGameModel_Tests' name='testElimanateAPlayer' time='0.067'/>
    <testcase classname='Impostor_Tests.ImpostorGameModel_Tests' name='testGameStatusChangedWhenDoneShowingSecretWords' time='0.001'/>
    <testcase classname='Impostor_Tests.ImpostorGameModel_Tests' name='testStartGameWithSamePlayers' time='0.003'/>
  </testsuite>
  <testsuite name='ImpostorUITests.ImpostorUITests' tests='1' failures='0'>
    <testcase classname='ImpostorUITests.ImpostorUITests' name='testScreenshots' time='31.804'/>
  </testsuite>
</testsuites>

And convert to STAT format.

With a script in the tools directory

Please fully document the Ruby API so it will show up at:

http://www.rubydoc.info/gems/structured-acceptance-test/0.0.6

Also document explicitly under Stat::to_json to make it clear that people should NOT use this output to the terminal.

See this mistake: https://github.com/fulldecent/web-puc/blob/master/lib/web-puc.rb#L111-L113

Also please make a simple use example in the Ruby implementation README file.

Referred to from https://github.com/fulldecent/structured-acceptance-test/blob/master/Stat-Input.md

Currently our Ruby implementation only supports JSON output. This is great for consumers, but linters prefer a more friendly looking output by default.

Let's make a great looking output option and add it into the Ruby implementation. It should look so good that linters will want to use our STAT implementation just to get this beautiful output.

Work plan

• Document all possible output situations and required output formatting
  • Processes
    • A process with only a name
      ⭐️ web-puc
    • A process with a name and a version
      ⭐️ web-puc, version 0.0.2
    • (Other information is not used.)
  • Findings
    • With failure set to true
      🛑 Rule, description (entire line is red)
    • With failure set to false
      ⚠️ Rule, description (entire line is yellow)
    • With rule, description and location (location only has file name)
      ⚠️ Rule, description (entire line is yellow)
      in index2.php
    • With rule, description and location (location is one line)
      ⚠️ Rule, description (entire line is yellow)
      in index2.php, line 24
    • With rule, description and location (location is multiple lines)
      ⚠️ Rule, description (entire line is yellow)
      in index2.php, lines 24-26
    • With rule, description and location (location is columns on one line)
      ⚠️ Rule, description (entire line is yellow)
      in index2.php, line 24:15-18
    • With rule, description and location (location is columns on multiple lines)
      ⚠️ Rule, description (entire line is yellow)
      in index2.php, line 24:15 to line 25:5
    • With recommendation
      ⚠️ Rule, description (entire line is yellow)
      RECOMMENDATION: Reword to avoid the passive voice.
    • (Other finding information is not used.)
  • Summaries
    • No errors, no warnings
      ✅ PASSED with no warnings (whole line is green)
    • No errors, with warnings
      ⚠️ PASSED with 5 warnings (whole line is yellow)
    • With errors, no warnings
      🛑 FAILED with 5 errors (whole line is red)
    • With errors, with warnings
      🛑 FAILED with 5 errors and 1 warning (whole line is red)
• Update our Ruby implement to add this outputting option
• Update example consumer from #25 to use this new outputting format

Follow-on work

• Create pull requests for other projects to use our new Gem and use our pretty output format by default
  • https://github.com/mcandre/lili
  • https://github.com/mcandre/gtdlint
  • https://github.com/mcandre/cowl
  • https://github.com/mcandre/enlint
  • https://github.com/mcandre/aspelllint
  • web-puc
• Begin looking for other projects that we may recommend to adopt STAT.

Will to see

https://mail.google.com/mail/u/0/#inbox/FMfcgxwDsFgnZgZcVnMCjLRDVQjhCjfn

See codeclimate/platform#31

THIS ISSUE IS INCOMPLETE

A STAT data structure is created in #15. Extend this to add methods for incrementally reading a STAT data stream.

When you are reading, call backs will pass back data for the main STAT object and each finding.

The whole point is for this to be asynchronous.

Please design the API and recommend how it should look in an idiosyncratic ruby implementation.

An example computer program is produced that reads and reports the above output

It can do any type of reporting with it

Program should be in Ruby and very simple.

In Ruby, in a new tools directory.

Depends on: #1

http://jsonlines.org/examples/

Remove producer schema. Emit only findings. Standardize file name. Make that file loadable in VS Code.

Our primary supported format is JSON. Only if there is a standard, well documented translation from XML to JSON should we consider this.

If so, then let's also create a XML schema so that valid XML documents can be translated into JSON documents that are valid per JSON schema in #1.

The Atom editor has a [https://github.com/AtomLinter/linter-clang](built in tool for validating c code with clang) as you type. It takes the error output and displays it inline in the editor. Atom is consuming the compiler output.

We want to create a consumer that inhales the same information into a STAT data structure. Then this program will simply print out the STAT in json format.

• Identify which program Atom uses by default to lint c code
• Make a simple c program with a few errors and generate the same linting output
• Write a new tool stat-from-clang it will read clang input and produce stat output
• Validate the output
• Add a README.md

See the working program already in tools/example-process/

Work plan

• Rewrite example-process in Ruby using our gem
• Add a README.md in there

Include pretty format with emoji rendering

This work will go in a new implementations/ruby folder. Eventually this may moved to a new repository.

• Create STAT data structures that Ruby programs can use
• The data structures should support all required and optional parameters designated in the specification
• It should be extensible

Also, please read the specification regarding incremental output. Please add three methods that allow incremental output:

• PRINT HEADER -- prints from the beginning of the output ({) up to the first finding ("findings":[), ends with a new line, and flushes output
• PRINT FINDING -- prints one finding, ends with a new line and flushes output
• PRINT FOOTER -- prints the matching ] and } and flushes output

Then when this is done:

• Make a Gemfile so that people can use this data structure in their projects
• Publish this Gemfile in Rubygems so everyone can use it

Incremental reading will be added to this data structure in a separate issue.

Yup

Depends on #24

Create an Atom linter that uses the example-process program from Atom.

Documentation: https://github.com/steelbrain/linter/tree/v1/docs

Discussion: steelbrain/linter#1298

Then people with STAT-compliant programs will have a VERY simple way to make their program compatible with Atom. They will only need to copy our example linter and change the name of the program that is run.

Work plan

• Make copy of https://github.com/sirbrillig/linter-mixed-indent and get it working in Atom
• Find a javascript library that can read JSON streams
• Call our example-process from the linter
  • You can probably find a better example than linter-mixed-indent which already handles calling an external program
• Update README.md so it is clear how others can copy and easily extend our work to use their own STAT compliant tool

Verify that this tool works correctly by using Atom and creating an animated gif (I use licecap). It should show one finding after 1 second, a second finding after a further 2 seconds and the final finding is shown after a further 3 seconds. It should NOT wait 6 seconds and show all three findings.

Discussion is here http://www.jstorimer.com/blogs/workingwithcode/7766119-when-to-use-stderr-instead-of-stdout

Should this apply to us?