dyegocosta / what_to_run Goto Github PK

View Code? Open in Web Editor NEW

26.0 5.0 3.0 444 KB

Regression test selection for Ruby

Ruby 96.60% C 3.40%

regression-tests ruby test-runner testing minitest rspec

what_to_run's People

Contributors

Stargazers

Watchers

Forkers

jstotz mbijon povilasjurcys

what_to_run's Issues

Conditionally collect coverage info during test runs

Currently recommendation

require 'what_to_run/rspec' if ENV['COLLECTION']

We need to make this unnecessary

Add smoke test

Currently the test suite lack of smoke tests. We need at least one smoke test for each supported framework.

Currently I'm envisioning two simple projects as git submodules, a shell script to copy those project to a safe dir, run its test suite, modify a file and run what_to_run and a rspec wrapper to assert its output.

Any input of how to write those tests are very welcome.

Make it work with other Ruby versions

Currently it's only working with Ruby 2.3 (which is currently trunk Ruby)

Argument list too long error

$ vagrant@local:/vagrant/core$ what_to_run rspec
$ /var/lib/gems/2.1.0/gems/what_to_run-1.0.0/lib/what_to_run/runner.rb:17:in `exec': Argument list too long - bundle exec rspec -e <RSpec extensive spec list> (Errno::E2BIG)
    from /var/lib/gems/2.1.0/gems/what_to_run-1.0.0/lib/what_to_run/runner.rb:17:in `run'
    from /var/lib/gems/2.1.0/gems/what_to_run-1.0.0/lib/what_to_run/cli.rb:15:in `run'
    from /var/lib/gems/2.1.0/gems/what_to_run-1.0.0/bin/what_to_run:5:in `<top (required)>'
    from /usr/local/bin/what_to_run:23:in `load'
    from /usr/local/bin/what_to_run:23:in `<main>'

Improve test collection interface

Goes from this COLLECT=1 bundle exec rspec to something like this what_to_run rspec --collect

Predict test to run when uncovered lines are modified

We need to embrace the false positive cases in order not miss tests that depends on lines evaluated on require time

Run all tests that uses this file when the not indicated lines are modified (yes, almost every line)

class Omg
  include Foo::Bar

  method_evaluated_on_require :foo

  def yey
    hey = 'hey' # => don't run "all" test when this line is modified
    puts hey    # => don't run "all" test when this line is modified
  end
end

Apparently there is no way to scape from this 😭

Handle concurrency properly

Currently what_to_run is not working properly when collecting coverage information on a parallel_tests run

Stop using /tmp

Create a .what_to_run directory in the project directory or use the project under test tmp

Add Minitest cli interface

Follow the path introduced on #3. Would be nice to have the refactoring mentioned on that PR

Improve scalability

Make it work regardless the codebase size

Compatibility with simplecov/coveralls

This project uses Coveralls to track code coverage. Coveralls uses simplecov and what_to_run seems to conflict with it. Running bundle exec rspec with require 'what_to_run/rspec' in the spec helper results in the following exception from simplecov:

/Users/jstotz/.rbenv/versions/2.1.5/lib/ruby/gems/2.1.0/gems/simplecov-0.9.2/lib/simplecov.rb:44:in `result': coverage measurement is not enabled (RuntimeError)
    from /Users/jstotz/.rbenv/versions/2.1.5/lib/ruby/gems/2.1.0/gems/simplecov-0.9.2/lib/simplecov.rb:44:in `result'
    from /Users/jstotz/.rbenv/versions/2.1.5/lib/ruby/gems/2.1.0/gems/simplecov-0.9.2/lib/simplecov/configuration.rb:158:in `block in at_exit'
    from /Users/jstotz/.rbenv/versions/2.1.5/lib/ruby/gems/2.1.0/gems/simplecov-0.9.2/lib/simplecov/defaults.rb:54:in `call'
    from /Users/jstotz/.rbenv/versions/2.1.5/lib/ruby/gems/2.1.0/gems/simplecov-0.9.2/lib/simplecov/defaults.rb:54:in `block in <top (required)>'