knapsackpro / knapsack Goto Github PK

View Code? Open in Web Editor NEW

507.0 7.0 99.0 1.06 MB

Knapsack splits tests evenly across parallel CI nodes to run fast CI build and save you time.

Home Page: https://knapsackpro.com

License: MIT License

Ruby 99.48% Gherkin 0.52%

ruby rspec cucumber minitest spinach turnip parallelism testing-tools auto-balancing ci-server

knapsack's People

Contributors

Stargazers

Watchers

Forkers

grzester shaley91 comjf seanknox bellyabc123 tmorris-fiksu lynxnathan ciembor chalkdotcom greggroth bf4 hirefrederick quangduc dkniffin cultureamp urbanautomaton harishreddyreddy rlhh jdoconnor rbloor-mdsol spyrbri radwo jnraine greenny sammerset designblooz edwinv tevio k2nr gaurav2728 yyang42 grepme olivierlacan synth radubogdan yosi-dediashvili sherrysarkar rdalin82 rbramwell asu126 500px pebsconsulting vinceano nine-inch-rails calendly aterris root-app booqable tsmmark omergez colinxfleming jho406 trevorrjohn nicolasleger jhnvz naw shipt svpersteve edysarwo vip21 brodock 2rba doboy caalberts autouncle imamfzn shelmire dyet92k prostko mrzasa niborg gogainda hacker0x01 tnir mleonidas ioquatix enzymecorp syllogy isabella232 chetankothari tonywok hixdevs lapoelisacci awallsr rnje-sghx zer0cloud iq-scm mattbrictson emeritus-tech

knapsack's Issues

Workflow for adding/deleting specs?

Am I correct in assuming that If I add or delete a spec, I need to re-create the knapsack report by running the entire test suite?

`uninitialized constant ActiveSupport` with Minitest runner

Hi -

I'm getting the below error running bundle exec rake knapsack:minitest in both my CI and development environments:

Traceback (most recent call last):
	4: from /Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/rake_test_loader.rb:5:in `<main>'
	3: from /Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/rake_test_loader.rb:5:in `select'
	2: from /Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/rake_test_loader.rb:17:in `block in <main>'
	1: from /Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/rake_test_loader.rb:17:in `require'
/Users/ruben/projects/budgit/test/models/article_test.rb:1:in `<top (required)>': uninitialized constant ActiveSupport (NameError)
rake aborted!
Command failed with status (1): [ruby -I"lib:test" -I"/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib" "/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/rake_test_loader.rb" "test/models/article_test.rb" ]
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/testtask.rb:130:in `block (3 levels) in define'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/file_utils.rb:57:in `sh'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/file_utils.rb:105:in `ruby'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/testtask.rb:117:in `block (2 levels) in define'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/file_utils_ext.rb:59:in `verbose'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/testtask.rb:111:in `block in define'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `block in execute'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `each'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `execute'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/task.rb:213:in `block in invoke_with_call_chain'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/task.rb:193:in `invoke_with_call_chain'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/task.rb:182:in `invoke'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/knapsack-1.16.0/lib/knapsack/runners/minitest_runner.rb:28:in `run'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/knapsack-1.16.0/lib/tasks/knapsack_minitest.rake:5:in `block (2 levels) in <main>'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `block in execute'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `each'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `execute'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/task.rb:213:in `block in invoke_with_call_chain'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/task.rb:193:in `invoke_with_call_chain'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/task.rb:182:in `invoke'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/application.rb:160:in `invoke_task'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/application.rb:116:in `each'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/application.rb:116:in `block in top_level'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/application.rb:125:in `run_with_threads'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/application.rb:110:in `top_level'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/application.rb:83:in `block in run'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/application.rb:186:in `standard_exception_handling'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/application.rb:80:in `run'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/rake-12.3.1/exe/rake:27:in `<top (required)>'
/Users/ruben/.rbenv/versions/2.5.1/bin/rake:23:in `load'
/Users/ruben/.rbenv/versions/2.5.1/bin/rake:23:in `<top (required)>'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.16.5/lib/bundler/cli/exec.rb:74:in `load'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.16.5/lib/bundler/cli/exec.rb:74:in `kernel_load'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.16.5/lib/bundler/cli/exec.rb:28:in `run'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.16.5/lib/bundler/cli.rb:424:in `exec'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.16.5/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.16.5/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.16.5/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.16.5/lib/bundler/cli.rb:27:in `dispatch'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.16.5/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.16.5/lib/bundler/cli.rb:18:in `start'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.16.5/exe/bundle:30:in `block in <top (required)>'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.16.5/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
/Users/ruben/.rbenv/versions/2.5.1/lib/ruby/gems/2.5.0/gems/bundler-1.16.5/exe/bundle:22:in `<top (required)>'
/Users/ruben/.rbenv/versions/2.5.1/bin/bundle:23:in `load'
/Users/ruben/.rbenv/versions/2.5.1/bin/bundle:23:in `<main>'
Tasks: TOP => knapsack:minitest_run

I am on Rails 5.2.1, knapsack 1.16.0 and Ruby 2.5.1.

Here is my Minitest configuration:

#test_helper.rb
ENV['RAILS_ENV'] ||= 'test'
require_relative '../config/environment'
require 'rails/test_help'

require 'knapsack'
knapsack_adapter = Knapsack::Adapters::MinitestAdapter.bind
knapsack_adapter.set_test_helper_path(__FILE__)

class ActiveSupport::TestCase
end

I am not running into any problems with the Rspec runner.

Generate report on rails 5

We recently upgraded to rails 5 which no longer uses rake and it seems that the reports are no longer generating. We're using minitest so we would run:

# Step for Minitest
$ KNAPSACK_GENERATE_REPORT=true bundle exec rake test

Now when we run the updated command the reports are not generated:

# Step for Minitest
$ KNAPSACK_GENERATE_REPORT=true bundle exec rails test

We cannot use rake because it no longer works with our tests. Is there another way to generate a report?

Knapsack does handle signals

When i start knapsack rake task rake knapsack:rspecand finish it via ctrl-c capybara webserver is not terminated properly

Knapsack misses tests due to different glob result ordering

We have been using Knapsack to split RSpec on our builds on Travis, and have experienced an issue where some leftover specs would not be run on any of the jobs of a single build. Here is an example of the issue that we are having.

Assume we have the following specs:

spec/a.rb
spec/b.rb
spec/c.rb
spec/d.rb
spec/e.rb

When running, we might get something like the following on run one...

$ export CI_NODE_TOTAL=2
$ export CI_NODE_INDEX=0
$ bundle exec rake "knapsack:rspec[--profile 10]"
...
Leftover specs:
spec/a.rb
spec/c.rb
spec/e.rb

However, on job two, we get something like this...

$ export CI_NODE_TOTAL=2
$ export CI_NODE_INDEX=1
$ bundle exec rake "knapsack:rspec[--profile 10]"
...
Leftover specs:
spec/c.rb
spec/a.rb

Note that "b" is not run on either spec, and that "a" is run on both. Also, note that "a" comes before "c" in the first run, but after "c" in the second run.

The best reason I can come up with is the fact that when Knapsack globs for tests, it does not then order them. This doesn't matter for the tests with times reported, since the report distributor appears to use the test run time to sort before assigning to nodes. However, the order in which files are returned would matter when assigning leftover tests.

I still am not sure exactly why in two separate jobs, Travis is returning the list of files with different orders (I would think that both running on the same operating system, they would return the same order). However, this can all be avoided by sorting the test files in a consistent way, either when they are first retrieved or when determining the leftover tests. That way, Knapsack is not relying on the Dir method's ordering (which according to the documentation "depends on your system") for proper distribution.

Incompatibility with earlier versions of minitest

In earlier version of minitest, Minitest.after_run isn't available. Instead, the method for doing that was Minitest::Unit.after_tests. I will submit a PR soon with a fix for this.

Incremental report generation

As our test suite becomes increasingly large and increasingly parallel, generating a full knapsack report becomes increasingly time-consuming and difficult.

It would be great to be able to run a subset of the tests and to only update their entries in the knapsack report without clearing the rest.

Generate multiple reports and aggregate them

I would like to start coding and contribute to the project the mentioned feature.
Take this scenarios as example:

Having 5 parallel CI containers running the same commit at first time, I would expect that every container generates execution for the files they are executing, and store in a specific folder, that will be aggregated at the end of the proccess.

At the next execution, instead of loading a single report file, it will load every file it can locates on that folder, and aggregate all values, so even the first run will be fast enough.

Compatibility with Turnip Features

I have a test suite that is a mixture of spec files in spec/**/*_spec.rb and Turnip features in spec/**/*.feature. I'm trying to generate the report on TravisCI and with the pattern spec/**/*{_spec.rb,.feature} it correctly runs all of the specs and turnip features; however the generated report doesn't include any of the features. I've looked through the source of this project and didn't spot anything obvious that I thought could be to blame. Do you have any thoughts on how I could go about troubleshooting this?

This is the command I'm using to run the tests:

KNAPSACK_TEST_FILE_PATTERN="spec/**/*{_spec.rb,.feature}" KNAPSACK_GENERATE_REPORT=true bundle exec rspec spec

custom configuration for the rake tasks

I see in the README that you can provide custom configuration in spec_helper.rb, test_helper.rb, etc.

This affects the configuration when running tests and generating a report.

However, how do you provide configuration when running the knapsack rake tasks?

For example, I'm using rspec, and have a custom configuration to change the report_path.

This correctly saves the report where I want it to when I run rspec to generate the report, but how do I tell rake knapsack:rspec to use the alternate report location?

knapsack:rspec passing in --fail-fast option?

Hi,

We are using knapsack on Semaphore CI and loves it. When our dev push their branches, we ran parallel tests on the suite, but for each test, we also want it to have --fail-fast option, stop running test on first failing test. How can I pass in that option?

Thanks.

Segmentation error under ruby 2.5.0

https://github.com/ArturT/knapsack/blob/4d2fb4d283db03c2fb3b20afb697a1e536555dd6/lib/knapsack/presenter.rb#L82

pry(Knapsack::Presenter)> seconds
=> 700.0878098011017

time = Time.at(seconds).gmtime.strftime('%Hh %Mm %Ss') gives me error

gems/knapsack-1.16.0/lib/knapsack/presenter.rb:82: [BUG] Segmentation fault at 0x0000000000004001

send complete test coverage report from circleCI building specs to codeclimate

Hello, I want to send my complete test coverate report to codeclimate, but with knapsak, the tests are splitted, and the report is not complete.

Hava anyway to resolve this problem? 🤔

Tests failing with Rails 5.2 RC1

Currently upgrading an app from Rails 5.1 to Rails 5.2 RC1 and have started to run into some issues that don't exist on our primary/production branches. It appears that there are some issues with routes, as several tests are failing with No route matches /whatever although the routes are working fine (when looking at them in development environment), and Rails 5.1 branches pass these tests with no issues.

Something to note, when running the tests locally (they have some basic assert_routing statements) using bin/rails test test_name, they pass, however they fail when the same tests are run using bundle exec ruby -Itest. How is Minitest within Knapsack running the tests? Is there an internal issue with Knapsack and Rails 5.2?

Hoping you can help, thanks!

run rspec need to add bundle exec

when I use knapsack with rspec. When try to run it with command rspec, it give errors:

/Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/activesupport-4.1.14.1/lib/active_support/dependencies.rb:247:in `require': cannot load such file -- knapsack (LoadError)
    from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/activesupport-4.1.14.1/lib/active_support/dependencies.rb:247:in `block in require'
    from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/activesupport-4.1.14.1/lib/active_support/dependencies.rb:232:in `load_dependency'
    from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/activesupport-4.1.14.1/lib/active_support/dependencies.rb:247:in `require'
    from /Users/fakeuser/Sites/turbo/spec/spec_helper.rb:4:in `<top (required)>'
    from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/activesupport-4.1.14.1/lib/active_support/dependencies.rb:247:in `require'
    from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/activesupport-4.1.14.1/lib/active_support/dependencies.rb:247:in `block in require'
    from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/activesupport-4.1.14.1/lib/active_support/dependencies.rb:232:in `load_dependency'
    from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/activesupport-4.1.14.1/lib/active_support/dependencies.rb:247:in `require'
    from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/rspec-core-3.3.2/lib/rspec/core/configuration.rb:1280:in `block in requires='
    from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/rspec-core-3.3.2/lib/rspec/core/configuration.rb:1280:in `each'
    from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/rspec-core-3.3.2/lib/rspec/core/configuration.rb:1280:in `requires='
    from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/rspec-core-3.3.2/lib/rspec/core/configuration_options.rb:109:in `block in process_options_into'
    from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/rspec-core-3.3.2/lib/rspec/core/configuration_options.rb:108:in `each'
    from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/rspec-core-3.3.2/lib/rspec/core/configuration_options.rb:108:in `process_options_into'
    from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/rspec-core-3.3.2/lib/rspec/core/configuration_options.rb:21:in `configure'
    from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/rspec-core-3.3.2/lib/rspec/core/runner.rb:101:in `setup'
    from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/rspec-core-3.3.2/lib/rspec/core/runner.rb:88:in `run'
    from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/rspec-core-3.3.2/lib/rspec/core/runner.rb:73:in `run'
    from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/rspec-core-3.3.2/lib/rspec/core/runner.rb:41:in `invoke'
    from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/rspec-core-3.3.2/exe/rspec:4:in `<top (required)>'
    from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/activesupport-4.1.14.1/lib/active_support/dependencies.rb:241:in `load'
    from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/activesupport-4.1.14.1/lib/active_support/dependencies.rb:241:in `block in load'
    from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/activesupport-4.1.14.1/lib/active_support/dependencies.rb:232:in `load_dependency'
    from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/activesupport-4.1.14.1/lib/active_support/dependencies.rb:241:in `load'
    from /Users/fakeuser/.rvm/gems/ruby-2.1.5/gems/spring-commands-rspec-1.0.4/lib/spring/commands/rspec.rb:18:in `call'
    from /Users/fakeuser/.rvm/rubies/ruby-2.1.5/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
    from /Users/fakeuser/.rvm/rubies/ruby-2.1.5/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
    from -e:1:in `<main>'

but when run it with bundle exec rspec, it works like a charm.
have anyone know the reason?

Knapsack doesn't work with spring binstubs

Our app has binstubs generated by rails/spring in the application root/bin folder. So for example, here is Rails.root/bin/rspec

#!/usr/bin/env ruby
begin
  load File.expand_path('../spring', __FILE__)
rescue LoadError => e
  raise unless e.message.include?('spring')
end
require 'bundler/setup'
load Gem.bin_path('rspec-core', 'rspec')

So, we execute all commands with the bin/ prefix.

bin/rspec spec
bin/rails c

When we run knapsack

bundle exec knapsack:rspec

We get loads of errors, including undefined_method errors

How can we get Knapsack to play nice with our binstubs?

Allow knapsack to ignore certain specs

I would love to be able to setup knapsack to with a series of tags. This would allow me to skip certain specs.

The reason I want this is because I have a series of specs that are testing an interface with an external service. These specs take 20 minutes to run and I don't want to run them on every push (I use travis).

Basically I want to know if the best way to do this would be to setup knapsack to pass arguments to it through to rspec.
So if I called bundle exec rake knapsack:rspec -t ~really_slow_running_specs it would then call rspec with bundle exec rspec --default-path #{allocator.spec_dir} -- #{allocator.stringify_node_specs} -t ~really_slow_running_specs

Or would it be better to set it in the config?

Want a recipe for working with rails systemtests in your docs?

Hey @ArturT , thanks much for this, it's a lovely library and a huge time-saver for us at DCAF Engineering (and by extension, me personally).

We recently ported some integration tests over to system test (in minitest), and it took a little bit of poking around to get them to show up in the report -- KNAPSACK_GENERATE_REPORT=true bundle exec rake test test:system in CI wound up doing the trick.

I thought I'd make a PR to the docs describing how to get systemtests to show up in the report to pay it forward, but wanted to check with you first and make sure this wouldn't be intrusive. Let me know if this is something that would be helpful.

Merge reports feature

Are there plans to add a merge_report feature so knapsack will update the report file instead of overwriting it?

I'd be happy to put together a PR is the feature is wanted :)

Bug some of the tests in the report are not being executed.

we have "spec/requests/api/artists_spec.rb": 5.043759346008301, in the report but its not actually executed in our CI. I was wondering is there a way to check what's the files being executed?(is it only possible to look through the output in the CI pipeline) And we do find that the problem is fixed after we regenerate the report again so not sure what's wrong(the line in the report was updated). By the way we are using minitest.

cannot load such file -- knapsack After install

Followed the install instructions. I'm able to run regular rspec and generate the knapsack report, but when I try to run the knapsack rake task I get the load error:

Rails 4.2

LoadError:
  cannot load such file -- knapsack

Gemfile

group :development, :test do
...
gem 'knapsack'
...

Rakefile

require File.expand_path('../config/application', __FILE__)

Myapp::Application.load_tasks
Knapsack.load_tasks if defined?(Knapsack)

rails_helper.rb

ENV['RAILS_ENV'] = 'test'

require File.expand_path("../../config/environment", __FILE__)

require 'knapsack'
Knapsack::Adapters::RSpecAdapter.bind

Has anyone else experienced this?

Trailing slash should be removed from `allocator.test_dir` in `Knapsack::Runners::SpinachRunner`

We, at GitLab, have an issue where the --features_path option passed to Spinach is set to features/: https://github.com/ArturT/knapsack/blob/e2e71d3974653bd3333fbf707fa20a8f0d95e209/lib/knapsack/runners/spinach_runner.rb#L15

The problem is that Spinach is not expecting a trailing slash for this option: https://github.com/codegram/spinach/blob/f561ec621dc309c87a60dd2c6ccca48a818ab298/lib/spinach/config.rb#L88

This ends in Spinach failing to require features/support/env.rb first: https://github.com/codegram/spinach/blob/f561ec621dc309c87a60dd2c6ccca48a818ab298/lib/spinach/runner.rb#L136-L139

I believe we should just call .chomp('/') on allocator.test_dir in Knapsack::Runners::SpinachRunner and move on with it. :)

For more detail about the issue we had, see https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10395#note_26752554.

wrong timing recordings

Whenever we run our cukes (roughly 3 minutes to run), we get this (anonymized) output in Travis:

554 scenarios (554 passed)
4545 steps (4545 passed)
2m6.315s


Knapsack report was generated. Preview:
{
  "features/**/*.feature": 1.003159761428833,
  "features/**/*.feature": 1.196033239364624,
  "features/**/recommend.feature": 57018183.49009848,
  "features/**/*.feature": 0.2130286693572998,
  "features/**/*.feature": 0.2160334587097168,
  "features/**/*.feature": 0.45173168182373047,
  "features/**/*.feature": 0.04832601547241211,
  "features/**/*.feature": 1.230623483657837,
  "features/**/*.feature": 0.7682235240936279,
  "features/**/*.feature": 0.10379314422607422,
  "features/**/*.feature": 0.5373373031616211,
  "features/**/*.feature": 2.7034032344818115,
  "features/**/*.feature": 0.32158589363098145,
  "features/**/*.feature": 0.19092035293579102,
  "features/**/*.feature": 0.6193909645080566,
  "features/**/*.feature": 0.5318620204925537,
  "features/**/*.feature": 0.7692575454711914,
  "features/**/*.feature": 0.7230222225189209,
  "features/**/*.feature": 0.495070219039917,
  "features/**/*.feature": 0.4870891571044922,
  "features/**/*.feature": 0.20952224731445312,
  "features/**/*.feature": 0.7473139762878418,
  "features/**/*.feature": 0.47346997261047363,
  "features/**/*.feature": 1.0329043865203857,
  "features/**/*.feature": 1.8328258991241455,
  "features/**/*.feature": 0.8722712993621826,
  "features/**/*.feature": 1.2147345542907715,
  "features/**/*.feature": 0.48812413215637207,
  "features/**/*.feature": 0.42723989486694336,
  "features/**/*.feature": 0.9765002727508545,
  "features/**/*.feature": 0.8365063667297363,
  "features/**/*.feature": 1.4077143669128418,
  "features/**/*.feature": 0.029282331466674805,
  "features/**/*.feature": 0.11276030540466309,
  "features/**/*.feature": 0.5681333541870117,
  "features/**/*.feature": 0.38565659523010254,
  "features/**/*.feature": 1.343261480331421,
  "features/**/*.feature": 0.6901326179504395,
  "features/**/*.feature": 1.1266040802001953,
  "features/**/*.feature": 2.198932647705078,
  "features/**/*.feature": 0.11878585815429688,
  "features/**/*.feature": 0.27373504638671875,
  "features/**/*.feature": 3.770233631134033,
  "features/**/*.feature": 0.2858238220214844,
  "features/**/*.feature": 0.5811190605163574,
  "features/**/*.feature": 0.6995368003845215,
  "features/**/*.feature": 0.6289098262786865,
  "features/**/*.feature": 0.4094107151031494,
  "features/**/*.feature": 0.74979567527771,
  "features/**/email.feature": 5391999.136254549,
  "features/**/*.feature": 0.33169007301330566,
  "features/**/*.feature": 0.2166881561279297,
  "features/**/*.feature": 0.03311800956726074,
  "features/**/*.feature": 0.030640840530395508,
  "features/**/*.feature": 0.17110943794250488,
  "features/**/*.feature": 0.48764514923095703,
  "features/**/*.feature": 0.8722720146179199,
  "features/**/*.feature": 0.555567741394043,
  "features/**/*.feature": 0.9950838088989258,
  "features/**/*.feature": 0.25239014625549316,
  "features/**/*.feature": 0.12649965286254883,
  "features/**/*.feature": 0.1031959056854248,
  "features/**/*.feature": 0.029646635055541992,
  "features/**/*.feature": 0.1288614273071289,
  "features/**/*.feature": 4.771716833114624,
  "features/**/export.feature": 26731273.147525072,
  "features/**/*.feature": 0.10891866683959961,
  "features/**/*.feature": 0.6129484176635742,
  "features/**/*.feature": 0.5339069366455078,
  "features/**/*.feature": 2.2400553226470947,
  "features/**/*.feature": 3.9317946434020996,
  "features/**/*.feature": 1.2024469375610352,
  "features/**/*.feature": 1.5093786716461182,
  "features/**/*.feature": 0.39945054054260254,
  "features/**/*.feature": 0.529149055480957,
  "features/**/*.feature": 0.4336683750152588,
  "features/**/*.feature": 1.2973487377166748,
  "features/**/*.feature": 0.11217498779296875,
  "features/**/*.feature": 1.6423816680908203,
  "features/**/*.feature": 0.3878803253173828,
  "features/**/*.feature": 0.5852077007293701,
  "features/**/*.feature": 1.1968224048614502,
  "features/**/*.feature": 1.0070815086364746,
  "features/**/*.feature": 1.324880599975586,
  "features/**/*.feature": 0.7847650051116943,
  "features/**/*.feature": 0.15419507026672363,
  "features/**/*.feature": 0.1375415325164795,
  "features/**/*.feature": 0.2437891960144043,
  "features/**/*.feature": 0.038895606994628906,
  "features/**/*.feature": 0.07857513427734375,
  "features/**/*.feature": 1.3422677516937256,
  "features/**/*.feature": 0.472412109375,
  "features/**/*.feature": 2.7329916954040527,
  "features/**/*.feature": 0.029615402221679688,
  "features/**/*.feature": 0.7670252323150635,
  "features/**/*.feature": 0.21507048606872559,
  "features/**/search.feature": 7807642.681318045,
  "features/**/*.feature": 3.499208450317383,
  "features/**/*.feature": 0.235215425491333,
  "features/**/*.feature": 0.5279958248138428,
  "features/**/*.feature": 0.21576786041259766,
  "features/**/*.feature": 0.09094381332397461,
  "features/**/*.feature": 0.37752389907836914,
  "features/**/*.feature": 1.363354206085205,
  "features/**/*.feature": 3.0626842975616455,
  "features/**/*.feature": 0.1384115219116211,
  "features/**/*.feature": 7.06137752532959,
  "features/**/*.feature": 1.7372627258300781,
  "features/**/*.feature": 1.0493583679199219,
  "features/**/*.feature": 0.25408387184143066,
  "features/**/*.feature": 4.52888560295105,
  "features/**/*.feature": 0.5256309509277344,
  "features/**/*.feature": 0.2931356430053711,
  "features/**/*.feature": 1.2082080841064453,
  "features/**/*.feature": 0.5562317371368408,
  "features/**/*.feature": 1.2475011348724365,
  "features/**/*.feature": 0.3939666748046875,
  "features/**/*.feature": 0.13701748847961426,
  "features/**/*.feature": 0.28044581413269043,
  "features/**/*.feature": 0.10454559326171875,
  "features/**/*.feature": 7.583114147186279,
  "features/**/*.feature": 3.2411201000213623,
  "features/**/*.feature": 0.423907995223999,
  "features/**/widget.feature": 7807685.043151617
}
Knapsack global time execution for tests: 11h 08m 15s

While some of those features take a while longer, they don't take that much longer, and also the "global time execution" reported by Knapsack is a long way off compared to the real Travis run.

Multiline declaration for circleci

Hello,
I know this is probably not the best place but I couldn't find a link to a discussion in the readme.

I can't get knapsack to work in my cucumber file with parralalism and custom arguments on a multiline yml

test:
  override:
    - mkdir -p $CIRCLE_TEST_REPORTS/cucumber
    # Run critical tests first
    - >
        KNAPSACK_REPORT_PATH=".circleci/knapsack_cucumber_critical_report.json"
        RAILS_ENV=test bundle exec rake "knapsack:cucumber[
          -t @critical
          --format junit --out $CIRCLE_TEST_REPORTS/cucumber/junit.xml
          --format json --out $CIRCLE_TEST_REPORTS/cucumber/tests.cucumber
        ]":
        parallel: true

Gives me errors when parsing on circleci. I've also tried

test:
  override:
    - mkdir -p $CIRCLE_TEST_REPORTS/cucumber
    # Run critical tests first
    - "KNAPSACK_REPORT_PATH=\".circleci/knapsack_cucumber_critical_report.json\"
        RAILS_ENV=test bundle exec rake \"knapsack:cucumber[
          -t @critical
          --format junit --out $CIRCLE_TEST_REPORTS/cucumber/junit.xml
          --format json --out $CIRCLE_TEST_REPORTS/cucumber/tests.cucumber
        ]\"":
        parallel: true

But I run into parsing errors. How did you manage to make multiline declarations with the parallel syntax for circleci ? I saw examples of custom arguments, and how to make the build parallel, but nos how to write that nicely in multiline.

Best error I got was

Don't know how to build task 'knapsack:cucumber[
  -t @critical
  --format junit --out /tmp/circle-junit.syU98Vs/cucumber/junit.xml
  --format json --out /tmp/circle-junit.syU98Vs/cucumber/tests.cucumber
]'

Generate report on minitest

I've got following errors:

$ KNAPSACK_GENERATE_REPORT=true bundle exec rake knapsack:minitest
rake aborted!
Knapsack report file knapsack_minitest_report.json doesn't exist. Please generate report first!
/Users/hong/.rbenv/versions/2.5.3/bin/bundle:23:in `load'
/Users/hong/.rbenv/versions/2.5.3/bin/bundle:23:in `<main>'

Caused by:
Errno::ENOENT: No such file or directory @ rb_sysopen - knapsack_minitest_report.json
/Users/hong/.rbenv/versions/2.5.3/bin/bundle:23:in `load'
/Users/hong/.rbenv/versions/2.5.3/bin/bundle:23:in `<main>'
Tasks: TOP => knapsack:minitest
(See full trace by running task with --trace)

It seemed there is no creating the report file at the first time, just open/write it, in codes. But this doc says like it should be created:
https://docs.knapsackpro.com/ruby/knapsack#common-step

Is it gonna fixed?

I solved this problem with creating an empty json report file manually

Build fails with custom KNAPSACK_TEST_FILE_PATTERN

https://github.com/ArturT/knapsack#how-can-i-run-tests-from-multiple-directories using the approach described here bundle exec rake knapsack:rspec fails with require': cannot load such file -- spec_helper (LoadError) when KNAPSACK_TEST_FILE_PATTERN contains other directories.

What does "leftover" mean?

Can you please update the README to explain the difference between Report specs: and leftover specs:

thanks

Using Knapsack with shoulda_context

Hiya!

Knapsack looks great, thanks!

On small problem - we're using the shoulda-context gem, and it seems like this is interfering with the test timing reporter. The JSON that's being generated is just:

{
  "/home/travis/.rvm/gems/ruby-2.1.4/gems/shoulda-context-1.2.1/lib/shoulda/context/context.rb": 183.92807483673096
}

Is there anything we can do to fix this up? Or is that what we're expecting?

Thanks!

Support for parallel_tests gem?

Would Knapsack work with CircleCI & parallel_tests gem?

Teach Knapsack about Timecop?

One of our longest specs has this line:
before(:all) do
Timecop.freeze(Time.new(2014, 6, 10, 11, 11, 11))
...
end

However, Knapsack them believes that the entire file takes 0 seconds to run.

Would it make sense (and be relatively easy?) for Knapsack to simply bypass Timecop completely when computing metrics? i.e. maybe shell out to the system to get the "real" time, regardless of the shenanigans Timecop is playing w/ Ruby's clock?

Minitest Adapter

This gem looks awesome and solve the problem of balancing tests but on our project we only run tests with minitest. I will be possible to get a minitest adapter?

Suggested practices for running parallel tagged specs?

Hey Artur!

I've read the README section on running specs with tags but I'm not 100% sure this fully describes our use case. Let me tell you how we roll 😄

Before configuring Knapsack we were already using rspec tags to parallelize our build. We have 4 different tags:

ordered - for legacy test that need to run in a predefined order 💀
random - for good tests
hive - for tests that integrate with hive
jruby - for a subset of tests that also has to run on JRuby

What I did when configuring Knapsack was I generated a separate report file for each tagged subset of specs. So we have 4 different files: knapsack_ordered.json, knapsack_hive.json, etc. Based on them we further parallelize each tagged rake task.

Is this a good setup for Knapsack or would you suggest something else? One of my concerns is that such a setup creates a lot of leftover specs and I'm not sure how Knapsack deals with them in mix with tags.

Test sequence random ordering

Apologies if this is not the right place to post this request.

It appears as though knapsack sequences tests in a particular order. The issue with this is it can hide caching issues an dependencies. A more ideal solution would be to randomise test ordering on each node and, potentially, across nodes. This would help catch cross test dependencies.

Cheers

Load Knapsack tasks inside Rakefile

I got some errors when I was using dis Gem. The rake tasks is not being loaded.

I found ou that on the example project you're loading the tasks there: https://github.com/KnapsackPro/rails-app-with-knapsack/blob/master/Rakefile

If that's required, you should update the Documentation.

Docs for circleci 2.0

Docs are helpful for circleci 1.0, but 2.0 has a lot of changes and weak documentation. This is just a request to update docs for those of us migrating to 2.0!

Knapsack crashes because exitstatus is nil

First of all, thank you very much for knapsack and maintaining it!

Since a few days we are getting crashes from knapsack in one of our builds because the exitstatus is nil:

rake aborted!

TypeError: no implicit conversion from nil to integer

/home/travis/build/the/project/vendor/bundle/ruby/2.3.0/gems/knapsack-1.13.1/lib/knapsack/runners/rspec_runner.rb:18:in `exit'

It crashes in rspec_runner.rb#L18 because exit can not handle nil itself.

~~I have yet to find the reason why the exitstatus is nil, but you have any ideas, that would be great.~~
The reason for the exitcode being nil was an infinite recursion in another dependency.

I also tried looking into writing a patch, but I am not sure how knapsack should behave if the exitstatus is nil.

Logger levels cannot be changed

We are using the Knapsack rake task to start our RSpec test suite:

rake knapsack:rspec

Knapsack is putting a lot of information into our build output, due to the Knapsack.logger.level begin set to INFO. I've tried changing this level to a lower setting, but because Knapsack is calling the logger from the rake task, there is no way to influence the logger level.

Is there any way to lower the logger level for the rake task? Maybe an environment variable can be introduced to influence the log level?

Add support for cucumber scenario outline

I will fix that soon.

Test not in the report

Hi,

Thanks for creating this gem, it's very useful. I have a questions regarding what happens when tests are not specified in the report. For example if I have an rspec report that contains:

{
  "spec/A_spec.rb": 3.047823905944824,
  "spec/B_spec.rb": 3.047823905944824
}

And if I add a new test file spec/C_spec.rb but I don't add it to the report will the tests in spec/C_spec.rb be run or will they be ignored and not run?

Thanks

Cucumber 3 support

I'm getting uninitialized constant Cucumber::RbSupport (NameError)

Ignore some test

Hey there!

You can selectively run some tests by specifying their directory, but is there a way to deselect some tests and keep the others?

Thanks!

Rails is not loaded in my spec_helper.rb

I'm using simplecov with my rspec suite, so I want to eager load all my application code so that my coverage stats actually include all the code in my application. Knapsack worked fine before I added the eager loading to my spec_helper, but once I added this, my test suite crashes because Rails is not defined:

in spec_helper.rb

if ENV['UPLOAD_CODECOV']
  require 'simplecov'
  require 'codecov'
  SimpleCov.start 'rails' do
    formatter SimpleCov::Formatter::Codecov
  end

  Rails.application.eager_load!
end

I get this error: uninitialized constant Rails (NameError).

I'm running Knapsack like this: bundle exec rake knapsack:rspec. Also tried with knapsack_pro, with the same results.

Any idea why Rails would not be defined here? Without this line, the specs will run, so at some point in the process Rails does get loaded, just not in my spec_helper.rb.

Thanks!

Support Jenkins

Any idea about having Jenkins support?

Minitest failed with status (1)

Coverage report generated for Unit Tests to /app/coverage. 5422 / 26782 LOC (20.24%) covered.

rake aborted!

Command failed with status (1)

/usr/local/bundle/gems/knapsack-1.15.0/lib/knapsack/runners/minitest_runner.rb:28:in `run'

/usr/local/bundle/gems/knapsack-1.15.0/lib/tasks/knapsack_minitest.rake:5:in `block (2 levels) in <top (required)>'

/usr/local/bundle/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'

/usr/local/bundle/bin/bundle:104:in `load'

/usr/local/bundle/bin/bundle:104:in `<main>'

Tasks: TOP => knapsack:minitest_run

(See full trace by running task with --trace)

Is there something wrong here?

Parallel executing inside Docker

Hi, we moved our app into Docker and are now running the tests inside the container via a compose file. Is it possible to still parallelize the execution? Currently we are doing something like: @docker-compose -f ./docker-compose.test.yml run -e RAILS_ENV=test platform bash -c "bundle exec rake db:create && bundle exec rake db:migrate && bundle exec rake knapsack:rspec
Previously we were able to run with 4 worker nodes on CirceCI 2.0, would love to get back to that.

How can this be used with the Kubernetes Jenkins Plugin that dynamically creates Jenkins slaves

I need to know if anyone has a documented solution for using this with the Kubernetes plugin for Jenkins.

Can't run tests with Cucumber

I tried to use with Cucumber but everytime I try, I get the following error:

undefined method `scenario_outline' for #<Cucumber::RunningTestCase::Scenario:0x0000000667af60> (NoMethodError)

I checked the Cucumber docs and scenario_outline doesn't exist on Cucumber::RunningTestCase::Scenario.
http://www.rubydoc.info/gems/cucumber/Cucumber/RunningTestCase/Scenario

And also, my report is being generated as

Knapsack report was generated. Preview:
{
}

Push Timecop to development dependencies

In the gemspec the Timecop gem is listed as a regular dependency:

  spec.add_dependency 'timecop', '>= 0.1.0'

You probably want to include it only as a development dependency since it's only being used in spec/knapsack/tracker_spec.rb:

https://github.com/ArturT/knapsack/search?utf8=%E2%9C%93&q=Timecop&type=

Rspec callbacks are not called

Hey guys,

I'm trying to use knapsack and semaphoreapp CI to spread the tests across multiple nodes and it looks like the before :each callback from spec_helper.rb is not running before the specs.

localhost

knapsack rspec                                                                                                       ✹chore/knapsack-split-specs ‹2.4.0›

Report specs:
[SOME IRRELEVANT SPEC PATHS]

Leftover specs:
[SOME IRRELEVANT SPEC PATHS]

Knapsack time offset warning enabled!

################## Rspec configure

bundler: failed to load command: rspec (/Users/raducroitoru/.rbenv/versions/2.4.0/bin/rspec)
CountryManager::CountryNotSet: The country_code is not set!
  /Users/raducroitoru/Coding/Work/A/a/lib/country_manager.rb:88:in `block in country'
  /Users/raducroitoru/.rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/request_store-1.3.2/lib/request_store.rb:47:in `fetch'
  /Users/raducroitoru/Coding/Work/A/a/lib/country_manager.rb:88:in `country'
  /Users/raducroitoru/Coding/Work/A/a/app/models/api/v5/search_criteria_adapter.rb:57:in `country_locale'
  /Users/raducroitoru/Coding/Work/A/a/app/models/api/v5/search_criteria_adapter.rb:11:in `criteria'

spec_helper.rb

require 'rubygems'

ENV["RAILS_ENV"] ||= 'test'

require File.expand_path("../../config/environment", __FILE__)
require 'knapsack'
require 'rspec/rails'
require 'email_spec'
require 'vcr'
require 'sidekiq/testing'
require "fakeredis/rspec"
require 'webmock/rspec'
require 'paper_trail/frameworks/rspec'
require 'shoulda/matchers'

# Requires supporting ruby files with custom matchers and macros, etc,
# in spec/support/ and its subdirectories.
Dir[Rails.root.join("spec/support/**/*.rb")].each {|f| require f}

Knapsack::Adapters::RSpecAdapter.bind

Shoulda::Matchers.configure do |config|
  config.integrate do |with|
    with.test_framework :rspec
    with.library :rails
  end
end

RSpec.configure do |config|
  puts "################## Rspec configure"

  config.infer_spec_type_from_file_location!
  config.mock_with :rspec
  config.use_transactional_fixtures = false
  config.include RequestHelpers
  config.include EmailSpec::Helpers
  config.include EmailSpec::Matchers
  config.include CountryHelper
  config.extend WithModel

  # For using focus tags
  config.filter_run :focus => true
  config.run_all_when_everything_filtered = true
  config.tty = true

  config.before :each do
    puts "################## Before each: setting country"

    CountryManager.country = :da
  end

  config.after :each, except: 'zipcode de' do
    puts "################## After each"

    # TODO Sanity stabilization experiment (2015-01, remove again soon)
    expect(CountryManager.country).to eq(:da)
  end
end

Knapsack: 1.13.2
Rbenv with ruby 2.4.0
Rails: 4.2.8
Rspec: 3.5.0
Rspec-rails: 3.5.2

It's working fine with normal rspec command. Is this supported?

How to run tests in parallel locally?

I have a machine with 8+ threads. How can I run my tests in parallel with these reports on my local system?

knapsackpro / knapsack Goto Github PK

knapsack's People

Contributors

Stargazers

Watchers

Forkers

knapsack's Issues

Recommend Projects

Recommend Topics

Recommend Org