voxpupuli / beaker Goto Github PK

View Code? Open in Web Editor NEW

367.0 201.0 325.0 13.93 MB

Puppet Acceptance Testing Harness

Home Page: https://groups.google.com/forum/#!forum/puppet-beaker

License: Apache License 2.0

Ruby 97.99% Puppet 0.15% Shell 0.25% XSLT 1.60%

hacktoberfest

beaker's Introduction

Beaker

Beaker is a test harness focused on acceptance testing via interactions between multiple (virtual) machines. It provides platform abstraction between different Systems Under Test (SUTs), and it can also be used as a virtual machine provisioner - setting up machines, running any commands on those machines, and then exiting.

Beaker runs tests written in Ruby, and provides additional Domain-Specific Language (DSL) methods. This gives you access to all standard Ruby along with acceptance testing specific commands.

Installation

See Beaker Installation.

Documentation

Documentation for Beaker can be found in this repository in the docs/ folder.

Tutorials take you by the hand through the steps to setup a beaker run. Start here if you’re new to Beaker or test development.
Concepts discuss key topics and concepts at a fairly high level and provide useful background information and explanation.
Rubydocs contains the technical reference for APIs and other aspects of Beaker. They describe how it works and how to use it but assume that you have a basic understanding of key concepts.
How-to guides are recipes. They guide you through the steps involved in addressing key problems and use-cases. They are more advanced than tutorials and assume some knowledge of how Beaker works.

Beaker Libraries

Beaker functionality has been extended through the use of libraries available as gems. See the complete list for available gems. See the beaker-template documentation for documentation on creating beaker-libraries.

Support & Issues

Please log tickets and issues at our Beaker Issue Tracker. In addition there is an active #puppet-dev channel on Freenode.

For additional information on filing tickets, please check out our CONTRIBUTOR doc, and for ticket lifecycle information, check out our ticket process doc.

Contributing

If you'd like to contribute improvements to Beaker, please see CONTRIBUTING.

Maintainers

For information on project maintainers, please check out our CODEOWNERS doc.

Transfer Notice

This plugin was originally authored by Puppet Inc. The maintainer preferred that Puppet Community take ownership of the module for future improvement and maintenance. Existing pull requests and issues were transferred over, please fork and continue to contribute here.

Previously: https://github.com/puppetlabs/beaker

License

This gem is licensed under the Apache-2 license.

Release information

To make a new release, please do:

update the version in the gemspec file
Install gems with bundle install --with release --path .vendor
generate the changelog with bundle exec rake changelog
Check if the new version matches the closed issues/PRs in the changelog
Create a PR with it
After it got merged, push a tag. GitHub actions will do the actual release to rubygems and GitHub Packages

beaker's People

Contributors

Stargazers

Watchers

Forkers

anodelman apchamberlain hunner jpartlow kbarber branan mosesmendoza justinstoller kylog zaphod42 pcarlisle nicklewis aperiodic alexharv074 joshcooper johnduarte jhoblitt sebastiansucker richardc colinpl nivensc simonswine shrug waynr cyberious liamjbennett carthik ngsdevops domcleal underscorgan jhaals stepanstipl thomasleyer smcclellan treydock nocturnalwarz jpadams frogbotherer ferventcoder cartesian-com iristyle diggalabs jantman jtopper leoc mayflower vertilon lunkdjedi greynolds123 paschdan kevpl michft dunse mylezeem rapaul jalvarezsamayoa ericwilliamson sschneid mcanevet mikaelsmith logicminds moutons jlambert121 patrobinson c10l madandroid chris-reeves auto-nation citkz tvpartytonight supercow frizop ticsucty carlossg alexjfisher chundu rlinehan dsbaars mcallaway tsiduia fiddyspence carlosvelarde stack72 leemachine jonnytdevops hohum bknowles burlito garrettrowell jeffmccune demophoon ofsole cwcowellshah dgolja nibalizer windowsrefund mullr nodog choffee foodity

beaker's Issues

beaker generates invalid junit xml content

Hi,

Not 100% sure if its beaker but in some cases it seems to create invalid junit xml content causing Jenkins junit parser to fail.
Details: https://gist.github.com/electrical/21e61b96b99778e40141

Docker support

Would be nice to have docker support in here.
We've seen that using vagrant + Virtualbox is a bad combo for doing parallel testing.
Docker should solve that.

Adding a utility method for adding host entries

As much as this is a infrastructure-smell in the linux world, adding
hosts entries is still all to common in the windows world. This method
allows host entries to be configured to meet that need.

It can also be used when developing with vagrant to make sure that the
master and agent vms can see each other properly.

User-Defined Environment Variables

Hi @anodelman,

Thanks for submitting a bug to fix the issue I had in #210. I've tried your patch, and while it's fixed the issue of Beaker not running at all, I'm still seeing some odd behaviour.

Below is what I have in my test file (the nodesets have specific versions of Ruby installed, and I specify what it is through the ruby_version key underneath CONFIG:):

apply_manifest_on hosts, manifest, { :environment => { 'PATH' => "/usr/local/rvm/gems/#{options['ruby_version']}/bin:/usr/local/rvm/gems/#{options['ruby_version']}@global/bin:/usr/local/rvm/rubies/#{options['ruby_version']}/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/rvm/bin:/root/bin"},
                                     :catch_failures => true }

The above results in this command being executed on my SUTs:

puppet-stack 17:32:01$  env PATH="/usr/local/rvm/gems/ruby-2.0.0-p451/bin:/usr/local/rvm/gems/ruby-2.0.0-p451@global/bin:/usr/local/rvm/rubies/ruby-2.0.0-p451/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/rvm/bin:/root/bin" PATH="/usr/bin:/opt/puppet-git-repos/hiera/bin:${PATH}" RUBYLIB="/opt/puppet-git-repos/hiera/lib:/opt/puppet-git-repos/hiera-puppet/lib:${RUBYLIB}" puppet apply --verbose --detailed-exitcodes /tmp/apply_manifest.pp.zoNoAW

It looks like PATH is being specified twice, and that what I set isn't taking precedence as I see these warnings still during my SUT's Puppet run.

Warning! PATH is not properly set up, '/usr/local/rvm/gems/ruby-2.0.0-p451/bin' is not at first place,
         usually this is caused by shell initialization files - check them for 'PATH=...' entries,
         it might also help to re-add RVM to your dotfiles: 'rvm get stable --auto-dotfiles',
         to fix temporarily in this shell session run: 'rvm use ruby-2.0.0-p451'.

Do the vagrant VM's really need to have 1GB memory?

Currently the Vagrant Vm is made with

 v_file << "    vb.customize [\"modifyvm\", :id, \"--memory\", \"1024\"]\n"

They should run fine with 512MB..

Several Puppet Labs internal settings are hard-coded

https://github.com/puppetlabs/beaker/blob/master/lib/beaker/host_prebuilt_steps.rb#L20-L23

archive-tar-minitar gem conflicts with minitar

When I try to install the latest beaker gem (1.12.1), archive-tar-minitar's "minitar" executable conflicts with minitar's executable of the same name. Archive-tar-minitar comes from beaker's docker-api dependency, while minitar comes from blimpy:

I haven't figured out what actual impact it has if you go ahead and overwrite it.

Example on ruby 2.1.2

[daniel@x220 beaker]$ rvm use 2.1
Using /home/daniel/.rvm/gems/ruby-2.1.2
[daniel@x220 beaker]$ gem install beaker
Fetching: net-ssh-2.9.1.gem (100%)
Successfully installed net-ssh-2.9.1
Fetching: net-scp-1.2.1.gem (100%)
Successfully installed net-scp-1.2.1
Fetching: inifile-2.0.2.gem (100%)
Successfully installed inifile-2.0.2
Fetching: trollop-2.0.gem (100%)
Successfully installed trollop-2.0
Fetching: builder-3.2.2.gem (100%)
Successfully installed builder-3.2.2
Fetching: nokogiri-1.5.10.gem (100%)
Building native extensions.  This could take a while...
Successfully installed nokogiri-1.5.10
Fetching: rbvmomi-1.8.1.gem (100%)
Successfully installed rbvmomi-1.8.1
Fetching: excon-0.34.0.gem (100%)
Successfully installed excon-0.34.0
Fetching: formatador-0.2.5.gem (100%)
Successfully installed formatador-0.2.5
Fetching: mime-types-1.25.1.gem (100%)
Successfully installed mime-types-1.25.1
Fetching: ruby-hmac-0.4.0.gem (100%)
Successfully installed ruby-hmac-0.4.0
Fetching: fog-1.11.1.gem (100%)
Successfully installed fog-1.11.1
Fetching: thor-0.19.1.gem (100%)
Successfully installed thor-0.19.1
Fetching: minitar-0.5.4.gem (100%)
Successfully installed minitar-0.5.4
Fetching: blimpy-0.6.7.gem (100%)
Successfully installed blimpy-0.6.7
Fetching: CFPropertyList-2.2.8.gem (100%)
Successfully installed CFPropertyList-2.2.8
Fetching: fission-0.5.0.gem (100%)
Successfully installed fission-0.5.0
Fetching: addressable-2.3.6.gem (100%)
Successfully installed addressable-2.3.6
Fetching: launchy-2.4.2.gem (100%)
Successfully installed launchy-2.4.2
Fetching: retriable-1.4.1.gem (100%)
Successfully installed retriable-1.4.1
Fetching: jwt-1.0.0.gem (100%)
Successfully installed jwt-1.0.0
Fetching: extlib-0.9.16.gem (100%)
Successfully installed extlib-0.9.16
Fetching: multi_json-1.10.1.gem (100%)
Successfully installed multi_json-1.10.1
Fetching: multipart-post-2.0.0.gem (100%)
Successfully installed multipart-post-2.0.0
Fetching: faraday-0.9.0.gem (100%)
Successfully installed faraday-0.9.0
Fetching: autoparse-0.3.3.gem (100%)
Successfully installed autoparse-0.3.3
Fetching: uuidtools-2.1.4.gem (100%)
Successfully installed uuidtools-2.1.4
Fetching: signet-0.5.0.gem (100%)
Successfully installed signet-0.5.0
Fetching: google-api-client-0.7.1.gem (100%)
Successfully installed google-api-client-0.7.1
Fetching: aws-sdk-1.42.0.gem (100%)
Successfully installed aws-sdk-1.42.0
Fetching: archive-tar-minitar-0.5.2.gem (100%)
archive-tar-minitar's executable "minitar" conflicts with minitar
Overwrite the executable? [yN]

Example on ruby 1.9.3

[daniel@x220 beaker]$ rvm use 1.9.3
Using /home/daniel/.rvm/gems/ruby-1.9.3-p547
[daniel@x220 beaker]$ rvm gemset create beaker-install
ruby-1.9.3-p547 - #gemset created /home/daniel/.rvm/gems/ruby-1.9.3-p547@beaker-install
ruby-1.9.3-p547 - #generating beaker-install wrappers.......r..
[daniel@x220 beaker]$ rvm gemset use beaker-install
Using ruby-1.9.3-p547 with gemset beaker-install
[daniel@x220 beaker]$ gem install beaker
Fetching: json-1.8.1.gem (100%)
Building native extensions.  This could take a while...
Successfully installed json-1.8.1
Fetching: net-ssh-2.9.1.gem (100%)
Successfully installed net-ssh-2.9.1
Fetching: net-scp-1.2.1.gem (100%)
Successfully installed net-scp-1.2.1
Fetching: inifile-2.0.2.gem (100%)
Successfully installed inifile-2.0.2
Fetching: trollop-2.0.gem (100%)
Successfully installed trollop-2.0
Fetching: builder-3.2.2.gem (100%)
Successfully installed builder-3.2.2
Fetching: nokogiri-1.5.10.gem (100%)
Building native extensions.  This could take a while...
Successfully installed nokogiri-1.5.10
Fetching: rbvmomi-1.8.1.gem (100%)
Successfully installed rbvmomi-1.8.1
Fetching: excon-0.34.0.gem (100%)
Successfully installed excon-0.34.0
Fetching: formatador-0.2.5.gem (100%)
Successfully installed formatador-0.2.5
Fetching: mime-types-1.25.1.gem (100%)
Successfully installed mime-types-1.25.1
Fetching: ruby-hmac-0.4.0.gem (100%)
Successfully installed ruby-hmac-0.4.0
Fetching: fog-1.11.1.gem (100%)
Successfully installed fog-1.11.1
Fetching: thor-0.19.1.gem (100%)
Successfully installed thor-0.19.1
Fetching: minitar-0.5.4.gem (100%)
Successfully installed minitar-0.5.4
Fetching: blimpy-0.6.7.gem (100%)
Successfully installed blimpy-0.6.7
Fetching: CFPropertyList-2.2.8.gem (100%)
Successfully installed CFPropertyList-2.2.8
Fetching: fission-0.5.0.gem (100%)
Successfully installed fission-0.5.0
Fetching: addressable-2.3.6.gem (100%)
Successfully installed addressable-2.3.6
Fetching: launchy-2.4.2.gem (100%)
Successfully installed launchy-2.4.2
Fetching: retriable-1.4.1.gem (100%)
Successfully installed retriable-1.4.1
Fetching: jwt-1.0.0.gem (100%)
Successfully installed jwt-1.0.0
Fetching: extlib-0.9.16.gem (100%)
Successfully installed extlib-0.9.16
Fetching: multi_json-1.10.1.gem (100%)
Successfully installed multi_json-1.10.1
Fetching: multipart-post-2.0.0.gem (100%)
Successfully installed multipart-post-2.0.0
Fetching: faraday-0.9.0.gem (100%)
Successfully installed faraday-0.9.0
Fetching: autoparse-0.3.3.gem (100%)
Successfully installed autoparse-0.3.3
Fetching: uuidtools-2.1.4.gem (100%)
Successfully installed uuidtools-2.1.4
Fetching: signet-0.5.0.gem (100%)
Successfully installed signet-0.5.0
Fetching: google-api-client-0.7.1.gem (100%)
Successfully installed google-api-client-0.7.1
Fetching: aws-sdk-1.42.0.gem (100%)
Successfully installed aws-sdk-1.42.0
Fetching: archive-tar-minitar-0.5.2.gem (100%)
archive-tar-minitar's executable "minitar" conflicts with minitar
Overwrite the executable? [yN]

GCE metadata value not specified

When I run the beaker using GCE as provisioner, I have the following error.

<Beaker::GoogleComputeHelper::GoogleComputeError: Bad Request: {

"error": {
"errors": [
{
"domain": "global",
"reason": "required",
"message": "Required field 'value' not specified"
}
],
"code": 400,
"message": "Required field 'value' not specified"
}
}

/Users/tongqingqiu/Documents/DevTools/beaker/lib/beaker/hypervisor/google_compute_helper.rb:154
/Users/tongqingqiu/Documents/DevTools/beaker/lib/beaker/hypervisor/google_compute_helper.rb:331
/Users/tongqingqiu/Documents/DevTools/beaker/lib/beaker/hypervisor/google_compute.rb:60
/Users/tongqingqiu/Documents/DevTools/beaker/lib/beaker/hypervisor/google_compute.rb:47

I think when constructing the metadata request, if value is missing, then that <key, value> should be ignored.

SCP errors with vagrant vmware_fusion provisioner

I setup Beaker to test a module (so unsure if this is a beaker or beaker-rspec error), and ran into an scp error. The setup is:

OS: 10.9 Mavericks
Vagrant: 1.4.3
Beaker: 1.9.1

The biggest thing that causes this is setting the vmware_fusion provider to vagrant to be the default provider by exporting the environment variable of: VAGRANT_DEFAULT_PROVIDER=vmware_fusion

When you kick off the beaker tests with bundle exec rspec spec/acceptance in the root of the module, you get scp blowing up according to this gist --> https://gist.github.com/glarizza/b2fc61a52a0c3f5edbff

For anyone googling for the error, it is:

/Library/Ruby/Gems/2.0.0/gems/net-scp-1.2.0/lib/net/scp.rb:365:in `block (3 levels) in start_command': SCP did not finish successfully (1):  (Net::SCP::Error)
    from /Library/Ruby/Gems/2.0.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/channel.rb:591:in `call'
    from /Library/Ruby/Gems/2.0.0/gems/net-ssh-2.8.0/lib/net/ssh/connection/channel.rb:591:in `do_close'

[DOC] Please add documentation on how to setup Windows boxes for Beaker Support

Now that we are supporting Windows. But it might even be more helpful to know what is needed so that a linux box could also interact with a windows box using beaker (the way Puppet Labs FOSS team does it now with our acceptance tests).

Docker requires "Privileged" for some operations

Using the Docker hypervisor support I've found that a container needs to be set with "Privileged" => true during creation or some parts of the tests will alway fail, such as creating iptables rules.

Example using centos6:centos based container

Error: Could not prefetch firewall provider 'iptables': Execution of '/sbin/iptables-save' returned 1: FATAL: Could not load /lib/modules/3.5.0-51-generic/modules.dep: No such file or directory
iptables-save v1.4.7: Cannot initialize: Permission denied (you must be root)

After applying this change, these errors are no more.

--- a/lib/beaker/hypervisor/docker.rb
+++ b/lib/beaker/hypervisor/docker.rb
@@ -32,7 +32,7 @@ module Beaker
         })

         @logger.debug("Starting container #{container.id}")
-        container.start({"PublishAllPorts" => true})
+        container.start({"PublishAllPorts" => true, "Privileged" => true})

         # Find out where the ssh port is from the container
         ip   = container.json["NetworkSettings"]["Ports"]["22/tcp"][0]["HostIp"]

Should this be a new option for the docker hypervisor, with default remaining 'false' for backwards compatibility, or would setting this 'true' be considered a bug fix?

Enable natresolver and natdnsproxy for vagrant.

There are cases where tests run will fail when trying to download
from the outside world due to dns issues. In order to resolve this
I configured the natdnsproxy1 and natdnshostresolver1 vagrant settings

installation issue

When I install beaker using gem install beaker
I have some errors when running beaker.

There is how to fix that

gem uninstall minitest
gem install minitest -v 4.7.5
gem uninstall jwt
gem install jwt -v 0.1.5

Not very familiar with all gem dependencies management.

Make SCP a bit smarter or replace it with rsync

One of the frequent issues we run into is that scp will take inordinate amounts of time, because there's a recursive symlink somewhere and it tries, obediently, and stupidly to follow it.

Another issue I found today is something like this:

centos-65-x64.local executed in 0.07 seconds
localhost $ scp /home/igalic/src/puppet/puppetlabs-mysql centos-65-x64.local:/etc/puppet/modules/mysql {}

Finished in 4.06 seconds
0 examples, 0 failures

Finished in 4.06 seconds
0 examples, 0 failures
/var/lib/gems/1.9.1/gems/net-scp-1.1.2/lib/net/scp/upload.rb:65:in `initialize': Permission denied - /home/igalic/src/puppet/puppetlabs-mysql/.vagrant/beaker_vagrant_files/ubuntu-server-12042-x64.yml/core (Errno::EACCES)
        from /var/lib/gems/1.9.1/gems/net-scp-1.1.2/lib/net/scp/upload.rb:65:in `open'
        from /var/lib/gems/1.9.1/gems/net-scp-1.1.2/lib/net/scp/upload.rb:65:in `upload_file_state'
        from /var/lib/gems/1.9.1/gems/net-scp-1.1.2/lib/net/scp/upload.rb:38:in `upload_current_state'
        from /var/lib/gems/1.9.1/gems/net-scp-1.1.2/lib/net/scp/upload.rb:104:in `next_item_state'
        from /var/lib/gems/1.9.1/gems/net-scp-1.1.2/lib/net/scp.rb:394:in `await_response_state'
        from /var/lib/gems/1.9.1/gems/net-scp-1.1.2/lib/net/scp.rb:363:in `block (3 levels) in start_command'
        from /var/lib/gems/1.9.1/gems/net-ssh-2.8.0/lib/net/ssh/connection/channel.rb:311:in `call'
        from /var/lib/gems/1.9.1/gems/net-ssh-2.8.0/lib/net/ssh/connection/channel.rb:311:in `process'
        from /var/lib/gems/1.9.1/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:222:in `block in preprocess'
        from /var/lib/gems/1.9.1/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:222:in `each'
        from /var/lib/gems/1.9.1/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:222:in `preprocess'
        from /var/lib/gems/1.9.1/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:205:in `process'
        from /var/lib/gems/1.9.1/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:169:in `block in loop'
        from /var/lib/gems/1.9.1/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:169:in `loop'
        from /var/lib/gems/1.9.1/gems/net-ssh-2.8.0/lib/net/ssh/connection/session.rb:169:in `loop'
        from /var/lib/gems/1.9.1/gems/net-ssh-2.8.0/lib/net/ssh/connection/channel.rb:269:in `wait'
        from /var/lib/gems/1.9.1/gems/net-scp-1.1.2/lib/net/scp.rb:279:in `upload!'
        from /var/lib/gems/1.9.1/gems/beaker-1.6.2/lib/beaker/ssh_connection.rb:167:in `scp_to'
        from /var/lib/gems/1.9.1/gems/beaker-1.6.2/lib/beaker/host.rb:194:in `do_scp_to'
        from /var/lib/gems/1.9.1/gems/beaker-1.6.2/lib/beaker/dsl/helpers.rb:190:in `scp_to'
        from /var/lib/gems/1.9.1/gems/beaker-rspec-2.1.1/lib/beaker-rspec/beaker_shim.rb:71:in `block in puppet_module_install'
        from /var/lib/gems/1.9.1/gems/beaker-rspec-2.1.1/lib/beaker-rspec/beaker_shim.rb:70:in `each'
        from /var/lib/gems/1.9.1/gems/beaker-rspec-2.1.1/lib/beaker-rspec/beaker_shim.rb:70:in `puppet_module_install'
        from /home/igalic/src/puppet/puppetlabs-mysql/spec/spec_helper_acceptance.rb:20:in `block (2 levels) in <top (required)>'
        from /var/lib/gems/1.9.1/gems/rspec-core-2.14.7/lib/rspec/core/hooks.rb:21:in `instance_eval'
        from /var/lib/gems/1.9.1/gems/rspec-core-2.14.7/lib/rspec/core/hooks.rb:21:in `run'
        from /var/lib/gems/1.9.1/gems/rspec-core-2.14.7/lib/rspec/core/hooks.rb:85:in `block in run'
        from /var/lib/gems/1.9.1/gems/rspec-core-2.14.7/lib/rspec/core/hooks.rb:85:in `each'
        from /var/lib/gems/1.9.1/gems/rspec-core-2.14.7/lib/rspec/core/hooks.rb:85:in `run'
        from /var/lib/gems/1.9.1/gems/rspec-core-2.14.7/lib/rspec/core/hooks.rb:446:in `run_hook'
        from /var/lib/gems/1.9.1/gems/rspec-core-2.14.7/lib/rspec/core/command_line.rb:27:in `block in run'
        from /var/lib/gems/1.9.1/gems/rspec-core-2.14.7/lib/rspec/core/reporter.rb:58:in `report'
        from /var/lib/gems/1.9.1/gems/rspec-core-2.14.7/lib/rspec/core/command_line.rb:25:in `run'
        from /var/lib/gems/1.9.1/gems/rspec-core-2.14.7/lib/rspec/core/runner.rb:80:in `run'
        from /var/lib/gems/1.9.1/gems/rspec-core-2.14.7/lib/rspec/core/runner.rb:17:in `block in autorun'

Caused by a core-file from a crashed VirtualBox process.

In rsync it would be comparatively trivial to exclude such things.

Can't ssh [email protected] via vagrant if root login disabled in /etc/ssh/sshd_config

I've found this bug when using beaker via vagrant with debian 6(squeeze), so simple solution is to change PermitRootLogin option in sshd_config before sshing to vm box as root

Plugin Changes in Vagrantfile

Some Vagrant plugins are on by default, and require explicit turning off.

It would be nice if we could have an option to turn off a Vagrant plugings option to speed up Beaker runs:

if option[:no_vbguest]
   v_file << "c.vbguest.auto_update = false"
end

Support snapshotting SUT

It would be really nice to be able to snapshot SUTs. There's a Vagrant plugin for VirtualBox that does it quite nicely, though I haven't tried the cross-provider ones.

I think that the ideal implementation for me would be to just let the user handle all of it, i.e. simply expose the functionality via the DSL and let me, in my tests, take_snapshot(:step1) and restore_snapshot(:step1).

I know it's a pretty big undertaking and I hate submitting issues that I have little to no intent of trying to fix, but this came up when I was trying to test voxpupuli/puppet-rabbitmq#221 (where the package simply can't be cleanly upgraded/downgraded).

Adding support to vagrant for box_version and box_check_update.

If using versioned boxes from vagrant cloud and a new update is
published, by default vagrant will attempt to update that box to
the latest version. This may be undesired behaviour. Like any other
versioned object you way wish to fix that initial download to a
fixed version or a version range.

Issues with new options...

So, with the change to using hashes for puppet_apply_opts is awesome for setting up puppet, but it causes issues with parameters that are not "option=value"

For example:

apply_manifest(pp, :catch_failures => true, :noop => true)

Results in:

Error: Could not parse application options: needless argument: --noop=true

I've fixed the test in the helpers_spec.rb to catch this issue:

    it 'can set the --noops flag' do
      subject.should_receive( :create_remote_file ).and_return( true )
      expect( subject ).to receive( :on ).with {|h, command, opts|
        cmdline = command.cmd_line( h )
        expect( h ).to be == agent
        expect( cmdline ).to include('puppet apply')
        expect( cmdline ).to include('--detailed-exitcodes')
        expect( cmdline ).to include('--verbose')
        expect( cmdline ).to include('--noop')
        expect( cmdline ).to_not include('--noop=')
      }
      subject.apply_manifest_on(
        agent,
        'class { "boo": }',
        :acceptable_exit_codes => (1..5),
        :noop                  => true,
        :expect_failures       => true,
      )
    end

Trying to work out how to fix it 👍

Docker provider does not ensure which is installed

The centos:centos6 image does not have which installed, leading to a nice stack trace:

centos-65-x64 23:21:40$  which curl  
bash: which: command not found

centos-65-x64 executed in 15.23 seconds
Exited: 127
Failed: errored in validate
#<Beaker::Host::CommandFailure: Host 'centos-65-x64' exited with 127 running:
  which curl  
Last 10 lines of output were:
    bash: which: command not found>
/home/ekohl/.gem/ruby/1.9.1/gems/beaker-1.12.0/lib/beaker/host.rb:202
/home/ekohl/.gem/ruby/1.9.1/gems/beaker-1.12.0/lib/beaker/host/unix/pkg.rb:11
/home/ekohl/.gem/ruby/1.9.1/gems/beaker-1.12.0/lib/beaker/host_prebuilt_steps.rb:86
/home/ekohl/.gem/ruby/1.9.1/gems/beaker-1.12.0/lib/beaker/host_prebuilt_steps.rb:85
/home/ekohl/.gem/ruby/1.9.1/gems/beaker-1.12.0/lib/beaker/host_prebuilt_steps.rb:85
/home/ekohl/.gem/ruby/1.9.1/gems/beaker-1.12.0/lib/beaker/host_prebuilt_steps.rb:83
/home/ekohl/.gem/ruby/1.9.1/gems/beaker-1.12.0/lib/beaker/host_prebuilt_steps.rb:83
/home/ekohl/.gem/ruby/1.9.1/gems/beaker-1.12.0/lib/beaker/host_prebuilt_steps.rb:83
/home/ekohl/.gem/ruby/1.9.1/gems/beaker-1.12.0/lib/beaker/hypervisor.rb:97
/home/ekohl/.gem/ruby/1.9.1/gems/beaker-1.12.0/lib/beaker/network_manager.rb:65
/home/ekohl/.gem/ruby/1.9.1/gems/beaker-1.12.0/lib/beaker/network_manager.rb:64
/home/ekohl/.gem/ruby/1.9.1/gems/beaker-1.12.0/lib/beaker/network_manager.rb:64
/home/ekohl/.gem/ruby/1.9.1/gems/beaker-rspec-2.2.4/lib/beaker-rspec/beaker_shim.rb:36
/home/ekohl/.gem/ruby/1.9.1/gems/beaker-rspec-2.2.4/lib/beaker-rspec/spec_helper.rb:39
/home/ekohl/.gem/ruby/1.9.1/gems/rspec-core-2.14.8/lib/rspec/core.rb:120
/home/ekohl/.gem/ruby/1.9.1/gems/beaker-rspec-2.2.4/lib/beaker-rspec/spec_helper.rb:5
/home/ekohl/dev/puppetlabs-apache/spec/spec_helper_acceptance.rb:1
/home/ekohl/dev/puppetlabs-apache/spec/spec_helper_acceptance.rb:1
/home/ekohl/dev/puppetlabs-apache/spec/acceptance/apache_parameters_spec.rb:1
/home/ekohl/dev/puppetlabs-apache/spec/acceptance/apache_parameters_spec.rb:1
/home/ekohl/.gem/ruby/1.9.1/gems/rspec-core-2.14.8/lib/rspec/core/configuration.rb:896
/home/ekohl/.gem/ruby/1.9.1/gems/rspec-core-2.14.8/lib/rspec/core/configuration.rb:896
/home/ekohl/.gem/ruby/1.9.1/gems/rspec-core-2.14.8/lib/rspec/core/configuration.rb:896
/home/ekohl/.gem/ruby/1.9.1/gems/rspec-core-2.14.8/lib/rspec/core/configuration.rb:896
/home/ekohl/.gem/ruby/1.9.1/gems/rspec-core-2.14.8/lib/rspec/core/command_line.rb:22
/home/ekohl/.gem/ruby/1.9.1/gems/rspec-core-2.14.8/lib/rspec/core/runner.rb:80
/home/ekohl/.gem/ruby/1.9.1/gems/rspec-core-2.14.8/lib/rspec/core/runner.rb:17
Failed: errored in validate
#<Beaker::Host::CommandFailure: Host 'centos-65-x64' exited with 127 running:
  which curl  
Last 10 lines of output were:
    bash: which: command not found>
/home/ekohl/.gem/ruby/1.9.1/gems/beaker-1.12.0/lib/beaker/host.rb:202
/home/ekohl/.gem/ruby/1.9.1/gems/beaker-1.12.0/lib/beaker/host/unix/pkg.rb:11
/home/ekohl/.gem/ruby/1.9.1/gems/beaker-1.12.0/lib/beaker/host_prebuilt_steps.rb:86
/home/ekohl/.gem/ruby/1.9.1/gems/beaker-1.12.0/lib/beaker/host_prebuilt_steps.rb:85
/home/ekohl/.gem/ruby/1.9.1/gems/beaker-1.12.0/lib/beaker/host_prebuilt_steps.rb:85
/home/ekohl/.gem/ruby/1.9.1/gems/beaker-1.12.0/lib/beaker/host_prebuilt_steps.rb:83
/home/ekohl/.gem/ruby/1.9.1/gems/beaker-1.12.0/lib/beaker/host_prebuilt_steps.rb:83
/home/ekohl/.gem/ruby/1.9.1/gems/beaker-1.12.0/lib/beaker/host_prebuilt_steps.rb:83
/home/ekohl/.gem/ruby/1.9.1/gems/beaker-1.12.0/lib/beaker/hypervisor.rb:97
/home/ekohl/.gem/ruby/1.9.1/gems/beaker-1.12.0/lib/beaker/network_manager.rb:65
/home/ekohl/.gem/ruby/1.9.1/gems/beaker-1.12.0/lib/beaker/network_manager.rb:64
/home/ekohl/.gem/ruby/1.9.1/gems/beaker-1.12.0/lib/beaker/network_manager.rb:64
/home/ekohl/.gem/ruby/1.9.1/gems/beaker-rspec-2.2.4/lib/beaker-rspec/beaker_shim.rb:36
/home/ekohl/.gem/ruby/1.9.1/gems/beaker-rspec-2.2.4/lib/beaker-rspec/spec_helper.rb:39
/home/ekohl/.gem/ruby/1.9.1/gems/rspec-core-2.14.8/lib/rspec/core.rb:120
/home/ekohl/.gem/ruby/1.9.1/gems/beaker-rspec-2.2.4/lib/beaker-rspec/spec_helper.rb:5
/home/ekohl/dev/puppetlabs-apache/spec/spec_helper_acceptance.rb:1
/home/ekohl/dev/puppetlabs-apache/spec/spec_helper_acceptance.rb:1
/home/ekohl/dev/puppetlabs-apache/spec/acceptance/apache_parameters_spec.rb:1
/home/ekohl/dev/puppetlabs-apache/spec/acceptance/apache_parameters_spec.rb:1
/home/ekohl/.gem/ruby/1.9.1/gems/rspec-core-2.14.8/lib/rspec/core/configuration.rb:896
/home/ekohl/.gem/ruby/1.9.1/gems/rspec-core-2.14.8/lib/rspec/core/configuration.rb:896
/home/ekohl/.gem/ruby/1.9.1/gems/rspec-core-2.14.8/lib/rspec/core/configuration.rb:896
/home/ekohl/.gem/ruby/1.9.1/gems/rspec-core-2.14.8/lib/rspec/core/configuration.rb:896
/home/ekohl/.gem/ruby/1.9.1/gems/rspec-core-2.14.8/lib/rspec/core/command_line.rb:22
/home/ekohl/.gem/ruby/1.9.1/gems/rspec-core-2.14.8/lib/rspec/core/runner.rb:80
/home/ekohl/.gem/ruby/1.9.1/gems/rspec-core-2.14.8/lib/rspec/core/runner.rb:17
/home/ekohl/.gem/ruby/1.9.1/gems/beaker-1.12.0/lib/beaker/host.rb:202:in `exec': Host 'centos-65-x64' exited with 127 running: (Beaker::Host::CommandFailure)
  which curl  
Last 10 lines of output were:
    bash: which: command not found
    from /home/ekohl/.gem/ruby/1.9.1/gems/beaker-1.12.0/lib/beaker/host/unix/pkg.rb:11:in `check_for_package'
    from /home/ekohl/.gem/ruby/1.9.1/gems/beaker-1.12.0/lib/beaker/host_prebuilt_steps.rb:86:in `block in validate_host'
    from /home/ekohl/.gem/ruby/1.9.1/gems/beaker-1.12.0/lib/beaker/host_prebuilt_steps.rb:85:in `each'
    from /home/ekohl/.gem/ruby/1.9.1/gems/beaker-1.12.0/lib/beaker/host_prebuilt_steps.rb:85:in `validate_host'
    from /home/ekohl/.gem/ruby/1.9.1/gems/beaker-1.12.0/lib/beaker/host_prebuilt_steps.rb:83:in `block in validate_host'
    from /home/ekohl/.gem/ruby/1.9.1/gems/beaker-1.12.0/lib/beaker/host_prebuilt_steps.rb:83:in `map'
    from /home/ekohl/.gem/ruby/1.9.1/gems/beaker-1.12.0/lib/beaker/host_prebuilt_steps.rb:83:in `validate_host'
    from /home/ekohl/.gem/ruby/1.9.1/gems/beaker-1.12.0/lib/beaker/hypervisor.rb:97:in `validate'
    from /home/ekohl/.gem/ruby/1.9.1/gems/beaker-1.12.0/lib/beaker/network_manager.rb:65:in `block in validate'
    from /home/ekohl/.gem/ruby/1.9.1/gems/beaker-1.12.0/lib/beaker/network_manager.rb:64:in `each_key'
    from /home/ekohl/.gem/ruby/1.9.1/gems/beaker-1.12.0/lib/beaker/network_manager.rb:64:in `validate'
    from /home/ekohl/.gem/ruby/1.9.1/gems/beaker-rspec-2.2.4/lib/beaker-rspec/beaker_shim.rb:36:in `validate'
    from /home/ekohl/.gem/ruby/1.9.1/gems/beaker-rspec-2.2.4/lib/beaker-rspec/spec_helper.rb:39:in `block in <top (required)>'
    from /home/ekohl/.gem/ruby/1.9.1/gems/rspec-core-2.14.8/lib/rspec/core.rb:120:in `configure'
    from /home/ekohl/.gem/ruby/1.9.1/gems/beaker-rspec-2.2.4/lib/beaker-rspec/spec_helper.rb:5:in `<top (required)>'
    from /home/ekohl/dev/puppetlabs-apache/spec/spec_helper_acceptance.rb:1:in `require'
    from /home/ekohl/dev/puppetlabs-apache/spec/spec_helper_acceptance.rb:1:in `<top (required)>'
    from /home/ekohl/dev/puppetlabs-apache/spec/acceptance/apache_parameters_spec.rb:1:in `require'
    from /home/ekohl/dev/puppetlabs-apache/spec/acceptance/apache_parameters_spec.rb:1:in `<top (required)>'
    from /home/ekohl/.gem/ruby/1.9.1/gems/rspec-core-2.14.8/lib/rspec/core/configuration.rb:896:in `load'
    from /home/ekohl/.gem/ruby/1.9.1/gems/rspec-core-2.14.8/lib/rspec/core/configuration.rb:896:in `block in load_spec_files'
    from /home/ekohl/.gem/ruby/1.9.1/gems/rspec-core-2.14.8/lib/rspec/core/configuration.rb:896:in `each'
    from /home/ekohl/.gem/ruby/1.9.1/gems/rspec-core-2.14.8/lib/rspec/core/configuration.rb:896:in `load_spec_files'
    from /home/ekohl/.gem/ruby/1.9.1/gems/rspec-core-2.14.8/lib/rspec/core/command_line.rb:22:in `run'
    from /home/ekohl/.gem/ruby/1.9.1/gems/rspec-core-2.14.8/lib/rspec/core/runner.rb:80:in `run'
    from /home/ekohl/.gem/ruby/1.9.1/gems/rspec-core-2.14.8/lib/rspec/core/runner.rb:17:in `block in autorun'

Ubuntu 14.04 fails with beaker

Hi,

Been trying to get beaker to run with an ubuntu 14.04 host and running into the following issue:

ubuntu-14-04 23:03:31$  apt-get install  -y rubygems  
Reading package lists...
Building dependency tree...
Reading state information...
Package rubygems is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
However the following packages replace it:
  ruby

E: Package 'rubygems' has no installation candidate

Looks like we shouldn't need to install it or at least install the 'ruby' package.
Will see if i can make a fix for this.

Setting PATH

Submitting this issue after talking with @anodelman on IRC.

I'm using a RVM Ruby installation to run Puppet on my SUTs, and it seems like beaker is forcing a set path before every Puppet run:

env PATH="/usr/bin:/opt/puppet-git-repos/hiera/bin:${PATH}" RUBYLIB="/opt/puppet-git-repos/hiera/lib:/opt/puppet-git-repos/hiera-puppet/lib:${RUBYLIB}" puppet apply  --verbose --detailed-exitcodes /tmp/apply_manifest.pp.NQ3AbV

This is causing the following warnings during my tests:

centos-65-x64 02:09:29$ env PATH="/usr/bin:/opt/puppet-git-repos/hiera/bin:${PATH}" RUBYLIB="/opt/puppet-git-repos/hiera/lib:/opt/puppet-git-repos/hiera-puppet/lib:${RUBYLIB}" puppet apply  --verbose --detailed-exitcodes /tmp/apply_manifest.pp.NQ3AbV
Info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/facter_dot_d.rb
Info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/root_home.rb
Info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/puppet_vardir.rb
Info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/pe_version.rb
Info: Loading facts in /etc/puppet/modules/concat/lib/facter/concat_basedir.rb
Info: Loading facts in /etc/puppet/modules/rvm/lib/facter/rvm_installed.rb
Info: Loading facts in /etc/puppet/modules/rvm/lib/facter/rvm_version.rb
Info: Loading facts in /etc/puppet/modules/firewall/lib/facter/iptables_persistent_version.rb
Info: Loading facts in /etc/puppet/modules/firewall/lib/facter/iptables_version.rb
Info: Loading facts in /etc/puppet/modules/firewall/lib/facter/ip6tables_version.rb
Warning! PATH is not properly set up, '/usr/local/rvm/gems/ruby-2.0.0-p451/bin' is not at first place,
         usually this is caused by shell initialization files - check them for 'PATH=...' entries,
         it might also help to re-add RVM to your dotfiles: 'rvm get stable --auto-dotfiles',
         to fix temporarily in this shell session run: 'rvm use ruby-2.0.0-p451'.
Warning: Config file /etc/puppet/hiera.yaml not found, using Hiera defaults
Notice: Compiled catalog for centos-65-x64.shott.lan in environment production in 4.28 seconds
Info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/facter_dot_d.rb
Info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/root_home.rb
Info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/puppet_vardir.rb
Info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/pe_version.rb
Info: Loading facts in /etc/puppet/modules/concat/lib/facter/concat_basedir.rb
Info: Loading facts in /etc/puppet/modules/rvm/lib/facter/rvm_installed.rb
Info: Loading facts in /etc/puppet/modules/rvm/lib/facter/rvm_version.rb
Info: Loading facts in /etc/puppet/modules/firewall/lib/facter/iptables_persistent_version.rb
Info: Loading facts in /etc/puppet/modules/firewall/lib/facter/iptables_version.rb
Info: Loading facts in /etc/puppet/modules/firewall/lib/facter/ip6tables_version.rb
Warning! PATH is not properly set up, '/usr/local/rvm/gems/ruby-2.0.0-p451/bin' is not at first place,
         usually this is caused by shell initialization files - check them for 'PATH=...' entries,
         it might also help to re-add RVM to your dotfiles: 'rvm get stable --auto-dotfiles',
         to fix temporarily in this shell session run: 'rvm use ruby-2.0.0-p451'.
Info: Applying configuration version '1396948171'
Warning: Could not match Warning! PATH is not properly set up, '/usr/local/rvm/gems/ruby-2.0.0-p451/bin' is not at first place,
Warning: Could not match Warning! PATH is not properly set up, '/usr/local/rvm/gems/ruby-2.0.0-p451/bin' is not at first place,
Warning: Could not match Warning! PATH is not properly set up, '/usr/local/rvm/gems/ruby-2.0.0-p451/bin' is not at first place,
Notice: Finished catalog run in 6.12 seconds

centos-65-x64 executed in 13.92 seconds
  should work with no errors
removing temporory ssh-config files per-vagrant box
Destroying vagrant boxes
==> centos-65-x64: Forcing shutdown of VM...
==> centos-65-x64: Destroying VM and associated drives...

I've tried setting a custom path using the :environment key in the options hash for apply_manifest, however it doesn't seem to respect it:

# pp being a variable, which holds my manifest
apply_manifest pp, { :environment => {'PATH' => '/usr/local/rvm/gems/ruby-2.0.0-p451/bin:/usr/local/rvm/gems/ruby-2.0.0-p451@global/bin:/usr/local/rvm/gems/ruby-2.0.0-p451/bin:/usr/local/rvm/gems/ruby-2.0.0-p451@global/bin:/usr/local/rvm/rubies/ruby-2.0.0-p451/bin:/usr/bin:/opt/puppet-git-repos/hiera/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/local/rvm/bin' }, :catch_failures => true }

A snippet of the result:

centos-65-x64 10:25:45$  mktemp -t apply_manifest.pp.XXXXXX  
/tmp/apply_manifest.pp.mAdZlR

centos-65-x64 executed in 0.14 seconds
localhost $ scp /var/folders/w9/c36_jf6d2kvfsvvy_5mw5s0c0000gn/T/beaker20140408-68015-9op1c3 centos-65-x64:/tmp/apply_manifest.pp.mAdZlR {}

centos-65-x64 10:25:45$ env PATH="/usr/bin:/opt/puppet-git-repos/hiera/bin:${PATH}" RUBYLIB="/opt/puppet-git-repos/hiera/lib:/opt/puppet-git-repos/hiera-puppet/lib:${RUBYLIB}" puppet apply  --verbose --detailed-exitcodes {:environment=>{"PATH"=>"/usr/local/rvm/gems/ruby-2.0.0-p451/bin:/usr/local/rvm/gems/ruby-2.0.0-p451@global/bin:/usr/local/rvm/gems/ruby-2.0.0-p451/bin:/usr/local/rvm/gems/ruby-2.0.0-p451@global/bin:/usr/local/rvm/rubies/ruby-2.0.0-p451/bin:/usr/bin:/opt/puppet-git-repos/hiera/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/local/rvm/bin"}} /tmp/apply_manifest.pp.mAdZlR
bash: /usr/local/rvm/gems/ruby-2.0.0-p451/bin:/usr/local/rvm/gems/ruby-2.0.0-p451@global/bin:/usr/local/rvm/gems/ruby-2.0.0-p451/bin:/usr/local/rvm/gems/ruby-2.0.0-p451@global/bin:/usr/local/rvm/rubies/ruby-2.0.0-p451/bin:/usr/bin:/opt/puppet-git-repos/hiera/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/local/rvm/bin}}: No such file or directory

centos-65-x64 executed in 0.14 seconds
Exited: 1
  should work with no errors (FAILED - 1)

Failures:

  1) puppet_stack should work with no errors
     Failure/Error: apply_manifest pp, { :environment => {'PATH' => '/usr/local/rvm/gems/ruby-2.0.0-p451/bin:/usr/local/rvm/gems/ruby-2.0.0-p451@global/bin:/usr/local/rvm/gems/ruby-2.0.0-p451/bin:/usr/local/rvm/gems/ruby-2.0.0-p451@global/bin:/usr/local/rvm/rubies/ruby-2.0.0-p451/bin:/usr/bin:/opt/puppet-git-repos/hiera/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/local/rvm/bin' }, :catch_failures => true }
     Beaker::Host::CommandFailure:
       Host 'centos-65-x64' exited with 1 running:
        env PATH="/usr/bin:/opt/puppet-git-repos/hiera/bin:${PATH}" RUBYLIB="/opt/puppet-git-repos/hiera/lib:/opt/puppet-git-repos/hiera-puppet/lib:${RUBYLIB}" puppet apply  --verbose --detailed-exitcodes {:environment=>{"PATH"=>"/usr/local/rvm/gems/ruby-2.0.0-p451/bin:/usr/local/rvm/gems/ruby-2.0.0-p451@global/bin:/usr/local/rvm/gems/ruby-2.0.0-p451/bin:/usr/local/rvm/gems/ruby-2.0.0-p451@global/bin:/usr/local/rvm/rubies/ruby-2.0.0-p451/bin:/usr/bin:/opt/puppet-git-repos/hiera/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/local/rvm/bin"}} /tmp/apply_manifest.pp.mAdZlR
       Last 10 lines of output were:
        bash: /usr/local/rvm/gems/ruby-2.0.0-p451/bin:/usr/local/rvm/gems/ruby-2.0.0-p451@global/bin:/usr/local/rvm/gems/ruby-2.0.0-p451/bin:/usr/local/rvm/gems/ruby-2.0.0-p451@global/bin:/usr/local/rvm/rubies/ruby-2.0.0-p451/bin:/usr/bin:/opt/puppet-git-repos/hiera/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/local/rvm/bin}}: No such file or directory
     # ./spec/acceptance/puppet_stack_spec.rb:16:in `block (2 levels) in <top (required)>'

Finished in 6.69 seconds
1 example, 1 failure

Failed examples:

As you can see, it doesn't recognize :environment, and just appends it wholesale to the puppet apply command.

Doing multiple runs at the same time causes vagrant VM's all to take port 2222

In allot of cases when i run multiple test instances via jenkins they all want to take port 2222.
Would it be an idea to be able to set the SSH Port number via a config or an ENV variable?
Or even chose a random number would help.

if it can be via an EVN variable I could have some kind of increment thing happening

Beaker's Vagrantfile - base_mac

I seem to be unable to run any rake spec/acceptance tests when my beaker Vagrantfile contains an entry for base_mac.

Example:

Vagrant.configure("2") do |c|
  c.vm.define 'centos-65-x64' do |v|
    v.vm.hostname = 'centos-65-x64'
    v.vm.box = 'boxname' # Value changed
    v.vm.box_url = 'file://local.box' # Value changed
    v.vm.base_mac = '08002757F8C5' # Problem line
    v.vm.network :private_network, ip: "10.255.130.2", :netmask => "255.255.0.0"
  end
  c.vm.provider :virtualbox do |vb|
    vb.customize ["modifyvm", :id, "--memory", "1024"]
  end
end

The test will eventually timeout and give an error about being unable to SSH. If I add vb.gui = true, and login during its attempt to SSH, I can see with ifconfig that there is only a lo adapter. However, if I comment out v.vm.base_mac, everything works as expected. Is this is a known issue, or am I falling into a common pitfall?

Just for the record , my box's internal Vagrantfile looks like this:

# The contents below were provided by the Packer Vagrant post-processor

Vagrant.configure("2") do |config|
  config.vm.base_mac = "080027947974"
end


# The contents below (if any) are custom contents provided by the
# Packer template during image build.
# Load include vagrant file if it exists after the auto-generated
# so it can override any of the settings
include_vagrantfile = File.expand_path("../include/_Vagrantfile", __FILE__)
load include_vagrantfile if File.exist?(include_vagrantfile)

Edit:
Versions:
vagrant - 1.5.2
beaker - 1.9.1

hosts_file_parser.rb:30:in `block in parse_hosts_file': undefined method `[]' for nil:NilClass (NoMethodError)

Seems that there is a parser error issue.

beaker --log-level debug --hosts sample.cfg 
/Users/lmo0/.rvm/gems/ruby-2.1.1/gems/beaker-1.12.0/lib/beaker/options/hosts_file_parser.rb:30:in `block in parse_hosts_file': undefined method `[]' for nil:NilClass (NoMethodError)
    from /Users/lmo0/.rvm/gems/ruby-2.1.1/gems/beaker-1.12.0/lib/beaker/options/hosts_file_parser.rb:29:in `each_key'
    from /Users/lmo0/.rvm/gems/ruby-2.1.1/gems/beaker-1.12.0/lib/beaker/options/hosts_file_parser.rb:29:in `parse_hosts_file'
    from /Users/lmo0/.rvm/gems/ruby-2.1.1/gems/beaker-1.12.0/lib/beaker/options/parser.rb:161:in `parse_args'
    from /Users/lmo0/.rvm/gems/ruby-2.1.1/gems/beaker-1.12.0/lib/beaker/cli.rb:14:in `initialize'
    from /Users/lmo0/.rvm/gems/ruby-2.1.1/gems/beaker-1.12.0/bin/beaker:6:in `new'
    from /Users/lmo0/.rvm/gems/ruby-2.1.1/gems/beaker-1.12.0/bin/beaker:6:in `<top (required)>'
    from /Users/lmo0/.rvm/gems/ruby-2.1.1/bin/beaker:23:in `load'
    from /Users/lmo0/.rvm/gems/ruby-2.1.1/bin/beaker:23:in `<main>'
    from /Users/lmo0/.rvm/gems/ruby-2.1.1/bin/ruby_executable_hooks:15:in `eval'
    from /Users/lmo0/.rvm/gems/ruby-2.1.1/bin/ruby_executable_hooks:15:in `<main>'

sample.cfg from http://tech.opentable.co.uk/blog/2014/04/04/testing-puppet-with-beaker/

HOSTS:
  ubuntu-server-12042-x64:
  roles:
    - master
  platform: ubuntu-12.04-amd64
  box : ubuntu-server-12042-x64-vbox4210-nocm
  box_url : http://puppet-vagrant-boxes.puppetlabs.com/ubuntu-server-12042-x64-vbox4210-nocm.box
  hypervisor : vagrant
CONFIG:
  log_level: verbose
  type: git

Docker provider does not check if image is given

In case the user specifies no image in the nodeset, an invalid Dockerfile is created with just FROM instead of FROM centos:centos6 (for example).

supporting packer-windows boxes

The current way of creating windows base boxes is using the packer-windows project.

The first (and current) problem with this is that they are only installed with OpenSSH and not a full cygwin installation which means much of the windows functionality within beaker is currently not working. There are ways around this (I will be submitting a PR shortly).

The second issue is that soon the packer-windows project will be (rightly so) dropping OpenSSH as the communication method in favour of winrm. Beaker will need to support this also.

Typo in method name ('has' instead of 'hash') in options_hash.rb (line 101)

https://github.com/puppetlabs/beaker/blob/master/lib/beaker/options/options_hash.rb#L101

if (base[key].is_a?(Hash) || base[key].is_a?(OptionsHash)) && (hash[key].is_a?(Hash) || has[key].is_a?(OptionsHash))

s/has\[/hash[/g

Unable to run beaker from MAC

OSX 10.9.3
beaker-1.12.0

$ beaker --help
/Library/Ruby/Gems/2.0.0/gems/beaker-1.12.0/lib/beaker/test_case.rb:47:in <class:TestCase>': uninitialized constant MiniTest (NameError) from /Library/Ruby/Gems/2.0.0/gems/beaker-1.12.0/lib/beaker/test_case.rb:37:inmodule:Beaker'
from /Library/Ruby/Gems/2.0.0/gems/beaker-1.12.0/lib/beaker/test_case.rb:26:in <top (required)>' from /Library/Ruby/Site/2.0.0/rubygems/core_ext/kernel_require.rb:73:inrequire'
from /Library/Ruby/Site/2.0.0/rubygems/core_ext/kernel_require.rb:73:in require' from /Library/Ruby/Gems/2.0.0/gems/beaker-1.12.0/lib/beaker/test_suite.rb:6:inblock in <top (required)>'
from /Library/Ruby/Gems/2.0.0/gems/beaker-1.12.0/lib/beaker/test_suite.rb:4:in each' from /Library/Ruby/Gems/2.0.0/gems/beaker-1.12.0/lib/beaker/test_suite.rb:4:in<top (required)>'
from /Library/Ruby/Site/2.0.0/rubygems/core_ext/kernel_require.rb:73:in require' from /Library/Ruby/Site/2.0.0/rubygems/core_ext/kernel_require.rb:73:inrequire'
from /Library/Ruby/Gems/2.0.0/gems/beaker-1.12.0/lib/beaker.rb:6:in block in <module:Beaker>' from /Library/Ruby/Gems/2.0.0/gems/beaker-1.12.0/lib/beaker.rb:4:ineach'
from /Library/Ruby/Gems/2.0.0/gems/beaker-1.12.0/lib/beaker.rb:4:in <module:Beaker>' from /Library/Ruby/Gems/2.0.0/gems/beaker-1.12.0/lib/beaker.rb:2:in<top (required)>'
from /Library/Ruby/Site/2.0.0/rubygems/core_ext/kernel_require.rb:73:in require' from /Library/Ruby/Site/2.0.0/rubygems/core_ext/kernel_require.rb:73:inrequire'
from /Library/Ruby/Gems/2.0.0/gems/beaker-1.12.0/bin/beaker:4:in <top (required)>' from /usr/bin/beaker:23:inload'
from /usr/bin/beaker:23:in `

debian metadata was out of date

When using docker the layers are cached and never updated
so when doing an apt-get install it will use the old data until an apt-get update is ran.
In my case it caused for the initial package install to fail.
perhaps an idea to run apt-get update anyway after curl and ntp checks?

Do the vagrant VM's really need to have 1GB memory?

Running Beaker on Windows

For reasons we shall not discuss (as it involves All of Government Purchasing policy, Public Records Act, red tape, and other-non-technical-issues-that-are-harder-than-make-it-work-in-windows) my $work requires that I develop puppet modules in a Windows environment. I'd really like to add beaker acceptance tests to my modules and run them locally.

I have Ruby 2.0, Vagrant, Puppet, Virtualbox, and VMware Workstation all working fine in Windows 7 x64, but installing beaker fails.

C:\Development\vagrant\ruby [master +0 ~1 -0]> gem install beaker
ERROR:  Error installing beaker:
        The 'nokogiri' native gem requires installed build tools.

Please update your PATH to include build tools or download the DevKit
from 'http://rubyinstaller.org/downloads' and follow the instructions
at 'http://github.com/oneclick/rubyinstaller/wiki/Development-Kit'

However installing Nokogiri itself works, as the later version bundles binaries & libraries with the gem:

C:\Development\vagrant\ruby [master +0 ~1 -0]> gem install nokogiri
Fetching: mini_portile-0.6.0.gem (100%)
Successfully installed mini_portile-0.6.0
Fetching: nokogiri-1.6.3.1-x64-mingw32.gem (100%)
Nokogiri is built with the packaged libraries: libxml2-2.8.0, libxslt-1.1.28, zlib-1.2.8, libiconv-1.14.
Successfully installed nokogiri-1.6.3.1-x64-mingw32
Parsing documentation for mini_portile-0.6.0
Installing ri documentation for mini_portile-0.6.0
Parsing documentation for nokogiri-1.6.3.1-x64-mingw32
unable to convert "\x90" from ASCII-8BIT to UTF-8 for lib/nokogiri/2.0/nokogiri.so, skipping
unable to convert "\x90" from ASCII-8BIT to UTF-8 for lib/nokogiri/2.1/nokogiri.so, skipping
Installing ri documentation for nokogiri-1.6.3.1-x64-mingw32
2 gems installed

Would it be possible to check that beaker works with the later Nokogiri version, and have it's dependency changed to allow the use of this version.

No doubt, once this is sorted, I'll find another problem... probably symbolic links...

...I'm also expecting some comments about how gems bundling libraries and binaries is bad, because it is...

Fix bug connecting to windows vagrant host

When running multiple hosts in vagrant with an linux master and windows
agent it seems that vagrant has difficulty connecting to box when the
ssh_config file contains 127.0.0.1 for the host. This change fixes the
issue by replacing it with the host ip if it has been specified in the
nodeset.

This might need a little more manual testing.

gem install beaker fails for ruby 1.9.3 - Unable to resolve dependency nokogiri

It seems the current beaker version tree is inconsistent:

$ gem install beaker
ERROR:  While executing gem ... (Gem::DependencyError)
    Unable to resolve dependencies: beaker requires nokogiri (= 1.5.10); aws-sdk requires nokogiri (>= 1.4.4); fog requires nokogiri (>= 1.5.11, ~> 1.5); rbvmomi requires nokogiri (>= 1.4.1)

I'm not a ruby programmer, but if I understand things correctly it's impossible to satisfy all of these dependencies at the same time (= 1.5.10, >= 1.5.11). Note that I have none of the mentioned gems installed.

$ruby --version
ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-darwin12.5.0]

Maybe related to #292

User defined roles

User defined roles not work for me. I added two roles "loader" and "graphite"

on database, "echo hello" # works
on dashboard, "echo hello" #works
on loader, "echo hello" #undefined local variable or method `loader'

Could you please let me know to correctly define and use customized roles?

Here is the host cfg

HOSTS:
centos-6-master:
roles:
- master
- agent
- database
- dashboard
platform: centos-6-xxx
hypervisor: google
ip: 107.178.220.183
centos-6-agent-01:
roles:
- agent
- loader
platform: centos-6-xxx
hypervisor: google
ip: 107.178.209.246
centos-6-agent-02:
roles:
- agent
- graphite
platform: centos-6-xxx
hypervisor: google
ip: 23.236.61.123
CONFIG:
nfs_server: none
consoleport: 443
gce_project : XXX
gce_keyfile : /path/to/gce.p12
gce_password: notasecret
gce_email : [email protected]

Configurable package mirrors

It would be really nice if Beaker could get a feature where people can expose their preferred package mirrors and have the host reconfigured to use those.

Right now the Debian Puppet Labs images all use http.us.debian.org which from the Netherlands is painfully slow, it takes about 2m alone to fetch that index. If I compare that to ftp.nl.debian.org I'm done in 10sec and that's mostly because the buffer is still being flushed. You can guess how those numbers scale when it actually starts fetching all the stuff it needs for Rubygems, compilers etc.

I've got an evil shell() with a nice sed -i that takes care of the issue right now but it's fiddly as depending on wether it is Debian or Ubuntu a different mirror needs to be set, it all becomes really nasty.

What I'd like is a hook of some sort to reconfigure package mirrors based on a list I provide or automate it by detecting the user's location through something like http://ipinfo.io and the country key it returns.

Check_for_package method should not use which command

For Unix::pkg, the method check_for_package is wrong.
Currently, the method check for a program instead of a package.

For example:
If we install libxml2-dev on a Debian os, we can't use check_for_package(master, 'libxml2-dev')

The real command to checking if a package is installed on a debian|ubuntu os is dpkg -s #{name}.
For others, I don't know a good command without grep or regex to check if package is installed.

Wiki Tutorial issue

On the tutorial for writing a module test, there's an issue.

If you use this nodeset:

--- 
CONFIG: 
  consoleport: 443 
  nfs_server: none
HOSTS: 
  ubuntu-10-04-4-x64-master: 
    box: ubuntu-server-10044-x64-vbox4210
    box_url: "http://puppet-vagrant-boxes.puppetlabs.com/ubuntu-server-10044-x64-vbox4210.box"
    hypervisor: vagrant
    ip: "192.168.20.20"
    platform: ubuntu-10.04-amd64
    roles: 
      - master
      - agent
      - dashboard
      - database

When you run rspec, the command on host, 'gem install puppet --no-ri --no-rdoc', it will fail on the above host with an error about the net/https gemfile missing. The fix is to install libopenssl-ruby. I would make this wiki change, but it wouldn't work for RHEL hosts for sure. But the tutorial does not work as-is and might be confusing to others.

support for additional disk

It would be nice to be able to attach an additional disk to the VM with something like:

config.vm.customize ['storageattach'...

for vagrant or

ec2-create-volume

for EC2 or ...

Refactoring run_agent_on so that it functions as expected.

When testing a module one would usually use the apply_manifest_on
method, which applies all the command line options and allows you
to validate the exits codes. When testing in a multi-node scenario
with one master host and one agent host you are unable to do that.
With windows, the multi-node test is a requirement.

The run_agent_on method does not provide the same flexibility in
apply detailed-exit-codes and being able to validate against them.
This change aims to solve that problem by making it function in the
same way as the apply_manifest_on method.

Add Proxy support for package managers

When running with a CI and having a large matrix it sometimes can be useful to have a caching proxy server for the packages.
Based on the different distro's it should be fairly easy to create the configs for that.
For example: https://github.com/elasticsearch/puppet-elasticsearch/blob/master/spec/spec_helper_acceptance.rb#L7-L21

Been thinking of the implementation in beaker to make this more universal.
As far as i could find the main provisioning steps are in the networkmanager file.
I could create an additional step called 'proxy' that sits between the validate and configure actions.
That proxy command then executes the one in the hypervisor code to keep things consistent.
That one calls a setup_proxy command which will be defined in host_prebuilt_steps
where the actual proxy config is setup per distro.

One thing im not sure of is how to set the proxy host it self.
Would like to be able to set it via the command line option and and env variable.

docker issues

I've been running with the docker hypervisor for a while now and did find some issues with the current code.

Tagging
At the moment we tag an image with the hostname after creation.
When doing a single run this can be useful but with multiple runs at the same time for the same host type its useless.
Next to that we don't need to have an image tagged to be able to launch a container. we can simply use the image ID.
Image removal
Docker tries to be very efficient with the images and layers of images by re-using them.
Seeing as the image stays the same in most cases it can efficiently re-use those.
Currently we remove the image after each run which causes several issues:
- Image creation gets messed up because in some cases its run multiple times.
- Image removal can fail in different ways ( unable to remove or image is already removed
- We can't re-use the built image ( cache )

I'm currently running tests with my jenkins jobs with a modified version of the hypervisor removing the 2 issues.
Will send a PR when ready.

Adding support to use bitvise ssh server as an alternative ssh provider on windows

The use-case in wishing to use bitvise ssh is that cygwin only emulates
a windows shell and bypasses some of the authentication mechanisms [1][2][3].
The impact of this is that certain commands/powershell scripts (such as
installing a windows feature) will fail. Using bitvise ssh server seems
to resolve this issue.

This change implements the host configuration option of communicator
(e.g. host['communicator']) which allows you to conditionally select the
appropriate commands. By implementing it this way it paves the way for
future communication providers such as winrm.

[1] https://cygwin.com/ml/cygwin/2004-09/msg00087.html
[2] https://cygwin.com/ml/cygwin/2006-06/msg00862.html
[3] http://thread.gmane.org/gmane.os.cygwin/128472

make "Last xx lines of output were" configurable or larger

Hi,

With most modules 10 lines won't be enough when the issue is at the beginning.
Perhaps an idea to show the whole stack?

Ruby 2.0.0 on host system

Seems on my OpenSuse 13.1 test system they have Ruby 2.0.0p247
And to make things easy for us programmers they put '2.0' behind each binary.
So 'facter' gets to be 'facter2.0', 'puppet' is 'puppet2.0'

add-el-extras (and possibly other CLI options) not accessible via DSL

There doesn't seem to be an intuitive way to use the add_el_extras method to install EPEL through the DSL or through the spec_helper - it appears to only be a command line option (--add-el-extras).

This would be really useful to have accessible at least via the spec_helper like install_puppet, or via the DSL.

Docker Hypervisor fails with BEAKER_provision=no

Hi, first let me appreciate beaker. It is amazingly helpful. 👍

I have switched from vagrant to docker. I like how the Docker images can be based on other images. I would like to create a simple image once with BEAKER_destroy=no. On further runs with BEAKER_destroy=no BEAKER_provision=no a new container should be created without having to run all those necessary apt-get calls.

But when I run BEAKER_destroy=no BEAKER_provision=no rake acceptance I get:

Hypervisor for debian-7-4 is none
Beaker::Hypervisor, found some none boxes to create

debian-7-4 17:24:13$  dpkg -s curl  
Warning: Try 1 -- Host debian-7-4 unreachable: getaddrinfo: Name or service not known
Warning: Trying again in 1 seconds
Warning: Try 2 -- Host debian-7-4 unreachable: getaddrinfo: Name or service not known
Warning: Trying again in 1 seconds
Warning: Try 3 -- Host debian-7-4 unreachable: getaddrinfo: Name or service not known
Warning: Trying again in 2 seconds
Warning: Try 4 -- Host debian-7-4 unreachable: getaddrinfo: Name or service not known
Warning: Trying again in 3 seconds
Warning: Try 5 -- Host debian-7-4 unreachable: getaddrinfo: Name or service not known
Warning: Trying again in 5 seconds
Warning: Try 6 -- Host debian-7-4 unreachable: getaddrinfo: Name or service not known
Warning: Trying again in 8 seconds
Warning: Try 7 -- Host debian-7-4 unreachable: getaddrinfo: Name or service not known
Warning: Trying again in 13 seconds
Warning: Try 8 -- Host debian-7-4 unreachable: getaddrinfo: Name or service not known
Warning: Trying again in 21 seconds
Warning: Try 9 -- Host debian-7-4 unreachable: getaddrinfo: Name or service not known
Warning: Trying again in 34 seconds
Warning: Try 10 -- Host debian-7-4 unreachable: getaddrinfo: Name or service not known
Warning: Trying again in 55 seconds
Warning: Try 11 -- Host debian-7-4 unreachable: getaddrinfo: Name or service not known
Warning: Trying again in 89 seconds
Failed to connect to debian-7-4
Failed: errored in validate
#<SocketError: getaddrinfo: Name or service not known>
/home/arthur/.rvm/gems/ruby-2.1.1/gems/beaker-rspec-2.2.6/lib/beaker-rspec/spec_helper.rb:41
/home/arthur/.rvm/gems/ruby-2.1.1/gems/beaker-rspec-2.2.6/lib/beaker-rspec/spec_helper.rb:5
/home/arthur/.rvm/gems/ruby-2.1.1/gems/beaker-rspec-2.2.6/lib/beaker-rspec.rb:5
/home/arthur/.rvm/gems/ruby-2.1.1/gems/beaker-rspec-2.2.6/lib/beaker-rspec.rb:5
/home/arthur/.rvm/gems/ruby-2.1.1/gems/beaker-rspec-2.2.6/lib/beaker-rspec.rb:1
/home/arthur/projects/puppet/puppet-puma/spec/spec_helper_acceptance.rb:1
/home/arthur/projects/puppet/puppet-puma/spec/spec_helper_acceptance.rb:1
/home/arthur/projects/puppet/puppet-puma/spec/acceptance/puma__app_spec.rb:1
/home/arthur/projects/puppet/puppet-puma/spec/acceptance/puma__app_spec.rb:1
/home/arthur/.rvm/gems/ruby-2.1.1/gems/net-ssh-2.9.1/lib/net/ssh/transport/session.rb:70:in `initialize': getaddrinfo: Name or service not known (SocketError)
    from /home/arthur/.rvm/gems/ruby-2.1.1/gems/net-ssh-2.9.1/lib/net/ssh/transport/session.rb:70:in `open'
    from /home/arthur/.rvm/gems/ruby-2.1.1/gems/net-ssh-2.9.1/lib/net/ssh/transport/session.rb:70:in `block in initialize'
    from /home/arthur/.rvm/rubies/ruby-2.1.1/lib/ruby/2.1.0/timeout.rb:91:in `block in timeout'
    from /home/arthur/.rvm/rubies/ruby-2.1.1/lib/ruby/2.1.0/timeout.rb:35:in `block in catch'
    from /home/arthur/.rvm/rubies/ruby-2.1.1/lib/ruby/2.1.0/timeout.rb:35:in `catch'
    from /home/arthur/.rvm/rubies/ruby-2.1.1/lib/ruby/2.1.0/timeout.rb:35:in `catch'
    from /home/arthur/.rvm/rubies/ruby-2.1.1/lib/ruby/2.1.0/timeout.rb:106:in `timeout'
    from /home/arthur/.rvm/rubies/ruby-2.1.1/lib/ruby/2.1.0/timeout.rb:127:in `timeout'
    from /home/arthur/.rvm/gems/ruby-2.1.1/gems/net-ssh-2.9.1/lib/net/ssh/transport/session.rb:67:in `initialize'
    from /home/arthur/.rvm/gems/ruby-2.1.1/gems/net-ssh-2.9.1/lib/net/ssh.rb:202:in `new'
    from /home/arthur/.rvm/gems/ruby-2.1.1/gems/net-ssh-2.9.1/lib/net/ssh.rb:202:in `start'
    from /home/arthur/.rvm/gems/ruby-2.1.1/gems/beaker-1.17.4/lib/beaker/ssh_connection.rb:41:in `connect'
    from /home/arthur/.rvm/gems/ruby-2.1.1/gems/beaker-1.17.4/lib/beaker/ssh_connection.rb:32:in `connect'
    from /home/arthur/.rvm/gems/ruby-2.1.1/gems/beaker-1.17.4/lib/beaker/host.rb:196:in `connection'
    from /home/arthur/.rvm/gems/ruby-2.1.1/gems/beaker-1.17.4/lib/beaker/host.rb:224:in `block in exec'
    from /home/arthur/.rvm/rubies/ruby-2.1.1/lib/ruby/2.1.0/benchmark.rb:294:in `realtime'
    from /home/arthur/.rvm/gems/ruby-2.1.1/gems/beaker-1.17.4/lib/beaker/host.rb:223:in `exec'
    from /home/arthur/.rvm/gems/ruby-2.1.1/gems/beaker-1.17.4/lib/beaker/host/unix/pkg.rb:30:in `check_for_package'
    from /home/arthur/.rvm/gems/ruby-2.1.1/gems/beaker-1.17.4/lib/beaker/host_prebuilt_steps.rb:94:in `block (2 levels) in validate_host'
    from /home/arthur/.rvm/gems/ruby-2.1.1/gems/beaker-1.17.4/lib/beaker/host_prebuilt_steps.rb:93:in `each'
    from /home/arthur/.rvm/gems/ruby-2.1.1/gems/beaker-1.17.4/lib/beaker/host_prebuilt_steps.rb:93:in `block in validate_host'
    from /home/arthur/.rvm/gems/ruby-2.1.1/gems/beaker-1.17.4/lib/beaker/shared/host_manager.rb:74:in `run_block_on'
    from /home/arthur/.rvm/gems/ruby-2.1.1/gems/beaker-1.17.4/lib/beaker/shared/host_manager.rb:71:in `block in run_block_on'
    from /home/arthur/.rvm/gems/ruby-2.1.1/gems/beaker-1.17.4/lib/beaker/shared/host_manager.rb:70:in `map'
    from /home/arthur/.rvm/gems/ruby-2.1.1/gems/beaker-1.17.4/lib/beaker/shared/host_manager.rb:70:in `run_block_on'
    from /home/arthur/.rvm/gems/ruby-2.1.1/gems/beaker-1.17.4/lib/beaker/dsl/patterns.rb:32:in `block_on'
    from /home/arthur/.rvm/gems/ruby-2.1.1/gems/beaker-1.17.4/lib/beaker/host_prebuilt_steps.rb:84:in `validate_host'
    from /home/arthur/.rvm/gems/ruby-2.1.1/gems/beaker-1.17.4/lib/beaker/hypervisor.rb:99:in `validate'
    from /home/arthur/.rvm/gems/ruby-2.1.1/gems/beaker-1.17.4/lib/beaker/network_manager.rb:61:in `block in validate'
    from /home/arthur/.rvm/gems/ruby-2.1.1/gems/beaker-1.17.4/lib/beaker/network_manager.rb:60:in `each_key'
    from /home/arthur/.rvm/gems/ruby-2.1.1/gems/beaker-1.17.4/lib/beaker/network_manager.rb:60:in `validate'
    from /home/arthur/.rvm/gems/ruby-2.1.1/gems/beaker-rspec-2.2.6/lib/beaker-rspec/beaker_shim.rb:36:in `validate'
    from /home/arthur/.rvm/gems/ruby-2.1.1/gems/beaker-rspec-2.2.6/lib/beaker-rspec/spec_helper.rb:41:in `block in <top (required)>'
    from /home/arthur/.rvm/gems/ruby-2.1.1/gems/rspec-core-2.99.2/lib/rspec/core.rb:154:in `configure'
    from /home/arthur/.rvm/gems/ruby-2.1.1/gems/beaker-rspec-2.2.6/lib/beaker-rspec/spec_helper.rb:5:in `<top (required)>'
    from /home/arthur/.rvm/gems/ruby-2.1.1/gems/beaker-rspec-2.2.6/lib/beaker-rspec.rb:5:in `require'
    from /home/arthur/.rvm/gems/ruby-2.1.1/gems/beaker-rspec-2.2.6/lib/beaker-rspec.rb:5:in `<module:BeakerRSpec>'
    from /home/arthur/.rvm/gems/ruby-2.1.1/gems/beaker-rspec-2.2.6/lib/beaker-rspec.rb:1:in `<top (required)>'
    from /home/arthur/projects/puppet/puppet-puma/spec/spec_helper_acceptance.rb:1:in `require'
    from /home/arthur/projects/puppet/puppet-puma/spec/spec_helper_acceptance.rb:1:in `<top (required)>'
    from /home/arthur/projects/puppet/puppet-puma/spec/acceptance/puma__app_spec.rb:1:in `require'
    from /home/arthur/projects/puppet/puppet-puma/spec/acceptance/puma__app_spec.rb:1:in `<top (required)>'
    from /home/arthur/.rvm/gems/ruby-2.1.1/gems/rspec-core-2.99.2/lib/rspec/core/configuration.rb:1065:in `load'
    from /home/arthur/.rvm/gems/ruby-2.1.1/gems/rspec-core-2.99.2/lib/rspec/core/configuration.rb:1065:in `block in load_spec_files'
    from /home/arthur/.rvm/gems/ruby-2.1.1/gems/rspec-core-2.99.2/lib/rspec/core/configuration.rb:1065:in `each'
    from /home/arthur/.rvm/gems/ruby-2.1.1/gems/rspec-core-2.99.2/lib/rspec/core/configuration.rb:1065:in `load_spec_files'
    from /home/arthur/.rvm/gems/ruby-2.1.1/gems/rspec-core-2.99.2/lib/rspec/core/command_line.rb:18:in `run'
    from /home/arthur/.rvm/gems/ruby-2.1.1/gems/rspec-core-2.99.2/lib/rspec/core/runner.rb:103:in `run'
    from /home/arthur/.rvm/gems/ruby-2.1.1/gems/rspec-core-2.99.2/lib/rspec/core/runner.rb:17:in `block in autorun'
/home/arthur/.rvm/rubies/ruby-2.1.1/bin/ruby -S rspec spec/acceptance failed

That is how my default.yml looks like:

HOSTS:
  debian-7-4:
    platform: debian-7.4-x64
    image: debian:7.4
    hypervisor: docker
CONFIG:
   type: foss

Is there no way to avoid the creation of a new image each run?

EDIT: Maybe I am misunderstanding things. But as far as I can see destroy really stops the current container and then deletes the image? Whereas provision creates the image and runs the necessary apt-get calls. Shouldn't those two steps be split up into something like create image, start container from image, stop and destroy container and destroy container and images.
I think the two steps in the middle should not be part of provision and destroy, no?

Difference in handling apply_manifest(pp, :expect_failures => true)

Im applying a manifest and i expect it to fail.
With the help of @hunner i've set the run to this:

apply_manifest(pp, :expect_failures => true)

On puppet 2.7 and 3.1 i get the following:

debian-70rc1-x64 $ env PATH="/usr/bin:/opt/puppet-git-repos/hiera/bin:${PATH}" RUBYLIB="/opt/puppet-git-repos/hiera/lib:/opt/puppet-git-repos/hiera-puppet/lib:${RUBYLIB}" puppet apply  --verbose --detailed-exitcodes /tmp/apply_manifest.pp.ZbMeaB
info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/puppet_vardir.rb
info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/facter_dot_d.rb
info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/root_home.rb
info: Loading facts in /etc/puppet/modules/stdlib/lib/facter/pe_version.rb
info: Applying configuration version '1391813873'
info: FileBucket adding {md5}da672da1bfe0db7baac2c2287b35b00f
info: /Stage[main]//Elasticsearch::Template[foo]/File[/etc/elasticsearch/templates_import/elasticsearch-template-foo.json]: Filebucketed /etc/elasticsearch/templates_import/elasticsearch-template-foo.json to puppet with sum da672da1bfe0db7baac2c2287b35b00f
notice: /Stage[main]//Elasticsearch::Template[foo]/File[/etc/elasticsearch/templates_import/elasticsearch-template-foo.json]/content: content changed '{md5}da672da1bfe0db7baac2c2287b35b00f' to '{md5}6c6e2f263dbb3bc2657b380309602dd7'
info: /Stage[main]//Elasticsearch::Template[foo]/File[/etc/elasticsearch/templates_import/elasticsearch-template-foo.json]: Scheduling refresh of Exec[delete_template_foo]
notice: /Stage[main]//Elasticsearch::Template[foo]/Exec[delete_template_foo]: Triggered 'refresh' from 1 events
info: /Stage[main]//Elasticsearch::Template[foo]/Exec[delete_template_foo]: Scheduling refresh of Exec[insert_template_foo]
err: /Stage[main]//Elasticsearch::Template[foo]/Exec[insert_template_foo]: Failed to call refresh: curl -sL -w "%{http_code}\n" -XPUT http://localhost:9200/_template/foo -d @/etc/elasticsearch/templates_import/elasticsearch-template-foo.json -o /dev/null | egrep "(200|201)" > /dev/null returned 1 instead of one of [0] at /etc/puppet/modules/elasticsearch/manifests/template.pp:110
notice: Finished catalog run in 30.97 seconds
err: /File[/var/lib/puppet/rrd]: Could not evaluate: Could not find group puppet
err: Could not send report: Got 1 failure(s) while initializing: Could not evaluate: Could not find group puppet

debian-70rc1-x64 executed in 32.93 seconds
Exited: 2
    run should fail (FAILED - 1)



Failures:

  1) elasticsearch template define: Insert a template with bad json content run should fail
     Failure/Error: apply_manifest(pp, :expect_failures => true)
     Beaker::Host::CommandFailure:
       Host 'debian-70rc1-x64' exited with 2 running:
        env PATH="/usr/bin:/opt/puppet-git-repos/hiera/bin:${PATH}" RUBYLIB="/opt/puppet-git-repos/hiera/lib:/opt/puppet-git-repos/hiera-puppet/lib:${RUBYLIB}" puppet apply  --verbose --detailed-exitcodes /tmp/apply_manifest.pp.ZbMeaB
       Last 10 lines of output were:
        info: FileBucket adding {md5}da672da1bfe0db7baac2c2287b35b00f
        info: /Stage[main]//Elasticsearch::Template[foo]/File[/etc/elasticsearch/templates_import/elasticsearch-template-foo.json]: Filebucketed /etc/elasticsearch/templates_import/elasticsearch-template-foo.json to puppet with sum da672da1bfe0db7baac2c2287b35b00f
        notice: /Stage[main]//Elasticsearch::Template[foo]/File[/etc/elasticsearch/templates_import/elasticsearch-template-foo.json]/content: content changed '{md5}da672da1bfe0db7baac2c2287b35b00f' to '{md5}6c6e2f263dbb3bc2657b380309602dd7'
        info: /Stage[main]//Elasticsearch::Template[foo]/File[/etc/elasticsearch/templates_import/elasticsearch-template-foo.json]: Scheduling refresh of Exec[delete_template_foo]
        notice: /Stage[main]//Elasticsearch::Template[foo]/Exec[delete_template_foo]: Triggered 'refresh' from 1 events
        info: /Stage[main]//Elasticsearch::Template[foo]/Exec[delete_template_foo]: Scheduling refresh of Exec[insert_template_foo]
        err: /Stage[main]//Elasticsearch::Template[foo]/Exec[insert_template_foo]: Failed to call refresh: curl -sL -w "%{http_code}\n" -XPUT http://localhost:9200/_template/foo -d @/etc/elasticsearch/templates_import/elasticsearch-template-foo.json -o /dev/null | egrep "(200|201)" > /dev/null returned 1 instead of one of [0] at /etc/puppet/modules/elasticsearch/manifests/template.pp:110
        notice: Finished catalog run in 30.97 seconds
        err: /File[/var/lib/puppet/rrd]: Could not evaluate: Could not find group puppet
        err: Could not send report: Got 1 failure(s) while initializing: Could not evaluate: Could not find group puppet
     # ./spec/acceptance/003_template_spec.rb:117:in `block (3 levels) in <top (required)>'

On Puppet 3.2/3.3/3.4 it works as expected and flags the test as passed ( as I'm expecting it to fail )

Can give more details via IRC.

Adding a utility method for configuring the puppet.conf file

There are many use cases for this, testing non-standard puppet
installations, testing switching environments etc.

Cannot run beaker on RHEL 6.X

I've tried to install beaker gem on RHEL 6.5 with ruby 1.8.7 (default in RHEL distribution) and I've got the error:

[root@host302 ~]# gem install beaker
ERROR: Error installing beaker:
nokogiri requires Ruby version >= 1.9.2.

I have the following environment:

[root@host302 beaker]# gem list

*** LOCAL GEMS ***

gem_plugin (0.2.3)
hocon (0.0.4)
inifile (2.0.2)
json (1.8.1, 1.5.5)
net-scp (1.2.1)
net-ssh (2.9.1)
rake (0.8.7)
[root@host302 beaker]# gem --version
1.3.7
[root@host302 beaker]# ruby --version
ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]

I've tried then with ruby193 from RHEL Software Collections. This time, the gem install finished successfully but the beaker --help give the error:
/opt/rh/ruby193/root/usr/local/bin/beaker --help
Warning: you should require 'minitest/autorun' instead.
Warning: or add 'gem "minitest"' before 'require "minitest/autorun"'
From:
/opt/rh/ruby193/root/usr/share/ruby/test/unit/assertions.rb:1:in <top (required)>' /opt/rh/ruby193/root/usr/local/share/gems/gems/beaker-1.16.0/lib/beaker/command_factory.rb:1:in<top (required)>'
/opt/rh/ruby193/root/usr/local/share/gems/gems/beaker-1.16.0/lib/beaker/host/windows.rb:2:in block in <top (required)>' /opt/rh/ruby193/root/usr/local/share/gems/gems/beaker-1.16.0/lib/beaker/host/windows.rb:1:ineach'
/opt/rh/ruby193/root/usr/local/share/gems/gems/beaker-1.16.0/lib/beaker/host/windows.rb:1:in <top (required)>' /opt/rh/ruby193/root/usr/local/share/gems/gems/beaker-1.16.0/lib/beaker/host.rb:281:inblock in module:Beaker'
/opt/rh/ruby193/root/usr/local/share/gems/gems/beaker-1.16.0/lib/beaker/host.rb:280:in each' /opt/rh/ruby193/root/usr/local/share/gems/gems/beaker-1.16.0/lib/beaker/host.rb:280:inmodule:Beaker'
/opt/rh/ruby193/root/usr/local/share/gems/gems/beaker-1.16.0/lib/beaker/host.rb:9:in <top (required)>' /opt/rh/ruby193/root/usr/local/share/gems/gems/beaker-1.16.0/lib/beaker/test_case.rb:2:inblock in <top (required)>'
/opt/rh/ruby193/root/usr/local/share/gems/gems/beaker-1.16.0/lib/beaker/test_case.rb:1:in each' /opt/rh/ruby193/root/usr/local/share/gems/gems/beaker-1.16.0/lib/beaker/test_case.rb:1:in<top (required)>'
/opt/rh/ruby193/root/usr/local/share/gems/gems/beaker-1.16.0/lib/beaker/test_suite.rb:5:in block in <top (required)>' /opt/rh/ruby193/root/usr/local/share/gems/gems/beaker-1.16.0/lib/beaker/test_suite.rb:4:ineach'
/opt/rh/ruby193/root/usr/local/share/gems/gems/beaker-1.16.0/lib/beaker/test_suite.rb:4:in <top (required)>' /opt/rh/ruby193/root/usr/local/share/gems/gems/beaker-1.16.0/lib/beaker.rb:6:inblock in module:Beaker'
/opt/rh/ruby193/root/usr/local/share/gems/gems/beaker-1.16.0/lib/beaker.rb:4:in each' /opt/rh/ruby193/root/usr/local/share/gems/gems/beaker-1.16.0/lib/beaker.rb:4:inmodule:Beaker'
/opt/rh/ruby193/root/usr/local/share/gems/gems/beaker-1.16.0/lib/beaker.rb:2:in <top (required)>' /opt/rh/ruby193/root/usr/local/share/gems/gems/beaker-1.16.0/bin/beaker:4:in<top (required)>'
/opt/rh/ruby193/root/usr/local/bin/beaker:23:in load' /opt/rh/ruby193/root/usr/local/bin/beaker:23:in

'
MiniTest::Unit::TestCase is now Minitest::Test. From /opt/rh/ruby193/root/usr/share/ruby/test/unit/testcase.rb:8:in <module:Unit>' /opt/rh/ruby193/root/usr/share/ruby/test/unit.rb:328:inautorun': uninitialized class variable @@installed_at_exit in Test::Unit::Runner (NameError)
from /opt/rh/ruby193/root/usr/share/ruby/test/unit.rb:640:in <top (required)>' from /opt/rh/ruby193/root/usr/share/rubygems/rubygems/custom_require.rb:36:inrequire'
from /opt/rh/ruby193/root/usr/share/rubygems/rubygems/custom_require.rb:36:in require' from /opt/rh/ruby193/root/usr/local/share/gems/gems/beaker-1.16.0/lib/beaker/test_case.rb:10:in<top (required)>'
from /opt/rh/ruby193/root/usr/share/rubygems/rubygems/custom_require.rb:36:in require' from /opt/rh/ruby193/root/usr/share/rubygems/rubygems/custom_require.rb:36:inrequire'
from /opt/rh/ruby193/root/usr/local/share/gems/gems/beaker-1.16.0/lib/beaker/test_suite.rb:5:in

block in <top (required)>' from /opt/rh/ruby193/root/usr/local/share/gems/gems/beaker-1.16.0/lib/beaker/test_suite.rb:4:in

each'
from /opt/rh/ruby193/root/usr/local/share/gems/gems/beaker-1.16.0/lib/beaker/test_suite.rb:4:in <top (required)>' from /opt/rh/ruby193/root/usr/share/rubygems/rubygems/custom_require.rb:36:inrequire'
from /opt/rh/ruby193/root/usr/share/rubygems/rubygems/custom_require.rb:36:in require' from /opt/rh/ruby193/root/usr/local/share/gems/gems/beaker-1.16.0/lib/beaker.rb:6:inblock in module:Beaker'
from /opt/rh/ruby193/root/usr/local/share/gems/gems/beaker-1.16.0/lib/beaker.rb:4:in each' from /opt/rh/ruby193/root/usr/local/share/gems/gems/beaker-1.16.0/lib/beaker.rb:4:inmodule:Beaker'
from /opt/rh/ruby193/root/usr/local/share/gems/gems/beaker-1.16.0/lib/beaker.rb:2:in <top (required)>' from /opt/rh/ruby193/root/usr/share/rubygems/rubygems/custom_require.rb:36:inrequire'
from /opt/rh/ruby193/root/usr/share/rubygems/rubygems/custom_require.rb:36:in require' from /opt/rh/ruby193/root/usr/local/share/gems/gems/beaker-1.16.0/bin/beaker:4:in<top (required)>'
from /opt/rh/ruby193/root/usr/local/bin/beaker:23:in load' from /opt/rh/ruby193/root/usr/local/bin/beaker:23:in'

What is the recommended distribution to run beaker?