Code Monkey home page Code Monkey logo

kitchen-inspec's Introduction

Kitchen::InSpec - A Test Kitchen Verifier for InSpec

  • Project State: Active
  • Issues Response SLA: 3 business days
  • Pull Request Response SLA: 3 business days

For more information on project states and SLAs, see this documentation.

Build Status Master Gem Version

This is the kitchen driver for InSpec. To see the project in action, we have the following test-kitchen examples available:

Installation

Note: kitchen-inspec ships as part of ChefDK. Installation is not necessary for DK users.

Add this line to your application's Gemfile:

gem 'kitchen-inspec'

And then execute:

bundle

Or install it yourself as:

gem install kitchen-inspec

Usage

In your kitchen.yml include

verifier:
  name: inspec

Optionally specify sudo and sudo_command

verifier:
  name: inspec
  sudo: true
  sudo_command: 'skittles'

You can also specify the host and port to be used by InSpec when targeting the node. Otherwise, it defaults to the hostname and port used by kitchen for converging.

verifier:
  name: inspec
  host: 192.168.56.40
  port: 22

Expected Directory Structure

By default kitchen-inspec expects test to be in test/integration/%suite% directory structure (we use Chef as provisioner here):

.
├── Berksfile
├── Gemfile
├── README.md
├── metadata.rb
├── recipes
│   ├── default.rb
│   └── nginx.rb
└── test
    └── integration
        └── default
            └── web_spec.rb

Directory Structure with complete profile

A complete profile is used here, including a custom InSpec resource named gordon_config:

.
├── Berksfile
├── Gemfile
├── README.md
├── metadata.rb
├── recipes
│   ├── default.rb
│   └── nginx.rb
└── test
    └── integration
        └── default
            ├── controls
            │   └── gordon.rb
            ├── inspec.yml
            └── libraries
                └── gordon_config.rb

Combination with other testing frameworks

If you need support with other testing frameworks, we recommend to place the tests in test/integration/%suite%/inspec:

.
├── Berksfile
├── Gemfile
├── README.md
├── metadata.rb
├── recipes
│   ├── default.rb
│   └── nginx.rb
└── test
    └── integration
        └── default
            └── inspec
                └── web_spec.rb

Specifying the Sudo Command

You can enable/disable sudo and set your own custom sudo command.

verifier:
  name: inspec
  sudo: true
  sudo_command: 'skittles'

Custom Host Settings

You can also specify the host, port, and proxy settings to be used by InSpec when targeting the node. Otherwise, it defaults to the hostname and port used by kitchen for converging.

verifier:
  name: inspec
  host: 192.168.56.40
  port: 22
  proxy_command: ssh [email protected] -W %h:%p

Custom Outputs

If you want to customize the output file per platform or test suite you can use template format for your output variable. Current flags supported:

  • %{platform}
  • %{suite}
verifier:
  name: inspec
  reporter:
    - cli
    - junit:path/to/results/%{platform}_%{suite}_inspec.xml

You can also decide to only run specific controls, instead of a full profile. This is done by specifying a list of controls:

suites:
  - name: supermarket
    run_list:
      - recipe[apt]
      - recipe[ssh-hardening]
    verifier:
      inspec_tests:
        - name: dev-sec/ssh-baseline
      controls:
        - sshd-46
    ...

Use remote InSpec profiles

In case you want to reuse tests across multiple cookbooks, they should become an extra artifact independent of a Chef cookbook, called InSpec profiles. Those can be easily added to existing local tests as demonstrated in previous sections. To include remote profiles, adapt the verifier attributes in kitchen.yml

suites:
  - name: default
    verifier:
      inspec_tests:
        - name: ssh-hardening
          url: https://github.com/dev-sec/tests-ssh-hardening

inspec_tests accepts all values that inspec exec profile would expect. We support:

  • local directory eg. path: /path/to/profile
  • github url git: https://github.com/dev-sec/tests-ssh-hardening.git
  • Chef Supermarket name: hardening/ssh-hardening # defaults to supermarket (list all available profiles with inspec supermarket profiles)
  • Chef Compliance name: ssh compliance: base/ssh

The following example illustrates the usage in a kitchen.yml

suites:
  - name: contains_inspec
    run_list:
      - recipe[apt]
      - recipe[yum]
      - recipe[ssh-hardening]
      - recipe[os-hardening]
    verifier:
      inspec_tests:
        - path: path/to/some/local/tests
        - name: ssh-hardening
          url: https://github.com/dev-sec/tests-ssh-hardening/archive/master.zip
        - name: os-hardening
          git: https://github.com/dev-sec/tests-os-hardening.git
  - name: supermarket
    run_list:
      - recipe[apt]
      - recipe[yum]
      - recipe[ssh-hardening]
    verifier:
      inspec_tests:
        - name: hardening/ssh-hardening  # name only defaults to supermarket
        - name: ssh-supermarket  # alternatively, you can explicitly specify that the profile is from supermarket in this way
          supermarket: hardening/ssh-hardening
          supermarket_url: http://supermarket.example.com
  # before you are able to use the compliance plugin, you need to run
  # insecure is only required if you use self-signed certificates
  # $ inspec compliance login https://compliance.test --user admin --insecure --token ''
  - name: compliance
    run_list:
      - recipe[apt]
      - recipe[yum]
      - recipe[ssh-hardening]
    verifier:
      inspec_tests:
        - name: ssh
          compliance: base/ssh

Use inputs with your InSpec profiles

To run a profile with inputs defined inline, you can adapt your kitchen.yml:

    verifier:
      inspec_tests:
        - path: test/integration/attributes
      inputs:
        user: bob
        password: secret

You can also define your inputs in external files. Adapt your kitchen.yml to point to those files:

    verifier:
      inspec_tests:
        - path: test/integration/attributes
      input_files:
        - test/integration/profile-attribute.yml

Use waivers with your InSpec profiles

You can define your waivers in external files:

    verifier:
      inspec_tests:
        - path: test/integration/attributes
      input_files:
        - test/integration/profile-attribute.yml
      waiver_files:
        - test/integration/control-waiver-01.yml

Use inspec plugins

By default, the verifier loads Inspec plugins such as additional Reporter or Input plugins. This adds a small delay as the system scans for plugins. If you know you are not using special Reporters or Inputs, you can disable plugin loading:

    verifier:
      load_plugins: false

Some Inspec plugins allow further configuration. You can supply these settings as well with InSpec 4.26 or newer:

    verifier:
      plugin_config:
        example_plugin_name:
          example_setting: "Example value"

When using Input plugins, please be aware that input values get cached between suites. If you want to re-evaluate these values for every suite, you can deactivate the cache:

    verifier:
      cache_inputs: false

Chef InSpec Backend Cache

Chef InSpec uses a cache when executing commands and accessing files on the remote target. The cache is enabled by default. To disable the cache:

    verifier:
      backend_cache: false

Development

After checking out the repo, run bin/setup to install dependencies. Then, run rake spec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/inspec/kitchen-inspec.

License

Apache 2.0 (see LICENSE)

kitchen-inspec's People

Contributors

adamleff avatar alexpop avatar arlimus avatar brettlangdon avatar cheesesashimi avatar chef-ci avatar chef-expeditor[bot] avatar chris-rock avatar clintoncwolfe avatar coderanger avatar dependabot-preview[bot] avatar fnichol avatar i5pranay93 avatar jayashrig158 avatar jaym avatar jeremymv2 avatar jkeiser avatar jquick avatar miah avatar mrwong99 avatar nik08 avatar nikhil2611 avatar sawanoboly avatar srenatus avatar swatu15 avatar tas50 avatar tduffield avatar thheinen avatar tyler-ball avatar wiebe avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

kitchen-inspec's Issues

Add way to define alternate test-path

Right now, the path kitchen-inspec searches for tests is hard-coded.(https://github.com/chef/kitchen-inspec/blob/master/lib/kitchen/verifier/inspec.rb#L72)
It takes the base path (e.g. test/integration/) and then appends the suite-name (default).

Now we have the problem that we want to share tests between different suites (dev-sec/ansible-ssh-hardening#61):
The tests are in test/integration/default/inspec/, but kitchen-inspec searches in test/integration/ansible-latest/inspec/.

Therefore I propose to let the user define the test search path, either as a relative variable to the base path (in this case default/ or as an absolute path (test/integration/default).

Custom inspec resources not loaded: 'undefined method'

I'm trying to implement some kitchen-inspec tests using a custom inspec resource in the spirit of this blog post. Thus I have created a file named test/integration/my-suite/libraries/query.rb with the following content:

module Inspec::Resources
  class QueryResource < Inspec.resource(1)
    name 'query'
    desc 'dummy inspec resource'

    def initialize(args)
      @args = args
    end

    def bar?
      @args == 'foo'
    end
  end
end

My test case test/integration/my-suite/default_spec.rb is

describe query('foo') do
  it { should be_bar }
end

When running kitchen via kitchen verify my-suite the new resource doesn't seem to be available:

$ kitchen verify my-suite
Ignoring unf_ext-0.0.7.2 because its extensions are not built.  Try: gem pristine unf_ext --version 0.0.7.2
-----> Starting Kitchen (v1.13.2)
-----> Verifying <zabbix-resources-ubuntu-1404>...
       Use `/Users/robert/work/westernacher/chef-repo/cookbooks/zabbix_wps/test/integration/zabbix-resources` for testing
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: 1 actions failed.
>>>>>>     Failed to complete #verify action: [undefined method `query' for #<#<Class:0x007ffd8ad6a058>:0x007ffd8ad697c0>] on zabbix-resources-ubuntu-1404
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration

I also tried putting the inspec resource definition directly in my test-case with the same result.

Thanks in advance!

How to verify with a local profile

I would like my verifier to go against a local profile that is available on my hard drive. Something like:

verifier: inspec
  profile: ../../profiles/my-profile

Is this currently possible?

Load directory from single inspec directory

As highlighted in chef/chef#4606 we need to support single inspec subdirectories as well.

acceptance git:(master) tree
.
├── Gemfile
├── README.md
├── basics
│   └── test
│       └── integration
│           └── chef-current-install
│               └── serverspec
│                   ├── chef_client_spec.rb
│                   └── spec_helper.rb
├── top-cookbooks
├── trivial
│   └── test
│       └── integration
│           └── chef-current-install
│               └── inspec
│                   └── chef_client_spec.rb
└── windows-service
    └── test
        └── integration
            └── chef-windows-service
                └── inspec
                    └── chef_windows_service_spec.rb

16 directories, 6 files

To fix that we need to add:

`kitchen verify` fails on Windows

I have the latest ChefDK installed and installed kitchen-inspec via "chef gem install kitchen-inspec".

When I run kitchen verify, I get the following error:

>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: Failed to complete #verify action: [undefined method `endpoint' for nil:NilClass]
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration

I'm running Test Kitchen on Windows 10 and the machine I am trying to test is Windows 2012 R2.

inspec fails with kitchen verify -c N

Executing multiple simultaneous kitchen verify runs using concurrency (-c 5, for example) fails, apparently using the same configuration for all nodes (note target in the attached log has the same port for each run, when the boxes are all running on different ports).

When running serially instead of concurrently, it works fine.

inspec-concurrent-kitchen.log.txt

Make it easier to filter output

Consider the scenario where you have a large number of compliance failures (for example, from the CIS benchmarks) and you want to tackle a few at a time.

With local remediation through Test Kitchen, you can scan temporary instances directly against a full compliance profile. kitchen verify potentially produces a huge volume of output, which can be difficult to wade through.

In the Learn Chef tutorials (CentOS) (Ubuntu), we have the user grep the output of kitchen verify to find the errors they care about.

What makes this difficult is that you need to grep for the matcher code itself. For example:

$ kitchen verify > verify.txt 2>&1
$ cat verify.txt | grep -A 10 '\d*).*should match /^Status: active$/'

It can be painful running grep because you need to escape the matcher code itself (because you're grepping for a regex from a regex tool.)

A few ways that might make this easier are to:

  1. Add the name of the rule to the output (for example, Set SSH Protocol to 2). This would make gripping easier. If the user is scaring against multiple profiles, perhaps also include the profile name as a way to namespace things (perhaps overkill.)

  2. Add a way for the user to specify in their .kitchen.yml file which rules they care about. Then scan against only those rules. For example:

    verifier:
      name: inspec
      inspec_tests:
        - compliance://cis/cis-centos7-level1
      filter_rules:
        - 'Set SSH Protocol to 2'
        - 'Ensure Firewall is active'
    
  3. Enable the user to create a .inspec.yml file (or similar) that defines specific rules to include or exclude, similar to how .rubocop.yml works.

These options would produce much leaner output when you run kitchen verify, and would eliminate the need to redirect the output to a file and then process that file.

Those are just a few ideas that came to mind. Perhaps there are better alternatives.

Output file support for multiple platforms and multiple test suites

There is no way to retrieve the output of InSpec from kitchen when you have multiple platforms and multiple test suites in your kitchen setup.

For example if you have support for centos6 and centos7 and multiple test suites defined, default, datacenter1 and datacenter2 you'll have 6 InSpec executions so you'll have 6 different inspec output files but there is no way to specify them.

Skipped tests report as failures in kitchen

When using the default tests created by chef generate cookbook, if I leave them in, and they fail, even though they are marked skip, they report as a failure in kitchen.

Test:

unless os.windows?
  describe user('root') do
    it { should exist }
    skip 'This is an example test, replace with your own test.'
  end
end

describe port(80) do
  it { should_not be_listening }
  skip 'This is an example test, replace with your own test.'
end

If the machine does reply on port 80, this is the output of kitchen verify:

  User root
     ✔  should exist
     ↺  This is an example test, replace with your own test.
  Port 80
     ∅  should not be listening
     expected `Port 80.listening?` to return false, got true
     ↺  This is an example test, replace with your own test.

Test Summary: 1 successful, 1 failures, 2 skipped
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: 1 actions failed.
>>>>>>     Verify failed on instance <default-centos-72>.  Please see .kitchen/logs/default-centos-72.log for more details
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration

kitchen inspec-shell default-ubuntu-1404

I often find myself looking up the ssh command-line parameters used in kitchen login [instance] to run inspec shell against it. Wouldn't it be great if this could be easier?

Using profiles with kitchen-inspec

I'm not sure if this is a duplicate of #25 but I'm having trouble getting kitchen-inspec to work with profiles.

I have the following directory structure:

test
└── integration
    ├── linux
    │   ├── controls
    │   │   ├── ntp_spec.rb
    │   │   └── selinux_spec.rb
    │   └── inspec.yml
    └── pm-inbound
        └── default_spec.rb

My test/pm-inbound/default_spec.rb looks like this:

require_controls 'linux' do

  control 'ntp-config'

end

inspec seems to work (obviously tests fail here b/c I haven't passed a host):

[git] $ inspec exec test/integration/pm-inbound/default_spec.rb --profiles-path test/integration
FFFF

Failures:

  1) System Package ntp should be installed
     Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }
       expected that `System Package ntp` is installed
     # controls/ntp_spec.rb:8:in `block (3 levels) in load'

  2) Service ntpd should be enabled
     Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }
       expected that `Service ntpd` is enabled
     # controls/ntp_spec.rb:12:in `block (3 levels) in load'

  3) Service ntpd should be installed
     Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }
       expected that `Service ntpd` is installed
     # controls/ntp_spec.rb:13:in `block (3 levels) in load'

  4) Service ntpd should be running
     Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }
       expected that `Service ntpd` is running
     # controls/ntp_spec.rb:14:in `block (3 levels) in load'

Finished in 0.40791 seconds (files took 0.39686 seconds to load)
4 examples, 4 failures

Failed examples:

rspec  # System Package ntp should be installed
rspec  # Service ntpd should be enabled
rspec  # Service ntpd should be installed
rspec  # Service ntpd should be running

However, kitchen-inspec won't work:

[git] $ kitchen verify
-----> Starting Kitchen (v1.6.0)
-----> Verifying <pm-inbound-centos-67>...
       Search `/Users/bkrein/git/ansible/test/integration/pm-inbound` for tests
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: Failed to complete #verify action: [undefined method `control' for #<Inspec::ProfileContext:0x007fbe14677c38>]
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration

zlib(finalizer): the stream was freed prematurely.

My .kitchen.yml:


---
driver:
  name: vagrant
  provision_command: sed -i '/tsflags=nodocks/d' /etc/yum.conf

provisioner:
  name: ansible_playbook
  hosts: localhost
  require_chef_for_busser: false
  require_ruby_for_busser: false
  requirements_path: requirements-test.txt

platforms:
  - name: centos-6.7
  - name: centos-7.1

verifier:
  name: inspec

suites:
  - name: pm-inbound
    provisioner:
      name: ansible_playbook
      hosts: pm-inbound-ord
      playbook: pm-inbound.yml
      additional_copy_path:
        - "."

Vague error if 'platform' value is missing in platform section and inspec profile platfrom/os-family is set

General info

My inspec profile had platform set to 'all'', while my platform section for debian7 did not specify 'platform' value at all. But on the other hand inspec profile has os-family set.

Thus, when invoking kitchen verify I got error like below:

Failed to complete #verify action: [no implicit conversion of nil into Array]

backtrace goes to /home/kaszpir/.rbenv/versions/2.1.3/lib/ruby/gems/2.1.0/gems/kitchen-inspec-0.15.0/lib/kitchen/verifier/inspec.rb:133:in +'`

where line 133 shows:

(local_suite_files + config[:inspec_tests]).compact

After adding some debugging messages I have found out that config[:inspec_tests] is nil

Then I verified platform on inspec profile and in kitchen.yml and they were different (well, actually platfrom was not set for debian in my case)

Steps to reproduce

I believe you can reproduce it with any driver, below is with LXC, also provisioner is rahter not important here.

Proper kitchen.yml - see comments

---
driver_plugin: lxc

driver:
  name: vagrant
  provider: lxc

transport:
  name: sftp # kitchen-sync does not work properly with test-kitchen 1.4.x
  compression_level: 9

provisioner:
  name: puppet_apply
  manifests_path: puppet/manifests
  modules_path: puppet/modules
  hiera_config_path: puppet/hiera.kitchen.yaml # different file mapping than files uploaded by boilerplate
  hiera_data_path: puppet/hiera
  puppet_verbose: true
  require_chef_for_busser: false

verifier:
  name: inspec

platforms:
# https://github.com/fgrehm/vagrant-lxc-base-boxes
# vagrant box add --name debian7lxc --provider lxc vagrant-lxc-wheezy-amd64.box

- name: debian.7-lxc
  platform: debian # without this line it breaks with error [no implicit conversion of nil into Array], so comment this line to the the error for free! :)
  driver:
    box: debian7lxc
  driver_config:
    username: root
    password: root

suites:
  - name: twemproxy
    driver:
    provisioner:
      manifest: twemproxy.pp
    verifier:
      inspec_tests:
        - inspec/twemproxy

inspec file:

name: twemproxy
title: InSpec Profile
maintainer: Michał Sochoń
copyright: Michał Sochoń
copyright_email: [email protected]
license: Apache 2 license
summary: An InSpec Profile for twemproxy node
version: 0.2.0
supports:
  - os-family: debian
    release: 7
  - platform: all

More info

Test Kitchen version 1.11.1
inspec version 0.32.0
kitchen-inspec (0.15.0)

Fix

Generate more sane message if test suite list is empty.

Support --no-color option

When I run kitchen verify --no-color, kitchen-inspec adds colorization markers to the output. I would expect kitchen-inspec to respect the --no-color flag.

...
       Compiling Cookbooks...
       Converging 0 resources

       Running handlers:
       Running handlers complete
       Chef Client finished, 0/0 resources updated in 03 seconds
       Finished converging <default-centos-72> (0m33.53s).
-----> Setting up <default-centos-72>...
       Finished setting up <default-centos-72> (0m0.00s).
-----> Verifying <default-centos-72>...
       Use `/root/learn-chef/cookbooks/webserver_test/test/recipes/default` for testing

Target:  ssh://[email protected]:2222

�[31m  ✖  System Package httpd should be installedexpected that `System Package httpd` is installed�[0m

Summary: �[32m0 successful�[0m, �[31m1 failures�[0m, �[37m0 skipped�[0m
$ chef --version
Chef Development Kit Version: 0.16.28
chef-client version: 12.12.15
delivery version: master (921828facad8a8bbbd767368bfc72f19bd30e7bd)
berks version: 4.3.5
kitchen version: 1.10.2
$ chef gem list | grep kitchen-inspec
kitchen-inspec (0.15.0)

Error with apache_conf on CentOS

I have attempted this with both Cent 6 and 7, using Apache 2.2 and 2.4.

describe apache_conf do
  its('ServerTokens') { should eq 'Prod' }
end

The error I get when running this in test kitchen is

>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: Failed to complete #verify action: [undefined method `run_command' for #<#<Class:0x4489548>:0x3ff0c48>] >>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration

I'm not sure if this is an issue with inspec or kitchen-inspec...if this should be reported to inspec, please let me know.

junit format does not always use proper classname.

verifier:
  name: inspec
  format: junit
  output: ../../build/junit/%{platform}_%{suite}_inspec.xml
$ cat test/integration/memcached-single-instance/inspec/memcached.rb
describe package('memcached') do
  it { should be_installed }
end

# workaround service() auto-detect
# https://github.com/chef/inspec/issues/1171
describe sysv_service('memcached') do
  it { should be_running }
  it { should be_enabled }
  before do
    skip unless os[:release] == '14.04' or os[:release] == '16.04'
  end
end

describe service('memcached') do
  it "is listening on port 11211" do
    expect(port(11211)).to be_listening
  end
end

describe file('/etc/memcached.conf') do
       its('content') { should match '-m 1G' }
end
$ cat ../../build//junit/ubuntu-16.04_memcached-single-instance_inspec.xml
<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="rspec" tests="5" failures="0" errors="0" time="0.197933" timestamp="2017-01-23T10:17:42+01:00">
  <!-- Randomized with seed 24234 -->
  <properties/>
  <testcase classname="usr.local.lib.ruby.gems.2.4.0.gems.inspec-1.9.0.lib.inspec.rule" name="System Package memcached should be installed" file="/usr/local/lib/ruby/gems/2.4.0/gems/inspec-1.9.0/lib/inspec/rule.rb" time="0.061945"/>
  <testcase classname="usr.local.lib.ruby.gems.2.4.0.gems.inspec-1.9.0.lib.inspec.rule" name="Service memcached should be running" file="/usr/local/lib/ruby/gems/2.4.0/gems/inspec-1.9.0/lib/inspec/rule.rb" time="0.077482"/>
  <testcase classname="usr.local.lib.ruby.gems.2.4.0.gems.inspec-1.9.0.lib.inspec.rule" name="Service memcached should be enabled" file="/usr/local/lib/ruby/gems/2.4.0/gems/inspec-1.9.0/lib/inspec/rule.rb" time="0.000192"/>
  <testcase classname="usr.local.lib.ruby.gems.2.4.0.gems.inspec-1.9.0.lib.inspec.rule" name="Service memcached is listening on port 11211" file="/usr/local/lib/ruby/gems/2.4.0/gems/inspec-1.9.0/lib/inspec/rule.rb" time="0.032800"/>
  <testcase classname="test.integration.memcached-single-instance.inspec.memcached" name="File /etc/memcached.conf content should match &quot;-m 1G&quot;" file="./test/integration/memcached-single-instance/inspec/memcached.rb" time="0.023982"/>
</testsuite>

Why does describe file come from memcached.rb, but describe package from /usr/local/lib/ruby/gems/2.4.0/gems/inspec-1.9.0/lib/inspec/rule.rb?

Can't verify when used with kitchen-digitalocean provider

Create and converge work without any problems, but verify fails with following error message:

>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: Failed to complete #verify action: [You must configure at least one authentication method for SSH: Password or key.]
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration

.kitchen.yml:

---
driver:
  name: digitalocean
  size: 512mb

platforms:
- name: centos-7-0-x64
- name: ubuntu-14-04-x64

provisioner:
  name: chef_zero

verifier:
  name: inspec

suites:
  - name: default
    run_list:
      - recipe[bla::default]

kitchen log:

I, [2016-04-14T22:51:36.877664 #19221]  INFO -- Kitchen: -----> Starting Kitchen (v1.6.0)
I, [2016-04-14T22:51:37.772431 #19221]  INFO -- Kitchen: -----> Converging <default-centos-7-0-x64>...
I, [2016-04-14T22:51:37.772636 #19221]  INFO -- Kitchen: -----> Converging <default-ubuntu-14-04-x64>...
I, [2016-04-14T22:52:13.444049 #19221]  INFO -- Kitchen: -----> Setting up <default-ubuntu-14-04-x64>...
I, [2016-04-14T22:52:13.447798 #19221]  INFO -- Kitchen: -----> Verifying <default-ubuntu-14-04-x64>...
I, [2016-04-14T22:52:33.439790 #19221]  INFO -- Kitchen: -----> Setting up <default-centos-7-0-x64>...
I, [2016-04-14T22:52:33.442952 #19221]  INFO -- Kitchen: -----> Verifying <default-centos-7-0-x64>...
E, [2016-04-14T22:52:33.449197 #19221] ERROR -- Kitchen: ------Exception-------
E, [2016-04-14T22:52:33.449383 #19221] ERROR -- Kitchen: Class: Kitchen::ActionFailed
E, [2016-04-14T22:52:33.449470 #19221] ERROR -- Kitchen: Message: Failed to complete #verify action: [You must configure at least one authentication method for SSH: Password or key.]
E, [2016-04-14T22:52:33.449545 #19221] ERROR -- Kitchen: ---Nested Exception---
E, [2016-04-14T22:52:33.449616 #19221] ERROR -- Kitchen: Class: Train::ClientError
E, [2016-04-14T22:52:33.449685 #19221] ERROR -- Kitchen: Message: You must configure at least one authentication method for SSH: Password or key.
E, [2016-04-14T22:52:33.449756 #19221] ERROR -- Kitchen: ------Backtrace-------
E, [2016-04-14T22:52:33.449824 #19221] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/r-train-0.10.3/lib/train/transports/ssh.rb:84:in `validate_options'
E, [2016-04-14T22:52:33.449897 #19221] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/r-train-0.10.3/lib/train/transports/ssh.rb:67:in `connection'
E, [2016-04-14T22:52:33.449968 #19221] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/inspec-0.14.8/lib/inspec/backend.rb:23:in `create'
E, [2016-04-14T22:52:33.450038 #19221] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/inspec-0.14.8/lib/inspec/runner.rb:46:in `configure_transport'
E, [2016-04-14T22:52:33.450109 #19221] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/inspec-0.14.8/lib/inspec/runner.rb:30:in `initialize'
E, [2016-04-14T22:52:33.450180 #19221] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/kitchen-inspec-0.12.3/lib/kitchen/verifier/inspec.rb:69:in `new'
E, [2016-04-14T22:52:33.450281 #19221] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/kitchen-inspec-0.12.3/lib/kitchen/verifier/inspec.rb:69:in `call'
E, [2016-04-14T22:52:33.450401 #19221] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.6.0/lib/kitchen/instance.rb:421:in `block in verify_action'
E, [2016-04-14T22:52:33.450474 #19221] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.6.0/lib/kitchen/instance.rb:513:in `call'
E, [2016-04-14T22:52:33.450544 #19221] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.6.0/lib/kitchen/instance.rb:513:in `synchronize_or_call'
E, [2016-04-14T22:52:33.450621 #19221] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.6.0/lib/kitchen/instance.rb:478:in `block in action'
E, [2016-04-14T22:52:33.450697 #19221] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/2.1.0/benchmark.rb:279:in `measure'
E, [2016-04-14T22:52:33.450773 #19221] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.6.0/lib/kitchen/instance.rb:477:in `action'
E, [2016-04-14T22:52:33.450863 #19221] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.6.0/lib/kitchen/instance.rb:415:in `verify_action'
E, [2016-04-14T22:52:33.450935 #19221] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.6.0/lib/kitchen/instance.rb:348:in `block in transition_to'
E, [2016-04-14T22:52:33.451008 #19221] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.6.0/lib/kitchen/instance.rb:347:in `each'
E, [2016-04-14T22:52:33.451080 #19221] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.6.0/lib/kitchen/instance.rb:347:in `transition_to'
E, [2016-04-14T22:52:33.451182 #19221] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.6.0/lib/kitchen/instance.rb:160:in `verify'
E, [2016-04-14T22:52:33.451281 #19221] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.6.0/lib/kitchen/command.rb:176:in `public_send'
E, [2016-04-14T22:52:33.451394 #19221] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.6.0/lib/kitchen/command.rb:176:in `block (2 levels) in run_action'
E, [2016-04-14T22:52:33.451521 #19221] ERROR -- Kitchen: ----------------------

Message: Could not load the 'inspec' verifier from the load path. Please ensure that your transport is installed as a gem or included in your Gemfile if using Bundler.

I'm having trouble running the tests for a chef cookbook. I've resolved many errors, but this one finally stumped me.

$ kitchen test
-----> Starting Kitchen (v1.13.2)
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ClientError
>>>>>> Message: Could not load the 'inspec' verifier from the load path. Please ensure that your transport is installed as a gem or included in your Gemfile if using Bundler.
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration

(.kitchen.yml)


---
driver:
  name: vagrant

provisioner:
  name: chef_solo

verifier:
  name: inspec

platforms:
  - name: ubuntu-15.04
    driver_config:
      box: opscode-ubuntu-15.04
      box_url: http://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_ubuntu-15.04_chef-provisionerless.box

suites:
  - name: default
    run_list:
      - recipe[my_hostname::default]
    verifier:
      inspec_tests:
        - test/recipes
    attributes:
$ chef -v && kitchen -v && chef gem list inspec
Chef Development Kit Version: 0.18.30
chef-client version: 12.14.89
delivery version: master (d86679335580be3de22996ef294b20d525889d8d)
berks version: 5.1.0
kitchen version: ERROR

Test Kitchen version 1.13.2

*** LOCAL GEMS ***
debug_inspector (0.0.2)
inspec (1.1.0, 1.0.0)
kitchen-inspec (0.15.2)

Any ideas as to what to try next?

Unable to test installed Gems

Hi,

I need to test that some Gems are installed(they have been installed via the chef_gem. However it looks like kitchen-inspec isn't able to pick up on any gem installation. In order to try and track down this issue I ended up installing rvm and some test gems to see if inspec would return those, it didn't seem to help. However I noticed when I ran inspec exec test/integration/server/default.rb -t ssh://[email protected] --password 'vagrant' it would seem to find my rvm gempath correctly. But this doesn't help my original issue of inspec not picking up on chef gems not being seen.

I'm not sure if this is an issue with inspec, kitchen-inspec or some misconfiguration on my end. Any advice to point me in the right direction would be great!

Thanks!

Kitchen verify/test fails when using the command resource with curl

If you write a test that looks similar to below and you use kitchen verify/test this will fail with the curl error curl: can't find host localhost

describe bash('curl -sS localhost:80/index.html') do
  its('stdout') { should_not match /HTTP Status 404/i }
  its('stderr') { should eq ''}
end

This does however work if you use the inspec cli and run inspec exec test.rb -t ssh://[email protected]:2222/

I don't think I am missing anything and I can't get any more detailed pulled from the kitchen logs. Is this expected behavior or did I miss up a configuration

Harmonize profile location targets

Right now we use

kitchen.yml (yml format)

suites:
  - name: contains_inspec
    run_list:
    ...
    verifier:
      inspec_tests:
        - https://github.com/dev-sec/tests-ssh-hardening
        - https://github.com/dev-sec/tests-os-hardening

This format is different from inspec.yml

name: meta-profile
title: Meta Compliance Profile
maintainer: InSpec Authors
copyright: InSpec Authors
copyright_email: [email protected]
license: Apache 2
summary: InSpec Profile that is only consuming dependencies
version: 0.2.0
depends:
  - name: hardening/ssh-hardening  # defaults to supermarket
  - name: os-hardening
    url: https://github.com/dev-sec/tests-os-hardening/archive/master.zip
  - git: https://github.com/dev-sec/ssl-benchmark.git
  - name: windows-patch-benchmark
    git: https://github.com/chris-rock/windows-patch-benchmark.git
  - name: linux
    compliance: base/linux

Therefore we should align those formats to easily allow copy&paste between tools:

suites:
  - name: contains_inspec
    run_list:
    ...
    verifier:
      inspec_tests:
        - name: hardening/ssh-hardening  # defaults to supermarket
        - name: os-hardening
          url: https://github.com/dev-sec/tests-os-hardening/archive/master.zip
        - git: https://github.com/dev-sec/tests-ssh-hardening.git
        - git: https://github.com/dev-sec/tests-os-hardening.git
        - name: hardening/ssh-hardening 
          supermarket_url: https://supermarket.my.com

See inspec/inspec#1227

Colo[u]r those dots and Fs!

It's pretty common now to see colo[u]rs for "." and "F"s in tests.

I'd like to ask that this is added to the runs, so something like the following would be colo[u]red.

       Search `/Users/jasghar/repo/cookbooks/hipchat-chef-plugin-cookbook/test/integration/default` for tests
......FF........

Failures:

This would also put the Failures in RED so you can could identify issues quickly.

Reuse the same test for multiple cookbooks

@coderanger and @cheeseplus pointed out, that it is not possible to reuse test across different cookbooks easily. He used shared tests like: https://github.com/poise/poise-service-spechelper/blob/master/lib/poise_service/spec_helper/helper.rb
in
https://github.com/poise/poise-service/blob/master/test/integration/default/serverspec/default_spec.rb#L17

We need to find a way to solve this problem with kitchen-inspec. Possible ways:

  • write busser implementation
  • allow test-kitchen to load a remote test suite

Any other ideas?

tests fail with inspec 0.9.5

$ .bin/rake test
/Users/stephan/.rbenv/versions/2.2.3/bin/ruby -I/Users/stephan/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/rspec-core-3.4.1/lib:/Users/stephan/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/rspec-support-3.4.1/lib /Users/stephan/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/rspec-core-3.4.1/exe/rspec --pattern spec/\*\*\{,/\*/\*\*\}/\*_spec.rb

Kitchen::Verifier::Inspec
  verifier api_version is 1
  plugin_version is set to Kitchen::Verifier::INSPEC_VERSION
  with an ssh transport
    constructs a Inspec::Runner using transport config data and state (FAILED - 1)
    adds *spec.rb test files to runner (FAILED - 2)
    calls #run on the runner (FAILED - 3)
  with an winrm transport
    constructs a Inspec::Runner using transport config data and state (FAILED - 4)
  with an unsupported transport
    #call raises a UserError

Failures:

  1) Kitchen::Verifier::Inspec with an ssh transport constructs a Inspec::Runner using transport config data and state
     Failure/Error: raise ActionFailed, "Inspec Runner returns #{exit_code}" unless exit_code == 0

     Kitchen::ActionFailed:
       Inspec Runner returns 
     # ./lib/kitchen/verifier/inspec.rb:57:in `call'
     # ./spec/kitchen/verifier/inspec_spec.rb:163:in `block (3 levels) in <top (required)>'

  2) Kitchen::Verifier::Inspec with an ssh transport adds *spec.rb test files to runner
     Failure/Error: raise ActionFailed, "Inspec Runner returns #{exit_code}" unless exit_code == 0

     Kitchen::ActionFailed:
       Inspec Runner returns 
     # ./lib/kitchen/verifier/inspec.rb:57:in `call'
     # ./spec/kitchen/verifier/inspec_spec.rb:177:in `block (3 levels) in <top (required)>'

  3) Kitchen::Verifier::Inspec with an ssh transport calls #run on the runner
     Failure/Error: raise ActionFailed, "Inspec Runner returns #{exit_code}" unless exit_code == 0

     Kitchen::ActionFailed:
       Inspec Runner returns 
     # ./lib/kitchen/verifier/inspec.rb:57:in `call'
     # ./spec/kitchen/verifier/inspec_spec.rb:184:in `block (3 levels) in <top (required)>'

  4) Kitchen::Verifier::Inspec with an winrm transport constructs a Inspec::Runner using transport config data and state
     Failure/Error: raise ActionFailed, "Inspec Runner returns #{exit_code}" unless exit_code == 0

     Kitchen::ActionFailed:
       Inspec Runner returns 
     # ./lib/kitchen/verifier/inspec.rb:57:in `call'
     # ./spec/kitchen/verifier/inspec_spec.rb:228:in `block (3 levels) in <top (required)>'

Finished in 0.08909 seconds (files took 0.16373 seconds to load)
7 examples, 4 failures

Failed examples:

rspec ./spec/kitchen/verifier/inspec_spec.rb:140 # Kitchen::Verifier::Inspec with an ssh transport constructs a Inspec::Runner using transport config data and state
rspec ./spec/kitchen/verifier/inspec_spec.rb:166 # Kitchen::Verifier::Inspec with an ssh transport adds *spec.rb test files to runner
rspec ./spec/kitchen/verifier/inspec_spec.rb:180 # Kitchen::Verifier::Inspec with an ssh transport calls #run on the runner
rspec ./spec/kitchen/verifier/inspec_spec.rb:213 # Kitchen::Verifier::Inspec with an winrm transport constructs a Inspec::Runner using transport config data and state

/Users/stephan/.rbenv/versions/2.2.3/bin/ruby -I/Users/stephan/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/rspec-core-3.4.1/lib:/Users/stephan/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/rspec-support-3.4.1/lib /Users/stephan/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/rspec-core-3.4.1/exe/rspec --pattern spec/\*\*\{,/\*/\*\*\}/\*_spec.rb failed

show test and result in the log

Hey, would it be possible to show the test and the result as it is with serverspec:

    File "/etc/apache2/httpd.conf"
       should exist
     Port "80"
       should be listening
     Command "curl http://localhost"
       stdout
         should match /Hello/

   Finished in 0.13462 seconds (files took 0.33277 seconds to load)
   3 examples, 0 failures

TTY issue with base AWS CentOS AMI

Is anyone else running into this on a base CentOS AMI?

>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: Failed to complete #verify action: [Sudo failed: Sudo requires a TTY. Please see the README on how to configure sudo to allow for non-interactive usage.]
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration

Kitchen verify fails with the kitchen-docker plugin

Not sure if this is an issue with kitchen-docker or kitchen-inspec. The issue returned comes with the following:

       Search `/Users/user/code/cookbook/test/integration/default` for tests
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: Failed to complete #verify action: [Sudo failed: Sudo requires a TTY. Please see the README on how to configure sudo to allow for non-interactive usage.]
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration

The kitchen file looks a such and the converge works easily. I am also using docker-beta which I am not sure if that plays into this at all.


---
driver:
  name: docker
  use_sudo: false

provisioner:
  name: chef_zero

verifier:
  name: inspec

platforms:
  - name: centos-6.7
  - name: centos-7.1

suites:
  - name: default
    run_list:
      - recipe[cookbook]

Pending messages are "inherited"/"propagated" down with several suites

In case you have several instances and one of them gives some pending messages like in my case, whey get propagated to the instance below:

$ kitchen list
Instance                   Driver   Provisioner  Verifier  Transport  Last Action
desktop-mint-172-cinnamon  Vagrant  ChefZero     Inspec    Ssh        Verified
desktop-ubuntu-1404        Vagrant  ChefZero     Inspec    Ssh        Verified

# single verify for mint
############################################################################
$ kitchen verify desktop-mint-172-cinnamon
-----> Starting Kitchen (v1.4.2)
-----> Verifying <desktop-mint-172-cinnamon>...
*............*.........*..*

Pending: (Failures listed here are expected and do not affect your suite's status)

  1) System Package crossover The `package` resource is not supported on your OS yet.
     # Not yet implemented
     # /home/artem/.chefdk/gem/ruby/2.1.0/gems/inspec-0.9.5/lib/inspec/runner.rb:114

  2) System Package x11vnc The `package` resource is not supported on your OS yet.
     # Not yet implemented
     # /home/artem/.chefdk/gem/ruby/2.1.0/gems/inspec-0.9.5/lib/inspec/runner.rb:114

  3) Service x11vnc The `service` resource is not supported on your OS yet.
     # Not yet implemented
     # /home/artem/.chefdk/gem/ruby/2.1.0/gems/inspec-0.9.5/lib/inspec/runner.rb:114

  4) System Package owncloud-client The `package` resource is not supported on your OS yet.
     # Not yet implemented
     # /home/artem/.chefdk/gem/ruby/2.1.0/gems/inspec-0.9.5/lib/inspec/runner.rb:114

Finished in 0.48911 seconds (files took 1.33 seconds to load)
27 examples, 0 failures, 4 pending

       Finished verifying <desktop-mint-172-cinnamon> (0m1.61s).
-----> Kitchen is finished. (0m4.43s)

# single verify for ubuntu
############################################################################
$ kitchen verify desktop-ubuntu-1404
-----> Starting Kitchen (v1.4.2)
-----> Setting up <desktop-ubuntu-1404>...
       Finished setting up <desktop-ubuntu-1404> (0m0.00s).
-----> Verifying <desktop-ubuntu-1404>...
............................

Finished in 0.94141 seconds (files took 1.36 seconds to load)
28 examples, 0 failures

       Finished verifying <desktop-ubuntu-1404> (0m2.09s).
-----> Kitchen is finished. (0m4.83s)

# verify both of them
############################################################################
$ kitchen verify
-----> Starting Kitchen (v1.4.2)
-----> Verifying <desktop-mint-172-cinnamon>...
*............*.........*..*

Pending: (Failures listed here are expected and do not affect your suite's status)

  1) System Package crossover The `package` resource is not supported on your OS yet.
     # Not yet implemented
     # /home/artem/.chefdk/gem/ruby/2.1.0/gems/inspec-0.9.5/lib/inspec/runner.rb:114

  2) System Package x11vnc The `package` resource is not supported on your OS yet.
     # Not yet implemented
     # /home/artem/.chefdk/gem/ruby/2.1.0/gems/inspec-0.9.5/lib/inspec/runner.rb:114

  3) Service x11vnc The `service` resource is not supported on your OS yet.
     # Not yet implemented
     # /home/artem/.chefdk/gem/ruby/2.1.0/gems/inspec-0.9.5/lib/inspec/runner.rb:114

  4) System Package owncloud-client The `package` resource is not supported on your OS yet.
     # Not yet implemented
     # /home/artem/.chefdk/gem/ruby/2.1.0/gems/inspec-0.9.5/lib/inspec/runner.rb:114

Finished in 0.49791 seconds (files took 1.35 seconds to load)
27 examples, 0 failures, 4 pending

       Finished verifying <desktop-mint-172-cinnamon> (0m1.64s).
-----> Verifying <desktop-ubuntu-1404>...
............................

Pending: (Failures listed here are expected and do not affect your suite's status)

  1) System Package crossover The `package` resource is not supported on your OS yet.
     # Not yet implemented
     # /home/artem/.chefdk/gem/ruby/2.1.0/gems/inspec-0.9.5/lib/inspec/runner.rb:114

  2) System Package x11vnc The `package` resource is not supported on your OS yet.
     # Not yet implemented
     # /home/artem/.chefdk/gem/ruby/2.1.0/gems/inspec-0.9.5/lib/inspec/runner.rb:114

  3) Service x11vnc The `service` resource is not supported on your OS yet.
     # Not yet implemented
     # /home/artem/.chefdk/gem/ruby/2.1.0/gems/inspec-0.9.5/lib/inspec/runner.rb:114

  4) System Package owncloud-client The `package` resource is not supported on your OS yet.
     # Not yet implemented
     # /home/artem/.chefdk/gem/ruby/2.1.0/gems/inspec-0.9.5/lib/inspec/runner.rb:114

Finished in 0.88895 seconds (files took 2.36 seconds to load)
55 examples, 0 failures, 4 pending

       Finished verifying <desktop-ubuntu-1404> (0m1.40s).
-----> Kitchen is finished. (0m5.93s)

Initially I thougth the second output is some kind of summary over failures, but the line Finished verifying <desktop-ubuntu-1404> is displayed after it

Software versions:

$ chef --version
Chef Development Kit Version: 0.10.0
chef-client version: 12.5.1
berks version: 4.0.1
kitchen version: 1.4.2
$ chef exec gem list inspec

*** LOCAL GEMS ***

inspec (0.9.5, 0.9.2)
kitchen-inspec (0.9.0)
$ inspec version
0.9.5

Specify inspec output formats in .kitchen.yml

If one were to run inspec exec, you can pass in something like inspec exec --format=documentation so that the output is more descriptive and meaningful. Presently, there appears to be no way to do that from within test-kitchen.

By way of example. The default output leaves something to be desired:

$ kitchen verify linux-rhel-71
-----> Starting Kitchen (v1.4.2)
-----> Verifying <linux-rhel-71>...
.......

Finished in 0.93712 seconds (files took 0.67007 seconds to load)
7 examples, 0 failures

       Finished verifying <linux-rhel-71> (0m1.30s).
-----> Kitchen is finished. (0m23.15s)

It would be nice if one could obtain output like this:

System Package ntp
  should be installed

System Package ntpdate
  should be installed

After some discussion on Slack, it appears that the right place to do that might be in https://github.com/chef/kitchen-inspec/blob/master/lib/kitchen/verifier/inspec.rb#L108, which I'm including for posterity. I don't mind trying to implement this myself and sending a PR, but I'm including the above link in the event I'm unable to do so.

kitchen-docker support?

I tried using this doing kitchen testing with docker utilizing the kitchen-docker gem and it doesn't seem to run the verify tests. I thought it'd work since it looks like kitchen-docker uses the SSH transport but it seems something more may be needed?

docker output:

$ kitchen list
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/httpclient-2.6.0.1/lib/httpclient/webagent-cookie.rb:458: warning: already initialized constant HTTPClient::CookieManager
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/httpclient-2.6.0.1/lib/httpclient/cookie.rb:8: warning: previous definition of CookieManager was here
Instance           Driver  Provisioner  Verifier  Transport  Last Action
default-centos-66  Docker  ChefZero     Inspec    Ssh        Verified
$ kitchen verify
-----> Starting Kitchen (v1.4.2)
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/httpclient-2.6.0.1/lib/httpclient/webagent-cookie.rb:458: warning: already initialized constant HTTPClient::CookieManager
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/httpclient-2.6.0.1/lib/httpclient/cookie.rb:8: warning: previous definition of CookieManager was here
-----> Verifying <default-centos-66>...
$$$$$$ Running legacy verify for 'Docker' Driver
       Preparing files for transfer
       Transferring files to <default-centos-66>
       Finished verifying <default-centos-66> (0m0.00s).
-----> Kitchen is finished. (0m0.95s)
$ ls test/integration/default/
node_prepare_spec.rb

vagrant/vbox output:

$ kitchen list
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/httpclient-2.6.0.1/lib/httpclient/webagent-cookie.rb:458: warning: already initialized constant HTTPClient::CookieManager
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/httpclient-2.6.0.1/lib/httpclient/cookie.rb:8: warning: previous definition of CookieManager was here
Instance       Driver   Provisioner  Verifier  Transport  Last Action
default-rhel6  Vagrant  ChefZero     Inspec    Ssh        Verified
$ kitchen verify
-----> Starting Kitchen (v1.4.2)
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/httpclient-2.6.0.1/lib/httpclient/webagent-cookie.rb:458: warning: already initialized constant HTTPClient::CookieManager
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/httpclient-2.6.0.1/lib/httpclient/cookie.rb:8: warning: previous definition of CookieManager was here
-----> Verifying <default-rhel6>...
FFFFF

Failures:

  1) System Package pcs should be installed
     Failure/Error: it { should be_installed }
       expected that `System Package pcs` is installed
     # ./test/integration/default/node_prepare_spec.rb:2:in `block (2 levels) in load'

  2) Service pcsd should be enabled
     Failure/Error: it { should be_enabled }
       expected that `Service pcsd` is enabled
     # ./test/integration/default/node_prepare_spec.rb:6:in `block (2 levels) in load'

  3) Service pcsd should be running
     Failure/Error: it { should be_running }
       expected that `Service pcsd` is running
     # ./test/integration/default/node_prepare_spec.rb:7:in `block (2 levels) in load'

  4) Service pacemaker should be running
     Failure/Error: it { should be_running }
       expected that `Service pacemaker` is running
     # ./test/integration/default/node_prepare_spec.rb:11:in `block (2 levels) in load'

  5) Service corosync should be running
     Failure/Error: it { should be_running }
       expected that `Service corosync` is running
     # ./test/integration/default/node_prepare_spec.rb:17:in `block (2 levels) in load'

Finished in 0.75939 seconds (files took 0.37673 seconds to load)
5 examples, 5 failures

Failed examples:

rspec  # System Package pcs should be installed
rspec  # Service pcsd should be enabled
rspec  # Service pcsd should be running
rspec  # Service pacemaker should be running
rspec  # Service corosync should be running

       Finished verifying <default-rhel6> (0m1.01s).
-----> Kitchen is finished. (0m2.10s)
zlib(finalizer): the stream was freed prematurely.
$ ls test/integration/default/
node_prepare_spec.rb

Thanks!

kitchen test destroys the instance with failing tests

$ kitchen test generic-centos-71
-----> Starting Kitchen (v1.4.2)
-----> Cleaning up any prior instances of <generic-centos-71>
-----> Destroying <generic-centos-71>...
       Finished destroying <generic-centos-71> (0m0.00s).
-----> Testing <generic-centos-71>
-----> Creating <generic-centos-71>...
       Bringing machine 'default' up with 'virtualbox' provider...
       ==> default: Importing base box 'opscode-centos-7.1'...
....
       Chef Client finished, 0/0 resources updated in 01 seconds
       Finished converging <generic-centos-71> (0m55.43s).
-----> Setting up <generic-centos-71>...
       Finished setting up <generic-centos-71> (0m0.00s).
-----> Verifying <generic-centos-71>...
F

Failures:

  1) Command /bin/false exit_status should eq 0
     Failure/Error: its('exit_status') { should eq 0 }

       expected: 0
            got: 1

       (compared using ==)
.....

Finished in 0.07612 seconds (files took 1 minute 32.92 seconds to load)
1 example, 1 failure

Failed examples:

rspec  # Command /bin/false exit_status should eq 0

       Finished verifying <generic-centos-71> (0m1.37s).
-----> Destroying <generic-centos-71>...
       ==> default: Forcing shutdown of VM...
       ==> default: Destroying VM and associated drives...
       Vagrant instance <generic-centos-71> destroyed.
       Finished destroying <generic-centos-71> (0m7.11s).
       Finished testing <generic-centos-71> (1m39.90s).
-----> Kitchen is finished. (1m42.84s)
$ echo $?
0

with default behaviour of --destroy=passing of kitchen test the instance should be not destroyed.
Probably some return code to the kitchen about failures is missing somewhere?

Software versions:

$ chef --version
Chef Development Kit Version: 0.10.0
chef-client version: 12.5.1
berks version: 4.0.1
kitchen version: 1.4.2
$ chef exec gem list inspec

*** LOCAL GEMS ***

inspec (0.9.5, 0.9.2)
kitchen-inspec (0.9.0)
$ inspec version
0.9.5

Probably this is related to #9

Duplicate testing when verifier specified in suite definition

Description

Using inspec - TestKitchen, I’m seeing duplicate inspec executions for the included tests. Single suite with path defined to test/recipes/ containing two files with one control in each but see four controls executed in the kitchen test/verify output.

  - name: master
    run_list:
      - recipe[example::install_master]
      - recipe[example::configure_master]
    verifier:
      inspec_tests:
        - test/recipes/master

One observation I've noted is the duplicated paths

-----> Verifying <master-ubuntu-1604>...
D      Initialize InSpec
       Use `/Users/droche/work/infra-auto/cookbooks/example/test/recipes/master` for testing
D      Running tests from: ["/Users/droche/work/infra-auto/cookbooks/example/test/recipes/master", "test/recipes/master"]

Recreate

Create a stock cookbook using chef generate cookbook duptest
Create a suite directory under test/recipes/<suite> and move the default_test.rb into this directory

Output

kitchen verify def --log-level=debug
-----> Starting Kitchen (v1.11.1)
D      [Vagrant command] BEGIN (vagrant --version)
D      [Vagrant command] END (0m0.47s)
D      Berksfile found at /Users/droche/work/infra-auto/cookbooks/duptest/Berksfile, loading Berkshelf
D      Berkshelf 4.3.5 library loaded
-----> Verifying <default-ubuntu-1604>...
D      Initialize InSpec
       Use `/Users/droche/work/infra-auto/cookbooks/duptest/test/recipes/default` for testing
D      Running tests from: ["/Users/droche/work/infra-auto/cookbooks/duptest/test/recipes/default", "test/recipes"]

Target:  ssh://[email protected]:2202

  ✔  User root should exist
  ✔  Port 80 should not be listening
  ○  User root should exist; User root This is an example test, r... (1 skipped)
     This is an example test, replace with your own test.
  ○  Port 80 should not be listening; Port 80 This is an example ... (1 skipped)
     This is an example test, replace with your own test.

Summary: 4 successful, 0 failures, 4 skipped
       Finished verifying <default-ubuntu-1604> (0m0.42s).
-----> Kitchen is finished. (0m1.78s)

Workaround

Removning the definition of the verifier removes the duplicate executions when the test/recipes/<suite> directory is used.

gem resource breaks if not using a different installation path

  gem package
     ✖  kitchen-ec2 should be installed
     expected that `gem package kitchen-ec2` is installed
vagrant@default-ubuntu-1604:/etc/systemd/system$ gem list -i kitchen-ec2
true
vagrant@default-ubuntu-1604:/etc/systemd/system$ which gem
/opt/chefdk/embedded/bin/gem

kitchen converge fails on ubuntu

Local gem list goes this way:-
kitchen-ec2 (1.0.0)
kitchen-inspec (0.12.5)
kitchen-vagrant (0.20.0)
test-kitchen (1.6.0)

Still facing the following issue

pavan@ubuntu:~/sample/git-cookbook$ kitchen converge
-----> Starting Kitchen (v1.6.0)
-----> Converging <default-windows-2012r2>...
       Preparing files for transfer
       Preparing dna.json
       Preparing current project directory as a cookbook
       Removing non-cookbook files before transfer
       Preparing validation.pem
       Preparing client.rb
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: Failed to complete #converge action: [undefined method `encoding' for nil:NilClass]
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration

Inspec Test Location

Is the default search location of kitchen-inspec going to be updated to match the standards direction that Chef is going in? Specifically discussed in chef-boneyard/chef-dk#1096.

Additionally, gathered from reading discussions on issues, and reading the source https://github.com/chef/kitchen-inspec/blob/master/lib/kitchen/verifier/inspec.rb#L121, it seems as if test/integration/suite_name is the only intended supported location.

The cookbook generator in the chef-dk creates /test/recipes so we have been storing our tests in /test/recipes/suite_name and kitchen-inspec finds them without having to explicitly point to them in kitchen.yml. https://github.com/chef/chef-dk/blob/627ff580d26c7452285de3d6c4f0692a762953f0/lib/chef-dk/skeletons/code_generator/recipes/cookbook.rb#L58

What's the intended location, which one of these things is the "future", and how can I help align these things so they make sense for the next person learning Chef?

Just to keep these all linked: chef/chef-vault-testfixtures#24

OS detection 2nd time with unknown value?

Description

After the upgrade of chefdk from 0.12 to chefdk 0.15 os['family'] returns unknown at some point. I see this only within test kitchen and kitchen-inspec. Inspec itself doesn't show this behavior.

InSpec and Platform Version

Inspec version:

$ chef gem list | grep inspec
debug_inspector (0.0.2)
inspec (0.26.0)
kitchen-inspec (0.14.0)
$ chef --version
Chef Development Kit Version: 0.15.15
chef-client version: 12.11.18
delivery version: 0.0.23 (bf89a6b776b55b89a46bbd57fcaa615c143a09a0)
berks version: 4.3.5
kitchen version: 1.10.0

Replication Case

Create some cookbook and simple kitchen configuration, configure kitchen to use inspec as verifier. Create following test file and invoke kitchen verify:

puts os['family']

Invoke kitchen verify:

$ kitchen verify packages-bento-debian-84
-----> Starting Kitchen (v1.10.0)
-----> Verifying <packages-bento-debian-84>...
       Detected alternative framework tests for `inspec`
       Use `chef-rkt/test/integration/packages/inspec` for testing
debian
unknown
...

Expected output:

$ kitchen verify packages-bento-debian-84
-----> Starting Kitchen (v1.10.0)
-----> Verifying <packages-bento-debian-84>...
       Detected alternative framework tests for `inspec`
       Use `chef-rkt/test/integration/packages/inspec` for testing
debian
...

Default to ( progress | documentation ) format for test-kitchen inspec verifier

It looks like the default inspec format for output has changed to not be as helpful when issues show up. For example lets say you have something like this:

describe file('/home/me/.ssh') do
  it { should exist }
  it { should be_file }
end

That would output a single line

✔  File /home/me/.ssh should exist

# And then a bunch of error with no context
... Blah

Not sure if this is something I am doing wrong on my system. Using inspec as a verifier is something we are trying to switch to but I would rather not add the following below:

verifier:
  name: inspec
  format: progress # <- This

Just trying to start a discussion on this, I can do the code. I believe it will be one line but not going to do that if its a waste of time and no one agrees.

Support InSpec attributes

InSpec introduced a new feature called attributes with InSpec 1.0 inspec/inspec#719. At this point, kitchen-inspec does not support this feature. My expectation is, that attributes work across the Chef tool chain.

a simple attribute https://github.com/chef/inspec/blob/master/examples/profile-attribute.yml file looks like

user: bob
password: secret

To support attributes in kitchen.yml, we could do the following:

suites:
  - name: default
    verifier:
      inspec_tests:
        - https://github.com/dev-sec/tests-ssh-hardening
      attributes:
        user: bob
        password: secret

Not sure if we want to support external yml files?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.