inspec / kitchen-inspec Goto Github PK
View Code? Open in Web Editor NEWTest-Kitchen Plugin for InSpec
License: Other
Test-Kitchen Plugin for InSpec
License: Other
Consider the scenario where you have a large number of compliance failures (for example, from the CIS benchmarks) and you want to tackle a few at a time.
With local remediation through Test Kitchen, you can scan temporary instances directly against a full compliance profile. kitchen verify
potentially produces a huge volume of output, which can be difficult to wade through.
In the Learn Chef tutorials (CentOS) (Ubuntu), we have the user grep the output of kitchen verify
to find the errors they care about.
What makes this difficult is that you need to grep for the matcher code itself. For example:
$ kitchen verify > verify.txt 2>&1
$ cat verify.txt | grep -A 10 '\d*).*should match /^Status: active$/'
It can be painful running grep
because you need to escape the matcher code itself (because you're grepping for a regex from a regex tool.)
A few ways that might make this easier are to:
Add the name of the rule to the output (for example, Set SSH Protocol to 2). This would make gripping easier. If the user is scaring against multiple profiles, perhaps also include the profile name as a way to namespace things (perhaps overkill.)
Add a way for the user to specify in their .kitchen.yml
file which rules they care about. Then scan against only those rules. For example:
verifier:
name: inspec
inspec_tests:
- compliance://cis/cis-centos7-level1
filter_rules:
- 'Set SSH Protocol to 2'
- 'Ensure Firewall is active'
Enable the user to create a .inspec.yml
file (or similar) that defines specific rules to include or exclude, similar to how .rubocop.yml
works.
These options would produce much leaner output when you run kitchen verify
, and would eliminate the need to redirect the output to a file and then process that file.
Those are just a few ideas that came to mind. Perhaps there are better alternatives.
I have the latest ChefDK installed and installed kitchen-inspec via "chef gem install kitchen-inspec".
When I run kitchen verify, I get the following error:
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: Failed to complete #verify action: [undefined method `endpoint' for nil:NilClass]
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration
I'm running Test Kitchen on Windows 10 and the machine I am trying to test is Windows 2012 R2.
Generated from inspec/inspec#338
Depends on inspec/inspec#719
ServerSpec is supporting test reusability and customisability via shared_examples. An example from elasticsearch cookbook:
After the upgrade of chefdk from 0.12 to chefdk 0.15 os['family']
returns unknown at some point. I see this only within test kitchen and kitchen-inspec. Inspec itself doesn't show this behavior.
Inspec version:
$ chef gem list | grep inspec
debug_inspector (0.0.2)
inspec (0.26.0)
kitchen-inspec (0.14.0)
$ chef --version
Chef Development Kit Version: 0.15.15
chef-client version: 12.11.18
delivery version: 0.0.23 (bf89a6b776b55b89a46bbd57fcaa615c143a09a0)
berks version: 4.3.5
kitchen version: 1.10.0
Create some cookbook and simple kitchen configuration, configure kitchen to use inspec as verifier. Create following test file and invoke kitchen verify:
puts os['family']
Invoke kitchen verify:
$ kitchen verify packages-bento-debian-84
-----> Starting Kitchen (v1.10.0)
-----> Verifying <packages-bento-debian-84>...
Detected alternative framework tests for `inspec`
Use `chef-rkt/test/integration/packages/inspec` for testing
debian
unknown
...
Expected output:
$ kitchen verify packages-bento-debian-84
-----> Starting Kitchen (v1.10.0)
-----> Verifying <packages-bento-debian-84>...
Detected alternative framework tests for `inspec`
Use `chef-rkt/test/integration/packages/inspec` for testing
debian
...
If one were to run inspec exec
, you can pass in something like inspec exec --format=documentation
so that the output is more descriptive and meaningful. Presently, there appears to be no way to do that from within test-kitchen.
By way of example. The default output leaves something to be desired:
$ kitchen verify linux-rhel-71
-----> Starting Kitchen (v1.4.2)
-----> Verifying <linux-rhel-71>...
.......
Finished in 0.93712 seconds (files took 0.67007 seconds to load)
7 examples, 0 failures
Finished verifying <linux-rhel-71> (0m1.30s).
-----> Kitchen is finished. (0m23.15s)
It would be nice if one could obtain output like this:
System Package ntp
should be installed
System Package ntpdate
should be installed
After some discussion on Slack, it appears that the right place to do that might be in https://github.com/chef/kitchen-inspec/blob/master/lib/kitchen/verifier/inspec.rb#L108, which I'm including for posterity. I don't mind trying to implement this myself and sending a PR, but I'm including the above link in the event I'm unable to do so.
When I run kitchen verify --no-color
, kitchen-inspec adds colorization markers to the output. I would expect kitchen-inspec to respect the --no-color
flag.
...
Compiling Cookbooks...
Converging 0 resources
Running handlers:
Running handlers complete
Chef Client finished, 0/0 resources updated in 03 seconds
Finished converging <default-centos-72> (0m33.53s).
-----> Setting up <default-centos-72>...
Finished setting up <default-centos-72> (0m0.00s).
-----> Verifying <default-centos-72>...
Use `/root/learn-chef/cookbooks/webserver_test/test/recipes/default` for testing
Target: ssh://[email protected]:2222
�[31m ✖ System Package httpd should be installedexpected that `System Package httpd` is installed�[0m
Summary: �[32m0 successful�[0m, �[31m1 failures�[0m, �[37m0 skipped�[0m
$ chef --version
Chef Development Kit Version: 0.16.28
chef-client version: 12.12.15
delivery version: master (921828facad8a8bbbd767368bfc72f19bd30e7bd)
berks version: 4.3.5
kitchen version: 1.10.2
$ chef gem list | grep kitchen-inspec
kitchen-inspec (0.15.0)
I have attempted this with both Cent 6 and 7, using Apache 2.2 and 2.4.
describe apache_conf do
its('ServerTokens') { should eq 'Prod' }
end
The error I get when running this in test kitchen is
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: Failed to complete #verify action: [undefined method `run_command' for #<#<Class:0x4489548>:0x3ff0c48>] >>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration
I'm not sure if this is an issue with inspec or kitchen-inspec...if this should be reported to inspec, please let me know.
I already filed two tickets for this issue. Here are their references:
I'm not sure if this is a duplicate of #25 but I'm having trouble getting kitchen-inspec to work with profiles.
I have the following directory structure:
test
└── integration
├── linux
│ ├── controls
│ │ ├── ntp_spec.rb
│ │ └── selinux_spec.rb
│ └── inspec.yml
└── pm-inbound
└── default_spec.rb
My test/pm-inbound/default_spec.rb
looks like this:
require_controls 'linux' do
control 'ntp-config'
end
inspec seems to work (obviously tests fail here b/c I haven't passed a host):
[git] $ inspec exec test/integration/pm-inbound/default_spec.rb --profiles-path test/integration
FFFF
Failures:
1) System Package ntp should be installed
Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }
expected that `System Package ntp` is installed
# controls/ntp_spec.rb:8:in `block (3 levels) in load'
2) Service ntpd should be enabled
Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }
expected that `Service ntpd` is enabled
# controls/ntp_spec.rb:12:in `block (3 levels) in load'
3) Service ntpd should be installed
Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }
expected that `Service ntpd` is installed
# controls/ntp_spec.rb:13:in `block (3 levels) in load'
4) Service ntpd should be running
Failure/Error: DEFAULT_FAILURE_NOTIFIER = lambda { |failure, _opts| raise failure }
expected that `Service ntpd` is running
# controls/ntp_spec.rb:14:in `block (3 levels) in load'
Finished in 0.40791 seconds (files took 0.39686 seconds to load)
4 examples, 4 failures
Failed examples:
rspec # System Package ntp should be installed
rspec # Service ntpd should be enabled
rspec # Service ntpd should be installed
rspec # Service ntpd should be running
However, kitchen-inspec won't work:
[git] $ kitchen verify
-----> Starting Kitchen (v1.6.0)
-----> Verifying <pm-inbound-centos-67>...
Search `/Users/bkrein/git/ansible/test/integration/pm-inbound` for tests
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: Failed to complete #verify action: [undefined method `control' for #<Inspec::ProfileContext:0x007fbe14677c38>]
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration
zlib(finalizer): the stream was freed prematurely.
My .kitchen.yml
:
---
driver:
name: vagrant
provision_command: sed -i '/tsflags=nodocks/d' /etc/yum.conf
provisioner:
name: ansible_playbook
hosts: localhost
require_chef_for_busser: false
require_ruby_for_busser: false
requirements_path: requirements-test.txt
platforms:
- name: centos-6.7
- name: centos-7.1
verifier:
name: inspec
suites:
- name: pm-inbound
provisioner:
name: ansible_playbook
hosts: pm-inbound-ord
playbook: pm-inbound.yml
additional_copy_path:
- "."
Not sure if this is an issue with kitchen-docker or kitchen-inspec. The issue returned comes with the following:
Search `/Users/user/code/cookbook/test/integration/default` for tests
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: Failed to complete #verify action: [Sudo failed: Sudo requires a TTY. Please see the README on how to configure sudo to allow for non-interactive usage.]
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration
The kitchen file looks a such and the converge works easily. I am also using docker-beta which I am not sure if that plays into this at all.
---
driver:
name: docker
use_sudo: false
provisioner:
name: chef_zero
verifier:
name: inspec
platforms:
- name: centos-6.7
- name: centos-7.1
suites:
- name: default
run_list:
- recipe[cookbook]
In case you have several instances and one of them gives some pending messages like in my case, whey get propagated to the instance below:
$ kitchen list
Instance Driver Provisioner Verifier Transport Last Action
desktop-mint-172-cinnamon Vagrant ChefZero Inspec Ssh Verified
desktop-ubuntu-1404 Vagrant ChefZero Inspec Ssh Verified
# single verify for mint
############################################################################
$ kitchen verify desktop-mint-172-cinnamon
-----> Starting Kitchen (v1.4.2)
-----> Verifying <desktop-mint-172-cinnamon>...
*............*.........*..*
Pending: (Failures listed here are expected and do not affect your suite's status)
1) System Package crossover The `package` resource is not supported on your OS yet.
# Not yet implemented
# /home/artem/.chefdk/gem/ruby/2.1.0/gems/inspec-0.9.5/lib/inspec/runner.rb:114
2) System Package x11vnc The `package` resource is not supported on your OS yet.
# Not yet implemented
# /home/artem/.chefdk/gem/ruby/2.1.0/gems/inspec-0.9.5/lib/inspec/runner.rb:114
3) Service x11vnc The `service` resource is not supported on your OS yet.
# Not yet implemented
# /home/artem/.chefdk/gem/ruby/2.1.0/gems/inspec-0.9.5/lib/inspec/runner.rb:114
4) System Package owncloud-client The `package` resource is not supported on your OS yet.
# Not yet implemented
# /home/artem/.chefdk/gem/ruby/2.1.0/gems/inspec-0.9.5/lib/inspec/runner.rb:114
Finished in 0.48911 seconds (files took 1.33 seconds to load)
27 examples, 0 failures, 4 pending
Finished verifying <desktop-mint-172-cinnamon> (0m1.61s).
-----> Kitchen is finished. (0m4.43s)
# single verify for ubuntu
############################################################################
$ kitchen verify desktop-ubuntu-1404
-----> Starting Kitchen (v1.4.2)
-----> Setting up <desktop-ubuntu-1404>...
Finished setting up <desktop-ubuntu-1404> (0m0.00s).
-----> Verifying <desktop-ubuntu-1404>...
............................
Finished in 0.94141 seconds (files took 1.36 seconds to load)
28 examples, 0 failures
Finished verifying <desktop-ubuntu-1404> (0m2.09s).
-----> Kitchen is finished. (0m4.83s)
# verify both of them
############################################################################
$ kitchen verify
-----> Starting Kitchen (v1.4.2)
-----> Verifying <desktop-mint-172-cinnamon>...
*............*.........*..*
Pending: (Failures listed here are expected and do not affect your suite's status)
1) System Package crossover The `package` resource is not supported on your OS yet.
# Not yet implemented
# /home/artem/.chefdk/gem/ruby/2.1.0/gems/inspec-0.9.5/lib/inspec/runner.rb:114
2) System Package x11vnc The `package` resource is not supported on your OS yet.
# Not yet implemented
# /home/artem/.chefdk/gem/ruby/2.1.0/gems/inspec-0.9.5/lib/inspec/runner.rb:114
3) Service x11vnc The `service` resource is not supported on your OS yet.
# Not yet implemented
# /home/artem/.chefdk/gem/ruby/2.1.0/gems/inspec-0.9.5/lib/inspec/runner.rb:114
4) System Package owncloud-client The `package` resource is not supported on your OS yet.
# Not yet implemented
# /home/artem/.chefdk/gem/ruby/2.1.0/gems/inspec-0.9.5/lib/inspec/runner.rb:114
Finished in 0.49791 seconds (files took 1.35 seconds to load)
27 examples, 0 failures, 4 pending
Finished verifying <desktop-mint-172-cinnamon> (0m1.64s).
-----> Verifying <desktop-ubuntu-1404>...
............................
Pending: (Failures listed here are expected and do not affect your suite's status)
1) System Package crossover The `package` resource is not supported on your OS yet.
# Not yet implemented
# /home/artem/.chefdk/gem/ruby/2.1.0/gems/inspec-0.9.5/lib/inspec/runner.rb:114
2) System Package x11vnc The `package` resource is not supported on your OS yet.
# Not yet implemented
# /home/artem/.chefdk/gem/ruby/2.1.0/gems/inspec-0.9.5/lib/inspec/runner.rb:114
3) Service x11vnc The `service` resource is not supported on your OS yet.
# Not yet implemented
# /home/artem/.chefdk/gem/ruby/2.1.0/gems/inspec-0.9.5/lib/inspec/runner.rb:114
4) System Package owncloud-client The `package` resource is not supported on your OS yet.
# Not yet implemented
# /home/artem/.chefdk/gem/ruby/2.1.0/gems/inspec-0.9.5/lib/inspec/runner.rb:114
Finished in 0.88895 seconds (files took 2.36 seconds to load)
55 examples, 0 failures, 4 pending
Finished verifying <desktop-ubuntu-1404> (0m1.40s).
-----> Kitchen is finished. (0m5.93s)
Initially I thougth the second output is some kind of summary over failures, but the line Finished verifying <desktop-ubuntu-1404>
is displayed after it
Software versions:
$ chef --version
Chef Development Kit Version: 0.10.0
chef-client version: 12.5.1
berks version: 4.0.1
kitchen version: 1.4.2
$ chef exec gem list inspec
*** LOCAL GEMS ***
inspec (0.9.5, 0.9.2)
kitchen-inspec (0.9.0)
$ inspec version
0.9.5
InSpec introduced a new feature called attributes with InSpec 1.0 inspec/inspec#719. At this point, kitchen-inspec does not support this feature. My expectation is, that attributes work across the Chef tool chain.
a simple attribute https://github.com/chef/inspec/blob/master/examples/profile-attribute.yml file looks like
user: bob
password: secret
To support attributes in kitchen.yml, we could do the following:
suites:
- name: default
verifier:
inspec_tests:
- https://github.com/dev-sec/tests-ssh-hardening
attributes:
user: bob
password: secret
Not sure if we want to support external yml files?
Hi. Can I use InSpec Profile? https://github.com/chef/inspec/blob/master/docs/profiles.rst
Where should I create directory for profiles?
I tried create profile in test/controls/*
and ran include_conrtols 'name'
in my test, but control not started.
Right now, the path kitchen-inspec searches for tests is hard-coded.(https://github.com/chef/kitchen-inspec/blob/master/lib/kitchen/verifier/inspec.rb#L72)
It takes the base path (e.g. test/integration/
) and then appends the suite-name (default
).
Now we have the problem that we want to share tests between different suites (dev-sec/ansible-ssh-hardening#61):
The tests are in test/integration/default/inspec/
, but kitchen-inspec searches in test/integration/ansible-latest/inspec/
.
Therefore I propose to let the user define the test search path, either as a relative variable to the base path (in this case default/
or as an absolute path (test/integration/default
).
When third-party transports are used, inspec fails to function. An example of this is kitchen-sync/issue#14 where the default verifier is 'rsync'.
The root cause of this issue is the hard coded suppported transports:
https://github.com/chef/kitchen-inspec/blob/master/lib/kitchen/verifier/inspec.rb#L38-L49
The expected behavior is to support graceful transport downgrades, and fall back to the default transports.
Local gem list goes this way:-
kitchen-ec2 (1.0.0)
kitchen-inspec (0.12.5)
kitchen-vagrant (0.20.0)
test-kitchen (1.6.0)
Still facing the following issue
pavan@ubuntu:~/sample/git-cookbook$ kitchen converge
-----> Starting Kitchen (v1.6.0)
-----> Converging <default-windows-2012r2>...
Preparing files for transfer
Preparing dna.json
Preparing current project directory as a cookbook
Removing non-cookbook files before transfer
Preparing validation.pem
Preparing client.rb
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: Failed to complete #converge action: [undefined method `encoding' for nil:NilClass]
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration
$ .bin/rake test
/Users/stephan/.rbenv/versions/2.2.3/bin/ruby -I/Users/stephan/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/rspec-core-3.4.1/lib:/Users/stephan/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/rspec-support-3.4.1/lib /Users/stephan/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/rspec-core-3.4.1/exe/rspec --pattern spec/\*\*\{,/\*/\*\*\}/\*_spec.rb
Kitchen::Verifier::Inspec
verifier api_version is 1
plugin_version is set to Kitchen::Verifier::INSPEC_VERSION
with an ssh transport
constructs a Inspec::Runner using transport config data and state (FAILED - 1)
adds *spec.rb test files to runner (FAILED - 2)
calls #run on the runner (FAILED - 3)
with an winrm transport
constructs a Inspec::Runner using transport config data and state (FAILED - 4)
with an unsupported transport
#call raises a UserError
Failures:
1) Kitchen::Verifier::Inspec with an ssh transport constructs a Inspec::Runner using transport config data and state
Failure/Error: raise ActionFailed, "Inspec Runner returns #{exit_code}" unless exit_code == 0
Kitchen::ActionFailed:
Inspec Runner returns
# ./lib/kitchen/verifier/inspec.rb:57:in `call'
# ./spec/kitchen/verifier/inspec_spec.rb:163:in `block (3 levels) in <top (required)>'
2) Kitchen::Verifier::Inspec with an ssh transport adds *spec.rb test files to runner
Failure/Error: raise ActionFailed, "Inspec Runner returns #{exit_code}" unless exit_code == 0
Kitchen::ActionFailed:
Inspec Runner returns
# ./lib/kitchen/verifier/inspec.rb:57:in `call'
# ./spec/kitchen/verifier/inspec_spec.rb:177:in `block (3 levels) in <top (required)>'
3) Kitchen::Verifier::Inspec with an ssh transport calls #run on the runner
Failure/Error: raise ActionFailed, "Inspec Runner returns #{exit_code}" unless exit_code == 0
Kitchen::ActionFailed:
Inspec Runner returns
# ./lib/kitchen/verifier/inspec.rb:57:in `call'
# ./spec/kitchen/verifier/inspec_spec.rb:184:in `block (3 levels) in <top (required)>'
4) Kitchen::Verifier::Inspec with an winrm transport constructs a Inspec::Runner using transport config data and state
Failure/Error: raise ActionFailed, "Inspec Runner returns #{exit_code}" unless exit_code == 0
Kitchen::ActionFailed:
Inspec Runner returns
# ./lib/kitchen/verifier/inspec.rb:57:in `call'
# ./spec/kitchen/verifier/inspec_spec.rb:228:in `block (3 levels) in <top (required)>'
Finished in 0.08909 seconds (files took 0.16373 seconds to load)
7 examples, 4 failures
Failed examples:
rspec ./spec/kitchen/verifier/inspec_spec.rb:140 # Kitchen::Verifier::Inspec with an ssh transport constructs a Inspec::Runner using transport config data and state
rspec ./spec/kitchen/verifier/inspec_spec.rb:166 # Kitchen::Verifier::Inspec with an ssh transport adds *spec.rb test files to runner
rspec ./spec/kitchen/verifier/inspec_spec.rb:180 # Kitchen::Verifier::Inspec with an ssh transport calls #run on the runner
rspec ./spec/kitchen/verifier/inspec_spec.rb:213 # Kitchen::Verifier::Inspec with an winrm transport constructs a Inspec::Runner using transport config data and state
/Users/stephan/.rbenv/versions/2.2.3/bin/ruby -I/Users/stephan/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/rspec-core-3.4.1/lib:/Users/stephan/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/rspec-support-3.4.1/lib /Users/stephan/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/rspec-core-3.4.1/exe/rspec --pattern spec/\*\*\{,/\*/\*\*\}/\*_spec.rb failed
This patch was included in 0.16, but that requires functionality only added to InSpec in 0.34. The current dependency on kitchen-inspec
is >= 0.22
, which is incorrect. We should yank the existing 0.16.0 and release a 0.16.1 with the correct InSpec dependency.
Create and converge work without any problems, but verify fails with following error message:
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: Failed to complete #verify action: [You must configure at least one authentication method for SSH: Password or key.]
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration
.kitchen.yml:
---
driver:
name: digitalocean
size: 512mb
platforms:
- name: centos-7-0-x64
- name: ubuntu-14-04-x64
provisioner:
name: chef_zero
verifier:
name: inspec
suites:
- name: default
run_list:
- recipe[bla::default]
kitchen log:
I, [2016-04-14T22:51:36.877664 #19221] INFO -- Kitchen: -----> Starting Kitchen (v1.6.0)
I, [2016-04-14T22:51:37.772431 #19221] INFO -- Kitchen: -----> Converging <default-centos-7-0-x64>...
I, [2016-04-14T22:51:37.772636 #19221] INFO -- Kitchen: -----> Converging <default-ubuntu-14-04-x64>...
I, [2016-04-14T22:52:13.444049 #19221] INFO -- Kitchen: -----> Setting up <default-ubuntu-14-04-x64>...
I, [2016-04-14T22:52:13.447798 #19221] INFO -- Kitchen: -----> Verifying <default-ubuntu-14-04-x64>...
I, [2016-04-14T22:52:33.439790 #19221] INFO -- Kitchen: -----> Setting up <default-centos-7-0-x64>...
I, [2016-04-14T22:52:33.442952 #19221] INFO -- Kitchen: -----> Verifying <default-centos-7-0-x64>...
E, [2016-04-14T22:52:33.449197 #19221] ERROR -- Kitchen: ------Exception-------
E, [2016-04-14T22:52:33.449383 #19221] ERROR -- Kitchen: Class: Kitchen::ActionFailed
E, [2016-04-14T22:52:33.449470 #19221] ERROR -- Kitchen: Message: Failed to complete #verify action: [You must configure at least one authentication method for SSH: Password or key.]
E, [2016-04-14T22:52:33.449545 #19221] ERROR -- Kitchen: ---Nested Exception---
E, [2016-04-14T22:52:33.449616 #19221] ERROR -- Kitchen: Class: Train::ClientError
E, [2016-04-14T22:52:33.449685 #19221] ERROR -- Kitchen: Message: You must configure at least one authentication method for SSH: Password or key.
E, [2016-04-14T22:52:33.449756 #19221] ERROR -- Kitchen: ------Backtrace-------
E, [2016-04-14T22:52:33.449824 #19221] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/r-train-0.10.3/lib/train/transports/ssh.rb:84:in `validate_options'
E, [2016-04-14T22:52:33.449897 #19221] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/r-train-0.10.3/lib/train/transports/ssh.rb:67:in `connection'
E, [2016-04-14T22:52:33.449968 #19221] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/inspec-0.14.8/lib/inspec/backend.rb:23:in `create'
E, [2016-04-14T22:52:33.450038 #19221] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/inspec-0.14.8/lib/inspec/runner.rb:46:in `configure_transport'
E, [2016-04-14T22:52:33.450109 #19221] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/inspec-0.14.8/lib/inspec/runner.rb:30:in `initialize'
E, [2016-04-14T22:52:33.450180 #19221] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/kitchen-inspec-0.12.3/lib/kitchen/verifier/inspec.rb:69:in `new'
E, [2016-04-14T22:52:33.450281 #19221] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/kitchen-inspec-0.12.3/lib/kitchen/verifier/inspec.rb:69:in `call'
E, [2016-04-14T22:52:33.450401 #19221] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.6.0/lib/kitchen/instance.rb:421:in `block in verify_action'
E, [2016-04-14T22:52:33.450474 #19221] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.6.0/lib/kitchen/instance.rb:513:in `call'
E, [2016-04-14T22:52:33.450544 #19221] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.6.0/lib/kitchen/instance.rb:513:in `synchronize_or_call'
E, [2016-04-14T22:52:33.450621 #19221] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.6.0/lib/kitchen/instance.rb:478:in `block in action'
E, [2016-04-14T22:52:33.450697 #19221] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/2.1.0/benchmark.rb:279:in `measure'
E, [2016-04-14T22:52:33.450773 #19221] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.6.0/lib/kitchen/instance.rb:477:in `action'
E, [2016-04-14T22:52:33.450863 #19221] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.6.0/lib/kitchen/instance.rb:415:in `verify_action'
E, [2016-04-14T22:52:33.450935 #19221] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.6.0/lib/kitchen/instance.rb:348:in `block in transition_to'
E, [2016-04-14T22:52:33.451008 #19221] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.6.0/lib/kitchen/instance.rb:347:in `each'
E, [2016-04-14T22:52:33.451080 #19221] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.6.0/lib/kitchen/instance.rb:347:in `transition_to'
E, [2016-04-14T22:52:33.451182 #19221] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.6.0/lib/kitchen/instance.rb:160:in `verify'
E, [2016-04-14T22:52:33.451281 #19221] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.6.0/lib/kitchen/command.rb:176:in `public_send'
E, [2016-04-14T22:52:33.451394 #19221] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.6.0/lib/kitchen/command.rb:176:in `block (2 levels) in run_action'
E, [2016-04-14T22:52:33.451521 #19221] ERROR -- Kitchen: ----------------------
I'm having trouble running the tests for a chef cookbook. I've resolved many errors, but this one finally stumped me.
$ kitchen test
-----> Starting Kitchen (v1.13.2)
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ClientError
>>>>>> Message: Could not load the 'inspec' verifier from the load path. Please ensure that your transport is installed as a gem or included in your Gemfile if using Bundler.
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration
(.kitchen.yml)
---
driver:
name: vagrant
provisioner:
name: chef_solo
verifier:
name: inspec
platforms:
- name: ubuntu-15.04
driver_config:
box: opscode-ubuntu-15.04
box_url: http://opscode-vm-bento.s3.amazonaws.com/vagrant/virtualbox/opscode_ubuntu-15.04_chef-provisionerless.box
suites:
- name: default
run_list:
- recipe[my_hostname::default]
verifier:
inspec_tests:
- test/recipes
attributes:
$ chef -v && kitchen -v && chef gem list inspec
Chef Development Kit Version: 0.18.30
chef-client version: 12.14.89
delivery version: master (d86679335580be3de22996ef294b20d525889d8d)
berks version: 5.1.0
kitchen version: ERROR
Test Kitchen version 1.13.2
*** LOCAL GEMS ***
debug_inspector (0.0.2)
inspec (1.1.0, 1.0.0)
kitchen-inspec (0.15.2)
Any ideas as to what to try next?
Using inspec - TestKitchen, I’m seeing duplicate inspec executions for the included tests. Single suite with path defined to test/recipes/ containing two files with one control in each but see four controls executed in the kitchen test/verify output.
- name: master
run_list:
- recipe[example::install_master]
- recipe[example::configure_master]
verifier:
inspec_tests:
- test/recipes/master
One observation I've noted is the duplicated paths
-----> Verifying <master-ubuntu-1604>...
D Initialize InSpec
Use `/Users/droche/work/infra-auto/cookbooks/example/test/recipes/master` for testing
D Running tests from: ["/Users/droche/work/infra-auto/cookbooks/example/test/recipes/master", "test/recipes/master"]
Create a stock cookbook using chef generate cookbook duptest
Create a suite directory under test/recipes/<suite>
and move the default_test.rb into this directory
kitchen verify def --log-level=debug
-----> Starting Kitchen (v1.11.1)
D [Vagrant command] BEGIN (vagrant --version)
D [Vagrant command] END (0m0.47s)
D Berksfile found at /Users/droche/work/infra-auto/cookbooks/duptest/Berksfile, loading Berkshelf
D Berkshelf 4.3.5 library loaded
-----> Verifying <default-ubuntu-1604>...
D Initialize InSpec
Use `/Users/droche/work/infra-auto/cookbooks/duptest/test/recipes/default` for testing
D Running tests from: ["/Users/droche/work/infra-auto/cookbooks/duptest/test/recipes/default", "test/recipes"]
Target: ssh://[email protected]:2202
✔ User root should exist
✔ Port 80 should not be listening
○ User root should exist; User root This is an example test, r... (1 skipped)
This is an example test, replace with your own test.
○ Port 80 should not be listening; Port 80 This is an example ... (1 skipped)
This is an example test, replace with your own test.
Summary: 4 successful, 0 failures, 4 skipped
Finished verifying <default-ubuntu-1604> (0m0.42s).
-----> Kitchen is finished. (0m1.78s)
Removning the definition of the verifier
removes the duplicate executions when the test/recipes/<suite>
directory is used.
If you write a test that looks similar to below and you use kitchen verify/test this will fail with the curl error curl: can't find host localhost
describe bash('curl -sS localhost:80/index.html') do
its('stdout') { should_not match /HTTP Status 404/i }
its('stderr') { should eq ''}
end
This does however work if you use the inspec cli and run inspec exec test.rb -t ssh://[email protected]:2222/
I don't think I am missing anything and I can't get any more detailed pulled from the kitchen logs. Is this expected behavior or did I miss up a configuration
Hey, would it be possible to show the test and the result as it is with serverspec:
File "/etc/apache2/httpd.conf"
should exist
Port "80"
should be listening
Command "curl http://localhost"
stdout
should match /Hello/
Finished in 0.13462 seconds (files took 0.33277 seconds to load)
3 examples, 0 failures
Is the default search location of kitchen-inspec going to be updated to match the standards direction that Chef is going in? Specifically discussed in chef-boneyard/chef-dk#1096.
Additionally, gathered from reading discussions on issues, and reading the source https://github.com/chef/kitchen-inspec/blob/master/lib/kitchen/verifier/inspec.rb#L121, it seems as if test/integration/suite_name
is the only intended supported location.
The cookbook generator in the chef-dk creates /test/recipes
so we have been storing our tests in /test/recipes/suite_name
and kitchen-inspec finds them without having to explicitly point to them in kitchen.yml. https://github.com/chef/chef-dk/blob/627ff580d26c7452285de3d6c4f0692a762953f0/lib/chef-dk/skeletons/code_generator/recipes/cookbook.rb#L58
What's the intended location, which one of these things is the "future", and how can I help align these things so they make sense for the next person learning Chef?
Just to keep these all linked: chef/chef-vault-testfixtures#24
As highlighted in chef/chef#4606 we need to support single inspec
subdirectories as well.
acceptance git:(master) tree
.
├── Gemfile
├── README.md
├── basics
│ └── test
│ └── integration
│ └── chef-current-install
│ └── serverspec
│ ├── chef_client_spec.rb
│ └── spec_helper.rb
├── top-cookbooks
├── trivial
│ └── test
│ └── integration
│ └── chef-current-install
│ └── inspec
│ └── chef_client_spec.rb
└── windows-service
└── test
└── integration
└── chef-windows-service
└── inspec
└── chef_windows_service_spec.rb
16 directories, 6 files
To fix that we need to add:
inspec
directoryIt looks like the default inspec format for output has changed to not be as helpful when issues show up. For example lets say you have something like this:
describe file('/home/me/.ssh') do
it { should exist }
it { should be_file }
end
That would output a single line
✔ File /home/me/.ssh should exist
# And then a bunch of error with no context
... Blah
Not sure if this is something I am doing wrong on my system. Using inspec as a verifier is something we are trying to switch to but I would rather not add the following below:
verifier:
name: inspec
format: progress # <- This
Just trying to start a discussion on this, I can do the code. I believe it will be one line but not going to do that if its a waste of time and no one agrees.
https://github.com/chef/kitchen-inspec/blob/master/lib/kitchen/verifier/inspec.rb#L85
Add a cli warning for all skipped files.
It's pretty common now to see colo[u]rs for "." and "F"s in tests.
I'd like to ask that this is added to the runs, so something like the following would be colo[u]red.
Search `/Users/jasghar/repo/cookbooks/hipchat-chef-plugin-cookbook/test/integration/default` for tests
......FF........
Failures:
This would also put the Failures in RED so you can could identify issues quickly.
I often find myself looking up the ssh command-line parameters used in kitchen login [instance]
to run inspec shell
against it. Wouldn't it be great if this could be easier?
Broken link in ReadMe
When running against multiple nodes, the last node seems to aggregate the failure output, but not flag which node they were really from.
I tried using this doing kitchen testing with docker utilizing the kitchen-docker gem and it doesn't seem to run the verify tests. I thought it'd work since it looks like kitchen-docker uses the SSH transport but it seems something more may be needed?
docker output:
$ kitchen list
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/httpclient-2.6.0.1/lib/httpclient/webagent-cookie.rb:458: warning: already initialized constant HTTPClient::CookieManager
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/httpclient-2.6.0.1/lib/httpclient/cookie.rb:8: warning: previous definition of CookieManager was here
Instance Driver Provisioner Verifier Transport Last Action
default-centos-66 Docker ChefZero Inspec Ssh Verified
$ kitchen verify
-----> Starting Kitchen (v1.4.2)
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/httpclient-2.6.0.1/lib/httpclient/webagent-cookie.rb:458: warning: already initialized constant HTTPClient::CookieManager
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/httpclient-2.6.0.1/lib/httpclient/cookie.rb:8: warning: previous definition of CookieManager was here
-----> Verifying <default-centos-66>...
$$$$$$ Running legacy verify for 'Docker' Driver
Preparing files for transfer
Transferring files to <default-centos-66>
Finished verifying <default-centos-66> (0m0.00s).
-----> Kitchen is finished. (0m0.95s)
$ ls test/integration/default/
node_prepare_spec.rb
vagrant/vbox output:
$ kitchen list
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/httpclient-2.6.0.1/lib/httpclient/webagent-cookie.rb:458: warning: already initialized constant HTTPClient::CookieManager
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/httpclient-2.6.0.1/lib/httpclient/cookie.rb:8: warning: previous definition of CookieManager was here
Instance Driver Provisioner Verifier Transport Last Action
default-rhel6 Vagrant ChefZero Inspec Ssh Verified
$ kitchen verify
-----> Starting Kitchen (v1.4.2)
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/httpclient-2.6.0.1/lib/httpclient/webagent-cookie.rb:458: warning: already initialized constant HTTPClient::CookieManager
/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/httpclient-2.6.0.1/lib/httpclient/cookie.rb:8: warning: previous definition of CookieManager was here
-----> Verifying <default-rhel6>...
FFFFF
Failures:
1) System Package pcs should be installed
Failure/Error: it { should be_installed }
expected that `System Package pcs` is installed
# ./test/integration/default/node_prepare_spec.rb:2:in `block (2 levels) in load'
2) Service pcsd should be enabled
Failure/Error: it { should be_enabled }
expected that `Service pcsd` is enabled
# ./test/integration/default/node_prepare_spec.rb:6:in `block (2 levels) in load'
3) Service pcsd should be running
Failure/Error: it { should be_running }
expected that `Service pcsd` is running
# ./test/integration/default/node_prepare_spec.rb:7:in `block (2 levels) in load'
4) Service pacemaker should be running
Failure/Error: it { should be_running }
expected that `Service pacemaker` is running
# ./test/integration/default/node_prepare_spec.rb:11:in `block (2 levels) in load'
5) Service corosync should be running
Failure/Error: it { should be_running }
expected that `Service corosync` is running
# ./test/integration/default/node_prepare_spec.rb:17:in `block (2 levels) in load'
Finished in 0.75939 seconds (files took 0.37673 seconds to load)
5 examples, 5 failures
Failed examples:
rspec # System Package pcs should be installed
rspec # Service pcsd should be enabled
rspec # Service pcsd should be running
rspec # Service pacemaker should be running
rspec # Service corosync should be running
Finished verifying <default-rhel6> (0m1.01s).
-----> Kitchen is finished. (0m2.10s)
zlib(finalizer): the stream was freed prematurely.
$ ls test/integration/default/
node_prepare_spec.rb
Thanks!
There is no way to retrieve the output of InSpec from kitchen when you have multiple platforms and multiple test suites in your kitchen setup.
For example if you have support for centos6 and centos7 and multiple test suites defined, default, datacenter1 and datacenter2 you'll have 6 InSpec executions so you'll have 6 different inspec output files but there is no way to specify them.
Right now we use
kitchen.yml (yml format)
suites:
- name: contains_inspec
run_list:
...
verifier:
inspec_tests:
- https://github.com/dev-sec/tests-ssh-hardening
- https://github.com/dev-sec/tests-os-hardening
This format is different from inspec.yml
name: meta-profile
title: Meta Compliance Profile
maintainer: InSpec Authors
copyright: InSpec Authors
copyright_email: [email protected]
license: Apache 2
summary: InSpec Profile that is only consuming dependencies
version: 0.2.0
depends:
- name: hardening/ssh-hardening # defaults to supermarket
- name: os-hardening
url: https://github.com/dev-sec/tests-os-hardening/archive/master.zip
- git: https://github.com/dev-sec/ssl-benchmark.git
- name: windows-patch-benchmark
git: https://github.com/chris-rock/windows-patch-benchmark.git
- name: linux
compliance: base/linux
Therefore we should align those formats to easily allow copy&paste between tools:
suites:
- name: contains_inspec
run_list:
...
verifier:
inspec_tests:
- name: hardening/ssh-hardening # defaults to supermarket
- name: os-hardening
url: https://github.com/dev-sec/tests-os-hardening/archive/master.zip
- git: https://github.com/dev-sec/tests-ssh-hardening.git
- git: https://github.com/dev-sec/tests-os-hardening.git
- name: hardening/ssh-hardening
supermarket_url: https://supermarket.my.com
Hi,
I need to test that some Gems are installed(they have been installed via the chef_gem
. However it looks like kitchen-inspec isn't able to pick up on any gem installation. In order to try and track down this issue I ended up installing rvm and some test gems to see if inspec would return those, it didn't seem to help. However I noticed when I ran inspec exec test/integration/server/default.rb -t ssh://[email protected] --password 'vagrant'
it would seem to find my rvm gempath correctly. But this doesn't help my original issue of inspec not picking up on chef gems not being seen.
I'm not sure if this is an issue with inspec, kitchen-inspec or some misconfiguration on my end. Any advice to point me in the right direction would be great!
Thanks!
gem package
✖ kitchen-ec2 should be installed
expected that `gem package kitchen-ec2` is installed
vagrant@default-ubuntu-1604:/etc/systemd/system$ gem list -i kitchen-ec2
true
vagrant@default-ubuntu-1604:/etc/systemd/system$ which gem
/opt/chefdk/embedded/bin/gem
Ability load compliance profiles from a remote storage.
Essentially implements test-kitchen/test-kitchen#434 for inspec
verifier:
name: inspec
format: junit
output: ../../build/junit/%{platform}_%{suite}_inspec.xml
$ cat test/integration/memcached-single-instance/inspec/memcached.rb
describe package('memcached') do
it { should be_installed }
end
# workaround service() auto-detect
# https://github.com/chef/inspec/issues/1171
describe sysv_service('memcached') do
it { should be_running }
it { should be_enabled }
before do
skip unless os[:release] == '14.04' or os[:release] == '16.04'
end
end
describe service('memcached') do
it "is listening on port 11211" do
expect(port(11211)).to be_listening
end
end
describe file('/etc/memcached.conf') do
its('content') { should match '-m 1G' }
end
$ cat ../../build//junit/ubuntu-16.04_memcached-single-instance_inspec.xml
<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="rspec" tests="5" failures="0" errors="0" time="0.197933" timestamp="2017-01-23T10:17:42+01:00">
<!-- Randomized with seed 24234 -->
<properties/>
<testcase classname="usr.local.lib.ruby.gems.2.4.0.gems.inspec-1.9.0.lib.inspec.rule" name="System Package memcached should be installed" file="/usr/local/lib/ruby/gems/2.4.0/gems/inspec-1.9.0/lib/inspec/rule.rb" time="0.061945"/>
<testcase classname="usr.local.lib.ruby.gems.2.4.0.gems.inspec-1.9.0.lib.inspec.rule" name="Service memcached should be running" file="/usr/local/lib/ruby/gems/2.4.0/gems/inspec-1.9.0/lib/inspec/rule.rb" time="0.077482"/>
<testcase classname="usr.local.lib.ruby.gems.2.4.0.gems.inspec-1.9.0.lib.inspec.rule" name="Service memcached should be enabled" file="/usr/local/lib/ruby/gems/2.4.0/gems/inspec-1.9.0/lib/inspec/rule.rb" time="0.000192"/>
<testcase classname="usr.local.lib.ruby.gems.2.4.0.gems.inspec-1.9.0.lib.inspec.rule" name="Service memcached is listening on port 11211" file="/usr/local/lib/ruby/gems/2.4.0/gems/inspec-1.9.0/lib/inspec/rule.rb" time="0.032800"/>
<testcase classname="test.integration.memcached-single-instance.inspec.memcached" name="File /etc/memcached.conf content should match "-m 1G"" file="./test/integration/memcached-single-instance/inspec/memcached.rb" time="0.023982"/>
</testsuite>
Why does describe file come from memcached.rb, but describe package from /usr/local/lib/ruby/gems/2.4.0/gems/inspec-1.9.0/lib/inspec/rule.rb?
$ kitchen test generic-centos-71
-----> Starting Kitchen (v1.4.2)
-----> Cleaning up any prior instances of <generic-centos-71>
-----> Destroying <generic-centos-71>...
Finished destroying <generic-centos-71> (0m0.00s).
-----> Testing <generic-centos-71>
-----> Creating <generic-centos-71>...
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'opscode-centos-7.1'...
....
Chef Client finished, 0/0 resources updated in 01 seconds
Finished converging <generic-centos-71> (0m55.43s).
-----> Setting up <generic-centos-71>...
Finished setting up <generic-centos-71> (0m0.00s).
-----> Verifying <generic-centos-71>...
F
Failures:
1) Command /bin/false exit_status should eq 0
Failure/Error: its('exit_status') { should eq 0 }
expected: 0
got: 1
(compared using ==)
.....
Finished in 0.07612 seconds (files took 1 minute 32.92 seconds to load)
1 example, 1 failure
Failed examples:
rspec # Command /bin/false exit_status should eq 0
Finished verifying <generic-centos-71> (0m1.37s).
-----> Destroying <generic-centos-71>...
==> default: Forcing shutdown of VM...
==> default: Destroying VM and associated drives...
Vagrant instance <generic-centos-71> destroyed.
Finished destroying <generic-centos-71> (0m7.11s).
Finished testing <generic-centos-71> (1m39.90s).
-----> Kitchen is finished. (1m42.84s)
$ echo $?
0
with default behaviour of --destroy=passing
of kitchen test
the instance should be not destroyed.
Probably some return code to the kitchen about failures is missing somewhere?
Software versions:
$ chef --version
Chef Development Kit Version: 0.10.0
chef-client version: 12.5.1
berks version: 4.0.1
kitchen version: 1.4.2
$ chef exec gem list inspec
*** LOCAL GEMS ***
inspec (0.9.5, 0.9.2)
kitchen-inspec (0.9.0)
$ inspec version
0.9.5
Probably this is related to #9
kitchen-inspec does not load supermarket urls
suites:
- name: default
run_list:
- recipe[prepare]
verifier:
inspec_tests:
- supermarket://hardening/ssh-hardening
Executing multiple simultaneous kitchen verify
runs using concurrency (-c 5
, for example) fails, apparently using the same configuration for all nodes (note target in the attached log has the same port for each run, when the boxes are all running on different ports).
When running serially instead of concurrently, it works fine.
As reported by @burtlo: With _bats_ we define a directory within the ./test/integration
directory. With _serverspec_ we define a similar directory. With _inspec_ we change the _verifier_ in the .kitchen.yml
and its convention (according to this one example: https://github.com/chef/inspec/tree/master/examples/test-kitchen/test/integration/default) uses the entire default test suite directory instead of sub-directory.
I'm trying to implement some kitchen-inspec tests using a custom inspec resource in the spirit of this blog post. Thus I have created a file named test/integration/my-suite/libraries/query.rb
with the following content:
module Inspec::Resources
class QueryResource < Inspec.resource(1)
name 'query'
desc 'dummy inspec resource'
def initialize(args)
@args = args
end
def bar?
@args == 'foo'
end
end
end
My test case test/integration/my-suite/default_spec.rb
is
describe query('foo') do
it { should be_bar }
end
When running kitchen via kitchen verify my-suite
the new resource doesn't seem to be available:
$ kitchen verify my-suite
Ignoring unf_ext-0.0.7.2 because its extensions are not built. Try: gem pristine unf_ext --version 0.0.7.2
-----> Starting Kitchen (v1.13.2)
-----> Verifying <zabbix-resources-ubuntu-1404>...
Use `/Users/robert/work/westernacher/chef-repo/cookbooks/zabbix_wps/test/integration/zabbix-resources` for testing
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: 1 actions failed.
>>>>>> Failed to complete #verify action: [undefined method `query' for #<#<Class:0x007ffd8ad6a058>:0x007ffd8ad697c0>] on zabbix-resources-ubuntu-1404
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration
I also tried putting the inspec resource definition directly in my test-case with the same result.
Thanks in advance!
When using the default tests created by chef generate cookbook
, if I leave them in, and they fail, even though they are marked skip, they report as a failure in kitchen.
Test:
unless os.windows?
describe user('root') do
it { should exist }
skip 'This is an example test, replace with your own test.'
end
end
describe port(80) do
it { should_not be_listening }
skip 'This is an example test, replace with your own test.'
end
If the machine does reply on port 80, this is the output of kitchen verify
:
User root
✔ should exist
↺ This is an example test, replace with your own test.
Port 80
∅ should not be listening
expected `Port 80.listening?` to return false, got true
↺ This is an example test, replace with your own test.
Test Summary: 1 successful, 1 failures, 2 skipped
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: 1 actions failed.
>>>>>> Verify failed on instance <default-centos-72>. Please see .kitchen/logs/default-centos-72.log for more details
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration
@coderanger and @cheeseplus pointed out, that it is not possible to reuse test across different cookbooks easily. He used shared tests like: https://github.com/poise/poise-service-spechelper/blob/master/lib/poise_service/spec_helper/helper.rb
in
https://github.com/poise/poise-service/blob/master/test/integration/default/serverspec/default_spec.rb#L17
We need to find a way to solve this problem with kitchen-inspec. Possible ways:
Any other ideas?
I would like my verifier to go against a local profile that is available on my hard drive. Something like:
verifier: inspec
profile: ../../profiles/my-profile
Is this currently possible?
Is anyone else running into this on a base CentOS AMI?
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: Failed to complete #verify action: [Sudo failed: Sudo requires a TTY. Please see the README on how to configure sudo to allow for non-interactive usage.]
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration
My inspec profile had platform set to 'all'', while my platform section for debian7 did not specify 'platform' value at all. But on the other hand inspec profile has os-family set.
Thus, when invoking kitchen verify I got error like below:
Failed to complete #verify action: [no implicit conversion of nil into Array]
backtrace goes to /home/kaszpir/.rbenv/versions/2.1.3/lib/ruby/gems/2.1.0/gems/kitchen-inspec-0.15.0/lib/kitchen/verifier/inspec.rb:133:in
+'`
where line 133 shows:
(local_suite_files + config[:inspec_tests]).compact
After adding some debugging messages I have found out that config[:inspec_tests]
is nil
Then I verified platform on inspec profile and in kitchen.yml and they were different (well, actually platfrom was not set for debian in my case)
I believe you can reproduce it with any driver, below is with LXC, also provisioner is rahter not important here.
Proper kitchen.yml - see comments
---
driver_plugin: lxc
driver:
name: vagrant
provider: lxc
transport:
name: sftp # kitchen-sync does not work properly with test-kitchen 1.4.x
compression_level: 9
provisioner:
name: puppet_apply
manifests_path: puppet/manifests
modules_path: puppet/modules
hiera_config_path: puppet/hiera.kitchen.yaml # different file mapping than files uploaded by boilerplate
hiera_data_path: puppet/hiera
puppet_verbose: true
require_chef_for_busser: false
verifier:
name: inspec
platforms:
# https://github.com/fgrehm/vagrant-lxc-base-boxes
# vagrant box add --name debian7lxc --provider lxc vagrant-lxc-wheezy-amd64.box
- name: debian.7-lxc
platform: debian # without this line it breaks with error [no implicit conversion of nil into Array], so comment this line to the the error for free! :)
driver:
box: debian7lxc
driver_config:
username: root
password: root
suites:
- name: twemproxy
driver:
provisioner:
manifest: twemproxy.pp
verifier:
inspec_tests:
- inspec/twemproxy
inspec file:
name: twemproxy
title: InSpec Profile
maintainer: Michał Sochoń
copyright: Michał Sochoń
copyright_email: [email protected]
license: Apache 2 license
summary: An InSpec Profile for twemproxy node
version: 0.2.0
supports:
- os-family: debian
release: 7
- platform: all
Test Kitchen version 1.11.1
inspec version 0.32.0
kitchen-inspec (0.15.0)
Generate more sane message if test suite list is empty.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.