test-kitchen / kitchen-ec2 Goto Github PK

View Code? Open in Web Editor NEW

221.0 26.0 203.0 1.12 MB

A Test Kitchen Driver for Amazon EC2

License: Other

Ruby 100.00%

aws test-kitchen ec2 ruby hacktoberfest managed-by-terraform ruby-gem testing

kitchen-ec2's Introduction

kitchen-ec2

A Test Kitchen Driver for Amazon EC2.

This driver uses the aws sdk gem to provision and destroy EC2 instances. Use Amazon's cloud for your infrastructure testing!

Quick Start

Install Chef Workstation. If testing things other than Chef Infra cookbooks, please consult your driver's documentation for information on what to install.
Install the AWS command line tools.
Run aws configure. This will set up your AWS credentials for both the AWS CLI tools and kitchen-ec2.
Add or edit the driver section of your kitchen.yml:
```
driver:
  name: ec2
```
Run kitchen test.

Requirements

There are no external system requirements for this driver. However you will need access to an AWS account. IAM users should have, at a minimum, permission to manage the lifecycle of an EC2 instance along with modifying components specified in kitchen driver configs. Consider using a permissive managed IAM policy like arn:aws:iam::aws:policy/AmazonEC2FullAccess or tailor one specific to your security requirements.

Configuration

By automatically applying reasonable defaults wherever possible, kitchen-ec2 does a lot of work to make your life easier. See the kitchen.ci kitchen-ec2 docs for a complete list of configuration options.

Development

Source hosted at GitHub
Report issues/questions/feature requests on GitHub Issues

Pull requests are very welcome! Make sure your patches are well tested. Ideally create a topic branch for every separate change you make. For example:

Fork the repo
Create your feature branch (git checkout -b my-new-feature)
Commit your changes (git commit -am 'Added some feature')
Push to the branch (git push origin my-new-feature)
Create new Pull Request

License

Apache 2.0 (see LICENSE)

kitchen-ec2's People

Contributors

Stargazers

Watchers

Forkers

agoddard jordandm dissonanz juliandunn higanworks johntdyer evertrue opsrockin ajmath atalanta arangamani nagliyvred fwiesel bozinsky coderanger spheromak mannytoledo simplefinance zerowolfgang anthroprose mdellanoce balanced-cookbooks naphthalene shalicke jigarjoshi anl sailthru-cookbooks benesch kavanista ekrupnik timurb dewdo ijin afiune justincampbell xtatsux gfawcettpq akretion sciurus scottnesmith kpettijohn chuckg r5v9 jghaines eslesarchuk engineyard nsdavidson charterj hroussez imagemetrics goodgamestudios dtoubelis rmoreira gfloyd localytics crowdworks volker-fr petecheslock igorshp listhub stevejmason jamesawesome caboteria jaym mfischer-zd acm1 grosendorf patcon lcanfield dsavinkov zl4bv gmiranda23 mirrors-ce yoshiwaan ustuehler whiteley jkeiser trevorrowe erkolson nellshamrell stift mauriciosilva proffalken rhass jlyheden yuzu-es s2t2 wjordan tas50 rtacconi harvard-atg yyuu dlukman jcastillocano criteo-forks kierandoonan chrisduong aogail eipplusone freimer

kitchen-ec2's Issues

net-scp version is 1.0.4

This is probably pulled in by Fog 1.9.0, this is out of sync with test-kitchen/1.0's version. Once the next Fog release is out, this will go away and perhaps we can lock it down to keep them in sync.

License missing from gemspec

Some companies will only use gems with a certain license.
The canonical and easy way to check is via the gemspec,

via e.g.

spec.license = 'MIT'
# or
spec.licenses = ['MIT', 'GPL-2']

Even for projects that already specify a license, including a license in your gemspec is a good practice, since it is easily
discoverable there without having to check the readme or for a license file. For example, it is the field that rubygems.org uses to display a gem's license.

For example, there is a License Finder gem to help companies ensure all gems they use
meet their licensing needs. This tool depends on license information being available in the gemspec. This is an important enough
issue that even Bundler now generates gems with a default 'MIT' license.

If you need help choosing a license (sorry, I haven't checked your readme or looked for a license file), github has created a license picker tool.

In case you're wondering how I found you and why I made this issue, it's because I'm collecting stats on gems (I was originally looking for download data) and decided to collect license metadata,too, and make issues for gemspecs not specifying a license as a public service :).

I hope you'll consider specifying a license in your gemspec. If not, please just close the issue and let me know. In either case, I'll follow up. Thanks!

p.s. I've written a blog post about this project

Issue with amazon linux 32 bit and SCP failing >>>>>> Message: Failed to complete #converge action: [SCP did not finish successfully (127): ]

snip from the tk.yml

  - name: amazon-ami-2014.09-32
    driver:
      image_id: ami-16852c7e
      flavor_id: c1.medium
      username: ec2-user
    run_list:
      - recipe[yum]

kitchen converge default-amazon-ami-201409-32
-----> Starting Kitchen (v1.2.1)
-----> Creating <default-amazon-ami-201409-32>...
       Creating <>...
       If you are not using an account that qualifies under the AWS
       free-tier, you may be charged to run these suites. The charge
       should be minimal, but neither Test Kitchen nor its maintainers
       are responsible for your incurred costs.
       EC2 instance <i-c1f4a93a> created.
.....................       (server ready)
       Waiting for ec2-54-88-70-203.compute-1.amazonaws.com:22...
       Waiting for ec2-54-88-70-203.compute-1.amazonaws.com:22...
       Waiting for ec2-54-88-70-203.compute-1.amazonaws.com:22...
       Waiting for ec2-54-88-70-203.compute-1.amazonaws.com:22...
       Waiting for ec2-54-88-70-203.compute-1.amazonaws.com:22...
       Waiting for ec2-54-88-70-203.compute-1.amazonaws.com:22...
       Waiting for ec2-54-88-70-203.compute-1.amazonaws.com:22...
       (ssh ready)

       Finished creating <default-amazon-ami-201409-32> (1m2.42s).
-----> Converging <default-amazon-ami-201409-32>...
       Preparing files for transfer
       Resolving cookbook dependencies with Berkshelf 3.2.1...
       Removing non-cookbook files before transfer
-----> Installing Chef Omnibus (11.16.4)
       downloading https://www.getchef.com/chef/install.sh
         to file /tmp/install.sh
       trying curl...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                               Dload  Upload   Total   Spent    Left  Spee       d
100   178  100   178    0     0    611      0 --:--:-- --:--:-- --:--:--   61       1
100 18265  100 18265    0     0  36728      0 --:--:-- --:--:-- --:--:-- 3672       8
       Downloading Chef 11.16.4 for el...
       downloading https://www.chef.io/chef/metadata?v=11.16.4&prerelease=false&nightlies=false&p=el&pv=6&m=i686
         to file /tmp/install.sh.1249/metadata.txt
       trying curl...
       url      https://opscode-omnibus-packages.s3.amazonaws.com/el/6/i686/chef-11.16.4-1.el6.i686.rpm
       md5      da7bbe41d9510de62adf6afe89ed2ecd
       sha256   545075be04de512c780c961b3aa3809c4540fd68c47c167eced67a8daa0821b7
       downloaded metadata file looks valid...
       downloading https://opscode-omnibus-packages.s3.amazonaws.com/el/6/i686/chef-11.16.4-1.el6.i686.rpm
         to file /tmp/install.sh.1249/chef-11.16.4-1.el6.i686.rpm
       trying curl...
       Comparing checksum with sha256sum...
       Installing Chef 11.16.4
       installing with rpm...
warning:        /tmp/install.sh.1249/chef-11.16.4-1.el6.i686.rpm: Header V4 DSA/SHA1 Signature, key ID 83ef826a: NOKEY
Preparing...                                 ################################# [100%]
       Updating / installing...
   1:chef-11.16.4-1.el6                      ################################# [100%]
       Thank you for installing Chef!
       Transfering files to <default-amazon-ami-201409-32>
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: Failed to complete #converge action: [SCP did not finish successfully (127): ]
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration

I, [2015-01-11T17:01:40.179529 #40963]  INFO -- Kitchen: -----> Starting Kitchen (v1.2.1)
I, [2015-01-11T17:01:42.041765 #40963]  INFO -- Kitchen: -----> Creating <default-amazon-ami-201409-32>...
I, [2015-01-11T17:02:44.468030 #40963]  INFO -- Kitchen: -----> Converging <default-amazon-ami-201409-32>...
E, [2015-01-11T17:02:52.606947 #40963] ERROR -- Kitchen: ------Exception-------
E, [2015-01-11T17:02:52.606997 #40963] ERROR -- Kitchen: Class: Kitchen::ActionFailed
E, [2015-01-11T17:02:52.607025 #40963] ERROR -- Kitchen: Message: Failed to complete #converge action: [SCP did not finish successfully (127): ]
E, [2015-01-11T17:02:52.607054 #40963] ERROR -- Kitchen: ---Nested Exception---
E, [2015-01-11T17:02:52.607078 #40963] ERROR -- Kitchen: Class: Net::SCP::Error
E, [2015-01-11T17:02:52.607101 #40963] ERROR -- Kitchen: Message: SCP did not finish successfully (127):
E, [2015-01-11T17:02:52.607122 #40963] ERROR -- Kitchen: ------Backtrace-------
E, [2015-01-11T17:02:52.607157 #40963] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/net-scp-1.2.1/lib/net/scp.rb:365:in `block (3 levels) in start_command'
E, [2015-01-11T17:02:52.607204 #40963] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/net-ssh-2.9.1/lib/net/ssh/connection/channel.rb:591:in `call'
E, [2015-01-11T17:02:52.607231 #40963] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/net-ssh-2.9.1/lib/net/ssh/connection/channel.rb:591:in `do_close'
E, [2015-01-11T17:02:52.607257 #40963] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/net-ssh-2.9.1/lib/net/ssh/connection/session.rb:586:in `channel_close'
E, [2015-01-11T17:02:52.607281 #40963] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/net-ssh-2.9.1/lib/net/ssh/connection/session.rb:465:in `dispatch_incoming_packets'
E, [2015-01-11T17:02:52.607305 #40963] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/net-ssh-2.9.1/lib/net/ssh/connection/session.rb:221:in `preprocess'
E, [2015-01-11T17:02:52.607345 #40963] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/net-ssh-2.9.1/lib/net/ssh/connection/session.rb:205:in `process'
E, [2015-01-11T17:02:52.607370 #40963] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/net-ssh-2.9.1/lib/net/ssh/connection/session.rb:169:in `block in loop'
E, [2015-01-11T17:02:52.607394 #40963] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/net-ssh-2.9.1/lib/net/ssh/connection/session.rb:169:in `loop'
E, [2015-01-11T17:02:52.607420 #40963] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/net-ssh-2.9.1/lib/net/ssh/connection/session.rb:169:in `loop'
E, [2015-01-11T17:02:52.607463 #40963] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/net-ssh-2.9.1/lib/net/ssh/connection/channel.rb:269:in `wait'
E, [2015-01-11T17:02:52.607488 #40963] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/net-scp-1.2.1/lib/net/scp.rb:284:in `upload!'
E, [2015-01-11T17:02:52.607521 #40963] ERROR -- Kitchen: /opt/chefdk/embedded/apps/test-kitchen/lib/kitchen/ssh.rb:70:in `upload!'
E, [2015-01-11T17:02:52.607545 #40963] ERROR -- Kitchen: /opt/chefdk/embedded/apps/test-kitchen/lib/kitchen/ssh.rb:76:in `upload_path!'
E, [2015-01-11T17:02:52.607576 #40963] ERROR -- Kitchen: /opt/chefdk/embedded/apps/test-kitchen/lib/kitchen/driver/ssh_base.rb:119:in `block in transfer_path'
E, [2015-01-11T17:02:52.607600 #40963] ERROR -- Kitchen: /opt/chefdk/embedded/apps/test-kitchen/lib/kitchen/driver/ssh_base.rb:119:in `each'
E, [2015-01-11T17:02:52.607638 #40963] ERROR -- Kitchen: /opt/chefdk/embedded/apps/test-kitchen/lib/kitchen/driver/ssh_base.rb:119:in `transfer_path'
E, [2015-01-11T17:02:52.607662 #40963] ERROR -- Kitchen: /opt/chefdk/embedded/apps/test-kitchen/lib/kitchen/driver/ssh_base.rb:46:in `block in converge'
E, [2015-01-11T17:02:52.607686 #40963] ERROR -- Kitchen: /opt/chefdk/embedded/apps/test-kitchen/lib/kitchen/ssh.rb:47:in `initialize'
E, [2015-01-11T17:02:52.607709 #40963] ERROR -- Kitchen: /opt/chefdk/embedded/apps/test-kitchen/lib/kitchen/driver/ssh_base.rb:43:in `new'
E, [2015-01-11T17:02:52.607746 #40963] ERROR -- Kitchen: /opt/chefdk/embedded/apps/test-kitchen/lib/kitchen/driver/ssh_base.rb:43:in `converge'
E, [2015-01-11T17:02:52.607771 #40963] ERROR -- Kitchen: /opt/chefdk/embedded/apps/test-kitchen/lib/kitchen/instance.rb:273:in `public_send'
E, [2015-01-11T17:02:52.607795 #40963] ERROR -- Kitchen: /opt/chefdk/embedded/apps/test-kitchen/lib/kitchen/instance.rb:273:in `block in perform_action'
E, [2015-01-11T17:02:52.607819 #40963] ERROR -- Kitchen: /opt/chefdk/embedded/apps/test-kitchen/lib/kitchen/instance.rb:308:in `call'
E, [2015-01-11T17:02:52.607991 #40963] ERROR -- Kitchen: /opt/chefdk/embedded/apps/test-kitchen/lib/kitchen/instance.rb:308:in `synchronize_or_call'
E, [2015-01-11T17:02:52.608025 #40963] ERROR -- Kitchen: /opt/chefdk/embedded/apps/test-kitchen/lib/kitchen/instance.rb:283:in `block in action'
E, [2015-01-11T17:02:52.608053 #40963] ERROR -- Kitchen: /opt/chefdk/embedded/lib/ruby/2.1.0/benchmark.rb:279:in `measure'
E, [2015-01-11T17:02:52.608085 #40963] ERROR -- Kitchen: /opt/chefdk/embedded/apps/test-kitchen/lib/kitchen/instance.rb:282:in `action'
E, [2015-01-11T17:02:52.608110 #40963] ERROR -- Kitchen: /opt/chefdk/embedded/apps/test-kitchen/lib/kitchen/instance.rb:273:in `perform_action'
E, [2015-01-11T17:02:52.608147 #40963] ERROR -- Kitchen: /opt/chefdk/embedded/apps/test-kitchen/lib/kitchen/instance.rb:256:in `converge_action'
E, [2015-01-11T17:02:52.608173 #40963] ERROR -- Kitchen: /opt/chefdk/embedded/apps/test-kitchen/lib/kitchen/instance.rb:246:in `block in transition_to'
E, [2015-01-11T17:02:52.608201 #40963] ERROR -- Kitchen: /opt/chefdk/embedded/apps/test-kitchen/lib/kitchen/instance.rb:245:in `each'
E, [2015-01-11T17:02:52.608226 #40963] ERROR -- Kitchen: /opt/chefdk/embedded/apps/test-kitchen/lib/kitchen/instance.rb:245:in `transition_to'
E, [2015-01-11T17:02:52.608276 #40963] ERROR -- Kitchen: /opt/chefdk/embedded/apps/test-kitchen/lib/kitchen/instance.rb:119:in `converge'
E, [2015-01-11T17:02:52.608308 #40963] ERROR -- Kitchen: /opt/chefdk/embedded/apps/test-kitchen/lib/kitchen/command.rb:109:in `public_send'
E, [2015-01-11T17:02:52.608353 #40963] ERROR -- Kitchen: /opt/chefdk/embedded/apps/test-kitchen/lib/kitchen/command.rb:109:in `block (2 levels) in run_action'
E, [2015-01-11T17:02:52.608381 #40963] ERROR -- Kitchen: ----------------------

using TK version

kitchen --version
Test Kitchen version 1.2.1

from Chef DK version:

chef --version
Chef Development Kit Version: 0.3.5

Has anyone been able to converge a TK run on a 32 bit Amazon Linux node?

amis.json out of date

Not sure when this happened, as I'm new, but noticed that the ubuntu-14.04 images were broken in all regions when searching on the official ami search. Confirmed that they aren't available here either: https://cloud-images.ubuntu.com/locator/ec2/

Kitchen attempts to log in before sshd is ready

This is on the plain vanilla CentOS 6.5 AMI from the marketplace. Kitchen fails to log in the first time i run 'kitchen converge', then the second time I run it succeeds. I think it should wait maybe 5-10 seconds after port 22 is open to allow the machine to become fully ready.

-----> Creating <frontend-2-centos-65>...
       Creating <>...
       If you are not using an account that qualifies under the AWS
       free-tier, you may be charged to run these suites. The charge
       should be minimal, but neither Test Kitchen nor its maintainers
       are responsible for your incurred costs.
       EC2 instance <i-99d44395> created.
.........       (server ready)
       Waiting for 54.149.191.16:22...
       Waiting for 54.149.191.16:22...
       (ssh ready)

       Finished creating <frontend-2-centos-65> (3m17.39s).
-----> Converging <frontend-2-centos-65>...
       Preparing files for transfer
       Resolving cookbook dependencies with Berkshelf 3.2.1...
       Removing non-cookbook files before transfer
>>>>>> Converge failed on instance <frontend-2-centos-65>.
>>>>>> Please see .kitchen/logs/frontend-2-centos-65.log for more details
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: Authentication failed for user [email protected]
>>>>>> ----------------------

Support for server.dns_name

Hi, there should be a switch option to use server.dns_name rather than server.public_ip_address. This is required when using AWS security group permissions within the same region where the CNAME needs to resolve to the internal, private IP.

#state[:hostname] = server.public_ip_address || server.private_ip_address
state[:hostname] = server.dns_name || server.private_ip_address

`kitchen login` fails if ssh_key is a relative path.

I'm not entirely sure where this bubbles up from (might be a test-kitchen issue), but kitchen login doesn't seem to like relative paths for ssh keys.

Full path:

- name: ec2-ubuntu-12.04
  driver_name: ec2
  driver_plugin: ec2
  driver_config:
    image_id: ami-ad3660c4 # 64-bit instance-store backed
    username: ubuntu
    ssh_key: ~/Users/markbate/.ec2/test-kitchen.pem

The following should work fine:

kitchen converge default-ec2-ubuntu-1204
kitchen login default-ec2-ubuntu-1204

If you set it a relative path:

- name: ec2-ubuntu-12.04
  driver_name: ec2
  driver_plugin: ec2
  driver_config:
    image_id: ami-ad3660c4 # 64-bit instance-store backed
    username: ubuntu
    ssh_key: ~/.ec2/test-kitchen.pem

The following will converge, but not login.

kitchen converge default-ec2-ubuntu-1204
kitchen login default-ec2-ubuntu-1204

Error message is:

Warning: Identity file ~/.ec2/test-kitchen.pem not accessible: No such file or directory.
Permission denied (publickey).

Tests should also work fine, it's just the login that doesn't expand the path.

Default to IAM Credentials if aws_access_key_id or aws_secret_access_key Not Provided

Currently, if you do not provide a value for aws_access_key_id, kitchen-ec2 will through an exception. I believe the better course of action here is to try and use any IAM credentials if available.

This would be useful for us in our CI environment which runs on EC2. It would allow us to remove any hard-coded AWS credentials from .kitchen.yml or the environment.

Can't find how to set EBS Volume Size

Hello, I can't find how to set EBS Volume size. It's always 1Gb

Here my conf

---
provisioner:
  name: chef_solo

driver:
  name: ec2
  aws_access_key_id: XXXXX
  aws_secret_access_key: YYYYYYY
  region: eu-west-1
  aws_ssh_key_id: GER
  availability_zone: eu-west-1a
  security_group_ids: ["sg-XXXXXX"]
  ssh_key: /root/XXXX.pem
  require_chef_omnibus: true

platforms:
  - name: debian6
    driver:
      image_id: ami-XXXXXX
      username: root
      ebs_volume_size: 40,
      ebs_delete_on_termination: 'true'
      ebs_device_name: '/dev/sda1'

suites:
  - name: default
    run_list:
      - recipe[ger::default]
    attributes:

AWS ENV vars not honored

It doesn't appear that AWS ENV vars are honored as described in README.
Even with the env vars set or by doing the following:

aws_access_key_id: <%= ENV['AWS_ACCESS_KEY'] %>
aws_secret_access_key: <%= ENV['AWS_SECRET_KEY'] %>
aws_ssh_key_id: <%= ENV['AWS_SSH_KEY_ID'] %>

I get this error:

-----> Starting Kitchen (v1.0.0.beta.3)
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::UserError
>>>>>> Message: Kitchen::Driver::Ec2<default-centos-64>#config[:aws_access_key_id] cannot be blank
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details

Why no releases since Feb?

Lots of useful new features in master right now but the last release was in Feb. What happened to continuous deployment? Any plans to cut a release in the near future?

Support HVM based virtualization

Support for HVM based virtualization for AWS EC2 would be really helpful.

Dead link in kitchen-ec2/README

Link to http://docs.kitchen-ci.org/drivers/usage at:

Please read the "Driver usage" page for more details.

is toast :(

Resolving dependencies on ec2 instead upload resolved cookbooks to ec2

... Because on travis for me upload take more than 10 minutes and travis terminate build

(Reading database ...        47505 files and directories currently installed.)
       Unpacking chef (from .../chef_11.10.4-1.ubuntu.12.04_amd64.deb) ...
       Setting up chef (11.10.4-1.ubuntu.12.04) ...
       Thank you for installing Chef!
       Processing triggers for initramfs-tools ...
       update-initramfs: Generating /boot/initrd.img-3.2.0-56-virtual
       Transfering files to <dev-ubuntu-1204>
No output has been received in the last 10 minutes, this potentially indicates a stalled build or something wrong with the build itself.
The build has been terminated

If I set log level to debug, then transfer complete successfully (because kitchen print each file that was transferred), but travis terminated with message like this:

The log length has exceeded the limit of 4 Megabytes (this usually means
that test suite is raising the same exception over and over).

I see two solution:

Resolving dependencies on ec2 instead upload resolved cookbooks to ec2. Because,
I can improve resolving/downloading speed on ec2 but not uploading speed on travis.
Print dots on in info/warn log level while files transferring.

Any ideas?

Slow file transference

Every time I do a converge, it takes a long time until all cookbooks are transferred. I think this is more of a kitchen + my internet connection issue but it shows up when using a remote driver like kitchen-ec2. As I could see, each file is copied one by one. Perhaps if everything was compressed in just one file we could improve on that. I'm not familiar with kitchen codebase, so I'd just like to know which component is exactly responsible for that, so I could try to work on it. Or could you suggest another way of solving that?

Can't get a public IP

I'm starting the instance while mentioning a subnet_id in my kitchen.yml file.
on AWS management console I see that the instance was initialized using the proper subnet, but it didn't received an external IP.

the driver is trying to login on the internal IP ( this can be considered as a bug? )
.......... (server ready)
Waiting for 10.0.0.64:22...
Waiting for 10.0.0.64:22...
Waiting for 10.0.0.64:22...
Waiting for 10.0.0.64:22...

Am I'm missing something?

Dynamically get info about all converged instances

Hello,

After converging appServer suite I need dynamically get network configuration in db suite to whitelist appServer IP address for my database. In dbServerRecipe recipe I have the following code to get appServer node configuration( this box is already converged):

appServerConfig = search('node',"recipes:appServer") # returns nothing
appServerConfig = search('node',"recipes:*") # returns only *db* config

But appServerConfig is NOT returning any configuration. I wonder what do I have to change in my logic to get this info dynamically?
At the same time I don't want to use preconfigured IP addresses in attributes to process this info.

I am sure it's not a bug - I just wondering if I can do it when use kitchen-ec2

suites:
  - name: appServer
    driver_config:
      image_id: amiName
      flavor_id: t2.micro
      tags: {
           "Name":"appServerName"
      }
    run_list:
      - recipe[cookbookName::appServerRecipe]
  - name: db
    driver_config:
      image_id: amiName
      flavor_id: t2.micro 
      tags: {
         "Name":"dbServerName"
      }
    run_list:
      - recipe[cookbookName::dbServerRecipe]

Default block_device_mapping fields

Currently if you want to overwrite the block_device_mappings, you are required to specify all 3 fields:

block_device_mappings:
  - ebs_device_name: /dev/sda1
    ebs_volume_size: 8
    ebs_delete_on_termination: true

We should default ebs_volume_size and ebs_delete_on_termination if they are not provided, so the following config would work:

block_device_mappings:
  - ebs_device_name: /dev/sda1
    # ebs_volume_size will be defaulted to 8
    # ebs_delete_on_termination will be defaulted to true

We should be able to specify tags for ec2 instances

This allows easy identification later.

I would imagine a driver_config tags: array to be the way to go.

kitchen-ec2 fails when setting associate_public_ip: false

Now that the test-kitchen windows-guest-support branch has been merged, I thought I'd take a look at updating the gems I'm using to try the latest (I have been successfully using a fork of the test-kitchen windows-guest-support branch and afiune's transport branch of kitchen-ec2).

First efforts have not been very promising :-(

It seems afiune's transport branch is no longer compatible with the master of test-kitchen, so I'm now trying the windows-guest-support branch of kitchen-ec2. The first problem I've hit is that we create our test-kitchen instances inside a VPC on a private subnet without any associated elastic ip address. So in my .kitchen.yml file I use:

interface: private
associate_public_ip: false

Unfortunately this doesnt work due to the following lines of code:
https://github.com/test-kitchen/kitchen-ec2/blob/windows-guest-support/lib/kitchen/driver/ec2.rb#L95-99

I haven't tried to fix this myself yet. Would it be desirable to do something like the following instead so that it takes into account the interface configuration setting ?

ready? && !hostname(server).nil? && hostname(server) != '0.0.0.0'

User Data content should be base64 encoded when passed to aws sdk

When using 0.9.0 and supplying a file to user_data the create action throws an error:

------Exception-------
Class: Kitchen::ActionFailed
Message: Failed to complete #create action: [Invalid BASE64 encoding of user data]
---Nested Exception---
Class: Aws::EC2::Errors::InvalidParameterValue
Message: Invalid BASE64 encoding of user data

If the user data file used has base64 encoded content then the create action is successful

The content of the user data file should be base64 encoded when being provided to aws sdk

The specified wait_for timeout (600 seconds) was exceeded

I get 'The specified wait_for timeout (600 seconds) was exceeded' as a reply when running kitchen test -c when trying to use spot instances.

It appears that the problem is that 10 minutes is often an aggressive target for spots. If possible, this should be a configurable value to adjust for a variety of end user experiences.

Configuration option for naming EC2 instances?

Looking through the configuration options in the README.md I don't see an option to name EC2 instances that are created with kitchen-ec2. Am I missing something here?

With multiple people on my team starting to use kitchen-ec2 the AWS console is getting cluttered with unnamed EC2 instances. I currently have kitchen-ec2 creating tags but at a glance in the AWS console these can not be seen.

Limited Permissions - Failed to complete #create action: [You are not authorized to perform this operation.]

In my environment we have limited access which is obtained through roles. I get the following when trying to converge.
I am able to use the kitchen-ec2 0.8.0 version

09:45:28 {master} /c/dev/aw-p11tf12pm$ kitchen converge -l debug
-----> Starting Kitchen (v1.4.0)
D      Berksfile found at c:/dev/aw-p11tf12pm/Berksfile, loading Berkshelf
D      Berkshelf 3.2.4 library loaded
-----> Creating <default-centos-65>...
       Creating <>...
If you are not using an account that qualifies under the AWS
free-tier, you may be charged to run these suites. The charge
should be minimal, but neither Test Kitchen nor its maintainers
are responsible for your incurred costs.

D      Creating EC2 Instance..
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: Failed to complete #create action: [You are not authorized to perform this operation.]
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration

D      ------Exception-------
D      Class: Kitchen::ActionFailed
D      Message: Failed to complete #create action: [You are not authorized to perform this operation.]
D      ---Nested Exception---
D      Class: Aws::EC2::Errors::UnauthorizedOperation
D      Message: You are not authorized to perform this operation.
D      ------Backtrace-------
D      c:/Users/tim/.chefdk/gem/ruby/2.1.0/gems/aws-sdk-core-2.0.42/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call'
D      c:/Users/tim/.chefdk/gem/ruby/2.1.0/gems/aws-sdk-core-2.0.42/lib/seahorse/client/plugins/param_conversion.rb:22:in `call'

D      c:/Users/tim/.chefdk/gem/ruby/2.1.0/gems/aws-sdk-core-2.0.42/lib/aws-sdk-core/plugins/response_paging.rb:10:in `call'
D      c:/Users/tim/.chefdk/gem/ruby/2.1.0/gems/aws-sdk-core-2.0.42/lib/seahorse/client/plugins/response_target.rb:18:in `call' D      c:/Users/tim/.chefdk/gem/ruby/2.1.0/gems/aws-sdk-core-2.0.42/lib/seahorse/client/request.rb:70:in `send_request'
D      c:/Users/tim/.chefdk/gem/ruby/2.1.0/gems/aws-sdk-core-2.0.42/lib/seahorse/client/base.rb:216:in `block (2 levels) in define_operation_methods'
D      c:/Users/tim/.chefdk/gem/ruby/2.1.0/gems/aws-sdk-resources-2.0.42/lib/aws-sdk-resources/request.rb:24:in `call'
D      c:/Users/tim/.chefdk/gem/ruby/2.1.0/gems/aws-sdk-resources-2.0.42/lib/aws-sdk-resources/operations.rb:41:in `call'
D      c:/Users/tim/.chefdk/gem/ruby/2.1.0/gems/aws-sdk-resources-2.0.42/lib/aws-sdk-resources/operations.rb:61:in `call'
D      c:/Users/tim/.chefdk/gem/ruby/2.1.0/gems/aws-sdk-resources-2.0.42/lib/aws-sdk-resources/resource.rb:147:in `load'
D      c:/Users/tim/.chefdk/gem/ruby/2.1.0/gems/aws-sdk-resources-2.0.42/lib/aws-sdk-resources/resource.rb:120:in `data'
D      c:/Users/tim/.chefdk/gem/ruby/2.1.0/gems/aws-sdk-resources-2.0.42/lib/aws-sdk-resources/resource.rb:223:in `block in add_data_attribute'
D      c:/Users/tim/.chefdk/gem/ruby/2.1.0/gems/kitchen-ec2-0.9.0/lib/kitchen/driver/aws/instance_generator.rb:117:in `debug_if_root_device'
D      c:/Users/tim/.chefdk/gem/ruby/2.1.0/gems/kitchen-ec2-0.9.0/lib/kitchen/driver/aws/instance_generator.rb:106:in `block_device_mappings'
D      c:/Users/tim/.chefdk/gem/ruby/2.1.0/gems/kitchen-ec2-0.9.0/lib/kitchen/driver/aws/instance_generator.rb:55:in `ec2_instance_data'
D      c:/Users/tim/.chefdk/gem/ruby/2.1.0/gems/kitchen-ec2-0.9.0/lib/kitchen/driver/ec2.rb:283:in `submit_server'
D      c:/Users/tim/.chefdk/gem/ruby/2.1.0/gems/kitchen-ec2-0.9.0/lib/kitchen/driver/ec2.rb:185:in `create'
D      c:/opscode/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.4.0/lib/kitchen/instance.rb:424:in `public_send'
D      c:/opscode/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.4.0/lib/kitchen/instance.rb:424:in `block in perform_action'

D      c:/opscode/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.4.0/lib/kitchen/instance.rb:488:in `call'
D      c:/opscode/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.4.0/lib/kitchen/instance.rb:488:in `synchronize_or_call'
D      c:/opscode/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.4.0/lib/kitchen/instance.rb:453:in `block in action'
D      c:/opscode/chefdk/embedded/lib/ruby/2.1.0/benchmark.rb:279:in `measure'
D      c:/opscode/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.4.0/lib/kitchen/instance.rb:452:in `action'
D      c:/opscode/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.4.0/lib/kitchen/instance.rb:424:in `perform_action'
D      c:/opscode/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.4.0/lib/kitchen/instance.rb:352:in `create_action'
D      c:/opscode/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.4.0/lib/kitchen/instance.rb:341:in `block in transition_to' D      c:/opscode/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.4.0/lib/kitchen/instance.rb:340:in `each'
D      c:/opscode/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.4.0/lib/kitchen/instance.rb:340:in `transition_to'
D      c:/opscode/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.4.0/lib/kitchen/instance.rb:138:in `converge'
D      c:/opscode/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.4.0/lib/kitchen/command.rb:176:in `public_send'
D      c:/opscode/chefdk/embedded/lib/ruby/gems/2.1.0/gems/test-kitchen-1.4.0/lib/kitchen/command.rb:176:in `block (2 levels) in run_action'
D      ----------------------

Support spot-instances

Periodic failures in kitchen-ec2

Periodically I get failures starting up EC2 machines, like this:

borkbork:~/Dropbox/devel/github/juliandunn/java (travis-ci-demo)$ kitchen test oracle-7-fedora-18
-----> Starting Kitchen (v1.0.0.alpha.7)
-----> Cleaning up any prior instances of <oracle-7-fedora-18>
-----> Destroying <oracle-7-fedora-18>
       Finished destroying <oracle-7-fedora-18> (0m0.00s).
-----> Testing <oracle-7-fedora-18>
-----> Creating <oracle-7-fedora-18>
Called 'load_file' without the :safe option -- defaulting to safe mode.
       EC2 instance <i-511f7733> created.
.............       (server ready)
..       (ssh ready)

       Finished creating <oracle-7-fedora-18> (0m57.97s).
-----> Converging <oracle-7-fedora-18>
>>>>>> Converge failed on instance <oracle-7-fedora-18>.
>>>>>> Please see .kitchen/logs/oracle-7-fedora-18.log for more details
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: ec2-user
>>>>>> ----------------------

However, the machine is actually created; if I immediately do "kitchen converge oracle-7-fedora-18", then kitchen successfully logs into the machine and starts converging.

Perhaps there's a race condition in here somewhere? Or kitchen is trying to connect to the SSH port even though it's really not quite ready?

ebs_delete_on_termination is not working

I've got ebs_delete_on_termination set to true, when I run "kitchen test", the instance gets created, the whole thing runs, and then the instance is destroyed and terminated.
The volume stays behind.

kitchen-ec2 - iam_profile_name fog not passing through

Attempting to generate an EC2 instance with a IAM role attached to it is not working. I am expected the ec2 instance to come up and have a IAM role attached to it (for auth and cli usage to happen without credentials).
Using the following versions of gems:
test-kitchen-1.3.1
fog-1.27.0
kitchen-ec2-0.8.0

I am setting the iam_profile_name (as well as trying iam_instance_profile_arn) and this is not passed to the server create.

kitchen file (redacted):

driver:
name: ec2
aws_access_key_id: <%= ENV['AWS_ACCESS_KEY'] %>
aws_secret_access_key: <%= ENV['AWS_SECRET_KEY'] %>
aws_ssh_key_id: <%= ENV['AWS_SSH_KEY_ID'] %>
ssh_key: <%= File.expand_path('~/.ssh/id_rsa') %>
username: ec2-user
iam_profile_name: ec2-role
iam_instance_profile_arn: <%= ENV['AWS_INTANCE_PROFILE_ARN'] %>
http_proxy: <%= ENV['AWS_HTTP_PROXY'] %>
https_proxy: <%= ENV['AWS_HTTPS_PROXY'] %>

provisioner:
name: chef_solo
chef_omnibus_url: http://www.getchef.com/chef/install.sh
require_chef_omnibus: true

platforms:

name: amazon
driver_pluing: ec2
driver_config:
image_id: <%= ENV['AWS_AMI_IMAGE'] %>
region: us-west-2
availability_zone: us-west-2a
security_group_ids: "[<%= ENV['AWS_SG_ID'] %>]"
vpc_mode: true
subnet_id: "<%= ENV['AWS_SUBNET_ID] %>"
vpc: "<%= ENV['AWS_SUBNET_ID] %>"
flavor_id: "m1.small"
tags:
Name: testing
created-by: test-kitchen

Debug output, for section as instance is create, no IAM information display in the output, as expected and the excon.request contains no iam information at all:

-----> Starting Kitchen (v1.3.1)
D Berksfile found at ./Berksfile, loading Berkshelf
D Berkshelf 3.2.3 library loaded
D Berksfile found at ./Berksfile, loading Berkshelf
D Berkshelf 3.2.3 previously loaded
-----> Cleaning up any prior instances of
-----> Destroying ...
Finished destroying (0m0.00s).
-----> Testing
-----> Creating ...
D ec2:region 'us-west-2'
D ec2:availability_zone 'us-west-2a'
D ec2:flavor_id 'm1.small'
D ec2:ebs_optimized 'false'
D ec2:image_id 'ami-XXXXXXXX'
D ec2:security_group_ids '["sg-XXXXXXXX"]'
D ec2:tags '{:Name=>"testing", :"created-by"=>"test-kitchen"}'
D ec2:key_name 'development-preprod'
D ec2:subnet_id 'subnet-XXXXXXXX'
excon.request
:body => "Action=RunInstances&EbsOptimized=false&ImageId=ami-XXXXXXXX&InstanceType=m1.small&KeyName=development-preprod&MaxCount=1&MinCount=1&Placement.AvailabilityZone=us-west-2a&SecurityGroupId.1=sg-XXXXXXXX&SubnetId=subnet-XXXXXXX&Version=2014-10-01"
:chunk_size => 1048576
:ciphers => "HIGH:!SSLv2:!aNULL:!eNULL:!3DES"
:connect_timeout => 60
:debug_request => true
:debug_response => true
:expects => 200
:headers => {
"Authorization" => "REDACTED"
"Content-Type" => "application/x-www-form-urlencoded"
"Host" => "ec2.us-west-2.amazonaws.com"
"User-Agent" => "fog/1.27.0 fog-core/1.28.0"
"x-amz-date" => "20150211T165725Z"
}
:host => "ec2.us-west-2.amazonaws.com"
:hostname => "ec2.us-west-2.amazonaws.com"
:idempotent => false
:instrumentor => Excon::StandardInstrumentor
:instrumentor_name => "excon"
:method => "POST"
:middlewares => [
Excon::Middleware::ResponseParser
Excon::Middleware::Expects
Excon::Middleware::Idempotent
Excon::Middleware::Instrumentor
Excon::Middleware::Mock
]
:mock => false
:nonblock => true
:omit_default_port => false
:path => "/"
:persistent => nil
:port => 443
:query => nil
:read_timeout => 60
:response_block => #<Proc:0x007fa9e9452be0@/opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/fog-xml-0.1.1/lib/fog/xml/sax_parser_connection.rb:31 (lambda)>
:retries_remaining => 4
:retry_limit => 4
:scheme => "https"
:ssl_verify_peer => true
:tcp_nodelay => false
:thread_safe_sockets => true
:uri_parser => URI
:versions => "excon/0.44.1 (x86_64-darwin12.0) ruby/2.1.4"
:write_timeout => 60

encrypted_data_bag_secret not found

I am using encrypted data bag secret in my recipe. So when I run it using this plugin its breaking due to following error

Errno::ENOENT

No such file or directory - file not found '/tmp/kitchen/encrypted_data_bag_secret'

Any thoughts on how to make it work ?

it always invokes tests as root

in my .kitchen.yml file I have

username: some-user-name

and I have a test

@test "whoami" {
  whoami > /tmp/1.txt
}

after this if I do kitchen login and cat /tmp/1.txt it prints root

when I do kitchen login it logs in as some-user-name

block_device_mappings setting is not optional

In the current HEAD the block_device_mappings is required. This was not the case before which I personally would have preferred.

Beside this I run into errors when I do not set all block_device_mapping configuration settings. In my humble opinion some default settings should be assumed for all entire block_device_mappings settings.

This configuration works:

        block_device_mappings:
            - ebs_device_name: /dev/sda1
              ebs_volume_size: 8
              ebs_delete_on_termination: true

No block_device_mappings et al:

% kitchen converge ec2
-----> Starting Kitchen (v1.3.1)
-----> Creating <default-ec2>...
       Creating <>...
       If you are not using an account that qualifies under the AWS
       free-tier, you may be charged to run these suites. The charge
       should be minimal, but neither Test Kitchen nor its maintainers
       are responsible for your incurred costs.
>>>>>> Create failed on instance <default-ec2>.
>>>>>> Please see .kitchen/logs/default-ec2.log for more details
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: InvalidBlockDeviceMapping => Missing device name

Only ebs_device_name:

        block_device_mappings:
            - ebs_device_name: /dev/sda1
#              ebs_volume_size: 8
#              ebs_delete_on_termination: true

leads to

% kitchen converge ec2
-----> Starting Kitchen (v1.3.1)
/Users/myuser/.chefdk/gem/ruby/2.1.0/gems/kitchen-ec2-0.8.1.dev/lib/kitchen/driver/ec2.rb:93:in `block (2 levels) in <class:Ec2>': Every :block_device_mapping must include the keys :ebs_volume_size, :ebs_delete_on_termination and :ebs_device_name (RuntimeError)
    from /Users/myuser/.chefdk/gem/ruby/2.1.0/gems/kitchen-ec2-0.8.1.dev/lib/kitchen/driver/ec2.rb:89:in `each'
[...]

No ebs_delete_on_termination:

        block_device_mappings:
            - ebs_device_name: /dev/sda1
              ebs_volume_size: 8
#              ebs_delete_on_termination: true

leads to

% kitchen converge ec2
-----> Starting Kitchen (v1.3.1)
/Users/myuser/.chefdk/gem/ruby/2.1.0/gems/kitchen-ec2-0.8.1.dev/lib/kitchen/driver/ec2.rb:93:in `block (2 levels) in <class:Ec2>': Every :block_device_mapping must include the keys :ebs_volume_size, :ebs_delete_on_termination and :ebs_device_name (RuntimeError)
    from /Users/myuser/.chefdk/gem/ruby/2.1.0/gems/kitchen-ec2-0.8.1.dev/lib/kitchen/driver/ec2.rb:89:in `each'
[...]

busser bats tests don't run

When I run kitchen test, my .bats files don't run.

My integration test directory is like

▾ test/
  ▾ integration/
    ▾ default/
      ▾ bats/
          verify_installed_packages.bats

My .kitchen.yml is like this:


---
driver_plugin: ec2
driver_config:
  require_chef_omnibus: true

platforms:
- name: centos-6.4
  driver_config:
...
    region: us-east-1
    availability_zone: us-east-1b
    flavor_id: t1.micro
    # username: root
    require_chef_omnibus: true
    groups: ssh-only

suites:
- name: default
  run_list: ["recipe[spark-base]"]
  attributes: {}

This is for test-kitchen (~> 1.0.0.beta.3)

Any ideas of why this might happen?

Failure to specify username leads to confusing error message

If you leave the username out from the ec2 driver configuration, the following non-helpful error message could result:

[SSH] connection failed, retrying in 1 seconds (#<Net::SSH::AuthenticationFailed: Authentication failed for user #<Proc:0x00000004302470@.../ec2.rb:54>@REDACTED>

The documentation states that the default login name is "root" but that doesn't appear to result here.

This is test-kitchen 0.8.0 BTW.

SSH private key config value needed

Otherwise we're implicitly depending on an SSH agent session to SSH into the instances.

Failing authentication for some larger instances.

I'm having what appears to be a timing issue with some larger EC2 instance types (that's the pattern I'm seeing anyway). I have some cases where I need larger instances (e.g. m3.large). Most of the time, when kitchen attempts to build them, an authentication failure happens after the "(ssh ready)" message:

-----> Starting Kitchen (v1.2.1)
-----> Cleaning up any prior instances of <foo-centos-65>
-----> Destroying <foo-centos-65>...
       Finished destroying <foo-centos-65> (0m0.00s).
-----> Testing <foo-centos-65>
-----> Creating <foo-centos-65>...
       EC2 instance <i-56fe0c5d> created.
..............       (server ready)
       Waiting for ec2-XXX-XXX-XXX-XXX.us-west-2.compute.amazonaws.com:22...
       Waiting for ec2-XXX-XXX-XXX-XXX.us-west-2.compute.amazonaws.com:22...
       (ssh ready)
       Finished creating <foo-centos-65> (1m49.42s).
-----> Converging <foo-centos-65>...
       Preparing files for transfer
       Resolving cookbook dependencies with Berkshelf 3.1.1...
       Removing non-cookbook files before transfer
-----> Destroying <foo-centos-65>...
       EC2 instance <i-56fe0c5d> destroyed.
       Finished destroying <foo-centos-65> (0m1.09s).
>>>>>> Converge failed on instance <foo-centos-65>.
>>>>>> Please see .kitchen/logs/foo-centos-65.log for more details
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: Authentication failed for user [email protected]
>>>>>> ----------------------

This is happening fairly consistently for me (at least with m3.large instances). If I edit the ec2 driver to add a 15 second delay (note: 15 seconds is an arbitrary number; I didn't try anything lower) like the following, everything seems to work better. I haven't had a failure with the delay added.

def create(state)
  return if state[:server_id]
  server = create_server
  state[:server_id] = server.id

  info("EC2 instance <#{state[:server_id]}> created.")
  server.wait_for { print '.'; ready? }
  print '(server ready)'
  state[:hostname] = hostname(server)
  wait_for_sshd(state[:hostname], config[:username])
  sleep 15 # <---- Added this delay
  print '(ssh ready)\n'
  debug("ec2:create '#{state[:hostname]}'")
rescue Fog::Errors::Error, Excon::Errors::Error => ex
  raise ActionFailed, ex.message
end

I'm not quite sure what the problem is, but it just looks like AWS is returning a "running" state that isn't quite ssh-able. I'm going to try to dig in a little bit further, but I figured I'd open up an issue in case someone else knows exactly how to fix this faster than I can take a look at it. :-)

release 0.8.0 doesn't properly honor instance_type

If you specify instance_type but not flavor_id in the ec2 driver configuration, the default instance type of m1.small will be used, even if the instance_type is different.

This bug is in 0.8.0.

iam_profile_name not being added to ec2

I am attempting to specify an AWS IAM role in my .kitchen.yml but when the EC2 instance is created the role it not attached to the instance which is being verified in the AWS console.

test-kitchen (1.2.1)
kitchen-ec2 (0.8.0)

Here is my .kitchen.yml


---
driver:
  name: ec2
  aws_access_key_id: A....A
  aws_secret_access_key: S....v
  aws_ssh_key_id: s...s
  ssh_key: ~/.ssh/s....pem
  region: us-west-2
  availability_zone: us-west-2b
  require_chef_omnibus: true
  subnet_id: subnet-2...c 
  security_group_ids: ["sg-09...c", "sg-b...dd", "sg-...ad7", "sg-...46c"]
  iam_profile_name: chef-manager
  tags:
    Name: marketplace-test-kitchen
    created-by: test-kitchen

platforms:
  - name: AWS
    driver:
      image_id: ami-5....5
      username: ec2-user

suites:
  - name: default
    run_list:
      - role[base]
      - marketplace-shopatron
      - newrelic
    attributes:

What am I missing here?

Might be leaving orphaned EBS volumes

I ended up with 51 unattached volumes after a day of testing.

Redundant parameters when using VPC

When Amazon VPC is set up, such parameters as region, availability_zone and security_group_ids are associated with subnet and I had to match them in .kitchen.yml as follows:

subnet_id: subnet-...
region: us-east-1
availability_zone: us-east-1a
security_group_ids: []

Would it be possible to change logic to the following: if subnet_id is provided the parameters region, availability_zone and security_group_ids are not allowed in configuration file and this info is pulled from Amazon via API.

This is the way vagrant-ec2 plugin works and it kind of makes sense.

fails to connect to ec2 if ip/host is not in known_hosts entry

it startsup ec2 instance when it attempts to ssh into it, it fails with

>>>>>> Converge failed on instance <default-centos-64>.
>>>>>> Please see .kitchen/logs/default-centos-64.log for more details 
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: Authentication failed for user root@someip

before it attempts if I manually install entry in known_hosts list it goes further

doesn't invoke test while using ec2 driver

This is my .kitchen.yml looks like, if I switch driver back to vagrant it is able to create virtual box instance and invoke the tests on it


---
driver:
name: ec2
aws_access_key_id: SOME_KEY
aws_secret_access_key: SOME_KEY
aws_ssh_key_id: id_rsa-aws
ssh_key: /path/to/jigaraws.pem
security_group_ids: ["SOME_GROUP"]
region: us-west-2
availability_zone: us-west-2c
require_chef_omnibus: true
subnet_id: SOME_SUBNET

provisioner:
  name: chef_solo

platforms:
  - name: centos-6.4

suites:
  - name: default
    run_list:
      - recipe[ci-somerecipe::default]
    attributes:

but if I use ec2 driver it just exists with exit code 0 in less than a second and with following output

-----> Starting Kitchen (v1.2.1)
-----> Cleaning up any prior instances of <default-centos-64>
-----> Destroying <default-centos-64>...
       [Dummy] Destroy on instance=#<Kitchen::Instance:0x000001018bf248> with state={}
       Finished destroying <default-centos-64> (0m0.00s).
-----> Testing <default-centos-64>
-----> Creating <default-centos-64>...
       [Dummy] Create on instance=#<Kitchen::Instance:0x000001018bf248> with state={:my_id=>"default-centos-64-1392858588"}
       Finished creating <default-centos-64> (0m0.00s).
-----> Converging <default-centos-64>...
       [Dummy] Converge on instance=#<Kitchen::Instance:0x000001018bf248> with state={:my_id=>"default-centos-64-1392858588", :last_action=>"create"}
       Finished converging <default-centos-64> (0m0.00s).
-----> Setting up <default-centos-64>...
       [Dummy] Setup on instance=#<Kitchen::Instance:0x000001018bf248> with state={:my_id=>"default-centos-64-1392858588", :last_action=>"converge"}
       Finished setting up <default-centos-64> (0m0.00s).
-----> Verifying <default-centos-64>...
       [Dummy] Verify on instance=#<Kitchen::Instance:0x000001018bf248> with state={:my_id=>"default-centos-64-1392858588", :last_action=>"setup"}
       Finished verifying <default-centos-64> (0m0.00s).
-----> Destroying <default-centos-64>...
       [Dummy] Destroy on instance=#<Kitchen::Instance:0x000001018bf248> with state={:my_id=>"default-centos-64-1392858588", :last_action=>"verify"}
       Finished destroying <default-centos-64> (0m0.00s).
       Finished testing <default-centos-64> (0m0.01s).
-----> Kitchen is finished. (0m0.80s)

Did I miss something in configuration ? I am expecting it to spin up an instance on ec2 and execute bats test against it

Use IAM role to authenticate with AWS

It would be nice to be able to have kitchen-ec2 create ec2 instances using permissions derived from an IAM role. This feature would be useful in situations where test-kitchen/kitchen-ec2 is being run on a jenkins CI server in EC2 which is a common setup since you cannot run vagrant on top of a hypervisor.

The underlying aws library fog already has this capability in the form of the use_iam_profile options attribute so it would just be a matter of passing along this flag as well as removing the need to make the aws access key and secret a required config.

Just wanted to see if anyone else thinks having the ability to use IAM roles to create EC2 test-kitchen instances would be useful.

how do you pass user data to the instance?

maybe i am in just too much of a hurry and i just don't see it.

The plugin does not create /etc/chef/ohai/hints/ec2.json file

It was stated in ohai discussion that reliably detecting cloud info is very difficult and it seems that common practice now create a file in /etc/chef/ohai/hints directory.

Could you please create an empty /etc/chef/ohai/hints/ec2.json file after chef client is installed?

Here are relevant discussions:

Unable to SSH into VPC Instance

I am attempting to create an instance inside of a VPC with the following configuration, however it just hangs attempting to connect and ultimately times out. While it's attempting to connect, in another terminal window I am able to manually SSH in without a problem.

What am I doing wrong?

---
driver_config:
  aws_access_key_id: <%= ENV['AWS_ACCESS_KEY_ID'] %>
  aws_secret_access_key: <%= ENV['AWS_SECRET_ACCESS_KEY'] %>
  aws_ssh_key_id: <%= ENV['AWS_KEYPAIR_NAME'] %>
  username: ec2-user

provisioner:
  name: chef_zero
  require_chef_omnibus: 11.16.4
  environments_path: "test/integration/default/environments"
  data_bags_path: "test/integration/default/data_bags"
  client_rb:
    environment: aws
    ssl_verify_mode: verify_peer

platforms:
- name: rhel-7.0
  driver_plugin: ec2
  driver_config:
    region: us-west-1
    availability_zone: us-west-1a
    security_group_ids: ["sg-XXXXXXXX"]
    subnet_id: subnet-XXXXXXXX
    associate_public_ip: false
    iam_profile_name: test-role
    image_id: ami-35cdd870
    flavor_id: c3.large

suites:
- name: default
  run_list: ["recipe[test]"]

tag_server: tag key needs to be cast to string

By default, the type of "k" is a symbol and the SDK requires it to be a string. I may be setting up my config incorrectly but if I cast k to a string, the SDK is happy and accepts it.

Here's my config option for tags:
tags:
"Name": "test-kitchen"

Here's what I did to fix the issue:

def tag_server(server)
  # tag assignation on the instance.
  config[:tags].each do |k, v|
    server.tag(*k.to_s*, :value => v)
  end
  server
end

Instance created but nothing happens from there

I'm trying to move from kitchen-vagrant to kitchen-ec2. In my test cookbook I have the following .kitchen.yml

---
driver:
  name: ec2
  aws_access_key_id: "xxxxxx"
  aws_secret_access_key: "xxxxx"
  aws_ssh_key_id: "xxxxx"
  ssh_key: <%= File.expand_path('~/.ssh/xxxx.pem') %>
  region: "us-east-1"
  security_group_ids: ["sg-xxxx", "sg-xxxx", "sg-xxxx", "sg-xxxx", "sg-xxxx"]
  require_chef_omnibus: true
  iam_profile_name: "test-kitchen"
  require_chef_omnibus: true
provisioner:
  name: chef_solo
platforms:
  - name: centos-6.4
    driver:
      image_id: "ami-xxxxxx"
      username: "cloud-user"
suites:
  - name: default
    run_list:
      - recipe[system::default]
      - recipe[affect_utils::base_linux_users]
      - recipe[system::default]
      - recipe[system::nodejs]
      - recipe[system::rvm_user]
      - recipe[system::sunjava6]
      - recipe[system::tomcat6]
      - recipe[minitest-handler::default]

    attributes:
<%= File.read("./attributes.yaml") %>

When I run kitchen-test, I get:

[banderson@banderson system ]$ kitchen test default
-----> Starting Kitchen (v1.2.1)
-----> Cleaning up any prior instances of <default-centos-64>
-----> Destroying <default-centos-64>...
       Finished destroying <default-centos-64> (0m0.00s).
-----> Testing <default-centos-64>
-----> Creating <default-centos-64>...
[fog][WARNING] Unable to load the 'unf' gem. Your AWS strings may not be properly encoded.
       EC2 instance <i-7628515c> created.
.........................       (server ready)

And then it just sits. I left it for 15 minutes and there was no further progress (side note: I have installed unf, yet that message persists)

Any ideas what might be wrong?

I've confirmed that the instance is created successfully, has the correct security groups, and can be connected to from the workstation that is running the kitchen test.

Error running kitchen-ec2 0.6.0

I'm using the release 0.6.0 version which depends on test-kitchen > 1.0.0.beta.1 and having some problems. When I run using 1.0.0.beta.1 I get the following error:

D      ------Exception-------
D      Class: Kitchen::ActionFailed
D      Message: Failed to complete #create action: [wrong number of arguments (1 for 2)]
D      ---Nested Exception---
D      Class: ArgumentError
D      Message: wrong number of arguments (1 for 2)
D      ------Backtrace-------
D      /Users/user/.rvm/gems/ruby-1.9.3-p374@p141/gems/test-kitchen-1.0.0.beta.1/lib/kitchen/ssh.rb:40:in `initialize'
D      /Users/user/.rvm/gems/ruby-1.9.3-p374@p141/gems/test-kitchen-1.0.0.beta.1/lib/kitchen/driver/ssh_base.rb:126:in `new'
D      /Users/user/.rvm/gems/ruby-1.9.3-p374@p141/gems/test-kitchen-1.0.0.beta.1/lib/kitchen/driver/ssh_base.rb:126:in `wait_for_sshd'
D      /Users/user/.rvm/gems/ruby-1.9.3-p374@p141/gems/kitchen-ec2-0.6.0/lib/kitchen/driver/ec2.rb:53:in `create'

When I use the latest test-kitchen, 1.0.0.beta.2 I get this error:

D      ------Exception-------
D      Class: Kitchen::ActionFailed
D      Message: Failed to complete #create action: [comparison of String with 0 failed]
D      ---Nested Exception---
D      Class: ArgumentError
D      Message: comparison of String with 0 failed
D      ------Backtrace-------
D      /Users/user/.rvm/rubies/ruby-1.9.3-p374/lib/ruby/1.9.1/logger.rb:365:in `<'
D      /Users/user/.rvm/rubies/ruby-1.9.3-p374/lib/ruby/1.9.1/logger.rb:365:in `add'
D      /Users/user/.rvm/gems/ruby-1.9.3-p374@p141/gems/test-kitchen-1.0.0.beta.2/lib/kitchen/ssh.rb:89:in `wait'
D      /Users/user/.rvm/gems/ruby-1.9.3-p374@p141/gems/test-kitchen-1.0.0.beta.2/lib/kitchen/driver/ssh_base.rb:126:in `wait_for_sshd'
D      /Users/user/.rvm/gems/ruby-1.9.3-p374@p141/gems/kitchen-ec2-0.6.0/lib/kitchen/driver/ec2.rb:53:in `create'

In both cases the ec2 instance gets created correctly and it begins to wait for ssh then dies. I'm new to test-kitchen so it might be something I'm doing wrong, I just can't figure out what.

Ohai attribute node[:ec2] is nil

I'm trying to use kitchen-ec2 as the driver for my test suites running on our CI server (Bamboo), and a few of my recipes rely on node[:ec2] not being nil to do certain tasks (attach EBS volumes, for example). When running Test-Kitchen with EC2 driver, node[:ec2] is not set, it seems.

Before bootstrapping the instance, kitchen-ec2 should touch /etc/chef/ohai/hints/ec2.json inside it to enable EC2 data collection by Ohai, per OHAI-267.

allow to hide aws keys from kitchen.yml

it allows to read it from environment variable, it would be nice if it allows to read it from encrypted databags, or some form of encrypted version