chef / cheffish Goto Github PK

View Code? Open in Web Editor NEW

39.0 64.0 29.0 977 KB

Resources and tools for testing and interacting with Chef and Chef Server.

License: Apache License 2.0

Ruby 99.74% Shell 0.26%

cheffish's Introduction

Cheffish

Umbrella Project: Chef Infra

Project State: Active

Issues Response Time Maximum: 14 days

Pull Request Response Time Maximum: 14 days

This library provides a variety of convergent resources for interacting with the Chef Server; along the way, it happens to provide some very useful and sophisticated ways of running Chef resources as recipes in RSpec examples.

This document may have errors, but it should have enough pointers to get you oriented.

There are essentially 3 collections here:

Resource/Provider Pairs for Manipulating Chef Servers

You'd use these in recipes/cookbooks. They are documented on the main Chef docs site.

Base/Helper Classes

To support the resource/provider pairs.

RSpec Support

Most of these RSpec...things were developed for testing the resource/provider pairs above; however, you can also require cheffish/rspec/chef_run_support for any RSpec expects you'd like, as we do for chef-provisioning and its drivers (especially chef-provisioning-aws).

The awesomeness here is that instead of instantiating a run_context and a node and a resource as Ruby objects, you can test your resources in an actual recipe:

when_the_chef_12_server "exists", organization: 'some-org', server_scope: :context, port: 8900..9000 do
  file "/tmp/something_important.json" do
    content "A resource in its native environment."
  end
end

An enclosing context that spins up chef-zero (local mode) Chef servers as dictated by server_scope. Chef::Config will be set up with the appropriate server URLs (see the with_* operators below).

server_scope:

:context
:example [default?]
?

port:

port number (8900 is the default)
port range (server will continue trying up this range until it finds a free port)

expect_recipe {
  # unquoted recipe DSL here.
}.to be_truthy    # or write your own matchers.

Converges the recipe using expect() (parentheses), which tests for a value and cannot be used with raise_error.

expect_converge {
  # unquoted recipe DSL here.
}.to raise_error(ArgumentException)

Converges the recipe using expect{ } (curly brackets), which wraps the block in a begin..rescue..end to detect when the block raises an exception; hence, this is only for raise_error.

The blocks for the following appear to be mostly optional: what they actually do is set the Chef::Config variable in the name to the given value, and if you provide a block, the change is scoped to that block. Probably this would be clearer if it were aliased to (and preferring) using rather than with.

with_chef_server(server_url, options = {}, &block)
with_chef_local_server(options, &block)
with_chef_environment(name, &block)
with_chef_data_bag_item_encryption(encryption_options, &block)
with_chef_data_bag(name)
- Takes a block, though this is not noted in the method signature.

get_private_key(name)

RSpec matchers

These are used with expect_recipe or expect_converge:

expect_recipe {
  file "/tmp/a_file.json" do
    content "Very important content."
  end
}.to be_idempotent.and emit_no_warnings_or_errors

be_idempotent

Runs the provided recipe again (expect_(recipe|converge) ran it the first time) and asks the Chef run if it updated anything (using updated?, which appears to be defined on Chef::Resource instead of Chef::Client, so there's some clarification to be done there); the matcher is satisfied if the answer is "no."

emit_no_warnings_or_errors

Greps the Chef client run's log output for WARN/ERROR lines; matcher is satisfied if there aren't any.

have_updated

Sifts the recipe's event stream(!) to determine if any resources were updated; matcher is satisfied is the answer is "yes."
This is not the opposite of be_idempotent.

partially_match

cheffish's People

Contributors

Stargazers

Watchers

cheffish's Issues

community cookbook download resource

`get_private_key_with_path` doesn't work well with all possible key names

If you have a private key with period characters in the name, you get down around here:

cheffish/lib/cheffish.rb

Line 103 in 087f683

if ext == '' || ext == '.pem'

and then get confused about what you're looking for.

Probably an elsif File.exists? key block down at the end would fix it.

chef_server_api does not correctly handle MergedConfig

53c8a24

This change causes a few issues:

1 - chef_server_api is expecting a MergedConfig object. When a MergedConfig object gets passed in, it dies on the merge! method because MergedConfig is not a hash. Like so:

NoMethodError
-------------
machine_batch[default] (/tmp/kitchen/cache/cookbooks/provisioner/providers/cluster.rb line 52) had an error:
NoMethodError: chef_client[QA1DLABTESTUBU01] (basic_chef_client::block line 132) had an error:
NoMethodError: Unexpected method merge! for MergedConfig with arguments [{:api_version=>"0"}]

2 - chef_server[:options] ||= {} is null coalescing to a hash, when it should be null coalescing to an empty MergedConfig.

As a result, since chef-provisioning has a dependency on Cheffish of cheffish ~> 1.1, this breaks a number of resources in the live version of chef-provisioning.

raw_json property of chef_data_bag_item is not used

Description

The raw_json property of the chef_data_bag_item resource is not used/respected when creating a data bag item

Chef Version

v16.11.7

Platform Version

Ubuntu-20.04

Replication Case

The following recipe can be used to replicate the issue.

my_data_bag_item = {}
my_data_bag_item['id'] = 'adam'
my_data_bag_item['real_name'] = 'Adam Brent Jacob'

chef_data_bag_item 'my_data_bag/adam' do
  raw_json my_data_bag_item
  action :create
  compile_time true
end

example = data_bag_item('my_data_bag', 'adam')

log example.inspect

Client Output

As you can see, the data bag only contains an id which itself is computed from the name of the resource. Nothing provided via raw_json gets added to the data bag.

-----> Starting Test Kitchen (v2.10.0)
-----> Converging <default-ubuntu-2004>...
       Preparing files for transfer
       Installing cookbooks for Policyfile /Users/gschofield/Workspace/progress/web_server/Policyfile.rb using `chef install`
       Installing cookbooks from lock
       Installing web_server 0.1.3
       Preparing dna.json
       Exporting cookbook dependencies from Policyfile /var/folders/s5/1w2wn2cs3v12x1pjv09gky8h0000gn/T/default-ubuntu-2004-sandbox-20210406-59850-mjt40t...
       Exported policy 'web_server' to /var/folders/s5/1w2wn2cs3v12x1pjv09gky8h0000gn/T/default-ubuntu-2004-sandbox-20210406-59850-mjt40t

       To converge this system with the exported policy, run:
         cd /var/folders/s5/1w2wn2cs3v12x1pjv09gky8h0000gn/T/default-ubuntu-2004-sandbox-20210406-59850-mjt40t
         chef-client -z
       Removing non-cookbook files before transfer
       Preparing data_bags
       Preparing validation.pem
       Preparing client.rb
       chef installation detected
       install_strategy set to 'once'
       Nothing to install
       Transferring files to <default-ubuntu-2004>
       Starting Chef Infra Client, version 16.11.7
       Patents: https://www.chef.io/patents
       Using policy 'web_server' at revision 'c66cf3e426227ae1bb20fef2f4dba2ab656d52aa2f0fa8d992c688120dc2df2c'
       resolving cookbooks for run list: ["web_server::[email protected] (58cfa44)"]
       Synchronizing Cookbooks:
         - web_server (0.1.3)
       Installing Cookbook Gems:
       Compiling Cookbooks...
       Recipe: web_server::default
         * chef_data_bag_item[my_data_bag/adam] action create
           - create data bag item adam at chefzero://localhost:1
         Converging 2 resources
         * chef_data_bag_item[my_data_bag/adam] action nothing (skipped due to action :nothing)
         * log[data_bag_item["my_data_bag", "adam", {"id"=>"adam"}]] action write

       Running handlers:
       Running handlers complete
       Chef Infra Client finished, 1/3 resources updated in 01 seconds
       Downloading files from <default-ubuntu-2004>
       Finished converging <default-ubuntu-2004> (0m5.76s).

Ruby version bump is completely arbitrary

In commit ca752a7 the required version of ruby was bumped to 2.3.3. Chef-zero bumped its ruby requirement to 2.3.1 in commit chef/chef-zero@1f3952e, I've opened chef/chef-zero#268 to try to get that reverted, but 2.3.3 isn't even in chefdk! I can (somewhat) understand requiring 2.3.1 since that's what's shipped with chefdk, but 2.3.3 doesn't make any sense. Can this please be reverted or at least adjusted? Thanks!

chef_acl provider updates the acls unnecessarily

All of our chef runs are updating ACLs every single time because the acl values are arrays and they are compared without sorting

Example chef run output -

update acl environments/environment at environments/environment
15:08:34.045 create: update groups from ["admins", "prod", "clients", "our"] to ["our", "admins", "prod", "clients"]�
15:08:34.045 �read: update groups from ["clients", "admins", "users", "our"] to ["our", "clients", "admins", "users"]�
15:08:34.046 �update, delete, grant: update groups from ["admins", "our"] to ["our", "admins"]�

Solution: Make the comparision insensitive to order

Deprecation warning on encrypt and encryption_version

Deprecated features used!
  encrypt nil currently does not overwrite the value of encrypt. This will change in Chef 13, and the value will be set to nil instead. Please change your code to explicitly accept nil using "property :encrypt, [MyType, nil]", or stop setting this value to nil. at 1 location:
    - /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/cheffish-1.5.0/lib/chef/provider/chef_data_bag_item.rb:115:in `load_current_resource'
  encryption_version nil currently does not overwrite the value of encryption_version. This will change in Chef 13, and the value will be set to nil instead. Please change your code to explicitly accept nil using "property :encryption_version, [MyType, nil]", or stop setting this value to nil. at 1 location:
    - /opt/chefdk/embedded/lib/ruby/gems/2.1.0/gems/cheffish-1.5.0/lib/chef/provider/chef_data_bag_item.rb:116:in `load_current_resource'

Just documenting a dep warning I'm seeing.
Versions can be seen in the output.

ffi-yajl version incompatibility with Chef 12.1.2

Chef 12.1 was a while ago now, so perhaps the solution here is to end support for it in Cheffish. (Or explore earlier versions of chef-zero, but that seems like a less good idea.)

Bundler could not find compatible versions for gem "ffi-yajl":
  In Gemfile:
    chef-zero (>= 0) ruby depends on
      ffi-yajl (~> 2.2) ruby

    chef (>= 0) ruby depends on
      ffi-yajl (~> 1.2) ruby

    chef-zero (>= 0) ruby depends on
      ffi-yajl (~> 2.2) ruby

chef_user isn't fully idempotent due to password attribute

The chef_user will not work unless I specify :password attribute. I'll get 400 bad request error if I try to comment attribute out.

Net::HTTPServerException
                                   ------------------------
                                   400 "Bad Request"

                                   Resource Declaration:
                                   ---------------------
                                   # In /var/chef/cache/cookbooks/bb_chef_server_cluster/recipes/manage_rbac.rb

                                    63:     chef_user user do
                                    64:       display_name user
                                    65:       email "#{user}@#{org}.com"
                                    66:       #password "pass0wrd"
                                    67:       source_key_path key_path
                                    68:       chef_server url_root
                                    69:       action :nothing
                                    70:       subscribes :create, 'ruby_block[only modify rbac content on master backend]', :immediately
                                    71:     end

However even though the user is already created it keeps trying to update password in sequential chef-client runs. I can not get this resource to skip and report (update to date).

* chef_user[poliva] action create
 - update user poliva at https://127.0.0.1/users
 -   update password from nil to "passw0rd"
* chef_user[anbrown] action create
 - update user anbrown at https://127.0.0.1/users
 -   update password from nil to "passw0rd"
* chef_user[bbuczynski] action create
 - update user bbuczynski at https://127.0.0.1/users
 -   update password from nil to "passw0rd"
* chef_user[eyurchenko] action create
 - update user eyurchenko at https://127.0.0.1/users
 -   update password from nil to "passw0rd"

I noticed related comment here, so maybe known already, but didn't see any issue reported on this.
https://github.com/chef/cheffish/blob/master/lib/chef/resource/chef_user.rb#L25

chef_user resource docs not clear for source_key related attributes

I found the documentation was unclear on how to set users key on :create action. The use in conjunction phrase is very misleading. After reviewing the code it seems you either specify source_key or source_key_pass_phrase and source_key_path. You can not use all three in combination as the documentation implies.

https://docs.chef.io/resource_chef_user.html

source_key Use to copy a public or private key, but apply a different format and password. Use in conjunction with source_key_pass_phrase` and source_key_path.

source_key_pass_phrase The pass phrase for the public key. Use in conjunction with source_key` and source_key_path.

source_key_path The path to the public key. Use in conjunction with source_key` and source_key_pass_phrase.

Chef::Config.ssl_verify_mode :verify_none doesn't get passed on the Cheffish::ServerApi

The documented workaround in #37 doesn't seem to get passed through the HTTP client.

I posted a wrapper cookbook against the chef-server cookbook to bootstrap users (and organizations eventually) in https://gist.github.com/aespinosa/62bf4801ac570ab76f47a644b793b194

I have done various ways of adding ssl_verify_mode and none seem to work. So far I have tried the following

put Chef::Config.ssl_verify_mode inside a ruby_block
put the whole chef_server parameter inside a lazy { Chef::Config.ssl_verify_mode ... } block

Need support for AWS tests

The expect_recipe matcher currently converges the recipe. The matchers inside the AWS integration tests run converge themselves. We need to expose a new matcher that basically calls expect( recipe {...}) without converging the recipe.

The be_idempotent matcher should check to see if the recipe that just ran had any failures. If it did, we shouldn't try to run it again - this will just cause the recipe to fail again and in the AWS case it will spin up more AWS resource. Inside be_idempotent it looks like we can check chef_run#converge_failed?

Finally, when running the AWS tests (even without expect_recipe) something is trying to converge a failing recipe an additional time. Adding a breakpoint here and you can see it getting called twice (if the recipe being converged raises an exception). A fix might be as simple as checking converge_failed? and not re-converge, but I would like to understand why the converge is being called a second time on a failure. Is it called a second time on success? Is it called a second time if using expect_recipe instead of expect(recipe {?

\cc @jkeiser @metadave

Newest version 2.0.1 do not parse the data_bag name correctly

Debugging a little bit I have discovered that the new_resource.data_bag has a wrong value. I am calling the resource:

     45: chef_data_bag_item 'keys/delivery_builder_keys' do
     46:   raw_data { DATA }
     47:   action :create
     48: end

And the result is this:

[5] pry(#<Chef::Resource::ChefDataBagItem action provider>)> Chef::Cheffish::VERSION
(pry):5: warning: toplevel constant Cheffish referenced by Chef::Cheffish
=> "2.0.1"
[6] pry(#<Chef::Resource::ChefDataBagItem action provider>)> new_resource.name
=> "keys/delivery_builder_keys"
[7] pry(#<Chef::Resource::ChefDataBagItem action provider>)> new_resource.data_bag
=> "k"
[8] pry(#<Chef::Resource::ChefDataBagItem action provider>)> new_resource.id
=> "delivery_builder_keys"

Dependabot can't parse your Gemfile

Dependabot couldn't parse the Gemfile found at /Gemfile.

The error Dependabot encountered was:

Dependabot only supports uninterpolated string arguments to eval_gemfile. Got `__FILE__ + ".local"`

chef_user resource does not provide display_name

The chef_user resource always fails when run against an Enterprise Chef server because it does not provide the display_name field in the API call, resulting in a 400 Bad Request from the server.

https://gist.github.com/charlesjohnson/38b20a18251b25a783b1

Display_name should be added to the resource and the provider.

Support groups and acls in all objects that support them

Break out the rspec helpers into a chef-rspec gem

Should probably live with chef/chef or follow the same release coordination as chef/chef and ohai will going forwards.

Having the rspec helpers in cheffish is weird as hell. Even the README.md makes it clear that its weird "here's a pile of resources... oh BTW here some 'bonus' rspec helpers...." WTF?

Now cheffish is pulled into chef/chef because of the dep that chef/chef's integration tests have on cheffish's rspec helpers which is another tangled nightmare of tightly-coupled dependencies.

And most likely since there's apparently not a lot of unit testing around the rspec helpers (at least I was able to break the tests that use them in chef/chef without breaking any spec tests in this repo) that suggests that the compatibility between the rspec helpers and old versions of chef is most likely a lie -- certain its untested and at any point may become a lie.

chef policy resource

Users are not added to new group until 2nd convergence of recipe

chef_group will create new group in your organization but the users do not appear to be added to group until 2nd chef-client run.

recipe snippet

url_with_org = {
    :chef_server_url => URI.join(chef_server_url, "/organizations/blackberry").to_s,
    :options => {
      :client_name => "pivotal",
      :signing_key_filename => "/etc/opscode/pivotal.pem" } }

chef_group 'ops' do
      users  ["ops-blackberry", "anbrown"]
      complete true
      chef_server url_with_org
      action :create
end

1st chef-client run

* chef_group[ops] action create
 - create group ops at https://127.0.0.1/organizations/blackberry
 -   update users from [] to ["ops-blackberry", "anbrown"]

knife group show ops -c ~/.chef/knife-chef12.rb  (via knife-acl)

actors:
annotated_groups:
clients:
groupname:        ops
groups:
name:             ops
orgname:          blackberry
users:

2nd chef-client run

* chef_group[ops] action create
 - update group ops at https://127.0.0.1/organizations/blackberry
 -   update users from [] to ["ops-blackberry", "anbrown"]
 -   remove orgname

knife group show ops -c ~/.chef/knife-chef12.rb (via knife-acl)

actors:
  anbrown
  ops-blackberry
annotated_groups:
clients:
groupname:        ops
groups:
name:             ops
orgname:          blackberry
users:
  anbrown
  ops-blackberry

Support policies

Support for policies or cookbooks is necessary to have a delightful cluster creation experience.

chef_organization isn't fully idempotent

Similar problem to #83 but this time with chef_organization lwrp.

On subsequent runs it keeps reporting its removing guid for this organization.

* chef_organization[blackberry] action create
    - update organization blackberry at https://127.0.0.1:443
    -   remove guid

chef_organization

QUESTION - Is there a way to use cheffish to create a node attribute for the chef_orgnization? I would like to query a nodes chef_organization in a multi-tenancy environment.

Remove 'attributes'

It's a little bit confusing that

attribute 'x', 'y'

does not modify other attributes,

but attributes 'x' => 'y'

removes all other attributes.

Create resources for managing Chef analytics resources

As a user, I'd like to be able to specify analytics rules and notifications using cheffish opposed to creating them through the UI and/or a knife command.

I was thinking this might look something like this
e.g.

chef_analytics_rule 'watch-admins-group' do
  priority 0
  message :run_converge
  rule <<-EOH
  when
     #blah
  then
     #blah
  EOH
  notify 'blah'
end

chef_group isn't fully idempotent

Similar problem to #83 but this time with chef_group lwrp.

On subsequent runs it keeps reporting its removing orgname for this group.

* chef_group[noc] action create
    - update group noc at https://127.0.0.1/organizations/blackberry
    -   remove orgname

chef_node creates node resource that cannot update itself

Is there any way to create the node with default permissions, similar to how it's created via bootstrap?

encrypted ssh keys are overwritten if a pass_phrase is not defined

If you attempt to use an existing encrypted ssh_key but do not provide a pass_phrase, the keyformatter can't read key and is 'rescued' by returning nil.

The private_key class receives a 'nil' key and generates then overwrites the existing key.

No way to install new cheffish version in single chef-client run once it got activated

I used chef-client 12.4.1 which has cheffish version 1.2. I hit issue #50 when trying to create organization with members.

I then used chef_gem at compile time to install version 1.5.0 (1.6.0 conflicts with chef-client 12.4.1) however I can not activate 1.5.0 since chef-client seems to have already activated 1.2. Only workaround (which kind of stinks) I have is to stop chef-client after 1.5.0 is installed and then trigger a new chef-client run and it seems to pick up latest version. Same result even if you have a chef_gem resource removing 1.2 (doesn't matter as its already activated) before installing 1.5.0. Is there anyway to get chef-client to deactivate cheffish 1.2 during chef-client run?

[poliva-chef-server-be-1a] Starting Chef Client, version 12.4.1
 resolving cookbooks for run list: ["bb_chef_server_cluster::backend", "bb_chef_server_cluster::manage_rbac"]
 Synchronizing Cookbooks:
   - bb_apt
   - bb_chef_server_cluster
   - bb_lvm
   - bb_nsupdate
   - apt
   - lvm
   - chef-ingredient
   - apt-chef
   - yum-chef
   - yum
 Compiling Cookbooks...
 Recipe: bb_chef_server_cluster::default
   * chef_gem[install cheffish version 1.5.0] action install
     - install version 1.5.0 of package cheffish
 ["1.5.0", "1.2"]

   ================================================================================
   Recipe Compile Error in /var/chef/cache/cookbooks/bb_chef_server_cluster/recipes/manage_rbac.rb
   ================================================================================

   Gem::LoadError
   --------------
   can't activate cheffish-1.5.0, already activated cheffish-1.2

   Cookbook Trace:
   ---------------
     /var/chef/cache/cookbooks/bb_chef_server_cluster/recipes/manage_rbac.rb:19:in `from_file'

   Relevant File Content:
   ----------------------
   /var/chef/cache/cookbooks/bb_chef_server_cluster/recipes/manage_rbac.rb:

    12:  output = `#{File.join(RbConfig::CONFIG['bindir'], 'gem')} list | grep cheffish`.slice(/\(.*\)/)
    13:  versions = output[1..-2].split(',').collect{|x| x.strip} unless output.nil?
    14:  versions ||= []
    15:  p versions
    16:
    17:  require 'English'
    18:  #require 'cheffish'
    19>> gem 'cheffish' , '=1.5.0'
    20:  require 'cheffish'
    21:
    22:  ruby_block "only modify rbac content on master backend" do
    23:    block do
    24:    end
    25:    only_if { `/usr/bin/chef-server-ctl ha-status` && $CHILD_STATUS.to_i.eql?(0) }
    26:    only_if { `/usr/bin/chef-server-ctl ha-status | /bin/grep 'cluster status = master'` && $CHILD_STATUS.to_i.eql?(0) }
    27:  end
    28:

chef_environment support to append to environments

It would be good to see chef_environment append to an environment so it is not just overwritten completely.

as in one recipe could in theory just add a specific context into the environment (for example in normal attributes)

then in another recipe to the same environment append a different context instead of overwriting all of normal attributes with what it decided.

Not sure if even feasible but never hurts to ask

There is no documentation / README

I have no idea what this really does. Please add a more descriptive README with examples and whatever else documentation might be helpful :)

Thanks!

Chef acl updates permissions when it doesn't need to

On chef-client re-runs I noticed sometimes chef-acl would report (up to date) but then other times it reported its updating even though no acls need to be changed. Seems like the order it finds out about current acls actually matters as the from and to lists below have same values but from list is unsorted whereas to list is sorted alphabetically.

 * chef_acl[containers/sandboxes] action create
 - update acl containers/sandboxes at containers/sandboxes
 -   create, update, delete:  update groups from ["dev", "admins"] to ["admins", "dev"]
 -   read:  update groups from ["clients", "users", "dev", "admins", "ops", "noc"] to ["admins", "clients", "dev", "noc", "ops", "users"]

Support acls, groups, containers, org membership and invites

This differs from #12 in that these are primary objects (chef_acl '...' do)

Ruby 2.7 deprecation warning

From our chef/chef spes:

/opt/chef/embedded/lib/ruby/gems/2.7.0/gems/cheffish-16.0.8/lib/cheffish/chef_run.rb:42: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call

Docs around expect_recipe() are probably wrong as of Chef 12.4

chef-boneyard/chef-provisioning-aws#250 and #52 indicate that the behavior of expect_recipe

may be different from what the README says, and
may differ between Chef 12.3 and 12.4.

Resolve Net::HTTPServerException deprecations

Our spec runs are full of Net::HTTPServerException deprecation warnings. Now that we only support Ruby 2.6+ we should udpate all these exceptions to avoid warnings.

Support policies

Add a resource for updating configured Chef Server

Since with_chef_server is a helper as opposed to resource, arguments passed to can only be evaluated in the compile phase. There are situations where the Chef Server ip/hostname won't be known until halfway through the CCR...for example you might be provisioning the new Chef Server at the beginning of the CCR. By adding a Chef resource that updates the Chef server config like with_chef_server we will get things like lazy attributes out of the box!

/cc @jkeiser @afiune

A better Readme

I dont think I fully understand how this gem fits in. Are there other chef technologies other than chef provisioning that use it? For example, I recently (and its not the first time) uncovered a snafu of my usage of the complete attribute on the machine resource. This is not an attribute that hangs directly off of machine and appears to be defined here.

Usually I look to the chef gem for core chef workings but this gem seems to sit in the middle and it would help to understand the intent behind this gem to better attack provisioning troubleshooting.

Thanks!

chef_user hits 400 errors whenever you try to update any user fields (email, display_name, etc.)

Discovered that when you try to update an email of a existing user the chef_user blows up with 400 "Bad Request" error. Tried other fields like display_name and hit similar results.

Discovered using combination of cheffish 2.0.2 and chef-client 12.5.1.

 * chef_user[admin-helm] action create

 ["update user admin-helm at https://127.0.0.1/users", "  update email from \"[email protected]\" to \"[email protected]\""]
 {"name"=>"admin-helm", "username"=>"admin-helm", "display_name"=>"Admin Shared User for HELM Org", "admin"=>false, "json_class"=>"Chef::WebUIUser", "chef_type"=>"webui_user", "salt"=>nil, "password"=>nil, "openid"=>nil, "email"=>"[email protected]", "public_key"=>"-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAqNKOarIC4dUS2SUdOuOZ\n0ACVepwMeoHuzQYdUtwVA1+fDXjLqfZy63b4KObCIPx8m4bdrWXw15Fr9eMrjw36\nBf4tPHxU4Jq+CivD1pW3QVAV4Z2cZ89mruOD4xwLfjycQ7/LWydhmGs59hC7l6Fk\nxxmDSoWVs3pmNs2Ug9ZRmN6OsAltb0JbMpLqFrleg2j4RalfLls24rOQqkKC5n8K\nAmef0zt3F5HaufxXbuBWBG1wmNCxAzQswTzpFUd1ZSSdXxPg3KOkNiZdY6GEt1AY\nJW4ReFWo5MISnv7JyNSEZWr0eGOJqzTEURGPrbJUWS6UbkioFdwLtBC+AsLk3Cvo\nBQIDAQAB\n-----END PUBLIC KEY-----\n"}


     ================================================================================
     Error executing action `create` on resource 'chef_user[admin-helm]'
     ================================================================================

     Net::HTTPServerException
     ------------------------
     400 "Bad Request"

     Resource Declaration:
     ---------------------
     # In /var/chef/cache/cookbooks/bb_chef_server_cluster/recipes/manage_rbac.rb

      92:   chef_user user do
      93:     display_name values['display_name']
      94:     email values['email']
      95:     password lazy { determine_user_password(chef_server_url, user, values['password']) }
      96:     source_key values['source_key']
      97:     chef_server url_root
      98:     action :create
      99:     not_if { values['source_key'].nil? }
     100:     only_if { node.run_state['check_ha_status'] }
     101:   end
     102: end

     Compiled Resource:
     ------------------
     # Declared in /var/chef/cache/cookbooks/bb_chef_server_cluster/recipes/manage_rbac.rb:92:in `block in from_file'

     chef_user("admin-helm") do
       action [:create]
       retries 0
       retry_delay 2
       default_guard_interpreter :default
       chef_server {:chef_server_url=>"https://127.0.0.1:443", :options=>{:client_name=>"pivotal", :signing_key_filename=>"/etc/opscode/pivotal.pem", :api_version=>"0"}}
       declared_type :chef_user
       cookbook_name "bb_chef_server_cluster"
       recipe_name "manage_rbac"
       display_name "Admin Shared User for HELM Org"
       email "[email protected]"
       password #<ChefCompat::CopiedFromChef::Chef::DelayedEvaluator:0x0000000584e770@/var/chef/cache/cookbooks/bb_chef_server_cluster/recipes/manage_rbac.rb:95>
       source_key "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCo0o5qsgLh1RLZJR0645nQAJV6nAx6ge7NBh1S3BUDX58NeMup9nLrdvgo5sIg/Hybht2tZfDXkWv14yuPDfoF/i08fFTgmr4KK8PWlbdBUBXhnZxnz2au44PjHAt+PJxDv8tbJ2GYazn2ELuXoWTHGYNKhZWzemY2zZSD1lGY3o6wCW1vQlsykuoWuV6DaPhFqV8uWzbis5CqQoLmfwoCZ5/TO3cXkdq5/Fdu4FYEbXCY0LEDNCzBPOkVR3VlJJ1fE+Dco6Q2Jl1joYS3UBglbhF4VajkwhKe/snI1IRlavR4Y4mrNMRREY+tslRZLpRuSKgV3Au0EL4CwuTcK+gF"
       not_if { #code block }
       only_if { #code block }
     end

Here is the complete stack trace:

Generated at 2016-05-12 14:01:32 +0000
Net::HTTPServerException: chef_user[admin-helm] (bb_chef_server_cluster::manage_rbac line 92) had an error: Net::HTTPServerException: 400 "Bad Request"
/opt/chef/embedded/lib/ruby/2.1.0/net/http/response.rb:119:in `error!'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/http.rb:145:in `request'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/http.rb:118:in `put'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/cheffish-2.0.2/lib/cheffish/chef_actor_base.rb:21:in `block in create_actor'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/mixin/why_run.rb:52:in `call'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/mixin/why_run.rb:52:in `add_action'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/provider.rb:175:in `converge_by'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/cheffish-2.0.2/lib/cheffish/chef_actor_base.rb:20:in `create_actor'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/cheffish-2.0.2/lib/chef/resource/chef_user.rb:42:in `block in <class:ChefUser>'
(eval):2:in `block in action_create'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/compat_resource-12.9.1/files/lib/chef_compat/copied_from_chef/chef/provider.rb:122:in `instance_eval'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/compat_resource-12.9.1/files/lib/chef_compat/copied_from_chef/chef/provider.rb:122:in `compile_and_converge_action'
(eval):2:in `action_create'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/provider.rb:144:in `run_action'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource.rb:585:in `run_action'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/runner.rb:49:in `run_action'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/runner.rb:81:in `block (2 levels) in converge'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/runner.rb:81:in `each'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/runner.rb:81:in `block in converge'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/resource_list.rb:83:in `block in execute_each_resource'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/stepable_iterator.rb:116:in `call'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/stepable_iterator.rb:116:in `call_iterator_block'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/stepable_iterator.rb:104:in `iterate'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/stepable_iterator.rb:55:in `each_with_index'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/resource_collection/resource_list.rb:81:in `execute_each_resource'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/runner.rb:80:in `converge'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/client.rb:653:in `block in converge'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/client.rb:648:in `catch'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/client.rb:648:in `converge'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/client.rb:687:in `converge_and_save'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/client.rb:269:in `run'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application.rb:270:in `block in fork_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application.rb:258:in `fork'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application.rb:258:in `fork_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application.rb:224:in `block in run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/local_mode.rb:44:in `with_server_connectivity'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application.rb:212:in `run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application/client.rb:408:in `block in interval_run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application/client.rb:398:in `loop'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application/client.rb:398:in `interval_run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application/client.rb:388:in `run_application'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/lib/chef/application.rb:60:in `run'
/opt/chef/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.1/bin/chef-client:26:in `<top (required)>'
/usr/bin/chef-client:54:in `load'
/usr/bin/chef-client:54:in `<main>'

Docs and/or gemspec should specify required Chef version

Via IRC, it looks like the minimum version is 11.12 since thats when chef/http/remote_request_id was added.

chef group resource/provider seems broken against chef 12 server

When creating groups the server api endpoint used is https://localhost/groups/admins but I keep getting back 404 errors.

[2015-02-23T17:18:30+00:00] DEBUG: Re-raising exception: Net::HTTPServerException - chef_group[admins](bb_chef12::_manage_rbac line 65) had an error: Net::HTTPServerException: 404 "Not Found"

I think the organization endpoints used is wrong according to the docs.

https://docs.chef.io/api_chef_server.html#groups
(Request) GET /organizations/NAME/groups

https://docs.chef.io/api_chef_server.html#groups-group-name
(Request) GET /organizations/NAME/groups/GROUP_NAME

I think the LWRP needs to take in an organization name to figure out the endpoint correctly.

Idempotency of machine resource is broken due to v1 vs v0 differences in clients API

Idempotency of machine resource in chef-provisioning is broken when working against Hosted Chef due to the fact that v1 API now does not return public_key but the state comparison in cheffish checks for it.

Repro Steps:

Have any chef-provisioning recipe with a machine resource
Run it once with success.
Run it second time and it will fail with Since Server API v1, all keys must be updated via the keys endpoint.

Details:
The logic here will break when running against v1 API because (pry debug values from that point):

[2] pry(#<Chef::Provider::ChefClient>)> a = new_json
=> {"name"=>"chef-server-backend-1",
 "clientname"=>"chef-server-backend-1",
 "admin"=>false,
 "validator"=>false,
 "chef_type"=>"client",
 "orgname"=>"chef-infra",
 "public_key"=>
  "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA00gA2sz6/Ki3geC99K06\nAoHMThQTlIoop5TJAnKdG9SUzl2ENT8ZD5cV85hiVNjTLQxcBDPBLxn9rMKjSXKF\nhuOkTvoy/PLTKcGKtfmspEv9VsqKqMlrj0TOeYGL02HsaqJ4hTrjpC7AMT7GWPhK\n0JbL8U3gYU3UltZ+gIdIrB3qzvJLXdWqcRR2nb2tEHnOooGXNmVUNr4OTmhsZ+Pk\nL1LqUkHtwh2k2aqRyccjcZkALTXitINLxixEHQzkDIzr7goADrK/sxCJG+s84Dv0\ngBMhF9eBIGacL8Y/Swsgysc7EGBoK1Bxl5BRRGGZcEh5jpag1HUSCd8vUz8bPjIT\nZwIDAQAB\n-----END PUBLIC KEY-----\n"}
[4] pry(#<Chef::Provider::ChefClient>)> b = current_json
=> {"name"=>"chef-server-backend-1",
 "clientname"=>"chef-server-backend-1",
 "admin"=>false,
 "validator"=>false,
 "chef_type"=>"client",
 "orgname"=>"chef-infra"}
[5] pry(#<Chef::Provider::ChefClient>)>

Possible Fix:
Talking to @stevendanna we need to either pin to v0 or update it's calls to be compatible with v1.

Update would probably be to check if the new_json has any public_key and if it does call the /keys endpoint to obtain the current keys and compare them with the ones in current_json and if there are any differences call the /keys endpoint to update them.

Since this is somewhat complicated pinning to v0 might be a temporary fix until we fix this the right hard way.

Confused about implications of using groups attribute in chef_group resource

It appears that its possible create a group and add other groups to this group. However I am confused about the acl implications. In spec example below does any user in group 'g' have all acl permissions set for group 'g' plus those also set for group 'x'? Which acl permissions would take precedence if group 'g' and 'x' rules are in conflict? Is it some kind of union? Sorry this probably isn't a cheffish related issue but I couldn't find these answers on chef docs (https://docs.chef.io/server_orgs.html) and thought someone here might know.

context 'and has a group named x' do
      group 'g', {}
      group 'g2', {}
      group 'g3', {}
      group 'g4', {}
      user 'u', {}
      user 'u2', {}
      user 'u3', {}
      user 'u4', {}
      client 'c', {}
      client 'c2', {}
      client 'c3', {}
      client 'c4', {}

      group 'x', {
        'users' => %w(u u2),
        'clients' => %w(c c2),
        'groups' => %w(g g2)
      }

      it 'Converging chef_group "x" adds new users' do
        expect_recipe {
          chef_group 'x' do
            users 'u3'
            clients 'c3'
            groups 'g3'
          end
        }.to have_updated('chef_group[x]', :create)
        expect(get('groups/x')).to eq({
          'name' => 'x',
          'groupname' => 'x',
          'orgname' => 'foo',
          'actors'  => %w(c c2 c3 u u2 u3),
          'groups'  => %w(g g2 g3),
          'users'   => %w(u u2 u3),
          'clients' => %w(c c2 c3)
        })
      end

document how to disable ssl verification

using with with chef12 in most cases runs into problems with ssl verification, is there anything I can pass into the opts in Cheffish::ServerApi that would mimic adding ssl_verify_mode :verify_none to my client.rb?

Update to use Chef Server API v1

See #57

We pinned to v0 and we should update everything to use v1

Wrong digitalocean_client_id when drivers is used in knife.rb

I'm not sure which gem's fault this is but story is I'm setting up cluster with chef-provisioning, my knife.rb is:

drivers({
  'fog:DigitalOcean:staging' => {
    :driver_options => {
      :compute_options => {
        :digitalocean_api_key => '<hash>',
        :digitalocean_client_id => '<hash>'
      }
    }
  }
})

I've also proper profile defined I'm running chef-client and I'm receiving '401 Unauthorized' from DO. chef-provisioning-fog is using Cheffish to get result config, extracts client_id and api_key.

result = Cheffish::MergedConfig.new(new_config, config, new_defaults)
...
id = result[:driver_options][:compute_options][:digitalocean_client_id]

When I'm running inspect on result[:driver_options] I'm getting:

#<Cheffish::MergedConfig:0x007fe81fc63bf8 @configs=[{:compute_options=>{:provider=>"DigitalOcean", :digitalocean_client_id=>"staging"}}, {:compute_options=>{:digitalocean_api_key=>"<hash>", :digitalocean_client_id=>"<hash>"}}, {:compute_options=>{}}], @merge_arrays={}>

my driver id is used as 'digitalocean_client_id'. It should be digital_client_id hash.

Add validator_client_key* attributes to chef_organization

It would be nice to add following attributes to chef_organization as currently you can't set private key a new organization uses when creating its related validator.pem.

new attributes

attribute :validator_client_key # String or OpenSSL::PKey::*
attribute :validator_client_key_path , :kind_of => String
attribute :validator_client_key_pass_phrase

example usage

chef_organization orgname do
  validator_client_key_path 
end

Multiple cookbook_path: broken with recent cheffish (~> 0.8)

Trying out cheffish 0.8, this bit of code is breaking in ec-metal: https://github.com/opscode/ec-metal/blob/master/cookbooks/ec-harness/recipes/ec2.rb#L22-L23

with_chef_local_server :chef_repo_path => repo_path,
  :cookbook_path => [ File.join(harness_dir, 'cookbooks'),
                      File.join(repo_path, 'cookbooks'),
    File.join(repo_path, 'vendor', 'cookbooks') ],

namely that clients cannot find any cookbooks from the second cookbook_path directory.
old behavior (0.7.x):

[2014-09-09T07:54:02-07:00] INFO: Starting chef-zero on port 9010 with repository at repository at /oc/src/ec-metal-aws-ubuntu
  One version per cookbook
  clients at /oc/src/ec-metal-aws-ubuntu/chef-repo/clients
  cookbooks at /oc/src/ec-metal-aws-ubuntu/cookbooks, /oc/src/ec-metal-aws-ubuntu/chef-repo/cookbooks, /oc/src/ec-metal-aws-ubuntu/chef-repo/vendor/cookbooks
  data_bags at /oc/src/ec-metal-aws-ubuntu/chef-repo/data_bags
  environments at /oc/src/ec-metal-aws-ubuntu/chef-repo/environments
  nodes at /oc/src/ec-metal-aws-ubuntu/chef-repo/nodes
  roles at /oc/src/ec-metal-aws-ubuntu/chef-repo/roles

new behavior (0.8.x):

Compiling Cookbooks...
[2014-09-08T17:00:47-07:00] INFO: Starting chef-zero on port 9010 with repository at repository at /oc/src/ec-metal-aws-ubuntu/chef-repo
  One version per cookbook