tobami / littlechef Goto Github PK

For some reason Ubuntu 10.04 install of Fabric is 0.9.0. Using pip to install Fabric got me 1.0. I assumed backward compatibility and went with Fabric 1.0. But found API change at fabric.contrib.files.append. Instead of (text, filename.. it is now (filename, text...

Robin reported a problem on the discussion group.

If the ssh/config contains something like:

Host h1
    HostName <some_ip>
    User root 

fabric will not correctly resolve the host, because it uses host_string which LittleChef sets to h1.

That can be easily solved with the following patch:
https://github.com/tobami/littlechef/blob/1.1.0/littlechef/lib.py#L436

+    if 'hostname' in credentials:
+        credentials['host_string'] = credentials['hostname'] 

The problem is that from then on LittleChef will think that the node is called "some_ip", and will save a node file called "some_ip.json".

A better solution needs to be found to deal with this issue

There is no need to have a tests.py module in the source dir. That causes overhead in the form of having to build a kitchen in the source dir, and then having to clean it up. The tests dir has already a kitchen structure, so the tests could be run there.

The structure would be:
tests/
test_lib.py
test_cook.py
my_kitchen/
cookbooks/
...

Since ssh-config defaults to ~/.ssh/config when un-configured in auth.cfg, should keypair-file also be optional?

It seems to me there should really be no reason to specify a ssh-config or keypair-file. If you want to use different ssh settings for a given host, then update your .ssh/config with the appropriate Host entry.

However, I am sure there are valid use cases for this implementation. If anything I feel it is silly for me to not define a ssh-config, yet have to define a keypair-file (which I already have defined in my default ssh config).

with running 'cook new_kitchen' several empty README files are created (e.g. in cooksbooks/, in nodes/ etc.)
Although they might be there to describe future content of these directories, it would be great if they could contain some basic documentation as a placeholder.

For example cookbooks/README could tell people how and where to get the default cookbooks, data_bags/README could point people to the online docs about data_bags, and so on.

Also the root of the kitchen could contain a Quick-start README
All of these files should make clear that heir content is just a placeholder, and should be replaced once cooking is in action.

Chef offers 4 Search Index Names, of which node and role are currently of interest to LittleChef users. The automatic node data bag provides search capabilities over all nodes. We should do the same with roles.

Example of cookbook using this functionality: nagios

To be able to check in an empty kitchen into git, directories need to have at least a place holder file

The node data bag merges attributes taking into account the Chef attribute precedence order.
However, there are special cases, specifically for attributes at the same precedence level.

One such case is:

• A cookbook's default attribute is set to an array of numbers
• A role defines that attribute using the default level

LittleChef currently substitutes the value defined in the cookbook for the value defined in the role.
However, the two values attributes should be deep merged (that is the second array appended), as explained here

yashh raised the issue of being able to specify an SSH key for authentication.

It should already work, as my servers usually disallow pasword authentication.
A password is still needed because chef is executed as sudo (and also for deploy_chef).

If there is a need to speficy a particular key name, it can be added.

Knife allows to do:
knife ssh "role:webserver" "sudo chef-client"

A LittleChef equivalent could be:
fix nodes_with_role:webserver

Normal configure runs could be refactored to be triggered by:
fix node:mynode

Currently for rpm based distros RHEL 5 repository is used

Installed littlechef via easy_install and fabric installs pycrypto 2.3. But when I run

$ cook -l
Traceback (most recent call last):
File "/usr/local/bin/cook", line 4, in
import pkg_resources
File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 2655, in
working_set.require(requires)
File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 648, in require
needed = self.resolve(parse_requirements(requirements))
File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 546, in resolve
raise DistributionNotFound(req)
pkg_resources.DistributionNotFound: pycrypto<2.1

I had to manually delete pycrypto from dist-packages and install 2.0.3 from pypi. Needs to work with latest

When using latests littlechef from the repository on a deployment which has been created with older releases of littlechef, cook failes to push the cookbooks using rsync. The problem seem to be some kind of "permission denied" errors.
(I will investigate this further, this is more like a placeholder for a more detailed bugreport)

On a minimal Ubuntu install, not even wget is installed out of the box. The first deploy_chef fails with:

Are you sure you want to install Chef at the node 10.0.1.62, using "lucid" packages? [Y/n] 
[10.0.1.62] out: /bin/bash: wget: command not found
[10.0.1.62] out: gpg: no valid OpenPGP data found.

Fatal error: sudo() encountered an error (return code 2) while executing 'wget -qO - http://apt.opscode.com/[email protected] | sudo apt-key add -'

Aborting.
Disconnecting from 10.0.1.62... done.

Chef has the concept of Environments [0]. It would be great if littlechef could 'patch' chef-solo in such way that

1.) node.chef_environment can return sth. useful (other than "_default")
2.) search() knows how to handle values returned by node.chef_environment

and for bonus points

3.) it's possible to set "default_attributes" and "override_attributes" for each environment

Implementing this feature request requires:

• adding a "environment ''" line to littlechef/solo.rb
• 'patching' chef (chef-solo) such that
  • Chef::Environment.load() returns the content of "./environments/.json"
  • Chef::Environment.default_attributes and Chef::Envoronment.override_attributes return the values defined in the .json file

[0] http://wiki.opscode.com/display/chef/Environments

I was deploying chef via littlechef to my existing linux VM which just so happened to have RVM already installed and sourced via the users .bash_profile.

littlechef failed to deploy until i removed my call to "rvm use".

I'm not sure if it was merely rvm causing the issue or perhaps the "echo Using /usr/local/rvm/gems/ruby-1.9.2-p0" was confusing littlechef.

Just thought i'd report it in case anyone else encounters this bizarre issue.

For reference...

Result:

shinobi:chef jamesu$ cook node:192.168.1.78 deploy_chef

Are you sure you want to install Chef at the node 192.168.1.78, using "lucid" packages? [Y/n] Y
[192.168.1.78] out: Using /usr/local/rvm/gems/ruby-1.9.2-p0
[192.168.1.78] out: mv: cannot stat `opscode.list': No such file or directory

The .bash_profile:

source /usr/local/lib/rvm
rvm use 1.9.2

Users should be able to write their own commands, which may or may not use LittleChef functionality.

A good example is the task get_ips., which could be implemented as a plugin.

Proposal:

• plugin folder inside a kitchen
• command plugin:<plugin_name>

Currently for every configuration run, LittleChef builds cookbook and role dependencies, zips all needed cookbooks and roles to a file and copies it to the node.

Instead, rsync could be used to sync the whole cookbooks directory, which could be quicker.

Being a wrapper for fab forwarding --help to fab is almost fine, but

• Usage should not be "Usage: fab [options] [...]" but sth. like "Usage: fix [options] [...]
• for bonus points it would be great if cook specific commands, like listed in the README.md are shown

Wes Winham has proposed to implement support for multiple cookbook folders.

I have outlined how that could be done in a discussion thread

It appears littlechef doesn't respect ForwardAgent.

I have a chef cookbook which attempts to do a git checkout of a repository. Rather than having to generate a deploy key on the server, and adding that to github, you can enable ForwardAgent.

It seems this does not work. I can manually login to the server using ssh -A root@domain and git clone fine.
Setting ForwardAgent yes in my ~/.ssh/config doesn't have the same effect.

Any ideas?

Security fix: currently all kitchen information is left on the tmp directory of every node. The reason is speed: on the next run, rsync will only cope files that have changed.

Removing the whole kitchen would result in getting no benefits from rsync vs ssh push, and would be pretty slow for big cookbook repositories.

A compromise would be to only delete the really sensitive files: the automatic node data bag, and the node.json file. It is there where actual passwords can be found. Cookbooks shouldn't really hold "secrets".

Post-chef-solo run clean-up is the goal of this issue.

Without it, a minimal ubuntu installation won't pick up on the packages correctly:

cook node:192.168.121.137 deploy_chef
Are you sure you want to install Chef at the node 192.168.121.137, using "lucid" packages? [Y/n]
[192.168.121.137] out: Reading package lists... Done
[192.168.121.137] out: Building dependency tree
[192.168.121.137] out: Reading state information... Done
[192.168.121.137] out: Package wget is not available, but is referred to by anot[192.168.121.137] out:
[192.168.121.137] out: This may mean that the package is missing, has been obsol[192.168.121.137] out:
[192.168.121.137] out: is only available from another source
[192.168.121.137] out: E: Package wget has no installation candidate
[192.168.121.137] out:
Fatal error: sudo() encountered an error (return code 100) while executing 'apt-get --yes install wget'

Pull request #37 adds two new features:

1. Use setuptools.setup if it is available.
2. Add cook.cmd to scripts if littlechef is being installed on windows. The cook.cmd file just calls python.exe on the existing cook file, but has the effect of making all the cook commands just work.

To do:

• fix paths

Fairly new to Python. When I tried to use littlechef on Mac Lion, I got

ImportError: No module named paramiko.config

To fix it

pip install paramiko

What would be a good starting point to work on this?

Thanks.

N

I'd like to be able to import littlechef and call its commands from my own fabfiles (specifically the deploy_chef task. However, because littlechef calls _readconfig() (which checks sys.argv[3]) at runtime I have to fake sys.argv to make the import work.

I'd think a better place to do the bootstrapping would be in the cook binary; that'd allow littlechef itself to be used as a library.

When configuring many nodes execution time can be very long, as nodes will be configured sequentially. There should be an option to configure nodes in parallel, which would achieve a big speedup.

Fabric implements parallel execution of tasks in version 1.3.

Knife's ssh command has a concurrency option:
-C, --concurrency NUM The number of concurrent connections

LittleChef could mimic that syntax.

• fix -C 4 node:all
• fix --concurrency=4 node:all

both should configure all nodes using 4 parallel threads. There could be a default number so that
fix -C node:all works as well.

Add the possibility to configure all roles that have a particular chef_environment value.

For example:
fix nodes_in_env:<chef_environment>
or
fix env:staging nodes_with_role:<role>
or
fix env:staging node:all

with more standard syntax:
fix -e staging nodes_with_role:<role>

Hey so I just created a new EC2 instance with ubuntu 10.04 and ran deploy_chef:lucid.

Error #1:
Fatal error: sudo() encountered an error (return code 1) while executing 'rm /etc/apt/sources.list.d/opscode.list'

To by pass this I created a blank file on the server.

Error #2:
Traceback (most recent call last):
File "/Library/Python/2.6/site-packages/fabric/main.py", line 435, in main
commands[name](*args, **kwargs)
File "/Library/Python/2.6/site-packages/littlechef.py", line 109, in deploy_chef
_update_cookbooks()
TypeError: _update_cookbooks() takes exactly 1 argument (0 given)

I see that _update_cookbooks() is being called in line 113 without argument configfile.

$ cook list_recipes

Fatal error: Unable to find cookbook "README.md" with metadata.json

Aborting.

This error message doesn't explain why it's trying to find a "README.md" cookbook. As a grep -ra "README.md" * turns up no examples of this cookbook name in my kitchen, I have no idea what's causing this error or how to fix it.

There is no documentation or examples of what the files under roles/ and nodes/ should look like.

This might not be an issue and only my misunderstanding what should happen, but:

When I run fix node:domain.com recipe:vim the first time, the file nodes/domain.com.json is created with the content:

{
    "ipaddress": "10.192.221.110",
    "run_list": [
        "recipe[vim]"
    ]
}

When I run fix node:domain.com recipe:ruby_build thereafter, it installs ruby_build on the node but does not add it to the nodes run_list. When running all recipes with fix node:domain.com it then only runs the vim recipe again, since ruby_build is not in the run_list of the json file. Shouldn't other recipes be added to the run_list as well?

Thanks for updating littlechef so frequently and please excuse my many questions.

Robin

For testing purposes it is useful to be able to run fix with both a role and a recipe, like:
fix node:MYNODE role:SOME_ENVIRONMENT recipe:MYRECIPE
This way it's easy to test a recipe with all attributes and dependencies without needing to change or create a dedicated role.

I just tried to use littlechef to configure my local machine. But when running

fix node:localhost recipe:irssi

I encountered

Fatal error: sudo() encountered an error (return code 1) while executing 'chown root:root /etc/chef/solo.rb'

The command returns 1 because there is no group root, but only the group wheel.

When I ran fix get_ips from OS 10.7.2, the IPs detected had an extra '):' appended, e.g.

Saving node configuration to nodes/my.host.com.json...
Node my.host.com has IP 192.168.1.1):

This function sould probably use a regex to find the IP address in that first line of ping output.

In lib.py:89 you check if the recipes array is still empty after reading the cookbook metadata. If the cookbook has no recipes (for example, the openssl cookbook), littlechef seems to assume that there has been some error and stops the build.

I'm guessing it'd be better to do an explicit file existence check around line 65 then error out of none of the searched paths exist.

Not sure if there are many dependency-only cookbooks like openssl, but I'm sure others will hit this since mysql is a common cookbook to start with.

Once databag support is added to Chef Solo (0.10 release?) and to Little Chef, a proxy "node" databag could be created as follows:

• In addition to pushing needed cookbooks, roles and databags to the node, Little Chef could read all nodes/**.json files and aggregate them into a databags/node.json
• The new "node" databag would always get pushed to the nodes
• Because you will be able to search databags in Chef Solo (probably not in Chef 0.10!), this would now work in a recipe:
  search(:node, 'run_list:"recipe[foo::bar]"')

Hi Miquel,

installing chef on cenots dose not work, it seems that ELFF is not supported, i changed solo.py in this way

https://gist.github.com/1268251

the repo details are in here

http://rbel.frameos.org/, there is also a repo for centos 6

Cheers,
Cristi

When configuring larger systems with the help of littlechef you sometimes need to update a bunch of nodes. It would be great if the cook command could help with this taks by taking multible nodes, like

cook nodes:a.google.com,b.google.com,c.google.com configure

It would also be greate if the order in which the nodes get configured gets preserved.

(This is different to issue 45 which asks for configurations based on roles, but there won't be way to define the order in which the nodes get configured)

Tim Diggins has posted to the chef mailing list about a gist he wrote that converts ruby roles to json roles:
https://gist.github.com/834890

A possibility would be to use this rake task to be able to use roles (and metadata.rb?) written in ruby.

It would be great if shell completion work on the cook command.
This completion should not only work for commands like "list_nodes" etc. but also for recipe and node arguments, so

fix node:srv0

does the right thing.

This way littlechef would work flawlessly with other VCS systems like bazaar, hg, git.

Hi, I tried setting a different keyfile in the .ssh/config with

Host ec2-107-20-130-69.compute-1.amazonaws.com
  IdentityFile $HOME/.ec2/me.pem

However, neither $HOME nor ~ are replaced correctly. It fails with the following error:

Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/fabric/main.py", line 682, in main
    *args, **kwargs
  File "/Library/Python/2.7/site-packages/fabric/tasks.py", line 232, in execute
    task.run(*args, **new_kwargs)
  File "/Library/Python/2.7/site-packages/fabric/tasks.py", line 105, in run
    return self.wrapped(*args, **kwargs)
  File "/Library/Python/2.7/site-packages/littlechef/runner.py", line 182, in recipe
    chef.sync_node(data)
  File "/Library/Python/2.7/site-packages/littlechef/chef.py", line 76, in sync_node
    solo.configure(current_node)
  File "/Library/Python/2.7/site-packages/littlechef/solo.py", line 59, in configure
    if not exists(cache_dir):
  File "/Library/Python/2.7/site-packages/fabric/contrib/files.py", line 35, in exists
    return not func(cmd).failed
  File "/Library/Python/2.7/site-packages/fabric/network.py", line 343, in host_prompting_wrapper
    return func(*args, **kwargs)
  File "/Library/Python/2.7/site-packages/fabric/operations.py", line 948, in run
    return _run_command(command, shell, pty, combine_stderr)
  File "/Library/Python/2.7/site-packages/fabric/operations.py", line 866, in _run_command
    stdout, stderr, status = _execute(default_channel(), wrapped_command, pty,
  File "/Library/Python/2.7/site-packages/fabric/state.py", line 326, in default_channel
    chan = connections[env.host_string].get_transport().open_session()
  File "/Library/Python/2.7/site-packages/fabric/network.py", line 74, in __getitem__
    self[real_key] = connect(user, host, port)
  File "/Library/Python/2.7/site-packages/fabric/network.py", line 199, in connect
    look_for_keys=not env.no_keys
  File "/Library/Python/2.7/site-packages/ssh/client.py", line 332, in connect
    self._auth(username, password, pkey, key_filenames, allow_agent, look_for_keys)
  File "/Library/Python/2.7/site-packages/ssh/client.py", line 436, in _auth
    key = pkey_class.from_private_key_file(key_filename, password)
  File "/Library/Python/2.7/site-packages/ssh/pkey.py", line 198, in from_private_key_file
    key = cls(filename=filename, password=password)
  File "/Library/Python/2.7/site-packages/ssh/rsakey.py", line 51, in __init__
    self._from_private_key_file(filename, password)
  File "/Library/Python/2.7/site-packages/ssh/rsakey.py", line 163, in _from_private_key_file
    data = self._read_private_key_file('RSA', filename, password)
  File "/Library/Python/2.7/site-packages/ssh/pkey.py", line 279, in _read_private_key_file
    f = open(filename, 'r')
IOError: [Errno 2] No such file or directory: '$HOME/.ec2/me.pem'

The file exists and everything works fine when supplying the absolute path.

Implement a new var BASEDIR in settings a la Django:
BASEDIR = os.path.abspath(os.path.dirname(__file__).replace('\\','/') )

cook would set it to cwd and all would be the same. The advantage would be as a library, where you could then set the settings.BASEDIR var once.

(lchef)ubuntu@ubuntu-testbox:/littlechef2$ cook node:xtradb deploy_chef
Are you sure you want to install Chef at the node xtradb, using "lucid" packages? [Y/n] y
[xtradb] out: Reading package lists... Done
[xtradb] out: Building dependency tree
[xtradb] out: Reading state information... Done
[xtradb] out: wget is already the newest version.
[xtradb] out: 0 upgraded, 0 newly installed, 0 to remove and 93 not upgraded.
[xtradb] out: [xtradb] out: OK
[xtradb] out: [xtradb] sudo: DEBIAN_FRONTEND=noninteractive apt-get --yes install chef
[xtradb] out: Reading package lists... Done
[xtradb] out: Building dependency tree
[xtradb] out: Reading state information... Done
[xtradb] out: chef is already the newest version.
[xtradb] out: 0 upgraded, 0 newly installed, 0 to remove and 93 not upgraded.
[xtradb] out:
[xtradb] out: Removing any system startup links for /etc/init.d/chef-client ...
[xtradb] out:
Done.
Disconnecting from xtradb... done.
(lchef)ubuntu@ubuntu-testbox:/littlechef2$ cook node:xtradb recipe:subversion

== Executing recipe 'subversion' on node xtradb ==

Fatal error: No config file found for node 'xtradb'

Aborting.

When trying to run the recipe python::pip on Debian Lenny I get the error
No such file or directory - /var/chef-solo/cache/distribute_setup.py

The reason is that Igal's security patches made LittleChef so amazingly secure, that not even chef-solo can create the cache directory, which is removed on every run.

I love the idea of little chef. Very cool!! Would be a nice addition to print out an error if a cookbook that I want to run uses databags. Some way to let the user know that the cookbook cannot be run with chef-solo/little chef.

Sometimes it is convenient to symlink a cookbook directory from elsewhere into the cookbooks directory in order to test a cookbook maintained elsewhere. fix does not appear to follow symlinks under the cookbooks directory, only actual directories.

This failed with 'Cookbook rsyslog not found':

ln -s /some/path/to/rsyslog $KITCHEN/cookbooks/rsyslog
fix node=something recipe:rsyslog

While this worked:

cp -r /some/path/to/rsyslog  $KITCHEN/cookbooks/rsyslog
fix node=something recipe:rsyslog

When I try to apply a role to a host, if the knife cookbook metadata command in lib.py fails with an 'ERROR:' (instead of 'FATAL:'), I get a false positive, e.g.:

% fix debug node:my.host.com role:logwatch
Setting Chef Solo log level to 'debug'...

== Applying role 'logwatch' to my.host.com ==
Generated metadata.json for cookbooks/logwatch

Fatal error: Cookbook "logwatch" has no metadata.json

Aborting.