Code Monkey home page Code Monkey logo

omnibus's Introduction

omnibus Cookbook

Cookbook Build Status

Prepares a machine to be an Omnibus builder.

This project is managed by the CHEF Release Engineering team. For more information on the Release Engineering team's contribution, triage, and release process, please consult the CHEF Release Engineering OSS Management Guide.

Announcement

Starting with omnibus cookbook version 5.0.0:

This cookbook requires Chef Infra Client 14 or later

Starting with omnibus cookbook version 4.0.0:

  • Chef 11 is no longer supported, since 12.5.1 is required to use chef-ingredient. If needed, you can pin to omnibus version 3.2.x to preserve the old functionality.
  • Instead of compiling everything from source in this cookbook, the omnibus-toolchain package will be installed. This package contains patch, bash, make, git, ruby, rubygems, and bundler (built from omnibus-software definitions).

Requirements

This cookbook requires Chef 14+

For a full list of supported platforms and external cookbook requirements, please see the metadata.rb file at the root of the cookbook.

Recipes

The default recipe is the main entrypoint for the cookbook and does the following:

  • Ensures all required Omnibus-related directories are created and owned by the build user.
  • Ensures a sane build tool-chain is installed and configured
  • Ensures the necessary tools to run an Omnibus project (ruby, git, etc) are installed (using the omnibus-toolchain package)
  • Includes a platform-specific recipe to apply additional tweaks as appropriate.

All other recipes should be treated as "private" and are not meant to be used individually. They only exist to support the default recipe.

Attributes

Attribute Default Description
build_user omnibus The user to execute Omnibus builds as
base_dir Windows: C:/omnibus-ruby *nix: /var/cache/omnibus The "base" directory where Omnibus will store its data.

Resources

omnibus_build

This resource is used to execute a build of an Omnibus project.

Attributes

Attribute Default Description
project_name The name of the Omnibus project to build
project_dir The directory to install Omnibus
install_dir /opt/<PROJECT> The installation of the project being built
base_dir Windows: C:/omnibus-ruby *nix: /var/cache/omnibus The base directory for Omnibus
log_level :internal Log level used during the build. Valid values include: :internal, :debug, :info, :warn, :error, :fatal
config_file <PROJECT_DIR>/omnibus.rb Omnibus configuration file used for the build.
config_overrides {} Overrides for one or more Omnibus config options
expire_cache false Indicates the Omnibus cache (including git cache) should be wiped before building.
build_user node['omnibus']['build_user'] The user to execute the Omnibus build as.
environment {} Environment variables to set in the underlying build process.
live_stream false Indicates output of build process should be logged to Chef event stream.

Example Usage

omnibus_build 'harmony' do
  project\_dir 'https://github.com/chef/omnibus-harmony.git'
  log_level :internal
  config_override(
    append_timestamp: true
  )
end

Usage

Include the omnibus::default recipe in your node's run list and override the cookbook's default attributes as desired. At the very least you will want to override node['omnibus']['install_dir'] to match the installation directory of your Omnibus project.

Using Test Kitchen with Docker

The following assumes you are on a macOS workstation and have installed and started Kitematic.

  1. Set environment variables to point kitchen-docker at your local Kitematic instance:

Bash

export DOCKER_HOST=tcp://192.168.99.100:2376
export DOCKER_CERT_PATH=$HOME/.docker/machine/certs
export DOCKER_TLS_VERIFY=1

Fish

set -gx DOCKER_HOST "tcp://192.168.99.100:2376"
set -gx DOCKER_CERT_PATH "$HOME/.docker/machine/certs"
set -gx DOCKER_TLS_VERIFY 1
  1. Run Test Kitchen with the provided .kitchen.docker.yml:
KITCHEN_LOCAL_YAML=.kitchen.docker.yml kitchen verify languages-ruby-ubuntu-1204

License & Authors

Copyright:: 2012-2017, Chef Software, Inc. (<[email protected]>)

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

omnibus's People

Contributors

adamhjk avatar atomic-penguin avatar b-dean avatar bearnard avatar btm avatar chef-delivery avatar cwjohnston avatar databus23 avatar iennae avatar jaym avatar jaymalasinha avatar jkeiser avatar kwilczynski avatar lamont-granquist avatar legal90 avatar markan avatar mwrock avatar oferrigni avatar petecheslock avatar rhass avatar robbkidd avatar schisamo avatar scotthain avatar sersut avatar sethvargo avatar tas50 avatar thommay avatar tylercloke avatar xorimabot avatar yzl avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

omnibus's Issues

omnibus_build: can't modify frozen Hash

Cookbook version

5.2.0

Chef-client version

13.1.31

Platform Details

CentOS/RHEL 7.3

Scenario:

Converge the omnibus build node using the following recipe:

include_recipe 'omnibus::default'

omnibus_build 'myproj' do
  project_dir node['omnibus']['build_dir']
  log_level :internal
end

Actual Result:

Recipe: omnibus-myproj::build_package
  * omnibus_build[myproj] action execute
  Recipe: <Dynamically Defined Resource>
    * directory[/var/cache/omnibus/build/myproj/*.manifest] action delete (up to date)
    * directory[/var/cache/omnibus/pkg] action delete (up to date)
    * directory[/home/vagrant/omnibus-apica/pkg] action delete (up to date)
    * directory[/opt/myproj] action delete
      - delete existing directory /opt/myproj
    * directory[/var/cache/omnibus] action create (up to date)
    * directory[/opt/myproj] action create
      - create new directory /opt/myproj
      - change owner from '' to 'vagrant'
      - restore selinux security context

    ================================================================================
    Error executing action `execute` on resource 'omnibus_build[myproj]'
    ================================================================================

    RuntimeError
    ------------
    can't modify frozen Hash

    Cookbook Trace:
    ---------------
    /tmp/kitchen/cache/cookbooks/omnibus/libraries/omnibus_build.rb:153:in `environment'
    /tmp/kitchen/cache/cookbooks/omnibus/libraries/omnibus_build.rb:144:in `execute_with_omnibus_toolchain'
    /tmp/kitchen/cache/cookbooks/omnibus/libraries/omnibus_build.rb:77:in `block (2 levels) in <class:OmnibusBuild>'
    /tmp/kitchen/cache/cookbooks/omnibus/libraries/omnibus_build.rb:74:in `block in <class:OmnibusBuild>'

    Resource Declaration:
    ---------------------
    # In /tmp/kitchen/cache/cookbooks/omnibus-myproj/recipes/build_package.rb

      7: omnibus_build 'myproj' do
      8:   project_dir node['omnibus']['build_dir']
      9:   log_level :internal
     10: end

    Compiled Resource:
    ------------------
    # Declared in /tmp/kitchen/cache/cookbooks/omnibus-myproj/recipes/build_package.rb:7:in `from_file'

    omnibus_build("myproj") do
      action [:execute]
      default_guard_interpreter :default
      declared_type :omnibus_build
      cookbook_name "omnibus-myproj"
      recipe_name "build_package"
      project_dir "/home/vagrant/omnibus-apica"
      log_level :internal
      base_dir "/var/cache/omnibus"
      project_name "myproj"
      install_dir "/opt/myproj"
      config_file "/home/vagrant/omnibus-apica/omnibus.rb"
    end

    System Info:
    ------------
    chef_version=13.1.31
    platform=centos
    platform_version=7.3.1611
    ruby=ruby 2.4.1p111 (2017-03-22 revision 58053) [x86_64-linux]
    program_name=chef-client worker: ppid=10230;start=10:43:49;
    executable=/opt/chef/bin/chef-client

need apt cookbook

shouldn't this depend on the apt cookbook and include_recipe it on ubuntu so that we're up-to-date and can install software?

i'm running into issues with vagrant bento boxes having out of date apt-caches and blowing up trying to install libxml2-dev and such..

load-omnibus-toolchain.sh prints an error on Mac OS X

Cookbook version

5.3.3

Chef-client version

13.6.4

Platform Details

OS X 10.10 Yosemite

Scenario:

Converge a clean node with omnibus::default recipe.

Steps to Reproduce:

After the node is converged, log in there and try to initialize the environment:

source ~/load-omnibus-toolchain.sh

Actual Result:

========================================
= Environment
========================================

env: illegal option -- 0
usage: env [-i] [name=value ...] [utility [argument ...]]


========================================
= Tool Versions
========================================

Bash.........GNU bash, version 4.3.30(1)-release (x86_64-apple-darwin14.5.0)
Bundler......Bundler version 1.16.0
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
GCC..........Apple LLVM version 7.0.2 (clang-700.1.81)
Git..........git version 2.14.1
Make.........GNU Make 4.1
Ruby.........ruby 2.4.2p198 (2017-09-14 revision 59899) [x86_64-darwin14]
RubyGems.....2.6.14

Expected Result:

env is printed properly, no error message is displayed.

Description

This "error" is minor and actually doesn't affect the script behavior (except the printed output). But maybe it could be easily fixed just by adding an if condition to the script, so -0 will not be interpolated if the OS is Darwin? Here is how it will without -0:

$ source ~/load-omnibus-toolchain.sh

========================================
= Environment
========================================

SSL_CERT_FILE=/opt/omnibus-toolchain/embedded/ssl/certs/cacert.pem
TERM=xterm
SHELL=/opt/omnibus-toolchain/bin/bash
TMPDIR=/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/
SSH_CLIENT=10.211.55.2 56328 22
OMNIBUS_TOOLCHAIN_INSTALL_DIR=/opt/omnibus-toolchain
OLDPWD=/Users/vagrant
SSH_TTY=/dev/ttys000
USER=vagrant
MAIL=/var/mail/vagrant
PATH=/opt/omnibus-toolchain/bin:/usr/local/bin:/opt/omnibus-toolchain/bin:/usr/local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
PWD=/Users/vagrant/ast
SHLVL=1
HOME=/Users/vagrant
LOGNAME=vagrant
SSH_CONNECTION=10.211.55.2 56328 10.211.55.237 22
_=/usr/bin/env

<...>

source command bashism fails to load environment

The method execute_with_omnibus_toolchain attempts to load the environment, however fails as the execute resource invokes a bourne shell opposed to a bash shell.

https://github.com/chef-cookbooks/omnibus/blob/master/libraries/omnibus_build.rb#L136

The following scripts illustrates what is happening:

#!/usr/bin/env bash
# This file should be named /tmp/test-env.sh 
export FOO='bar'
#!/usr/bin/env sh

echo "Sourcing test environment file..."
source /tmp/test-env.sh

echo "FOO is set to: $FOO"

Running this results in the following:

# chmod +x /tmp/test.sh
/tmp/test.sh
Sourcing test environment file...
/tmp/test.sh: 4: /tmp/test.sh: source: not found
FOO is set to: 

However, when changing either the parent shell to bash or using . instead of source the error is resolved.

I have been working on a fix for this, however I am currently encountering a different error which is unclear to me at this time if it is caused by my fix or if there is another bug elsewhere which is now being exposed as result of my fix.

Require SSH

I'm not sure where this got changed, but since I updated to the new cookbook that uses omnibus-toolchain openssh-clients doesn't get installed which results in git clones not working.

The following shell command exited with status 128:

    $ git ls-remote "[email protected]:repo.git" "master*"

Output:

    (nothing)

Error:

    error: cannot run ssh: No such file or directory
fatal: unable to fork

Cookbook tries to install x86_64 toolchain package on i386 linux

Cookbook version

5.1.4

Chef-client version

12.19.36

Platform Details

Linux default-debian-wheezy-i386 4.4.0-66-generic #87-Ubuntu SMP Fri Mar 3 15:29:05 UTC 2017 i686 GNU/Linux
(Vagrant box using vagrant-lxc provider)

Scenario:

Using vagrant, berkshelf, and omnibus to build versions of a package in different environments. recipes/_omnibus_toolchain.rb seems to assume an x86_64 architecture unless you're on Windows.

Steps to Reproduce:

Vagrantfile:

  c.vm.define "debian-wheezy-i386" do |b|
    b.vm.box = "debian-wheezy-i386"
    b.vm.box_url = "file:///data/lxc-boxes/vagrant-lxc-debian-wheezy-i386.box"
    b.vm.hostname = "default-debian-wheezy-i386"
    b.vm.provision "chef_solo" do |chef|
      chef.add_recipe "apt::default"
      chef.add_recipe "omnibus::default"
      chef.add_recipe "cmake"
      chef.json = chef_json
    end
  end

Command: vagrant up debian-wheezy-i386

Expected Result:

Chef and the Omnibus toolchain are installed on the box.

Actual Result:

==> debian-wheezy-i386: Mixlib::ShellOut::ShellCommandFailed
==> debian-wheezy-i386:
==> debian-wheezy-i386: ------------------------------------
==> debian-wheezy-i386:
==> debian-wheezy-i386: Expected process to exit with [0], but received '1'
==> debian-wheezy-i386:
==> debian-wheezy-i386:       ---- Begin output of ["dpkg", "-i", "/var/chef/cache/omnibus-toolchain_1.1.61-1_amd64.deb"] ----
==> debian-wheezy-i386:
==> debian-wheezy-i386: STDOUT:
==> debian-wheezy-i386:
==> debian-wheezy-i386: STDERR: dpkg: error processing /var/chef/cache/omnibus-toolchain_1.1.61-1_amd64.deb (--install):
==> debian-wheezy-i386:
==> debian-wheezy-i386:  package architecture (amd64) does not match system (i386)
==> debian-wheezy-i386:       Errors were encountered while processing:
==> debian-wheezy-i386:
==> debian-wheezy-i386:
==> debian-wheezy-i386:  /var/chef/cache/omnibus-toolchain_1.1.61-1_amd64.deb
==> debian-wheezy-i386:
==> debian-wheezy-i386: ---- End output of ["dpkg", "-i", "/var/chef/cache/omnibus-toolchain_1.1.61-1_amd64.deb"] ----
==> debian-wheezy-i386:
==> debian-wheezy-i386: Ran ["dpkg", "-i", "/var/chef/cache/omnibus-toolchain_1.1.61-1_amd64.deb"] returned 1

Note that it is trying to install an amd64 package on an i386 container. I have a fix and will submit a pull request in a moment.

Unable to install ruby on Fedora

Fedora base images seem to have the same limitation as RHEL with regards to bzip not being installed. This causes the cookbook to fail on Fedora.

Install rsync for omnibus-ruby path_fetcher

In order for the path_fetcher of omnibus-ruby to work, we need to ensure that rsync is installed on the builder boxes. We've been lucky in the past by having rsync available on the bento boxes, but it appears that recent cleanup of the boxes has resulted in rsync getting removed and builds have started to fail during local testing.

Install of perl5 needed on freebsd

The version of the omnibus cookbook in the branch sethvargo/modernize, soon to be merged to master, requires that perl5 be installed on the node in advance. The omnibus cookbook does not currently manage this, but should.

The current behavior on the freebsd bento boxes is that perl5 is installed with pkg install as a dependency of the vmtools as part of building the boxes; however, neither

package 'perl5'

or

package 'perl5' do
  source 'ports'
end

detect that installation.

Ruby build from source

Cookbook seems to build ruby from source, takes a long time for new users.

How about we use the omnibus-rubies, or BrightBox UK's Ruby packages?

How do you guys solve this? pre-baked images?

PATH wrong on FreeBSD

Integration tests currently fail on FreeBSD:

Failures:

        1) $PATH Command "su - omnibus -c 'echo $PATH'" should return stdout /^\/usr\/local\/bin(.+)/
            Failure/Error: it { should return_stdout(%r{^/usr/local/bin(.+)}) }
      su - omnibus -c 'echo $PATH'
      /sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/omnibus/bin

      expected Command "su - omnibus -c 'echo $PATH'" to return stdout /^\/usr\/local\/bin(.+)/
            # /tmp/busser/suites/serverspec/default_spec.rb:55:in `block (3 levels) in <top (required)>'

      Finished in 0.17509 seconds
      11 examples, 1 failure

After some deeper digging it appears the generated .bashrc file is not being properly loaded.

Does this cookbook install the latest version of omnibus-ruby?

I'm just wondering if this cookbook will always try and install the latest version of the omnibus-ruby gem? Or each build of this cookbook is tied to a specific omnibus-ruby version?

I see Omnibus 3 came out last week (yay!) but using version 1.2.4 of this cookbook is giving me Omnibus 2.0.1. I'm just wondering if this is expected?

Ta

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.