Code Monkey home page Code Monkey logo

elasticsearch's Introduction

Elasticsearch Chef Cookbook

Cookbook Version

Please review the frequently asked questions and contributing guidelines before opening issues or submitting pull requests.

Looking for Elasticsearch 5.x or 6.x?

Please check out the previous 3.x.x releases of this cookbook. Please consider pinning your cookbook to '> 3.0' for support for Elasticsearch 6 and earlier, or '> 4.0' release for Elasticsearch 6 and beyond.

Resources

Notifications and Service Start/Restart

The resources provided in this cookbook do not automatically restart services when changes have occurred. They do start services by default when configuring a new service This has been done to protect you from accidental data loss and service outages, as nodes might restart simultaneously or may not restart at all when bad configuration values are supplied.

elasticsearch_service has a special service_actions parameter you can use to specify what state the underlying service should be in on each chef run (defaults to :enabled and :started). It will also pass through all of the standard service resource actions to the underlying service resource if you wish to notify it.

You must supply your desired notifications when using each resource if you want Chef to automatically restart services. Again, we don't recommend this unless you know what you're doing.

Resource names

Many of the resources provided in this cookbook need to share configuration values. For example, the elasticsearch_service resource needs to know the path to the configuration file(s) generated by elasticsearch_configure and the path to the actual ES binary installed by elasticsearch_install. And they both need to know the appropriate system user and group defined by elasticsearch_user.

Search order: In order to make this easy, all resources in this cookbook use the following search order to locate resources that apply to the same overall Elasticsearch setup:

  1. Resources that share the same resource name
  2. Resources that share the same value for instance_name
  3. Resources named default or resources named elasticsearch
    • This fails if both default and elasticsearch resources exist

Examples of more complicated resource names are left to the reader, but here we present a typical example that should work in most cases:

elasticsearch_user 'elasticsearch'
elasticsearch_install 'elasticsearch'
elasticsearch_configure 'elasticsearch'
elasticsearch_service 'elasticsearch'
elasticsearch_plugin 'x-pack'

elasticsearch_user

Actions: :create, :remove

Creates a user and group on the system for use by elasticsearch. Here is an example with many of the default options and default values (all options except a resource name may be omitted).

Examples:

elasticsearch_user 'elasticsearch'
elasticsearch_user 'elasticsearch' do
  username 'elasticsearch'
  groupname 'elasticsearch'
  shell '/bin/bash'
  comment 'Elasticsearch User'

  action :create
end

elasticsearch_install

Actions: :install, :remove

Downloads the elasticsearch software, and unpacks it on the system. There are currently three ways to install -- 'repository' (the default), which creates an apt or yum repo and installs from there, 'package', which downloads the appropriate package from elasticsearch.org and uses the package manager to install it, and 'tarball' which downloads a tarball from elasticsearch.org and unpacks it. This resource also comes with a :remove action which will remove the package or directory elasticsearch was unpacked into.

You may always specify a download_url and/or download_checksum, and you may include %s which will be replaced by the version parameter you supply.

Please be sure to consult the above attribute section as that controls how Elasticsearch version, download URL and checksum are determined if you omit them.

NOTE: The :remove action has not been implemented yet. Pull requests are very much welcome & encouraged, if you'd like to see this feature.

Examples:

elasticsearch_install 'elasticsearch'
elasticsearch_install 'my_es_installation' do
  type 'package'
  version '7.8.0'
  action :install
end
elasticsearch_install 'my_es_installation' do
  type 'tarball'
  dir '/usr/local' # where to install

  download_url "https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.7.2.tar.gz"
  # sha256
  download_checksum "6f81935e270c403681e120ec4395c28b2ddc87e659ff7784608b86beb5223dd2"

  action :install
end
elasticsearch_install 'my_es_installation' do
  type 'tarball'
  version '7.8.0'
  action :install # could be :remove as well
end
elasticsearch_install 'my_es_installation' do
  type 'package' # type of install
  download_url "https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.7.2.deb"
  # sha256
  download_checksum "791fb9f2131be2cf8c1f86ca35e0b912d7155a53f89c2df67467ca2105e77ec2"
  instance_name 'elasticsearch'
  action :install # could be :remove as well
end

elasticsearch_configure

Actions: :manage, :remove

Configures an elasticsearch instance; creates directories for configuration, logs, and data. Writes files log4j2.properties, elasticsearch.in.sh and elasticsearch.yml.

The main attribute for this resource is configuration, which is a hash of any elasticsearch configuration directives. The other important attribute is default_configuration -- this contains the minimal set of required defaults.

Note that these are both not a Chef mash, everything must be in a single level of keys and values. Any settings you pass in configuration will be merged into (and potentially overwrite) any default settings.

See the examples, as well as the attributes in the resource file, for more.

Examples:

With all defaults -

elasticsearch_configure 'elasticsearch'

With mostly defaults -

elasticsearch_configure 'elasticsearch' do
    allocated_memory '512m'
    configuration ({
      'cluster.name' => 'escluster',
      'node.name' => 'node01',
      'http.port' => 9201
    })
end

Very complicated -

elasticsearch_configure 'my_elasticsearch' do
  # if you override one of these, you probably want to override all
  path_home     "/opt/elasticsearch"
  path_conf     "/etc/opt/elasticsearch"
  path_data     "/var/opt/elasticsearch"
  path_logs     "/var/log/elasticsearch"
  path_pid      "/var/run/elasticsearch"
  path_plugins  "/opt/elasticsearch/plugins"
  path_bin      "/opt/elasticsearch/bin"

  # override logging parameters
  cookbook_log4j2_properties "my_wrapper_cookbook"
  template_log4j2_properties "my_log4j2.properties.erb"

  logging({:"action" => 'INFO'})

  allocated_memory '123m'

  jvm_options %w(
                -XX:+UseParNewGC
                -XX:+UseConcMarkSweepGC
                -XX:CMSInitiatingOccupancyFraction=75
                -XX:+UseCMSInitiatingOccupancyOnly
                -XX:+HeapDumpOnOutOfMemoryError
                -XX:+PrintGCDetails
              )

  configuration ({
    'node.name' => 'crazy'
  })

  action :manage
end

elasticsearch_service

Actions: :configure, :remove

Writes out a system service configuration of the appropriate type, and enables it to start on boot. You can override almost all of the relevant settings in such a way that you may run multiple instances. Most settings will be taken from a matching elasticsearch_config resource in the collection.

elasticsearch_service 'elasticsearch'

If you'd like to skip init scripts and systemd scripts, simply pass nil for the template file (init_source or systemd_source) and this cookbook will entirely skip trying to setup those scripts. Combined with changing the default service actions, this will have the same effect as action :nothing.

elasticsearch_plugin

Actions: :install, :remove

Installs or removes a plugin to a given elasticsearch instance and plugin directory. Please note that there is currently no way to upgrade an existing plugin using commandline tools, so we haven't exposed that feature here either. Furthermore, there isn't a way to determine if a plugin is compatible with ES or even what version it is. So once we install a plugin to a directory, we generally assume that is the desired one and we don't touch it further.

See #264 for more info. NB: You may encounter issues on certain distros with NSS 3.16.1 and OpenJDK 7.x.

Officially supported or commercial plugins require just the plugin name:

elasticsearch_plugin 'analysis-icu' do
  action :install
end
elasticsearch_plugin 'shield' do
  action :install
end

Plugins from GitHub require a URL of 'username/repository' or 'username/repository/version':

elasticsearch_plugin 'kopf' do
  url 'lmenezes/elasticsearch-kopf'
  action :install
end

elasticsearch_plugin 'kopf' do
  url 'lmenezes/elasticsearch-kopf/1.5.7'
  action :install
end

Plugins from Maven Central or Sonatype require 'groupId/artifactId/version':

elasticsearch_plugin 'mapper-attachments' do
  url 'org.elasticsearch/elasticsearch-mapper-attachments/2.6.0'
  action :install
end

Plugins can be installed from a custom URL or file location as follows:

elasticsearch_plugin 'mapper-attachments' do
  url 'http://some.domain.name//my-plugin-1.0.0.zip'
  action :install
end

elasticsearch_plugin 'mapper-attachments' do
  url 'file:/path/to/my-plugin-1.0.0.zip'
  action :install
end

The plugin resource respects the https_proxy or http_proxy (non-SSL) Chef settings unless explicitly disabled using chef_proxy false:

elasticsearch_plugin 'kopf' do
  url 'lmenezes/elasticsearch-kopf'
  chef_proxy false
  action :install
end

To run multiple instances per machine, an explicit plugin_dir location has to be provided:

elasticsearch_plugin 'x-pack' do
  plugin_dir '/usr/share/elasticsearch_foo/plugins'
end

If for some reason, you want to name the resource something else, you may provide the true plugin name using the plugin_name parameter:

elasticsearch_plugin 'xyzzy' do
  plugin_name 'kopf'
  url 'lmenezes/elasticsearch-kopf'
  action :install
end

License

This software is licensed under the Apache 2 license, quoted below.

    Copyright (c) 2015 Elasticsearch <https://www.elastic.co/>

    Licensed under the Apache License, Version 2.0 (the "License");
    you may not use this file except in compliance with the License.
    You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing, software
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.

elasticsearch's People

Contributors

brianvans avatar bwvoss avatar damacus avatar edgarasg avatar eheydrick avatar ernestas-poskus avatar gregkare avatar hrak avatar idolgirev avatar janko avatar jescholl avatar jmlrt avatar jperville avatar jtwarren avatar jwieringa avatar karmi avatar kitchen-porter avatar martinb3 avatar michaelklishin avatar mitch-roblox avatar momokatte avatar nathwill avatar nickmarden avatar nwadams avatar renovate[bot] avatar scalp42 avatar spuder avatar st331h0rs3 avatar tas50 avatar tmortensen avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

elasticsearch's Issues

Elasticsearch doesn't start after run 'sudo chef-client' over knife ssh

Elasticsearch doesn't start only if it's started by 'knife ssh' with sudo from managment workstation at another machine. PID file is created, but elasticsearch is not started.

If I try run sudo chef-client locally, it works without problems. If chef-client is run by knife ssh as root without sudo, it's ok too. So problem is only if I use knife ssh with sudo.

Same problem is when I try run only elasticsearch with knife ssh 'sudo /etc/init.d/elasticsearch start'.

E.g. knife ssh "name:*" "sudo /etc/init.d/elasticsearch start" -x testuser

OS is debian 6.0

elasticsearch::ebs fails if apt package cache is out of date

Given the current most recent EC2 AMI of Ubuntu 12.04 in us-east-1 (ami-d0f89fb9, according to http://cloud-images.ubuntu.com/locator/ec2/), any attempt to include the elasticsearch::ebs recipe in a run list before an 'apt-get update' has been issued results in a run failure.

This issue is due to the package cache being out of date when we try and upgrade the 'build-essential' package as the package currently referenced in the apt package cache results in HTTP 403s.

An apt-get update before upgrading the dependant packages solves this, but because the packages are being upgraded at compile-time, even including the 'apt' cookbook is no use. The compile-time run of these package installs happen before the 'apt' cookbook gets a chance to refresh the package cache.

As a temporary fix I've got a separate recipe with

execute('compiletime-apt-get-update') { command 'apt-get update' and action :nothing }.run_action(:run)

... but this of course runs every chef-client run, even though it's not necessary, and I'd imagine there is probably a more graceful solution that I have not thought of.

You must set ES_CLASSPATH var

Everything with the cookbook seems to have run fine, except when I try to run ES it says "You must set the ES_CLASSPATH var". I see that this is defined in both /usr/local/elasticsearch/bin/elasticsearch.in.sh AND /usr/local/etc/elasticsearch/elasticsearch.en.sh but I don't know where it's looking for this variable in terms of actually starting ES. Any help would be much appreciated.

I should addendum that I followed Karel's instructions on from ES.org, but I got it working using Chef Server, instead of Chef solo. I don't know if there is anything that I did that could have effected the process, but from what I can tell, everything is installed properly (monit, nginx, etc), all the files look correct, the data directories look ok as well, so I'm a bit befuddled.

Document bare minimum configuration for default recipe

I would, perhaps unfairly, assume that the default recipe without any attribute configuration should install and configure elasticsearch as a system service. I've been trying to get this working on a Vagrant VM (precise64 box). The VM is provisioned without error but elasticsearch is not running, nor will it run. Attempts to start the service always fail without any decent error messages.

Judging from the Vagrant file provided by the cookbook, there is a lot of configuration going on in order to get elasticsearch running properly. However, I can't distill from this file what exactly is the bare minimum to get a VM up and running.

Centos 5 / RHEL 5 Support

Hi,

we use the rpm installation for CentOS 6 / RHEL6 and it works fine. On CentOS5 / RHEL5 the rpm installation fails. Is it possible to install elasticsearch on CentOS 5 / RHEL 5 with this cookbook or with manual rpm installation?

Thanks in advance and kind regards
Aimo

Setting a custom installation directory doesn't work

If you set a custom installation directory using the :dir attribute, the cookbook still installs ElasticSearch in /usr/local/. This is because the call to ark that downloads and installs ElasticSearch doesn't include the necessary parameters to actually place the unpacked files in a custom location.

The version of elasticsearch can only be set via elasticsearch/settings databag

While trying to set the (non-default) version via a chef environment [1] attribute it does not actually install that version. It will install the default version, unless you set the version via the elasticsearch/settings.

To reproduce:

  1. If using chef-server, navigate to the node, edit, attributes (bottom left), select elasticsearch, delete (red minus button), save node. (this is to get the attributes to a clean state)
  2. Delete the version from the elasticsearch/settings data bag (or the entire data bag).
  3. Set the version to a non-default value via an environment (a role would probably show the same behavior).
  4. Run chef-client (with the -l debug to see the version)- it will install the default version, not the version specified by the environment.

This was reproduced with the 0.20.0 version of the cookbook.

[1]
"elasticsearch" => {
"version" => "0.20.5"

Creating new ebs volume is taking forever

Hi,

I'm using v0.1.0, trying to add an EBS volume, this is my "data" data bag:

{
  "id": "data",
  "_default": {
    "devices" : {
      "/dev/sda2" : {
        "file_system"      : "ext3",
        "mount_options"    : "rw,user",
        "mount_path"       : "/usr/local/var/data/elasticsearch/disk1",
        "format_command"   : "mkfs.ext3",
        "fs_check_command" : "dumpe2fs",
        "ebs"            : {
          "size"                  : 10,
          "delete_on_termination" : false,
          "type"                  : "io1",
          "iops"                  : 100
        }
      }
    }
  }
}

I'm runnig a m1.small instance.

From the logs:

[2013-01-30T09:41:24+00:00] INFO: Processing chef_gem[fog] action install (elasticsearch::ebs line 13)
[2013-01-30T09:41:24+00:00] INFO: Processing ruby_block[Create EBS volume on /dev/sda2 (size: 10GB)] action create (elasticsearch::ebs line 14)
[2013-01-30T09:41:29+00:00] INFO: Attaching volume: vol-SOMEID

Then just . . . . . forever (1 hour+ and counting)

Is it the formatting that takes a long time? I've only specified a 10 GB volume. Is it because I'm using only a m1.small? Is something wrong in my settings, I can see that the format command is specified with -F in the vagrant file in the source code..

Thanks,
Erling

Broken attempted aws plugin installation by default

I'm not including the aws recipe, just the default and plugins recipes. The aws plugin is attempted to install by default, and fails installation (possibly due to github no longer serving their downloads service?). The install plugin block fires every chef run because this plugin is still not installed (though the chef run completes cleanly), and also triggers the notification to restart elasticsearch (restarting elasticsearch every chef run is the main issue I see with this).

I'd be fine if either, the plugin install block didn't trigger a restart if the installation failed, or the default settings actually successfully completed the aws plugin install that they attempt.

I believe the cause of this plugin being installed by default is that the attributes/aws.rb sets a version for it though chef will parse and evaluate all attributes files even if the elasticsearch::aws recipe isn't included. (And of course presence of the key in the plugins attributes triggers the install)

Installation fails: Error executing action `start` on resource 'service[elasticsearch]'

I'm trying to install the elasticsearch cookbook using Chef Solo (Ubuntu 12.04 LTS 64bit) and it is failing during the execution this resource declaration (in the default recipe):

  * service[elasticsearch] action start
================================================================================
Error executing action `start` on resource 'service[elasticsearch]'
================================================================================


Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '127'
---- Begin output of /etc/init.d/elasticsearch start ----
STDOUT: Starting elasticsearch...
STDERR: bash: /usr/local/bin/elasticsearch: No such file or directory
---- End output of /etc/init.d/elasticsearch start ----
Ran /etc/init.d/elasticsearch start returned 127


Resource Declaration:
---------------------
# In /home/vincenzo/chef-solo/cookbooks-1/elasticsearch/recipes/default.rb

 61: service "elasticsearch" do
 62:   supports :status => true, :restart => true
 63:   action [ :enable ]
 64: end
 65:

because the path to the binary is wrong.
I'm stuck. Any ideas?

Thanks,
Vincenzo

Monit doesn't start after machine reboot

It may be in your sight down the road but I just wanted to highlight the fact that monit doesn't come back up after a system stop/start on aws.

You may instead choose to start monit from init via /etc/inittab.
If the monit dรฆmon should stop due to an error, the init program
will re-spawn a new monit process. Init will also automatically
start monit at machine boot time. To learn more on how to set-up
monit to run from init, read the monit man file.

If I get to it before you do, I'll be sure to write back or if this is actually a bug, then well ... consider it filed :)

Configure unicast_hosts dynamically on non-AWS clusters via `search`

Current cookbook only addresses unicast configuration for systems running under AWS and uses some cleaver logic to populate unicast_hosts using the AWS API. Great solution but it unfortunately leaves those on non-AWS environments without multicast on their networks out in the cold. I have a commit that allows setting the unicast_hosts via an attribute but that sort of defeats the beauty of chef.

I'm opening this issue and would like some feedback from others to solve this intelligently.

My current kludge to fix this is here:
benattar@fbbb82c

Thoughts?

version bump the metadata

I had previously installed the released cookbook version 0.1.0 and tried to pull in this one as it stood from master. Because the version is 0.0.4 chef server doesn't recognize it as an update and prefers the "newer" one. Please just bump the version in master.

When updating versions, the wrong version can be installed unless you manually clear node attributes (chef server only)

When updating versions, the wrong version can be installed unless you manually clear node attributes (chef server only).

Scenario:
Running a chef client node installed elasticsearch 0.20.1, then update the elasticsearch/settings to version 0.20.4 (a non-default version), then run chef-client. The standard out log will appear to install 0.20.4, but in reality it downloads 0.20.1 (the original version) and places version 0.20.1 in /usr/local/elasticsearch-0.20.4.

This is occurring because the node attributes are stale from prior runs. Looking at the node attributes on the chef server, the "version" is updated as it should be, but "filename" and "download_url" are stale (from a prior configuration).

The work around is easy...manually delete the node/elasticsearch attributes prior to updating, but this should not be necessary.

Changing nofile attribute is not idempotent

Changing the nofile attribute leads to duplicate line in /etc/security/limits.conf such as:

elasticsearch     -    nofile    1024
elasticsearch     -    memlock   512
elasticsearch     -    nofile    20000
elasticsearch     -    memlock   512

Don't seem to be able to change the version

I'm trying to install version 0.20.6 (for logstash 1.13) with the 0.3.0 cookbook version and I'm trying to set the version via a wrapper cookbook (to allow for multiple different elasticsearch clusters, not just for logstash). However the version doesn't seem to take effect. The wrapper cookbook has:

node.override["elasticsearch"]["version"] = "0.20.6"

and the output is:

[2013-06-17T21:40:32+00:00] INFO: Processing ark[elasticsearch] action install (elasticsearch::default line 71)
[2013-06-17T21:40:32+00:00] DEBUG: DEBUG: release_basename is elasticsearch-0.90.1.tar.gz
[2013-06-17T21:40:32+00:00] DEBUG: DEBUG: file_extension is tar.gz
[2013-06-17T21:40:32+00:00] DEBUG: path is /usr/local/elasticsearch-0.20.6
[2013-06-17T21:40:32+00:00] DEBUG: DEBUG: new_resource.release_file
[2013-06-17T21:40:32+00:00] DEBUG: DEBUG: cmd: /bin/tar xzf /tmp/vagrant-chef-1/elasticsearch.tar.gz --strip-components=1
[2013-06-17T21:40:32+00:00] INFO: Processing directory[/usr/local/elasticsearch-0.20.6] action create (/tmp/vagrant-chef-1/chef-solo-1/cookbooks/ark/providers/default.rb line 37)
[2013-06-17T21:40:32+00:00] INFO: Processing remote_file[/tmp/vagrant-chef-1/elasticsearch.tar.gz] action create (/tmp/vagrant-chef-1/chef-solo-1/cookbooks/ark/providers/default.rb line 43)
[2013-06-17T21:40:32+00:00] DEBUG: remote_file[/tmp/vagrant-chef-1/elasticsearch.tar.gz] checking for changes
[2013-06-17T21:40:32+00:00] DEBUG: Sending HTTP Request via GET to download.elasticsearch.org:80/elasticsearch/elasticsearch/elasticsearch-0.90.1.tar.gz
[2013-06-17T21:40:32+00:00] DEBUG: Streaming download from http://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.1.tar.gz to tempfile /tmp/chef-rest20130617-22513-u49c2g
[2013-06-17T21:40:58+00:00] DEBUG: remote_file[/tmp/vagrant-chef-1/elasticsearch.tar.gz] checking for file existence of /tmp/vagrant-chef-1/elasticsearch.tar.gz
[2013-06-17T21:40:58+00:00] DEBUG: remote_file[/tmp/vagrant-chef-1/elasticsearch.tar.gz] file exists at /tmp/vagrant-chef-1/elasticsearch.tar.gz
[2013-06-17T21:40:58+00:00] DEBUG: remote_file[/tmp/vagrant-chef-1/elasticsearch.tar.gz] target checksum: 01e7df982569ff7f13008cbef361c52f3cd8a3b289423483be5251e7ed4f1064
[2013-06-17T21:40:58+00:00] DEBUG: remote_file[/tmp/vagrant-chef-1/elasticsearch.tar.gz] source checksum: 2a5d4390a24a7feff12a0f1ac0e75c53b333316c44cf2e783dad9d7d7b03fb5b
[2013-06-17T21:40:58+00:00] DEBUG: remote_file[/tmp/vagrant-chef-1/elasticsearch.tar.gz] checksum changed from 01e7df982569ff7f13008cbef361c52f3cd8a3b289423483be5251e7ed4f1064 to 2a5d4390a24a7feff12a0f1ac0e75c53b333316c44cf2e783dad9d7d7b03fb5b
[2013-06-17T21:40:58+00:00] INFO: remote_file[/tmp/vagrant-chef-1/elasticsearch.tar.gz] backed up to /var/chef/backup/tmp/vagrant-chef-1/elasticsearch.tar.gz.chef-20130617214058
[2013-06-17T21:40:58+00:00] INFO: remote_file[/tmp/vagrant-chef-1/elasticsearch.tar.gz] updated

so it looks like the version attribute is set correctly (as evident by the directory name), but the filename doesn't use the updated attribute.

Using setup with ELB

Hi Karel,

With http basic authentication, it is not possible to use ELB, it does not has an option to health check with http basic.

My solution is simply adding another location to elasticsearch_proxy.conf.erb

location /health {
return 200;
}

ELB checks /health and works as expected. Perhaps it can be solved with a better way, mine is just fast and simple.

Cannot find a resource for create_ebs on amazon version 2012.09

I've been trying to get this cookbook working on an Amazon Linux EC2 instance. I'm hitting an error that seems to indicate the files in the extensions folder aren't being loaded.

[2013-01-08T00:47:53+00:00] INFO: HTTP Request Returned 404 Not Found: Cannot load data bag elasticsearch item users

================================================================================
Recipe Compile Error in /var/chef/cache/cookbooks/elasticsearch/recipes/ebs.rb
================================================================================

NameError
---------
Cannot find a resource for create_ebs on amazon version 2012.09

Cookbook Trace:
---------------
  /var/chef/cache/cookbooks/elasticsearch/recipes/ebs.rb:12:in `block in from_file'
  /var/chef/cache/cookbooks/elasticsearch/recipes/ebs.rb:7:in `each'
  /var/chef/cache/cookbooks/elasticsearch/recipes/ebs.rb:7:in `from_file'

Relevant File Content:
----------------------
/var/chef/cache/cookbooks/elasticsearch/recipes/ebs.rb:

  5:  # See the `attributes/data` file for instructions.
  6:  #
  7:  node.elasticsearch[:data][:devices].
  8:    reject do |device, params|
  9:      params[:ebs].keys.empty?
 10:    end.
 11:    each do |device, params|
 12>>     create_ebs device, params
 13:  end
 14:  

[2013-01-08T00:47:53+00:00] ERROR: Running exception handlers
[2013-01-08T00:47:53+00:00] FATAL: Saving node information to /var/chef/cache/failed-run-data.json
[2013-01-08T00:47:53+00:00] ERROR: Exception handlers complete
[2013-01-08T00:47:53+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2013-01-08T00:47:53+00:00] FATAL: NameError: Cannot find a resource for create_ebs on amazon version 2012.09

When I look on my chef-server in the cookbook - it doesn't seem to show either of the files in elasticsearch/libraries/extensions/

The UI shows the libraries directory with a single file under it (extensions.rb). No sub-folder called "extensions" containing the 2 files. On my local disk the files are there. I've even removed the cookbook from the server and re-uploaded it to be sure.

I am using version 0.0.5 of the cookbook, version 1.10.0 of cloud aws plugin - and my runlist for the node is:

run_list(
  "recipe[elasticsearch]", 
  "recipe[elasticsearch::plugins]",
  "recipe[elasticsearch::ebs]",
  "recipe[elasticsearch::data]",
  "recipe[elasticsearch::aws]"
)

I had some problems getting the data.json data bag formatted correctly - but this is what I have (that seems to work, although I haven't had a successful run yet to know for sure)

{
  "id": "data",
  "_default" : {
      "devices" : {
        "/dev/sda2" : {
        "file_system"      : "ext3",
          "mount_options"    : "rw,user",
          "mount_path"       : "/usr/local/var/data/elasticsearch/disk1",
          "format_command"   : "mkfs.ext3",
          "fs_check_command" : "dumpe2fs",
          "ebs"            : {
            "region"                : "us-west-2", // Optional: instance region is used by default
            "size"                  : 100,         // In GB
            "delete_on_termination" : true,
            "type"                  : "io1",
            "iops"                  : 2000
          }
        }
      }
    }
}

Any ideas what the problem is?

Conflict with nginx cookbook

I'm using the opscode nginx cookbook and I'm not loading the elasticsearch nginx recipe. But just including the elasticsearch cookbook causes problems with the nginx cookbook. The problem is that in attributes/nginx.rb, it sets nginx user to 'nginx' which overrides the 'default' user of www-data set by the nginx cookbook. As the 'nginx' user doesn't exist and as I'm not loading the elasticsearch nginx recipe, that user does not get created.

Should the attributes be using 'set' instead of 'default'? I could not even set a normal attribute on the node, and had to create override attributes for nginx user/group to set it back to www-data. I don't know the best way to avoid conflict with multiple default attributes set across cookbooks. Would default_unless for the attributes make it not override the nginx cookbook defaults?

you must set ES_CLASSPATH

I've just build a new server with your elasticsearch recipe - when I go to start it it says you must set ES_CLASSPATH - did I miss a step out? I don't think I did...

Failing installation test on master

When I use vagrant to provision the precise64 VM, I get a failing installation test.

Failure:
test_0004_creates configuration files(recipe::elasticsearch::default::Installation) [./etc/vagrant-chef/chef-solo-2/cookbooks/ace-homebase/cookbooks/elasticsearch/tests/installation_test.rb:35]:
Expected file '/usr/local/etc/elasticsearch/elasticsearch.yml' to include the specified content

If I ssh to the VM, the file exists and includes every line specified by the next test as well. Is this a race condition? How would I go about debugging this?

elasticsearch::test doesn't work in ec2 with chef server

The elasticsearch::test doesn't work in the Amazon ec2 and chef server. I can upload the cookbook using
knife cookbook upload cookbook-elasticseach but when the recipe runs it doesn't find the tests:

[2012-12-12T17:38:59+00:00] DEBUG: Will run these tests: []

getting an error trying to install plugin

---- Begin output of "bash" "/tmp/chef-script20120720-17103-166556x-0" ----
STDOUT:
STDERR: /usr/local/bin/plugin: 31: exec: -Xmx64m: not found
---- End output of "bash" "/tmp/chef-script20120720-17103-166556x-0" ----

AJAX requests and nginx proxy

So I setup an Angular app leveraging elastic-service to try to interact with my ES server. Since I used your fantastic cookbook, it installed the nginx proxy to wall off ES from external requests without a username and PW. Upon trying to search using the elastic-service, it fails due to failed authorization because the AJAX request is external to the server.

Is there an easy way around this? I'm guessing what I need to do is setup IP restrictions, so that only certain IP's can POST/DEL/PUT and whitelist GET requests, and require authorization for any external POST/DEL/PUT requests.

Ubuntu Tests Failing

Running this sequence of commands (taken from the readme):

$ git clone https://github.com/karmi/cookbook-elasticsearch.git elasticsearch
$ cd elasticsearch
$ berks install --shims ./tmp/cookbooks
$ vagrant up lucid32

Fails this test:

# Running tests:

[omitted tests that pass]

  1) Failure:
test_0002_installs_elasticsearch_jar(recipe::elasticsearch::default::Installation) [/opt/vagrant_ruby/lib/ruby/gems/1.8/gems/minitest-chef-handler-0.6.0/lib/minitest-chef-handler/resources.rb:33]:
The file does not have the expected owner.
Expected: "elasticsearch"
  Actual: nil

12 tests, 16 assertions, 1 failures, 0 errors, 0 skips
[Sun, 29 Jul 2012 10:53:40 +0200] INFO: Report handlers complete
[Sun, 29 Jul 2012 10:53:40 +0200] ERROR: Running exception handlers
[Sun, 29 Jul 2012 10:53:40 +0200] ERROR: Exception handlers complete
[Sun, 29 Jul 2012 10:53:40 +0200] DEBUG: Re-raising exception: RuntimeError - MiniTest failed with 1 failure(s)
/opt/vagrant_ruby/lib/ruby/gems/1.8/gems/minitest-chef-handler-0.6.0/lib/minitest-chef-handler.rb:43:in `report'
  /opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-0.10.10/bin/../lib/chef/client.rb:104:in `call'
  /opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-0.10.10/bin/../lib/chef/client.rb:104:in `run_completed_successfully'
  /opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-0.10.10/bin/../lib/chef/client.rb:103:in `each'
  /opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-0.10.10/bin/../lib/chef/client.rb:103:in `run_completed_successfully'
  /opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-0.10.10/bin/../lib/chef/client.rb:168:in `run'
  /opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-0.10.10/bin/../lib/chef/application/solo.rb:207:in `run_application'
  /opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-0.10.10/bin/../lib/chef/application/solo.rb:195:in `loop'
  /opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-0.10.10/bin/../lib/chef/application/solo.rb:195:in `run_application'
  /opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-0.10.10/bin/../lib/chef/application.rb:70:in `run'
  /opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-0.10.10/bin/chef-solo:25
  /opt/vagrant_ruby/bin/chef-solo:19:in `load'
  /opt/vagrant_ruby/bin/chef-solo:19
[Sun, 29 Jul 2012 10:53:40 +0200] FATAL: Stacktrace dumped to /etc/vagrant-chef/chef-stacktrace.out
[Sun, 29 Jul 2012 10:53:40 +0200] DEBUG: RuntimeError: MiniTest failed with 1 failure(s)
/opt/vagrant_ruby/lib/ruby/gems/1.8/gems/minitest-chef-handler-0.6.0/lib/minitest-chef-handler.rb:43:in `report'
/opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-0.10.10/bin/../lib/chef/client.rb:104:in `call'
/opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-0.10.10/bin/../lib/chef/client.rb:104:in `run_completed_successfully'
/opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-0.10.10/bin/../lib/chef/client.rb:103:in `each'
/opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-0.10.10/bin/../lib/chef/client.rb:103:in `run_completed_successfully'
/opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-0.10.10/bin/../lib/chef/client.rb:168:in `run'
/opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-0.10.10/bin/../lib/chef/application/solo.rb:207:in `run_application'
/opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-0.10.10/bin/../lib/chef/application/solo.rb:195:in `loop'
/opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-0.10.10/bin/../lib/chef/application/solo.rb:195:in `run_application'
/opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-0.10.10/bin/../lib/chef/application.rb:70:in `run'
/opt/vagrant_ruby/lib/ruby/gems/1.8/gems/chef-0.10.10/bin/chef-solo:25
/opt/vagrant_ruby/bin/chef-solo:19:in `load'
/opt/vagrant_ruby/bin/chef-solo:19
[Sun, 29 Jul 2012 10:53:40 +0200] FATAL: RuntimeError: MiniTest failed with 1 failure(s)

Chef never successfully completed! Any errors should be visible in the
output above. Please fix your recipes so that they properly complete.

I get the same error with lucid64. However, centos6 runs as expected. Did I miss something?

Plugins not working if aws recipe is used

Hi,

The include_attribute 'elasticsearch::plugins' in aws attributes is not working (Chef: 11.4.4). And node.default.elasticsearch[:plugins] ||= plugins in plugins attributes is not setting it up as it already has aws plugin in it.

Anyway, why use '||=' when '=' will simply merge attributes and everything will work fine.

Thanks,
Maciek

min_mem should be the same as max_mem

Using mlockall, you want the min_mem and max_mem to be the same, otherwise it doesn't lock all the memory into RAM at startup.

perhaps just use the ES_HEAP_SIZE (which sets both min and max) instead?

Unable to change elasticsearch version via role and version tag

If you try to change elastic search via:

override_attributes "elasticsearch" => {
    :version => '0.19.8'
}

only :version will have been changed, because chef first evaluates all default attributes:
Your code:

default.elasticsearch[:version]       = "0.20.1"
default.elasticsearch[:host]          = "http://download.elasticsearch.org"
default.elasticsearch[:repository]    = "elasticsearch/elasticsearch"
default.elasticsearch[:filename]      = "elasticsearch-#{node.elasticsearch    [:version]}.tar.gz" # BUG: version can't be changed
default.elasticsearch[:download_url]  = [node.elasticsearch[:host], node.elasticsearch[:repository], node.elasticsearch[:filename]].join('/')

download url, filename, stays untouched.

Appears to install nginx even in cases when it's not requested (no proxy)

Can't confirm, but I was seeing weird errors about missing nginx-.tar.gz without specifying an nginx version in the attribute list.

Not a huge problem, as I plan on using the nginx proxy support (excellent idea, by the way) - but weird when I was starting out. if I get chance, I might send a patch, but I'm quite new to Chef.

Error message when running start script

I get an error message when I run the start script manually:

 root@es-6246b:~# /etc/init.d/elasticsearch start
 Starting elasticsearch...
 -su: /root/.bash_profile: Permission denied

I traced this down to the -m flag to su in the init script. This causes the elasticsearch process to try to read ~root/.bash_profile, but it does not have read access to ~root. The simplest solution I can think of is to remove the -m flag, but I do not know if that would have any bad consequences.

Can't find Monit template?

The recipe seems to be looking for the Monit template in the wrong place?

/var/lib/gems/1.9.1/gems/chef-10.12.0/lib/chef/cookbook_version.rb:647:in `preferred_manifest_record'
/var/lib/gems/1.9.1/gems/chef-10.12.0/lib/chef/cookbook_version.rb:655:in `preferred_filename_on_disk_location'
/var/lib/gems/1.9.1/gems/chef-10.12.0/lib/chef/provider/template.rb:68:in `template_location'
/var/lib/gems/1.9.1/gems/chef-10.12.0/lib/chef/provider/template.rb:39:in `action_create'
/var/lib/gems/1.9.1/gems/chef-10.12.0/lib/chef/resource.rb:454:in `run_action'
/var/lib/gems/1.9.1/gems/chef-10.12.0/lib/chef/runner.rb:49:in `run_action'
/var/lib/gems/1.9.1/gems/chef-10.12.0/lib/chef/runner.rb:85:in `block (2 levels) in converge'
/var/lib/gems/1.9.1/gems/chef-10.12.0/lib/chef/runner.rb:85:in `each'
/var/lib/gems/1.9.1/gems/chef-10.12.0/lib/chef/runner.rb:85:in `block in converge'
/var/lib/gems/1.9.1/gems/chef-10.12.0/lib/chef/resource_collection.rb:94:in `block in execute_each_resource'
/var/lib/gems/1.9.1/gems/chef-10.12.0/lib/chef/resource_collection/stepable_iterator.rb:116:in `call'
/var/lib/gems/1.9.1/gems/chef-10.12.0/lib/chef/resource_collection/stepable_iterator.rb:116:in `call_iterator_block'
/var/lib/gems/1.9.1/gems/chef-10.12.0/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
/var/lib/gems/1.9.1/gems/chef-10.12.0/lib/chef/resource_collection/stepable_iterator.rb:104:in `iterate'
/var/lib/gems/1.9.1/gems/chef-10.12.0/lib/chef/resource_collection/stepable_iterator.rb:55:in `each_with_index'
/var/lib/gems/1.9.1/gems/chef-10.12.0/lib/chef/resource_collection.rb:92:in `execute_each_resource'
/var/lib/gems/1.9.1/gems/chef-10.12.0/lib/chef/runner.rb:80:in `converge'
/var/lib/gems/1.9.1/gems/chef-10.12.0/lib/chef/client.rb:330:in `converge'
/var/lib/gems/1.9.1/gems/chef-10.12.0/lib/chef/client.rb:163:in `run'
/var/lib/gems/1.9.1/gems/chef-10.12.0/lib/chef/application/solo.rb:207:in `block in run_application'
/var/lib/gems/1.9.1/gems/chef-10.12.0/lib/chef/application/solo.rb:195:in `loop'
/var/lib/gems/1.9.1/gems/chef-10.12.0/lib/chef/application/solo.rb:195:in `run_application'
/var/lib/gems/1.9.1/gems/chef-10.12.0/lib/chef/application.rb:70:in `run'
/var/lib/gems/1.9.1/gems/chef-10.12.0/bin/chef-solo:25:in `<top (required)>'
/usr/local/bin/chef-solo:19:in `load'
/usr/local/bin/chef-solo:19:in `<main>'
[2012-08-11T04:56:09+02:00] ERROR: Running exception handlers
[2012-08-11T04:56:09+02:00] ERROR: Exception handlers complete
[2012-08-11T04:56:09+02:00] FATAL: Stacktrace dumped to /home/vagrant/chef/chef-stacktrace.out
[2012-08-11T04:56:09+02:00] FATAL: Chef::Exceptions::FileNotFound: template[/etc/monit/conf.d/elasticsearch.conf] (elasticsearch::default line 9) had an error: Chef::Exceptions::FileNotFound: Cookbook 'monit' (0.7.0) does not contain a file at any of these locations:
  templates/ubuntu-12.04/elasticsearch.conf.erb
  templates/ubuntu/elasticsearch.conf.erb
  templates/default/elasticsearch.conf.erb

This cookbook _does_ contain: ['/home/vagrant/chef/cookbooks/monit/templates/default/monitrc.erb','/home/vagrant/chef/cookbooks/monit/templates/default/postfix.conf.erb','/home/vagrant/chef/cookbooks/monit/templates/default/ssh.conf.erb']

Plugins defined in databag do not get installed

Initially I had been installing plugins via the elasticsearch::plugin recipe with attributes set at role level but have attempted to transition these to a databag. I've created a databag containing:

{
  "id": "plugins",
  "aws-dev": {
    "plugins": {
      "karmi/elasticsearch-paramedic": {},
      "mobz/elasticsearch-head": {},
      "lukas-vlcek/bigdesk": { "version": "2.0.0" },
      "elasticsearch-cloud-aws": { "version": "1.10.0" }
    }
  }
}

The only plugin that gets installed is the 'cloud-aws' plugin (but with the wrong version - 1.11.0 which it appears to get from the attributes/aws.rb). Running chef-client with logging set to debug shows

DEBUG: Plugins list: ["karmi/elasticsearch-paramedic", "mobz/elasticsearch-head", "lukas-vlcek/bigdesk", "elasticsearch-cloud-aws"]

but they do not appear to be converged into node attributes correctly.

Need a more comprehensive max_mem calculation

On an AWS m1.small instance, the current max_mem calculation of:

max_mem = "#{(node.memory.total.to_i - (node.memory.total.to_i/3) ) / 1024}m"

yields:

rb(main):002:0> ("1697880kB".to_i - ("1697880kB".to_i/3) ) / 1024
=> 1105
irb(main):003:0> 1105*1024
=> 1131520

So -Xmx1105m isn't so bad on a m1.small (1.7GB memory) instance until I run sudo service elasticsearch status -v and see:

elasticsearch 0.19.3 running with PID 1119
...
Memory Total:        1.6gb
...

I know that your setup is perfect for m1.large but I would still like to see a more universal equation than the current one ... so I'll also try and strain my brain to come up with something that factors in permgen etc. before coming up with a max_mem number to use.

Without such a consideration Monit hits the restart if mem_usage > 90% for 15 cycles condition very easily on non m1.large instances like mine.

Nginx rpm install doesn't support chunkin module

This isn't really your issue, but I wanted to let you know and get your thoughts.

Here is where I am at.

I setup a node with this cookbook, and I setup nginx as a proxy to get basic auth.

My use case is that I want Hubot to push chat logs into elastic search
https://gist.github.com/4050748 # Work In Progress

The issue is that node seems to like streaming requests and doesn't set the Content-Length header.
This makes Nginx do the following

<head><title>411 Length Required</title></head>
<body bgcolor="white">
<center><h1>411 Length Required</h1></center>
<hr><center>nginx/1.0.15</center>
</body>
</html>

It seems like this would be a fairly common use case, which is why I am bringing this to you.

The work around

Currently, I am hand coding the json into a string, and setting the Content-Length header, which is ugly but working.

Alternatives?

http://www.lamnk.com/blog/computer/fix-nginx-411-length-required-error/

suggests compiling with the chunkin module, but then that goes back to community nginx. It was cleaned up a bit a few days ago, but it is still a pretty heavy dependency.

What are your thoughts or suggestions?

Thank You
Chris Lundquist

-Xss128k is too low

The - Xss128k seems to be too low when we try using this recipe with ES 0.19.* and SPM.
When we try running ES using this recipe and SPM, ES/JVM doesn't start.
By default, ES has it set to 256k (in elasticsearch.in.sh)

With -Xss256k things work. Could you please make this small increase?

Monit issues (template file name, internal issues)

First of all, I wanted to say: great cookbook! I've been hacking around with various other ElasticSearch cookbooks, and this was the only one that was current, works with AWS, and doesn't drag in a whole mess of other dependencies.

One thing I found on initially running the recipe is that the Monit conf template is "elasticsearch.monitrc.erb", but the monitrc recipe (v0.6) has a hard-coded assumption of "#{name}.conf.erb". I ended up renaming "elasticsearch.monitrc.erb" to "elasticsearch.conf.erb" to get past it, but I don't particularly like my solution as "elasticsearch.conf.erb" sounds too general.

Here's my exception:

[Fri, 09 Mar 2012 20:25:41 +0000] ERROR: template[/etc/monit/conf.d/elasticsearch.conf] (/var/cache/chef/cookbooks/monit/libraries/monitrc.rb:8:in `monitrc') had an error:
template[/etc/monit/conf.d/elasticsearch.conf] (elasticsearch::default line 8) had an error: Chef::Exceptions::FileNotFound: Cookbook 'elasticsearch' (0.0.1) does not contain a file at any of these locations:
  templates/ubuntu-11.10/elasticsearch.conf.erb
  templates/ubuntu/elasticsearch.conf.erb
  templates/default/elasticsearch.conf.erb

Also, I get an error for this line in the monit conf:

http://<%= node.monit[:http_auth]['username'] %>:<%= node.monit[:http_auth]['password_encoded'] %>@<%= node.cloud.public_hostname %>:2812/elasticsearch_cluster_health

Skimming the monit recipe, I don't see an http_auth attribute anywhere.

Discovery settings in elasticsearch.yml.erb

Hi Karel,

Very minor issue, but I hit this in elasticsearch.yml.erb at this section:

<% if node.elasticsearch[:discovery][:type] %>
discovery:
  type: <%= node.elasticsearch[:discovery][:type] %>
<% end %>
<% unless node.elasticsearch[:cloud][:ec2].values.compact.empty? %>
  ec2.groups: <%= node.elasticsearch[:cloud][:ec2][:security_group] %>
<% end %>

I actually have aws settings (and thus node.elasticsearch[:cloud][:ec2]), but I have not yet enabled discovery, which lead to an incorrect YAML directive. I think:

<% if node.elasticsearch[:discovery][:type] %>
discovery:
  type: <%= node.elasticsearch[:discovery][:type] %>
  <% unless node.elasticsearch[:cloud][:ec2].values.compact.empty? %>
  ec2.groups: <%= node.elasticsearch[:cloud][:ec2][:security_group] %>
  <% end %>
<% end %>

might be more appropriate.

Probable bugs in install_plugin.rb

I think there might be several scenarios where the logic in the following file may not be quite robust:
https://github.com/karmi/cookbook-elasticsearch/blob/master/definitions/install_plugin.rb

@karmi I'm currently not a very good student of Ruby so I wanted to bring the use-cases to your attention and ask for your opinion on the matter.

I have been adding additional ES plugins in a fork:
https://github.com/pulkitsinghal/cookbook-elasticsearch/commits/master
with the only difference being that each one of them is executed in a separate run_list otherwise, the ES restart gets in the way.

####
#Source @ https://gist.github.com/2793878
####
# install ES and aws plugin
time ssh -t $SSH_OPTIONS $HOST "sudo chef-solo --node-name elasticsearch-test-1 -j /tmp/run-1.json"
# install HEAD plugin
time ssh -t $SSH_OPTIONS $HOST "sudo chef-solo --node-name elasticsearch-test-1 -j /tmp/run-2.json"
# install jetty plugin
time ssh -t $SSH_OPTIONS $HOST "sudo chef-solo --node-name elasticsearch-test-1 -j /tmp/run-3.json"

By the time I get to the 3rd run for jetty, basically nothing works. I suspect that this may simply be because the following loop is a bit off (line 11 - 15) and thinks that plugin jetty is already present:
https://github.com/karmi/cookbook-elasticsearch/blob/master/definitions/install_plugin.rb#L11-15

Please let me know if I'm just barking up the wrong tree. Or if there is some truth to the matter.

Fog >= 1.11.0 breaks run with elasticsearch::ebs

Since the release of Fog 1.11.0 (and 1.11.1) the elasticsearch::ebs recipe causes the chef run to fail due to the latest version of fog installing a conflicting version of the json gem.

The run fails on a brand new machine with the error:

[2013-05-07T08:54:08+00:00] FATAL: Gem::LoadError: ruby_block[Create EBS volume on /dev/xvdf (size: 10GB)](elasticsearch::ebs line 16) had an error: Gem::LoadError: Unable to activate fog-1.11.1, because json-1.5.4 conflicts with json (~> 1.7)

Forcing the version of the fog gem down to 1.10.1 with the following resolves the issue as a band-aid, but there's probably a better way to do it.

chef_gem 'fog' do
  version '1.10.1'
  action :install
end

Proxy recipe has hardcoded localhost which fails if elasticsearch is not bound to that IP

I'm using node.elasticsearch[:custom_config][:network.host] to have Elasticsearch bind to a specific IP (which is not 127.0.0.1). This works great, but the proxy recipe fails because it has localhost hardcoded (https://github.com/elasticsearch/cookbook-elasticsearch/blob/master/templates/default/elasticsearch_proxy.conf.erb#L26, https://github.com/elasticsearch/cookbook-elasticsearch/blob/master/templates/default/elasticsearch_proxy.conf.erb#L50). Instead it should use

<%= node.elasticsearch[:custom_config][:network.host] ||
    node.elasticsearch[:custom_config][:network.bind_host] ||
    'localhost' %>

(or something like that).

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.