Create AWS environment in new VPC with at least two zones in one region and setup Elasticsearch cluster with autodiscovery: cluster will be automatically reconfigured after any changes in nodes amount. This scheme will work in most cases, maybe except cases when you need automatically set up new nodes after some monitoring events, but even in this cases there is some solutions like additional pipelines: internal aws solutions (or chef in opsworks instead of ansible) or external tools like StackStorm.

• Ansible
• boto
• AWS admin access

Ansible uses python-boto library to call AWS API, and boto needs AWS credentials in order to perform all the functions. There are many ways to configure your AWS credentials. The easiest way is to crate a .boto file under your user home directory:

vim ~/.boto

Then add the following:

[Credentials]
aws_access_key_id = <your_access_key_here>
aws_secret_access_key = <your_secret_key_here>

Edit the group_vars/all.yml vars file:

cluster: elasticsearch

# Tags
# if you'll change `Name` then change it in `setup_elasticsearch.yml` twice too ;)
tags:
  Name: "{{ cluster }}"

# ec2_key
key_name: "devops-key"
ssh_key_location: ~/.ssh/id_rsa.pub 

# VPC
vpc_region: us-east-1
vpc_name: "{{ cluster }}-vpc"
vpc_cidr_block: 172.22.0.0/16
vpc_subnets:
  - cidr: 172.22.1.0/24
    az: "{{ vpc_region }}a"
    resource_tags: { "Name":"{{ cluster }}" }
  - cidr: 172.22.2.0/24
    az: "{{ vpc_region }}c"
    resource_tags: { "Name":"{{ cluster }}" }
sg_name: "{{ cluster }}-security-group"

# EC2
image: ami-ff427095 # ubuntu trusty 14.04
instances_count: 1 # count per one subnet
instances_type: t2.micro
instances_data_size: 10
instances_device_name: /dev/sdb

# LB
lb_name: "{{ cluster }}-load-balancer"

# ElasticSearch
es_mount_point: /data
es_cluster_name: "{{ cluster }}"
es_data_dir: "{{ es_mount_point }}/elasticsearch/data"

ansible-playbook setup_aws.yml

What will be created:

• Ssh key pair
• VPC with subnets in different regions
• Network security group that allows any ssh traffic and internal vpc traffic
• IAM role with policy that allows ec2 discovery
• Ec2 instances in all vpc networks
• Internal load balancer

ansible-playbook -i inventory/ setup_elasticsearch.yml

Warning: repeated application will restart worked nodes in most cases. Use limit if you want apply it only to new hosts!

What will be created:

• Mount /dev/sdb as /data on instances
• Install Elasticsearch 2.x
• Install cloud-aws plugin for elasticsearch
• Set java heap size as half or ram for elasticsearch
• Configure autodiscovery via ec2 tags for elasticsearch nodes