snowplow / ansible-playbooks Goto Github PK

View Code? Open in Web Editor NEW

176.0 29.0 127.0 286 KB

Ansible playbooks to install common platforms and tools (e.g. JVM, Ruby, Postgres etc.)

Shell 100.00%

ansible-playbooks's People

Contributors

Stargazers

Watchers

Forkers

jhidalgo3 apeckham skynet robbertvanwaveren denistiago apostiglioni-ansible miguelperalvo jeffusan nmatyukov chuwy sayayusen pravasna ferrlin vogxn oldoni rajatguptarg archanachillala drgeb szz giappv issactran krajewf fwahlqvist ksnehalakshmi suteerth chocopowwwa gogobook hgomez gzxultra rbramwell stvhanna nishv-ls dixon629 raghavf1 aelkz ptumuluri8 devops98 igoros777 yangjunyan magnificrogue rsettlage mhadam vigneshavm rakeshkamisetti kotya vivekthakker jrluis chihseanhsu pakanativ narayudu anuwardeen naveenlj jawlitkp tariknaeem andytengca muriloaj jyothisrirad harivamsiengg devisetty99 dineshsandil kandepi143 sundeeplp zainul1114 hemanth4188 mikeaddison93 purplefox81 nullhappens adglow-fbermejo madamanchikiran rahman13dec4u babu003 armohamm vivekragupathy rvenkat455 jasonskidmore suresh58 cu12 scovl rameshmishra777 gopinadhmuthavarapu chandrakiran88 srikanth4424 claudiouzelac kot-begemot keonjeo nikolaidering herky1818 lovescott bds1965 prathyuveredic akshay9599 gaya3chandran1 davehowell devopsankur chanikdevops999 pandruszkow-foss-sourcemine ravishsubramanya madhuscm20 sorincucu84 manikanta1905

ansible-playbooks's Issues

Remove yui-compressor from snowplow-javascript-tracker.yml

Grunt takes care of compression so this is no longer needed.

Running the playbook without removing the line produces this error:

TASK: [yui-compressor | Ensure YUI Compressor is downloaded] ****************** 
failed: [127.0.0.1] => {"dest": "/tmp/", "failed": true, "gid": 0, "group": "root", "mode": "01777", "owner": "root", "response": "Request failed: <urlopen error [Errno 1] _ssl.c:504: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure>", "size": 4096, "state": "directory", "status_code": -1, "uid": 0, "url": "https://github.com/downloads/yui/yuicompressor/yuicompressor-2.4.2.zip"}
msg: Request failed

FATAL: all hosts have already failed -- aborting

Create playbook for zabbix-agent

ruby-rvm.yaml* is missing a pg dependency

Almost all our Ruby stuff requires PG access, and the pg gem cannot be installed without first:

sudo apt-get install libpq-dev

* or arguably snowplow-batch-pipeline.yaml instead

Can't get latest website installer working

From a clean dev-environment and ansible-playbooks:

vagrant@precise64:~$ ansible-playbook /vagrant/ansible-playbooks/snowplow-website.yml \
>
ansible/                   .bash_history              .bashrc                    postinstall.sh             .ssh/                      .vbox_version
ansible_hosts              .bash_logout               .cache/                    .profile                   .sudo_as_admin_successful  .veewee_version
> --inventory-file=/home/vagrant/ansible_hosts --connection=local

PLAY [vagrant] ****************************************************************

GATHERING FACTS ***************************************************************
ok: [127.0.0.1]

TASK: [base | Update apt cache] ***********************************************
ok: [127.0.0.1]

TASK: [base | Update all packages] ********************************************
changed: [127.0.0.1]

TASK: [base | install basic packages] *****************************************
changed: [127.0.0.1] => (item=curl,vim,libxslt-dev,libxml2-dev,unzip,python-pip)

TASK: [ruby-rvm | Ensure curl is installed (used to download RVM)] ************
ok: [127.0.0.1] => (item=curl)

TASK: [ruby-rvm | Ensure RVM is installed] ************************************
changed: [127.0.0.1]

TASK: [ruby-rvm | Ensure Ruby 1.9.3 is installed as default] ******************
changed: [127.0.0.1]

TASK: [ruby-rvm | Start RVM (otherwise only launched when the shell is rebooted)] ***
changed: [127.0.0.1]

TASK: [ruby-rvm | NOTICE! You will need to exit the terminal and open a fresh terminal before you can use RVM. \n(Or alternatively run '. /home/vagrant/.rvm/scripts/rvm] ***
changed: [127.0.0.1]

TASK: [jekyll | Ensure Jekyll's gem dependencies are installed] ***************
failed: [127.0.0.1] => (item={'version': u'1.4.3', 'flags': '--no-ri --no-rdoc', 'name': 'jekyll'}) => {"changed": true, "cmd": "gem install jekyll -v 1.4.3 --no-ri --no-rdoc ", "delta": "0:00:32.792299", "end": "2014-03-05 12:38:44.284301", "item": {"flags": "--no-ri --no-rdoc", "name": "jekyll", "version": "1.4.3"}, "rc": 1, "start": "2014-03-05 12:38:11.492002"}
stderr: ERROR:  While executing gem ... (Gem::FilePermissionError)
    You don't have write permissions into the /opt/vagrant_ruby/lib/ruby/gems/1.8 directory.

FATAL: all hosts have already failed -- aborting

PLAY RECAP ********************************************************************
           to retry, use: --limit @/home/vagrant/snowplow-website.retry

127.0.0.1                  : ok=9    changed=6    unreachable=0    failed=1

Problems auto-cloning snowplow-python tracker

Have tried:

Deleting dev-environment's own .git* files and folders
Installing with dest=/vagrant/
Installing with dest=/vagrant/snowplow-python-tracker

All give different error messages, suggesting getting closer to the answer (?), but no joy yet...

snowplow-js-tracker missing Java dependency

vagrant@precise64:/vagrant/snowplow-javascript-tracker$ grunt publish
Running "concat:dist" (concat) task
File "dist/snowplow.js" created.

Running "min:dist" (min) task
Warning: Command failed: /bin/sh: 1: java: not found
 Use --force to continue.

Aborted due to warnings.

node.js version check doesn't work

TASK: [nodejs | Node.js | Checking installed version of node.js] **************
failed: [127.0.0.1] => {"changed": true, "cmd": "/usr/bin/test "$(node -v 2> /dev/null)" = v0.10.25 ", "delta": "0:00:00.003899", "end": "2014-03-17 11:08:12.214885", "item": "", "rc": 1, "start": "2014-03-17 11:08:12.210986"}
...ignoring

Update ansible_cache references to use ansible_cache_dir

Add maven & ant to JVM playbooks

@alexanderdean - is this is a good idea?

sudo apt-get install maven

Create snowplow-python-tracker.yaml

Python 3.3
Virtualenv
pip install python-dateutil
pip install httmock
pip install freezegun

References:

snowplow-js-tracker should install grunt-cli

npm install grunt-cli

Fix bug in snowplow.github.com where -e is being added to bash

Something wrong with this line:

shell: echo -e "export LC_ALL=en_US.UTF-8 \nexport LANG=en_US.UTF-8" >> /home/vagrant/.bashrc

Add Android environment setup

Here are the exact build steps that you need to build the Android Tracker. I've tested it on a fresh Ubuntu 14.04 x64 VM.

Your VM needs 2GB RAM for the SDK to build the tracker

# Installing OpenJDK
apt-get update
apt-get install -y openjdk-7-jre openjdk-7-jdk

# Download the latest Android SDK
wget http://dl.google.com/android/android-sdk_r23.0.2-linux.tgz
tar -xvf android-sdk_r23.0.2-linux.tgz

# Setup the Android environment
export ANDROID_HOME=`pwd`/android-sdk-linux
export PATH="$ANDROID_HOME/tools:$ANDROID_HOME/platform-tools:$PATH"

# Adds the environment to your profile as well
echo "export ANDROID_HOME=`pwd`/android-sdk-linux
export PATH=\"\$ANDROID_HOME/tools:\$ANDROID_HOME/platform-tools:\$PATH\"" >> ~/.bashrc

# Downloads the Platform and Build tools, App Support and Play Services SDKs
( sleep 5 && while [ 1 ]; do sleep 1; echo y; done ) | android update sdk -u --all --filter platform-tool-19.1.0,build-tools-19.1.0,android-19,105,106,99,98,16,4,2

# Libraries required to build
apt-get install -y libc6-i386 lib32stdc++6 lib32gcc1 lib32ncurses5 zlib1g lib32z1

# Actual build steps
apt-get install -y git-core
git clone https://github.com/snowplow/snowplow-android-tracker.git
cd snowplow-android-tracker/
#git checkout develop
./gradlew build

Figure out why Neo4J complains about number of file opens

Neo4J complains about the maximum number of file opens on loading / restarting:

vagrant@precise64:/etc/init.d$ sudo ./neo4j-service restart
 * Restarting Neo4j Graph Database neo4j                                                                                                                WARNING: Max 1024 open files allowed, minimum of 40 000 recommended. See the Neo4j manual.
Using additional JVM arguments:  -server -XX:+DisableExplicitGC -Dorg.neo4j.server.properties=conf/neo4j-server.properties -Djava.util.logging.config.file=conf/logging.properties -Dlog4j.configuration=file:conf/log4j.properties -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled
Starting Neo4j Server...WARNING: not changing user
process [2168]... waiting for server to be ready..... OK.
http://localhost:7474/ is ready.

I've followed the instructions in the following blog post to increase the limit: http://www.delimited.io/blog/2014/1/15/getting-started-with-neo4j-on-ubuntu-server, but I still get the same error...

Update snowplow-python-tracker.yml for use in Vagrant quickstart

Add Golang setup

Add screen to the base install

Centralize ansible_cache_dir checks in base role

Add batch importer to neo4j playbook

http://maxdemarzi.com/2012/02/28/batch-importer-part-1/
https://github.com/jexp/batch-import

Bring Ansible coding conventions over to Snowplow wiki

https://github.com/edx/configuration/wiki/Ansible-Coding-Conventions

Snowplow Batch Pipeline - Failing on [ruby-rvm | Ensure Ruby 1.9.3 is installed as default]

Error with: ansible-playbook /vagrant/ansible-playbooks/snowplow-batch-pipeline.yml
--inventory-file=/home/vagrant/ansible_hosts --connection=local

Resolution - need to add a keyring for getting Ruby RVM:

gpg --keyserver hkp://keys.gnupg.net --recv-keys D39DC0E3

Make Java installs idempotent

Add maven to neo4j server

RVM operations should not be sudo'ed

Just confirmed that Jekyll does not work on a fresh build, even when making sure to log out between Ruby & snowplow.github.com installation.

vagrant@precise64:~$ jekyll --server
/home/vagrant/.rvm/rubies/ruby-1.9.3-p484/lib/ruby/site_ruby/1.9.1/rubygems/dependency.rb:298:in `to_specs': Could not find 'jekyll' (>= 0) among 7 total gem(s) (Gem::LoadError)
    from /home/vagrant/.rvm/rubies/ruby-1.9.3-p484/lib/ruby/site_ruby/1.9.1/rubygems/dependency.rb:309:in `to_spec'
    from /home/vagrant/.rvm/rubies/ruby-1.9.3-p484/lib/ruby/site_ruby/1.9.1/rubygems/core_ext/kernel_gem.rb:53:in `gem'
    from /home/vagrant/.rvm/rubies/ruby-1.9.3-p484/bin/jekyll:22:in `<main>'
    from /home/vagrant/.rvm/gems/ruby-1.9.3-p484@global/bin/ruby_executable_hooks:15:in `eval'
    from /home/vagrant/.rvm/gems/ruby-1.9.3-p484@global/bin/ruby_executable_hooks:15:in `<main>'

Install Ruby v 1.8.7 as part of the aws-cli tools

This is needed to run emr-cli

Figure out why running oh-my-zsh playbook stops Ansible working

Seems to be to do with the switch in shell, which changes the $PATH...

Update website yaml to include ruby.yaml as a dependency

Make Python Tracker playbook install Redis

Make all playbooks composable

Remove Gradle's dependency on Java

Because inter-role dependencies in Ansible are broken

Remove Android's dependency on Oracle Java

Because variables and dependencies don't work in Ansible.

Add Neo4J shell tools to neo4j role

https://github.com/jexp/neo4j-shell-tools#cypher-import

Update the Snowplow Python Tracker playbook to install Python version 2.7

This blocks snowplow/snowplow-python-tracker#42

Make Postgres open like Travis CI's

Travis CI has a very open PG which is great for tests.

To achieve the same:

$ sudo vi /etc/postgresql/8.4/main/pg_hba.conf

then add/replace:

local   all             postgres                                trust
host    all             postgres          127.0.0.1/32                      trust

then:

$ sudo /etc/init.d/postgresql reload

Add a PHP role and playbook

Josh can you add these?

Update play-2.yaml to formally include jvm dependency

Add terraform install

This needs tidying up, is hideous:

cd /tmp
wget https://dl.bintray.com/mitchellh/terraform/terraform_0.3.0_linux_amd64.zip
unzip terraform_0.3.0_linux_amd64.zip
sudo mv terraform* /usr/bin

Split Thrift out from jvm-*

Install mountebank to /opt or similar, not /vagrant

It shouldn't be installed into the project codebase path

Add vendor/snowplow-javascript-tracker

/cc @wiremine

Add Clojure/lein to jvm-6

Incompatible character encodings bug with jekyll --server

Having fixed the other issues, now left with:

vagrant@precise64:/vagrant/snowplow.github.com$ jekyll --server
Configuration from /vagrant/snowplow.github.com/_config.yml
Building site: . -> ./_site
Liquid Exception: incompatible character encodings: UTF-8 and ISO-8859-1 in blog-post
/home/vagrant/.rvm/gems/ruby-1.9.3-p484@global/gems/liquid-2.4.1/lib/liquid/block.rb:92:in `join'
/home/vagrant/.rvm/gems/ruby-1.9.3-p484@global/gems/liquid-2.4.1/lib/liquid/block.rb:92:in `render_all'
/home/vagrant/.rvm/gems/ruby-1.9.3-p484@global/gems/liquid-2.4.1/lib/liquid/block.rb:82:in `render'
/home/vagrant/.rvm/gems/ruby-1.9.3-p484@global/gems/liquid-2.4.1/lib/liquid/template.rb:124:in `render'
/home/vagrant/.rvm/gems/ruby-1.9.3-p484@global/gems/liquid-2.4.1/lib/liquid/template.rb:132:in `render!'
/home/vagrant/.rvm/gems/ruby-1.9.3-p484@global/gems/jekyll-0.12.1/lib/jekyll/convertible.rb:101:in `do_layout'
/home/vagrant/.rvm/gems/ruby-1.9.3-p484@global/gems/jekyll-0.12.1/lib/jekyll/post.rb:195:in `render'
/home/vagrant/.rvm/gems/ruby-1.9.3-p484@global/gems/jekyll-0.12.1/lib/jekyll/site.rb:200:in `block in render'
/home/vagrant/.rvm/gems/ruby-1.9.3-p484@global/gems/jekyll-0.12.1/lib/jekyll/site.rb:199:in `each'
/home/vagrant/.rvm/gems/ruby-1.9.3-p484@global/gems/jekyll-0.12.1/lib/jekyll/site.rb:199:in `render'
/home/vagrant/.rvm/gems/ruby-1.9.3-p484@global/gems/jekyll-0.12.1/lib/jekyll/site.rb:41:in `process'
/home/vagrant/.rvm/gems/ruby-1.9.3-p484@global/gems/jekyll-0.12.1/bin/jekyll:264:in `<top (required)>'
/home/vagrant/.rvm/gems/ruby-1.9.3-p484@global/bin/jekyll:23:in `load'
/home/vagrant/.rvm/gems/ruby-1.9.3-p484@global/bin/jekyll:23:in `<main>'
/home/vagrant/.rvm/gems/ruby-1.9.3-p484@global/bin/ruby_executable_hooks:15:in `eval'
/home/vagrant/.rvm/gems/ruby-1.9.3-p484@global/bin/ruby_executable_hooks:15:in `<main>'
Build Failed

Create playbook for zabbix-server

"EMR CLI tools installed into /vagrant/elastic-mapreduce-ruby"

incorrect, path is /vagrant/emr-cli

Write ElasticSearch playbook

You can use this as a starting point:

https://github.com/pigdata/snowplow/wiki/5---Elasticsearch-with-snowplow

Set JAVA_HOME in java playbooks

Needed e.g. for Android

Can't figure out how to make Ansible respect the Java 8 requirement

$ /vagrant/ansible-playbooks/roles/gradle/tasks$ ansible-playbook /vagrant/ansible-playbooks/java-8-gradle.yml --inventory-file=/vagrant/home/ansible/ansible_hosts --connection=local

When installing:

TASK: [oracle-java | Install Oracle Java Repo Installer Repository] ***********
ok: [127.0.0.1]

TASK: [oracle-java | Wizardry to bypass the Oracle License File prompt] *******
changed: [127.0.0.1]

TASK: [oracle-java | Install Oracle Java 6] ***********************************
skipping: [127.0.0.1]

TASK: [oracle-java | Install Oracle Java 7] ***********************************
skipping: [127.0.0.1]

TASK: [oracle-java | Install Oracle Java 8] ***********************************
ok: [127.0.0.1]

TASK: [oracle-java | Install PyCurl (required for apt-repository updates)] ****
ok: [127.0.0.1]

TASK: [oracle-java | Install Oracle Java Repo Installer Repository] ***********
ok: [127.0.0.1]

TASK: [oracle-java | Wizardry to bypass the Oracle License File prompt] *******
changed: [127.0.0.1]

TASK: [oracle-java | Install Oracle Java 6] ***********************************
skipping: [127.0.0.1]

TASK: [oracle-java | Install Oracle Java 7] ***********************************
ok: [127.0.0.1]

TASK: [oracle-java | Install Oracle Java 8] ***********************************
skipping: [127.0.0.1]

The first run through, for Java 8, successfully installs Java 8 and skips the others. When it comes to Gradle's turn, it seems to attempt to install Java 7, despite being run in an environment that specifically requests Java 8.

Maybe you know what's going on here?

node installation missing 2 dependencies

TASK: [nodejs | Node.js | Make] ***********************************************
failed: [127.0.0.1] => {"changed": true, "cmd": "/usr/bin/make ", "delta": "0:00:00.430994", "end": "2014-03-17 11:00:48.098899", "item": "", "rc": 2, "start": "2014-03-17 11:00:47.667905"}
stderr: make[1]: g++: Command not found
make[1]: *** [/tmp/node-v0.10.25/out/Release/obj.target/v8_base/deps/v8/src/accessors.o] Error 127
make: *** [node] Error 2
stdout: /usr/bin/make -C out BUILDTYPE=Release V=1
make[1]: Entering directory `/tmp/node-v0.10.25/out'
  g++ '-DENABLE_DEBUGGER_SUPPORT' '-DENABLE_EXTRA_CHECKS' '-DV8_TARGET_ARCH_X64' -I../deps/v8/src  -Wall -Wextra -Wno-unused-parameter -pthread -m64 -fno-strict-aliasing -O2 -fno-strict-aliasing -fno-tree-vrp -fno-omit-frame-pointer -fno-rtti -fno-exceptions -MMD -MF /tmp/node-v0.10.25/out/Release/.deps//tmp/node-v0.10.25/out/Release/obj.target/v8_base/deps/v8/src/accessors.o.d.raw  -c -o /tmp/node-v0.10.25/out/Release/obj.target/v8_base/deps/v8/src/accessors.o ../deps/v8/src/accessors.cc
make[1]: Leaving directory `/tmp/node-v0.10.25/out'

FATAL: all hosts have already failed -- aborting

Fix:

sudo apt-get install build-essential g++