Ambari is a tool which makes provisioning, managing, and monitoring of Apache Hadoop deployments easy. zData's Ambari Extensions builds atop Ambari to provide easy deployment and management of HAWQ, Chorus, and soon other Pivotal technologies.
The master branch contains code to work with Apache Ambari. There is a pivotal branch which works with Pivotal Ambari.
Visit the project's documentation for quick start guides and more information.
-
Requires the following plugins: vagrant-hostmanager, vagrant-reload
vagrant plugin install vagrant-hostmanager vagrant plugin install vagrant-reload vagrant plugin install vagrant-aws # Optional, provision on AWS vagrant plugin install vagrant-cachier # Optional, cache downloaded packages to speed up provisioning
-
Create boxes with Virtualbox
vagrant up # Bring up master, slave1Note: Copy
vagrant-env.conf.sampletovagrant-env.confand modify the values to change various vagrant settings such as the number of slave machines. -
Connect, vms created: master, slave0, slave1
master.ambaricluster.local slave1.ambaricluster.local vagrant ssh master vagrant ssh slave1
-
Install a dummy box:
vagrant box add dummy https://github.com/mitchellh/vagrant-aws/raw/master/dummy.box
-
Configure Vagrant with your unique Amazon access key and secret. Creating a user Vagrantfile in
~/.vagrant.d/Vagrantfilewith:Vagrant.configure('2') do |config| config.vm.provider :aws do |aws, override| aws.access_key_id = ENV['AWS_KEY'] aws.secret_access_key = ENV['AWS_SECRET'] end end
Add environmental variables with your AWS_KEY and AWS_SECRET in your
~/.bashrc:export AWS_KEY="THEKEY" export AWS_SECRET="THESECRET"
-
Configure Vagrant with a keypair it can use to communicate with the created boxes. Generate a new keypair. In your user Vagrantfile add the following lines in the
config.vm.provider :awsblock:aws.keypair_name = 'vagrant' override.ssh.private_key_path = '~/.ssh/aws-vagrant'
The variable
keypair_nameshould be the name of the keypair on AWS, the variableprivate_key_pathshould be the path to the private key on your local computer.Your user Vagrantfile should now look like:
Vagrant.configure('2') do |config| config.vm.provider :aws do |aws, override| aws.access_key_id = ENV['AWS_KEY'] aws.secret_access_key = ENV['AWS_SECRET'] aws.keypair_name = 'vagrant' override.ssh.private_key_path = '~/.ssh/aws-vagrant' end end
-
Create the boxes on AWS
vagrant up --provider=aws --no-parallel vagrant hostmanager
More information about getting started with Ambari using vagrant is available here.
Installs and manages the Pivotal Greenplum database software.
- Does not automatically create or setup XFS filesystem.
- Does not specifiy an IO scheduler of deadline.
- Does not configure read-ahead.
- Does not disable transparent hugepage.
Installs and manages zData Chorus.
- inimum_memory = 256M
- maximum_memory = 256M
- young_heap_size = 128M
- max_perm_size = 256M
Installs and manages the Pivotal HAWQ Hadoop SQL engine.
Installs and manages the Pivotal Extension Framework, patches it to work with Hortonwork's Hadoop.
Sometimes it's possible to use the same code for both Pivotal and Apache Ambari, when this is the case you can use git to help simplify merging a feature branch to both branches.
git checkout master
git checkout -b feature/##
# Do feature
# Pivotal port
git checkout -b feature/##_pivotal_port
git rebase --onto pivotal master feature/##_pivotal_port
git checkout pivotal
git merge --ff-only feature/##_pivotal_port
git branch -D feature/##_pivotal_port
# Merge to master
git checkout master
git merge feature/##_pivotal_port
Tests are written with unittest2, and they require the ambari source code be available.
-
Set an environment variable so unittests can find the Ambari's source directory:
git clone https://git-wip-us.apache.org/repos/asf/ambari.git ~/source-ambari export AMBARI_SOURCE="$HOME/source-ambari" -
Install necessary python packages
pip install unittest2 mock -
Run tests:
(cd tests; python -m unittest discover)
To install HAWQ you will need some files from Pivotal. You can find these files at https://network.pivotal.io/products/pivotal-hd. Create an account if you don't have one already, and place the downloaded files in the artifacts folder located in the project root.
- Pivotal HD 2.1 -> Pivotal HD 2.1.0
- Pivotal HD 2.1 -> PHD 2.1.0: Pivotal HAWQ 1.2.1.0
- 4.3.5.0 Database Server -> Greenplum Database 4.3.5.0 for Red Hat Enterprise Linux 5 and 6
Ambari is an open source deployment tool, users must follow license agreements provided by Hortonworks and Pivotal software.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent