Code Monkey home page Code Monkey logo

hadoop-mutinode's Introduction

Hadoop (YARN) Multinode Cluster with Docker.

The purpose of this project is to help developer quickly start multinode cluster with docker containers on their laptop.

There are multiple and betters ways to solve the problems and since I am not a DevOps guy please feel free to suggest , advice or contribute to it.

The first docker container hadoop-base provisions ubuntu with java and hadoop and does most of the leg work to setup the master and slave containter.The hadoop-base is extending from docker-serf. See DNSMASQ/SERF

The second container image hadoop-dn extends from hadoop-base and install the slave specific hadoop configuaration and it also installs daemontools to run the sshd , serf and dnsmasq so when we start the docker container in daemon mode it keep running instead of exiting immediatley after startup.

The third docker container hadoop-nn-dn extends from hadoop-base and install the master specific hadoop configuaration and it also installs daemontools to run the sshd , serf and dnsmasq.

I decided to start master container in foreground mode therefore when we startup master container it will take you to the bash shell prompt after successfull startup.

The example for demonstration is using 2 node cluster.The master node is also configured as slave node so we have 2 slave nodes and 1 master node.

Prerequisite

  1. Docker must be installed on the host computer / laptop.
  2. git clone https://github.com/alvinhenrick/docker-serf
  3. Change directiy to where you cloned the above repository.
  4. Run docker build -t alvinhenrick/serf .

Build Multinode Hadoop Cluster

git clone https://github.com/alvinhenrick/hadoop-mutinode

  • Build hadoop-base container

    • Change directory to hadoop-mutinode/hadoop-base.
    • Run docker build -t alvinhenrick/hadoop-base .
    • This will take a while to build the container go grab a cup of coffee or whatever drink you like :)
  • Build hadoop-dn Slave container (DataNode / NodeManager)

    • Change directory to hadoop-mutinode/hadoop-dn.
    • Run docker build -t alvinhenrick/hadoop-dn .
  • Build hadoop-nn-dn Master container (NameNode / DataNode / Resource Manager / NodeManager)

    • Change directory to hadoop-mutinode/hadoop-nn-dn.
    • Run docker build -t alvinhenrick/hadoop-nn-dn .

Sart the containers.

  • Change directory to hadoop-mutinode.
  • Run ./start-cluster.sh
  • On BASH shell prompt Run /usr/local/hadoop/bin/start-hadoop.sh
  • After startup on prompt type su - hduser
  • Run jps
  • Run hdfs dfs -ls /

hadoop-mutinode's People

Contributors

alvinhenrick avatar

Watchers

James Cloos avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.