Code Monkey home page Code Monkey logo

hdfs's Introduction

Build Status HA HDFS on Apache Mesos

Starts 1 active NameNode (with JournalNode and ZKFC), 1 standby NN (+JN,ZKFC), 1 JN, and everything else is DataNodes.

Building HDFS-Mesos

  1. ./bin/build-hdfs
  2. Run ./bin/build-hdfs nocompile to skip the mvn clean package step and just re-bundle the binaries.
  3. To remove the project build output and downloaded binaries, run ./bin/build-hdfs clean

Installing HDFS-Mesos on your Cluster

  1. Upload hdfs-mesos-*.tgz to a node in your Mesos cluster.
  2. Extract it with tar zxvf hdfs-mesos-*.tgz
  3. Customize configuration in hdfs-mesos-*/etc/hadoop/*-site.xml
  4. Check that hostname on that node resolves to a non-localhost IP; update /etc/hosts if necessary

Starting HDFS-Mesos

  1. cd hdfs-mesos-*
  2. ./bin/hdfs-mesos
  3. Check the Mesos web console to wait until all tasks are RUNNING (monitor status in JN sandboxes)

Using HDFS

See some of the many HDFS tutorials out there for more details and explore the web UI at
http://<ActiveNameNode>:50070.
Note that you can access commands through hdfs://<mesos.hdfs.framework.name>/ (default: hdfs://hdfs/). Also here is a quick sanity check:

  1. hadoop fs -ls hdfs://hdfs/ should show nothing for starters
  2. hadoop fs -put /path/to/src_file hdfs://hdfs/
  3. hadoop fs -ls hdfs://hdfs/ should now list src_file

Resource Reservation Instructions (Optional)

  1. In mesos-site.xml, change mesos.hdfs.role to hdfs.
  2. On master, add the role for HDFS, by running echo hdfs > /etc/mesos-master/role or by setting the โ€”-role=hdfs.
  3. Then restart the master by running sudo service mesos-master restart.
  4. On each slave where you want to reserve resources, add specific resource reservations for the HDFS role. Here is one example:
    cpus(*):4;cpus(hdfs):2;mem(*):8192;mem(hdfs):4096 > /etc/mesos-slave/resources
    or by setting โ€”-resources=cpus(*):4;cpus(hdfs):2;mem(*):8192;mem(hdfs):4096.
  5. On each slave with the new settings, stop the mesos slave by running
    sudo service mesos-slave stop
    .
  6. On each slave with the new settings, remove the old slave state by running
    rm -f /tmp/mesos/meta/slaves/latest
    .
    Note: This will also remove task state, so you will want to manually kill any running tasks as a precaution.
  7. On each slave with the new settings, start the mesos slave by running
    sudo service mesos-slave start
    .

hdfs's People

Contributors

abhay-agarwal avatar adam-mesos avatar brndnmtthws avatar elingg avatar lamdor avatar

Watchers

 avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.