Code Monkey home page Code Monkey logo

runtime-compose's Introduction

Docker compose based hadoop/spark cluster

This repository contains docker-compose files to demonstrate usage of the flokkr docker images with simple docker-compose files.

To start the containers go to a subdirectory and start the containers with

docker-compose up -d

To scale services

docker-compose scale datanode=1

Attributes

Topic Solution
Configuration management
Source of config files: docker-compose external environment variable file
Configuration preprocessing: envtoconf (Convert environment variables to configuration formats
Automatic restart on config change: Not supported, docker-compose up is required
Provisioning and scheduling
Multihost support NO
Requirements on the hosts docker daemon and docker-compose
Definition of the containers per host N/A, one docker-compose file for the local host
Scheduling (find hosts with available resource) NO, localhost only
Failover on host crash NO
Scale up/down: Easy with docker-compose scale datanode=3
Multi tenancy (multiple cluster) Partial (from multiple checkout directory, after port adjustment)
Network
Network between containers dedicated network per docker-compose file
DNS YES, handled by the docker network
Service discovery NO (DNS based)
Data locality NO
Availability of the ports Published according to the docker-compose files

runtime-compose's People

Contributors

adoroszlai avatar elek avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

runtime-compose's Issues

Initial job has not accepted any resources

Hi!, Sorry I edited my previous issue because I didn't understand something correctly.
I'm trying to run the spark/yarn example using master yarn but I'm getting this error:
Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

In the history server if I go to executors while the job is running I only see this:

image

Am I doing something wrong?

Thanks.

[Spark] Spark history throw exception: java.lang.NoClassDefFoundError: org/apache/log4j/spi/Filter

Steps:

  1. go to /spark
  2. docker-compose up -d
  3. Obseve

Observe:

Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/log4j/spi/Filter
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.apache.log4j.spi.Filter
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 7 more

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.