Apache Spark in a Docker container, based on java:8. Heavily inspired by gettyimages/docker-spark. Cluster deployed with Docker Compose and orchestrated with Docker Swarm.

Official image for Cassandra is used and installed on the same nodes as Spark slaves.

In scripts/, run :

./bootstrap.sh

This script will set up a key-value store using Consul, a Swarm cluster with a master and a slave, and an overlay network for multi-host networking.

Connect to the Swarm master with:

eval $(docker-machine env --swarm swarm-master)

The cluster configuration should be visible by running:

docker info

Deploy the containers on constrained nodes with:

docker-compose --x-networking --x-network-driver=overlay up -d

In scripts/, run:

./deploy_cassandra.sh

After deployment is complete, you can run cqlsh this way:

docker run -it --rm --net container:cass1 cassandra:2.2.4 cqlsh

A folder /data is mounted as a shared volume outside of the container to save Cassandra data.

• Multi-host networking: https://docs.docker.com/engine/userguide/networking/get-started-overlay/
• Networking in Compose: https://docs.docker.com/compose/networking/