./build-images.sh

Note: Run below commands from the directory where docker-compose.yml file is present.

docker-compose up -d

docker-compose stop

docker-compose start

docker-compose rm -f

docker-compose scale spark-slave=n where n is the new number of containers.

• HDFS NameNode container
  • Runs HDFS NameNode and DataNode services
  • docker exec -it sparkdockercluster_hdfs-namenode_1 /bin/bash
• HDFS DataNode container(s)
  • Runs HDFS DataNode service
  • There could be multiple instances of this container. To connect to n'th container
    • docker exec -it sparkdockercluster_hdfs-datanode__n_ /bin/bash
• Spark Master container
  • Runs Spark Master and Worker services
  • docker exec -it sparkdockercluster_spark-master_1 /bin/bash
• Spark Worker container
  • Runs Spark Worker service
  • There could be multiple instances of this container. To connect to n'th container
    • docker exec -it sparkdockercluster_spark-slave__n_ /bin/bash