Version 2.0.0 introduces uses wait_for_it script for the cluster startup
It is recommended to mount a local directory to the namenode to make working inside the docker container easier. You'll be able to learn about it here.
To deploy an example HDFS cluster, run:
docker-compose up -d
Run example wordcount job:
make wordcount
Or deploy in swarm:
docker stack deploy -c docker-compose-v3.yml hadoop
docker-compose
creates a docker network that can be found by running docker network list
, e.g. dockerhadoop_default
.
Run docker network inspect
on the network (e.g. dockerhadoop_default
) to find the IP the hadoop interfaces are published on. Access these interfaces with the following URLs:
- Namenode: http://<dockerhadoop_IP_address>:9870/dfshealth.html#tab-overview
- History server: http://<dockerhadoop_IP_address>:8188/applicationhistory
- Datanode: http://<dockerhadoop_IP_address>:9864/
- Nodemanager: http://<dockerhadoop_IP_address>:8042/node
- Resource manager: http://<dockerhadoop_IP_address>:8088/
Once deployed, run docker ps
and ensure that the containers are healthy. If they appear to be unhealthy, you can read the logs by using the command docker logs <container-id>
.
To exec into the namenode, run docker exec -it namenode /bin/bash
- "Library initialization failed - unable to allocate file descriptor table" (often resulting in namenode restarting) You can follow the instructions present here
- Resource manager keeps restarting because the namenode is in safemode
- Exec into the namenode and run
hdfs dfsadmin -safemode leave
- Then, run
hdfs fsck -delete
- Exec into the namenode and run
The configuration parameters can be specified in the hadoop.env file or as environmental variables for specific services (e.g. namenode, datanode etc.):
CORE_CONF_fs_defaultFS=hdfs://namenode:8020
CORE_CONF corresponds to core-site.xml. fs_defaultFS=hdfs://namenode:8020 will be transformed into:
<property><name>fs.defaultFS</name><value>hdfs://namenode:8020</value></property>
To define dash inside a configuration parameter, use triple underscore, such as YARN_CONF_yarn_log___aggregation___enable=true (yarn-site.xml):
<property><name>yarn.log-aggregation-enable</name><value>true</value></property>
The available configurations are:
- /etc/hadoop/core-site.xml CORE_CONF
- /etc/hadoop/hdfs-site.xml HDFS_CONF
- /etc/hadoop/yarn-site.xml YARN_CONF
- /etc/hadoop/httpfs-site.xml HTTPFS_CONF
- /etc/hadoop/kms-site.xml KMS_CONF
- /etc/hadoop/mapred-site.xml MAPRED_CONF
If you need to extend some other configuration file, refer to base/entrypoint.sh bash script.