hadoop's People
Forkers
baifendian typhoonzero yancey1989 wsxzhmzwdfxeds lupan2015 lapterchow anuragtomar hexiaofeng zbaolong view-sonic switchiihadoop's Issues
Is it possible to run HDFS in Docker?
Is it possible for us to have Ceph on Kubernetes in addition to HDFS
with init script, pod can not connect the service.
Env:Vagrant + coreos * 2
K8s version: 1.2.0
CoreOS version: Linux version 4.6.0-coreos ([email protected]) (gcc version 4.9.3 (Gentoo Hardened 4.9.3 p1.3, pie-0.6.3) ) #2 SMP Fri Jun 3 23:44:22 UTC 2016
- svc info
yancey@ yancey-macbook k8s$kubectl get svc
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hadoop-zookeeper 10.0.0.120 <none> 2181/TCP 1h
kubernetes 10.0.0.1 <none> 443/TCP 1h
- my pod can not telnet to the kubernets service
yancey@ yancey-macbook k8s$kubectl exec hadoop-zookeeper-x96rj telnet 10.0.0.1 443
telnet: can't connect to remote host (10.0.0.1): Connection refused
error: error executing remote command: Error executing command in container: Error executing in Docker Container: 1
- ifconfig && iptables info
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 10.1.4.1 netmask 255.255.255.0 broadcast 0.0.0.0
inet6 fe80::42:bcff:fef3:35ab prefixlen 64 scopeid 0x20<link>
ether 02:42:bc:f3:35:ab txqueuelen 0 (Ethernet)
RX packets 159 bytes 12380 (12.0 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 21 bytes 1242 (1.2 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.0.2.15 netmask 255.255.255.0 broadcast 10.0.2.255
inet6 fe80::a00:27ff:fea7:5e9e prefixlen 64 scopeid 0x20<link>
ether 08:00:27:a7:5e:9e txqueuelen 1000 (Ethernet)
RX packets 97026 bytes 76268827 (72.7 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 45284 bytes 3009253 (2.8 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.17.8.102 netmask 255.255.255.0 broadcast 172.17.8.255
inet6 fe80::a00:27ff:feda:478f prefixlen 64 scopeid 0x20<link>
ether 08:00:27:da:47:8f txqueuelen 1000 (Ethernet)
RX packets 39909 bytes 4961258 (4.7 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 40458 bytes 4070934 (3.8 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet 10.1.4.0 netmask 255.255.0.0 broadcast 0.0.0.0
inet6 fe80::b3:c3ff:fe13:4d94 prefixlen 64 scopeid 0x20<link>
ether 02:b3:c3:13:4d:94 txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 8 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1 (Local Loopback)
RX packets 2360 bytes 122960 (120.0 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2360 bytes 122960 (120.0 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
veth054ea6f: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450
inet6 fe80::60c5:d4ff:fe2e:b234 prefixlen 64 scopeid 0x20<link>
ether 62:c5:d4:2e:b2:34 txqueuelen 0 (Ethernet)
RX packets 159 bytes 14606 (14.2 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 29 bytes 1890 (1.8 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
core-v2-02 init # iptables-save -t nat
...
-A KUBE-SEP-OGNOLD2JUSLFPOMZ -s 10.0.2.15/32 -m comment --comment "default/kubernetes:https" -j KUBE-MARK-MASQ
-A KUBE-SEP-OGNOLD2JUSLFPOMZ -p tcp -m comment --comment "default/kubernetes:https" -m tcp -j DNAT --to-destination 10.0.2.15:6443
-A KUBE-SERVICES -d 10.0.0.1/32 -p tcp -m comment --comment "default/kubernetes:https cluster IP" -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES -m comment --comment "kubernetes service nodeports; NOTE: this must be the last rule in this chain" -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
...
- I think the address in iptables 10.0.2.15 should be insted of eth1(172.17.8.102), but I can not find the solution.
RM can not find the host of hadoop-slave-xxx
I'm tring to run the example WordCount, but the app state alwasy ACCEPTED.
In RM log:
16/05/08 15:56:36 ERROR scheduler.SchedulerApplicationAttempt: Error trying to assign container token and NM token to an allocated container container_1462716844418_0001_01_000001
java.lang.IllegalArgumentException: java.net.UnknownHostException: hadoop-slave-qjzbu
at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:374)
at org.apache.hadoop.yarn.server.utils.BuilderUtils.newContainerToken(BuilderUtils.java:256)
at org.apache.hadoop.yarn.server.resourcemanager.security.RMContainerTokenSecretManager.createContainerToken(RMContainerTokenSecretManager.java:220)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt.pullNewlyAllocatedContainersAndNMTokens(SchedulerApplicationAttempt.java:448)
at org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler.allocate(FairScheduler.java:921)
It looks like RM node can not connect the pod of NM node,such as hadoop-slave-xxxx.
So my question is :
I can use dns service with the component name such as hadoop-master, but when the hadoop-master want to connect the specific pod, such as hadoop-slave-xxx, it can not find the ip.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.