Hadoop集群问题dataNode成功启动却没有出现在web页面中?

Question

描述

• centos7
• hadoop-2.6.0-cdh5.15.1

•  #三台机器的ip和主机名 hserver1为nameNode
   192.168.107.134  hserver1
   192.168.107.108  hserver2
   192.168.107.133  hserver3

搭建hadoop集群后start-all.sh启动, 启动正常, jps正常
查看 http://hserver1:50070/dfshealth.html#tab-datanode
只显示了一个dataNode :

但是查看http://hserver1:8088/cluster/nodes却显示了两个hserver2:

当我执行MapReduce程序的时候问题出现了, 报错信息如下:

20/08/02 11:31:22 INFO mapreduce.Job: Task Id : attempt_1596337818851_0001_m_000000_2, Status : FAILED
Container launch failed for container_1596337818851_0001_01_000004 : java.net.ConnectException: Call From dockerhost/192.168.107.108 to hserver2:39133 failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
    at sun.reflect.GeneratedConstructorAccessor24.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791)
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:731)
    at org.apache.hadoop.ipc.Client.call(Client.java:1508)
    at org.apache.hadoop.ipc.Client.call(Client.java:1441)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
    at com.sun.proxy.$Proxy38.startContainers(Unknown Source)
    at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96)
    at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:258)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
    at com.sun.proxy.$Proxy39.startContainers(Unknown Source)
    at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:155)
    at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:379)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: 拒绝连接
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
    at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:494)
    at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:648)
    at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:744)
    at org.apache.hadoop.ipc.Client$Connection.access$3000(Client.java:396)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1557)
    at org.apache.hadoop.ipc.Client.call(Client.java:1480)
    ... 15 more

尝试

• 查看三个node的VERSION文件, 看起来没什么问题

# hserver1 Sun Aug 02 09:23:06 CST 2020
namespaceID=201224422 
clusterID=CID-932abe94-03c1-42a2-86d4-57a1a225dfa1 
cTime=0 
storageType=NAME\_NODE 
blockpoolID=BP-283975175-192.168.107.134-1595663794528 
layoutVersion=-60

# hserver2 Sun Aug 02 09:23:07 CST 2020
storageID=DS-0936e18c-86bb-4d9b-b8ef-e39e44e2ffe5
clusterID=CID-932abe94-03c1-42a2-86d4-57a1a225dfa1
cTime=0
datanodeUuid=8492b66c-198d-42f0-a253-02b0c42d1918
storageType=DATA_NODE
layoutVersion=-56

# hserver3 Sun Aug 02 09:23:07 CST 2020
storageID=DS-877f0c92-46bc-4644-b5cb-9c5fd1575468
clusterID=CID-932abe94-03c1-42a2-86d4-57a1a225dfa1
cTime=0
datanodeUuid=b87d9bd5-46e0-42f5-8c5c-ccd0362bd156
storageType=DATA_NODE
layoutVersion=-56

• netstat -ntlp 查看9000端口, 是存在的

[root@hserver1 Jar]# netstat -ntlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 192.168.107.134:9000    0.0.0.0:*               LISTEN      12517/java          
tcp        0      0 0.0.0.0:50090           0.0.0.0:*               LISTEN      12676/java          
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      1/systemd           
tcp        0      0 0.0.0.0:6000            0.0.0.0:*               LISTEN      7412/X              
tcp        0      0 192.168.122.1:53        0.0.0.0:*               LISTEN      7208/dnsmasq        
tcp        0      0 0.0.0.0:50070           0.0.0.0:*               LISTEN      12517/java          
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      6738/sshd           
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      6744/cupsd          
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      7046/master         
tcp6       0      0 :::3306                 :::*                    LISTEN      7543/mysqld         
tcp6       0      0 :::111                  :::*                    LISTEN      1/systemd           
tcp6       0      0 :::6000                 :::*                    LISTEN      7412/X              
tcp6       0      0 :::22                   :::*                    LISTEN      6738/sshd           
tcp6       0      0 ::1:631                 :::*                    LISTEN      6744/cupsd          
tcp6       0      0 192.168.107.134:8088    :::*                    LISTEN      12857/java          
tcp6       0      0 ::1:25                  :::*                    LISTEN      7046/master         
tcp6       0      0 192.168.107.134:8030    :::*                    LISTEN      12857/java          
tcp6       0      0 192.168.107.134:8031    :::*                    LISTEN      12857/java          
tcp6       0      0 192.168.107.134:8032    :::*                    LISTEN      12857/java          
tcp6       0      0 192.168.107.134:8033    :::*                    LISTEN      12857/java          
tcp6       0      0 :::2181                 :::*                    LISTEN      7410/docker-proxy-c 

• 三台机器也是可以互相ping通的

所以求指点啊, 多谢!