为什么hadoop中有很多超时错误？

Question

为什么hadoop的datanode节点中有很多如下如错误：

ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(192.168.10.100:50010, storageID=DS-406166940-192.168.10.100-50010-1373101
404581, infoPort=50075, ipcPort=50020):DataXceiver
java.net.SocketTimeoutException: 480000 millis timeout while waiting for channel to be ready for write. ch : java.nio.channels.SocketChannel[connected local=/192.168.10.100:50010 remote=/192.168.10.100:57748]
        at org.apache.hadoop.net.SocketIOWithTimeout.waitForIO(SocketIOWithTimeout.java:246)
        at org.apache.hadoop.net.SocketOutputStream.waitForWritable(SocketOutputStream.java:159)
        at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:198)
        at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendChunks(BlockSender.java:350)
        at org.apache.hadoop.hdfs.server.datanode.BlockSender.sendBlock(BlockSender.java:436)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:197)
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:99)
        at java.lang.Thread.run(Thread.java:662)

查询了之后说是因为上层客户端(上层用了hbase)查询慢导致的

另外日志中还有很多版本不匹配错误，这个不知道是怎么触发的？

ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(192.168.10.100:50010, storageID=DS-406166940-192.168.10.100-50010-1373101
404581, infoPort=50075, ipcPort=50020):DataXceiver
java.io.IOException: Version Mismatch
        at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:85)
        at java.lang.Thread.run(Thread.java:662)

上层的hbase则有不少连接被重置的错误

WARN org.apache.hadoop.ipc.HBaseServer: IPC Server listener on 60020: readAndProcess threw exception java.io.IOException: Connection reset by p
eer. Count of bytes read: 0
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcher.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
        at sun.nio.ch.IOUtil.read(IOUtil.java:171)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:245)
        at org.apache.hadoop.hbase.ipc.HBaseServer.channelRead(HBaseServer.java:1796)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:1179)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:748)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:539)
        at org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:514)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
        at java.lang.Thread.run(Thread.java:662)

那么最终原因是不是hadoop的datanode导致的呢？

datanode的读取超时是在什么情况下触发的呢？

Answer 1

其他节点也有，当时正在做合并，会不会有影响？

Answer 2

单节点的网络问题吧？其他节点同样有这问题？