datanode变动本地block存储路径后出现以下问题

Question

2017-11-16 17:31:44,693 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Need to move 17.96 TB to make the cluster balanced.
2017-11-16 17:31:44,693 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Decided to move 10 GB bytes from 10.129.198.32:10010 to 10.129.196.24:10010
2017-11-16 17:31:44,693 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Decided to move 10 GB bytes from 10.129.198.33:10010 to 10.129.196.23:10010
2017-11-16 17:31:44,693 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Decided to move 10 GB bytes from 10.129.198.21:10010 to 10.129.196.27:10010
2017-11-16 17:31:44,693 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Decided to move 10 GB bytes from 10.129.198.30:10010 to 10.129.196.21:10010
2017-11-16 17:31:44,693 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Decided to move 10 GB bytes from 10.129.198.27:10010 to 10.129.196.26:10010
2017-11-16 17:31:44,694 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Decided to move 10 GB bytes from 10.129.198.35:10010 to 10.129.196.25:10010
2017-11-16 17:31:44,694 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Decided to move 10 GB bytes from 10.129.198.31:10010 to 10.129.196.22:10010
2017-11-16 17:31:44,694 INFO org.apache.hadoop.hdfs.server.balancer.Balancer: Will move 70 GB in this iteration
2017-11-16 17:31:44,759 WARN org.apache.hadoop.hdfs.server.balancer.Balancer: Exception while getting block list
org.apache.hadoop.ipc.RemoteException(java.lang.ArrayIndexOutOfBoundsException): java.lang.ArrayIndexOutOfBoundsException

        at org.apache.hadoop.ipc.Client.call(Client.java:1411)
        at org.apache.hadoop.ipc.Client.call(Client.java:1364)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
        at com.sun.proxy.$Proxy12.getBlocks(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.NamenodeProtocolTranslatorPB.getBlocks(NamenodeProtocolTranslatorPB.java:98)
        at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy13.getBlocks(Unknown Source)
        at org.apache.hadoop.hdfs.server.balancer.Balancer$Source.getBlockList(Balancer.java:684)
        at org.apache.hadoop.hdfs.server.balancer.Balancer$Source.dispatchBlocks(Balancer.java:810)
        at org.apache.hadoop.hdfs.server.balancer.Balancer$Source.access$1600(Balancer.java:640)
        at org.apache.hadoop.hdfs.server.balancer.Balancer$Source$BlockMoveDispatcher.run(Balancer.java:647)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)

 

麻烦大家帮忙看一下怎么回事，这是namenode报的错误信息。

Answer 1

应该是数据备份数不一致的原因，通过bin/hadoop fsck /查看很多文件备份数不足3份，然后将全部文件备份数设置为3份后问题解决，设置备份数命令bin/hadoop dfs -setrep -w 3 -R /