flume 往hdfs 里写失败

Question

2016-04-03 14:50:21,897 (hdfs-k1-call-runner-17) [ERROR - org.apache.flume.sink.hdfs.AbstractHDFSWriter.hflushOrSync(AbstractHDFSWriter.java:267)] Error while trying to hflushOrSync!
2016-04-03 14:50:22,240 (ResponseProcessor for block BP-379782447-10.215.1.51-1450951413112:blk_1074017796_277848) [WARN - org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:871)] Slow ReadProcessor read fields took 59999ms (threshold=30000ms); ack: seqno: -2 status: SUCCESS status: SUCCESS status: ERROR downstreamAckTimeNanos: 0, targets: [10.215.1.53:50010, 10.215.1.54:50010, 10.215.1.52:50010]
2016-04-03 14:50:22,240 (ResponseProcessor for block BP-379782447-10.215.1.51-1450951413112:blk_1074017796_277848) [WARN - org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:954)] DFSOutputStream ResponseProcessor exception  for block BP-379782447-10.215.1.51-1450951413112:blk_1074017796_277848
java.io.IOException: Bad response ERROR for block BP-379782447-10.215.1.51-1450951413112:blk_1074017796_277848 from datanode 10.215.1.52:50010
    at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:897)
2016-04-03 14:50:31,898 (SinkRunner-PollingRunner-DefaultSinkProcessor) [WARN - org.apache.flume.sink.hdfs.BucketWriter.close(BucketWriter.java:370)] failed to close() HDFSWriter for file (hdfs://10.215.1.51:8020/flume/amq/care/2016/04/03/event160403.1459666160825.log.tmp). Exception follows.
java.io.IOException: Callable timed out after 10000 ms on file: hdfs://10.215.1.51:8020/flume/amq/care/2016/04/03/event160403.1459666160825.log.tmp
    at org.apache.flume.sink.hdfs.BucketWriter.callWithTimeout(BucketWriter.java:693)
    at org.apache.flume.sink.hdfs.BucketWriter.close(BucketWriter.java:367)
    at org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:559)
    at org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:418)
    at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
    at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.TimeoutException
    at java.util.concurrent.FutureTask.get(FutureTask.java:205)
    at org.apache.flume.sink.hdfs.BucketWriter.callWithTimeout(BucketWriter.java:686)

Answer 1

可能是集群负载高导致操作超时，可以对集群做个监控。

可以适当调大 flume 中 hdfs.callTimeout 参数

参考：

• https://flume.apache.org/FlumeUserGuide....