Flume-ng HDFS IO error, Callable timed out 예외 발생

이 두 대의flume-ng은 밤 9시~11시 flume에 이상이 발생했습니다.

25 Mar 2014 22:18:25,189 ERROR [hdfs-thrift_hdfsSink-roll-timer-0] (org.apache.flume.sink.hdfs.BucketWriter$2.call:257)  - Unexpected error
java.io.IOException: Callable timed out after 10000 ms on file: /logdata/2014/03/25/software_log/197a.1395757003934.tmp
     at org.apache.flume.sink.hdfs.BucketWriter.callWithTimeout(BucketWriter.java:550)
     at org.apache.flume.sink.hdfs.BucketWriter.doFlush(BucketWriter.java:353)
     at org.apache.flume.sink.hdfs.BucketWriter.flush(BucketWriter.java:319)
     at org.apache.flume.sink.hdfs.BucketWriter.close(BucketWriter.java:277)
     at org.apache.flume.sink.hdfs.BucketWriter$2.call(BucketWriter.java:255)
     at org.apache.flume.sink.hdfs.BucketWriter$2.call(BucketWriter.java:250)
     at java.util.concurrent.FutureTask.run(FutureTask.java:262)
     at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
     at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
     at java.lang.Thread.run(Thread.java:724)
Caused by: java.util.concurrent.TimeoutException
     at java.util.concurrent.FutureTask.get(FutureTask.java:201)
     at org.apache.flume.sink.hdfs.BucketWriter.callWithTimeout(BucketWriter.java:543)
     ... 11 more
25 Mar 2014 22:34:17,639 WARN  [ResponseProcessor for block BP-928773537-10.31.246.10-1392969615809:blk_-6973900543529394933_175021] (org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run:747)  - DFSOutputStream ResponseProcessor exception  for block BP-928773537-10.31.246.10-1392969615809:blk_-6973900543529394933_175021
java.io.EOFException: Premature EOF: no length prefix available
     at org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:171)
     at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:114)
     at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:694)

로그 피크 보기

18시부터 23시까지 최고조, hadoop 집단 백메가 대역폭을 뚜렷하게 볼 수 있으며, 로그 쓰기 최고조에 도달할 때 대역폭 상한선에 도달한다.hadoop 쪽에 아직 감시 도구를 배치하지 않았습니다 (-...-)

현재 솔루션:

근거http://blog.csdn.net/yangbutao/article/details/8845025,flumehdfs 클라이언트 시간 초과 수정..
hdfs를 수정합니다.call Timeout은 기본 10초에서 40초로 변경됩니다
전재: 에서http://blog.csdn.net/wsscy2004