Hadoop 管道程序抛出的 EOFException
首先，我是Hadoop的新手。
我有一个小型 Hadoop 管道程序，它会抛出 java.io.EOFException。该程序需要 作为输入一个小文本文件，并使用 hadoop.pipes.java.recordreader 和 hadoop.pipes.java.recordwriter。 输入非常简单，例如：
1 262144 42.8084 15.9157 4.1324 0.06 0.1
但是，Hadoop 会抛出 EOFException，我看不出原因。下面是 堆栈跟踪：
10/12/08 23:04:04 INFO mapred.JobClient: Running job: job_201012081252_0016
10/12/08 23:04:05 INFO mapred.JobClient: map 0% reduce 0%
10/12/08 23:04:16 INFO mapred.JobClient: Task Id : attempt_201012081252_0016_m_000000_0, Status : FAILED
java.io.IOException: pipe child exception
at org.apache.hadoop.mapred.pipes.Application.abort(Application.java:151)
at org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:101)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.io.EOFException
at java.io.DataInputStream.readByte(DataInputStream.java:267)
at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
at org.apache.hadoop.mapred.pipes.BinaryProtocol$UplinkReaderThread.run(BinaryProtocol.java:114)
顺便说一句，我在完全分布式模式（具有 3 个工作节点的集群）上运行了它。
任何帮助表示赞赏！谢谢
First of all, I am a newbie of Hadoop.
I have a small Hadoop pipes program that throws java.io.EOFException. The program takes
as input a small text file and uses hadoop.pipes.java.recordreader and hadoop.pipes.java.recordwriter.
The input is very simple like:
1 262144 42.8084 15.9157 4.1324 0.06 0.1
However, Hadoop will throw an EOFException, which I can't see the reason. Below is the
stack trace:
10/12/08 23:04:04 INFO mapred.JobClient: Running job: job_201012081252_0016
10/12/08 23:04:05 INFO mapred.JobClient: map 0% reduce 0%
10/12/08 23:04:16 INFO mapred.JobClient: Task Id : attempt_201012081252_0016_m_000000_0, Status : FAILED
java.io.IOException: pipe child exception
at org.apache.hadoop.mapred.pipes.Application.abort(Application.java:151)
at org.apache.hadoop.mapred.pipes.PipesMapRunner.run(PipesMapRunner.java:101)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.io.EOFException
at java.io.DataInputStream.readByte(DataInputStream.java:267)
at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
at org.apache.hadoop.mapred.pipes.BinaryProtocol$UplinkReaderThread.run(BinaryProtocol.java:114)
BTW, I ran this on a fully-distributed mode (a cluster with 3 work nodes).
Any help is appreciated! Thanks
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（2）
经验教训：无论如何，尽量确保你自己的程序中没有错误。
Lessons learned: by all means, try to make sure there is no bug in your own program.
此堆栈跟踪通常表明工作计算机中的可用文件描述符已用完。这是非常常见的，记录很少，这正是为什么我有两个 相关 问题关于这个问题。
如果您在所有计算机上都具有 root 访问权限，则应考虑通过编辑
/etc/sysctl.conf来提高 Hadoop 用户的文件描述符限制：或发出：
Ad infinitum。有关提高此限制的一般信息可在此处获取
不过，从长远规划的角度来看，这个策略有些虚假。如果您碰巧发现有关该问题的更多信息，也许您可以 帮助我帮助你帮助我们所有人？ [谢谢你，GLaDOS。 -Ed]
（编辑：请参阅下面的评论。）
This stacktrace is usually indicative of running out of available file descriptors within your worker machines. This is exceedingly common, documented sparsely, and precisely why I have two related questions on the subject.
If you have root access on all of the machines, you should consider raising the file descriptor limit for your Hadoop user by editing
/etc/sysctl.conf:Or issuing:
Ad infinitum. General information for raising this limit is available here.
However, from the perspective of long-term planning, this strategy is somewhat spurious. If you happen to discover more information on the problem, perhaps you can help me help you help us all? [Thank you, GLaDOS. -Ed]
(Edit: See commentary that follows.)