为什么Mapreduce中完成的任务数量减少了？
在运行 hadoop 作业时，我注意到有时已完成的任务数量会减少，而取消的任务数量会增加。
这怎么可能？为什么会发生这种情况？
When running hadoop jobs, I noticed that sometimes the number of completed tasks decreases and number of canceled tasks increases.
How is this possible? Why does this happen?
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
我只在我们的集群处于奇怪状态时经历过这种情况，所以我不确定这是否是同一个问题。基本上，映射任务将完成，然后减速器将启动......然后映射器将被重新处理。
我认为问题在于映射器输出挂在该数据节点上等待减速器拾取它。如果该节点出现问题或死亡，JobTracker 会决定需要再次重新运行该任务，即使该任务已经完成。我们的问题是，我们的 NameNode 所在的系统存在一些与 Hadoop 无关的问题，一旦这些问题得到解决，它似乎就会消失。
抱歉，如果我的经验与您的问题无关。也许，您可以发布更多详细信息吗？您看到任何错误消息吗？您的 JobTracker 或 NameNode 日志中有什么奇怪的吗？
I've only experienced this when our cluster was in a strange state, so I'm not sure if this is the same issue. Basically, map tasks would complete, and then the reducers would start... and then mappers would be reprocessed.
I believe that the problem is that mapper output hangs around on that data node waiting for reducers to pick it up. If that node has issues or it dies, the JobTracker decides that it needs to rerun that task again, even if it had completed. Our issue was that the system our NameNode was on was having some non-Hadoop related issues and once those were resolves it seemed to go away.
Sorry if my experience was not relevant to your issue. Perhaps, can you post more details? Do you see any error messages? Is there anything weird in your JobTracker or NameNode logs?