Hadoop：提交作业后客户端的 CPU 负载较高
在筛选一些 Hadoop 指南时，我找不到问题的答案：我通过客户端计算机上的 shell 脚本一次性提交各种 Hadoop 作业（最多 200 个）。每个作业都通过 JAR（相当大；大约 150 MB）启动。提交作业后，客户端计算机的 CPU 负载非常高（每个核心都为 100%），并且 RAM 很快就满了。这样，客户端就不再可用。我认为每个作业的计算完全在Hadoop框架内完成，作业运行时集群和客户端之间只交换一些状态信息。
那么，为什么客户会完全紧张呢？我是否以错误的方式提交 Hadoop 作业？每个 JAR 是否太大？
提前致谢。
I couldn't find an answer to my issue while sifting through some Hadoop guides: I am committing various Hadoop jobs (up to 200) in one go via a shell script on a client computer. Each job is started by means of a JAR (which is quite large; approx. 150 MB). Right after submitting the jobs, the client machine has a very high CPU load (each core on 100%) and the RAM is getting full quite fast. That way, the client is no longer usable. I thought that the computation of each job is entirely done within the Hadoop framework, and only some status information is exchanged between the cluster and the client while the job is running.
So, why is the client fully stretched? Am I committing Hadoop jobs the wrong way? Is each JAR too big?
Thanks in advance.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
这与罐子无关。客户端正在计算
InputSplits。因此，当每个作业有大量输入文件时，客户端计算机可能会承受大量负载。
但我猜当提交 200 个作业时，jobtracker 上的 RPC 处理程序会出现一些问题。 jobtracker 上有多少个 RPC 处理程序处于活动状态？
不管怎样，我会一次批量提交最多 10 或 20 个作业，然后等待它们完成。我猜你有默认的 FIFO 调度程序？因此，您也不会从一次提交全部 200 个作业中受益。
It is not about the jar. The client side is calculating the
InputSplits.So it can be possible that when having large number of input files for each job the client machine gets a lot of load.
But I guess when submitting 200 jobs the RPC Handler on the jobtracker has some problems. How many RPC handlers are active on the jobtracker?
Anyways, I would batch the submission up to 10 or 20 jobs at a time and wait for their completion. I guess you're having the default FIFO scheduler? So you won't benefit from submitting all 200 jobs at a time either.