EC2 上 Hadoop 集群中的按需从属生成
我计划在 EC2 上使用 Hadoop。由于我们必须为每个实例的使用付费，因此拥有比作业实际需要的固定数量的实例并不好。
在我们的应用程序中，许多作业是并发执行的，并且我们始终不知道从属要求。是否可以用最少的从站启动 hadoop 集群，然后根据需求管理可用性？
即按需创建/销毁从
站子问题：hadoop 集群可以同时管理多个作业吗？
谢谢
I am planning to use Hadoop on EC2. Since we have to pay per instance usage, it is not good to have fixed number of instances than what are actually required for the job.
In our application, many jobs are executed concurrently and we do not know the slave requirement all the time. Is it possible to start the hadoop cluster with minimum slaves and later on manage the availability based on requirement?
i.e. create/destroy slaves on demand
Sub question: Can hadoop cluster manage multiple jobs concurrently?
Thanks
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（3）
hadoop 中使用的默认调度程序是一个简单的 FIFO 调度程序，您可以考虑使用 FairScheduler，它将集群的份额分配给每个正在运行的作业，并具有广泛的配置来控制这些份额。
就 EC2 而言 - 您可以轻松地从一定数量的节点开始，然后一旦您发现队列中有太多任务并且集群中的所有插槽都被占用 - 添加更多节点。您只需启动一个实例并在其上启动一个任务跟踪器，该任务跟踪器将向作业跟踪器注册。
但是，您必须拥有自己的系统来管理这些节点的启动和关闭。
The default scheduler that is used in hadoop is a simple FIFO one, you can look into using FairScheduler which assigns a share of the cluster to each of the running jobs and has extensive configuration to control those shares.
As far as EC2 is concerned - you can easily start of with some number of nodes and then once you see that there are too many tasks in the queue and all the slots in the cluster are occupied - add more of them. You will simply have to start up an instance and launch a task tracker on it that will register with the jobtracker.
However you will have to have your own system that will manage startup and shutdown of these nodes.
这看起来很有希望
http://hadoop.apache.org/common/docs/r0。 17.1/hod.html
This seems promising
http://hadoop.apache.org/common/docs/r0.17.1/hod.html
只是想让您知道，我们正在 Apache Whirr 中对此进行一些工作。我们正在跟踪 WHIRR-214 的进展。投票或加入开发。 :)
Just want to let you know that we are doing some work on this in Apache Whirr. We are tracking progress in WHIRR-214. Vote or join development. :)