Hadoop 数据节点、名称节点、辅助名称节点、作业跟踪器和任务跟踪器
我是hadoop新手，所以我有一些疑问。如果主节点发生故障，hadoop 集群会发生什么情况？我们能否在没有任何损失的情况下恢复该节点？是否可以让一个辅助主节点在当前主节点故障时自动切换到主节点？
我们有namenode（Secondary namenode）的备份，因此当它出现故障时，我们可以从Secondary namenode恢复namenode。这样，当datanode出现故障时，我们如何恢复datanode中的数据呢？ secondary namenode是namenode的备份，只是不备份到datenode，对吗？如果某个节点在作业完成之前发生故障，因此作业跟踪器中有待处理的作业，那么该作业是继续还是从空闲节点中的第一个作业重新启动？
如果发生什么情况，如何恢复整个集群的数据呢？
我的最后一个问题是，我们可以在 Mapreduce 中使用 C 程序吗（例如，mapreduce 中的冒泡排序）？
提前致谢
I am new in hadoop so I have some doubts. If the master-node fails what happened the hadoop cluster? Can we recover that node without any loss? Is it possible to keep a secondary master-node to switch automatically to the master when the current one fails?
We have the backup of the namenode (Secondary namenode), so we can restore the namenode from Secondary namenode when it fails. Like this, How can we restore the data's in datanode when the datanode fails? The secondary namenode is the backup of namenode only not to datenode, right? If a node is failed before completion of a job, so there is job pending in job tracker, is that job continue or restart from the first in the free node?
How can we restore the entire cluster data if anything happens?
And my final question, can we use C program in Mapreduce (For example, Bubble sort in mapreduce)?
Thanks in advance
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（3）
虽然现在回答你的问题已经太晚了，但它可能会帮助其他人。
首先让我向您介绍一下辅助名称节点：
现在来谈谈您的担忧..
如果主节点发生故障，hadoop 集群会发生什么？
我们可以在没有任何损失的情况下恢复该节点吗？
是否可以让一个辅助主节点在当前主节点故障时自动切换到主节点？
我们有namenode（Secondary namenode）的备份，因此当它失败时我们可以从Secondary namenode恢复namenode。像这样，当datanode出现故障时，我们如何恢复datanode中的数据？
辅助namenode只是namenode的备份，不备份到datenode，对吗？
如果节点在作业完成之前发生故障，因此作业跟踪器中有待处理的作业，该作业是继续还是从空闲节点中的第一个作业重新启动？
出现问题如何恢复整个集群数据？
我的最后一个问题，我们可以在 Mapreduce 中使用 C 程序（例如，mapreduce 中的冒泡排序）吗？
我刚刚尝试过。希望它能帮助您和其他人。
*欢迎提出建议/改进。*
Although, It is too late to answer your question but just It may help others..
First of all let me Introduce you with Secondary Name Node:
Now coming to your concerns..
If the master-node fails what happened the hadoop cluster?
Can we recover that node without any loss?
Is it possible to keep a secondary master-node to switch automatically to the master when the current one fails?
We have the backup of the namenode (Secondary namenode), so we can restore the namenode from Secondary namenode when it fails. Like this, How can we restore the data's in datanode when the datanode fails?
The secondary namenode is the backup of namenode only not to datenode, right?
If a node is failed before completion of a job, so there is job pending in job tracker, is that job continue or restart from the first in the free node?
How can we restore the entire cluster data if anything happens?
And my final question, can we use C program in Mapreduce (For example, Bubble sort in mapreduce)?
I Just gave a try. Hope it will help you as well as others.
*Suggestions/Improvements are welcome.*
目前hadoop集群存在单点故障，即namenode。
关于辅助节点问题（来自 apache wiki）：
有一些棘手的方法可以克服这个单点故障。如果您使用的是cloudera发行版，请使用此处解释的方法之一。 Mapr 发行版有一种不同的方式来处理这种欺骗。
最后，您可以使用每种编程语言通过 hadoop 流 编写 MapReduce。
Currently hadoop cluster has a single point of failure which is namenode.
And about the secondary node isssue (from apache wiki) :
There are tricky ways to overcome this single point of failure. If you are using cloudera distribution, one of the ways explained here. Mapr distribution has a different way to handle to this spof.
Finally, you can use every single programing language to write map reduce over hadoop streaming.
虽然，现在回答你的问题已经太晚了，但它可能会帮助其他人..首先我们将讨论 Hadoop 1.X 守护进程的角色，然后讨论你的问题..
1.辅助名称Node的作用是什么
它不完全是一个备份节点。它定期读取编辑日志并为名称节点创建更新的 fsimage 文件。它定期从名称节点获取元数据并保留它并在名称节点发生故障时使用。
2.名称节点的作用是什么
它是所有守护进程的管理者。它的主jvm进程在主节点上运行。它与数据节点交互。
3.工作跟踪器的作用是什么
它接受作业并分发给任务跟踪器以在数据节点进行处理。它被称为映射过程
4。任务跟踪器的作用是什么
它将执行为处理数据节点处的现有数据而提供的程序。该过程称为映射。
hadoop 1.X 的局限性
这是名称节点，因此我们可以为名称节点维护高质量的硬件。如果名称节点失败，一切都将无法访问
解决方案
单点故障的解决方案是hadoop 2.X，它提供了高可用性。
hadoop 2.X 的高可用性
现在是您的主题....
我们如何恢复整个集群数据如果发生什么情况？
如果集群失败，我们可以重新启动它。
如果节点在作业完成之前发生故障，因此作业跟踪器中有待处理的作业，该作业是继续还是从空闲节点中的第一个节点重新启动？
我们有默认的 3 个数据副本（我的意思是块）来获得高可用性，这取决于管理员设置了多少副本...因此作业跟踪器将继续使用其他数据节点上的其他数据副本
我们可以使用Mapreduce中的C程序（例如mapreduce中的冒泡排序）？
基本上，mapreduce 是执行引擎，它将以（存储加处理）分布式方式解决或处理大数据问题。我们正在使用 MapReduce 编程进行文件处理和所有其他基本操作，因此我们可以使用任何可以根据要求处理文件的语言。
hadoop 1.X架构
hadoop 1.x 有 4 个基本守护进程
我刚刚尝试过。希望它能帮助您以及其他人。
欢迎提出建议/改进。
Although, It is too late to answer your question but just It may help others..firstly we will discuss role of Hadoop 1.X daemons and then your issues..
1. What is role of secondary name Node
it is not exactly a backup node. it reads a edit logs and create updated fsimage file for name node periodically. it get metadata from name node periodically and keep it and uses when name node fails.
2. what is role of name node
it is manager of all daemons. its master jvm proceess which run at master node. it interact with data nodes.
3. what is role of job tracker
it accepts job and distributes to task trackers for processing at data nodes. its called as map process
4. what is role of task trackers
it will execute program provided for processing on existing data at data node. that process is called as map.
limitations of hadoop 1.X
which is name node so we can maintain high quality hardware for the name node. if name node fails everything will be inaccessible
Solutions
solution to single point of failure is hadoop 2.X which provides high availability.
high availability with hadoop 2.X
now your topics ....
How can we restore the entire cluster data if anything happens?
if cluster fails we can restart it..
If a node is failed before completion of a job, so there is job pending in job tracker, is that job continue or restart from the first in the free node?
we have default 3 replicas of data(i mean blocks) to get high availability it depends upon admin that how much replicas he has set...so job trackers will continue with other copy of data on other data node
can we use C program in Mapreduce (For example, Bubble sort in mapreduce)?
basically mapreduce is execution engine which will solve or process big data problem in(storage plus processing) distributed manners. we are doing file handling and all other basic operations using mapreduce programming so we can use any language of where we can handle files as per the requirements.
hadoop 1.X architecture
hadoop 1.x has 4 basic daemons
I Just gave a try. Hope it will help you as well as others.
Suggestions/Improvements are welcome.