Flume 自动扩展和故障转移
我的公司正在考虑使用 Flume 进行一些相当大容量的日志处理。我们认为，出于容量（可扩展性）和故障转移（可靠性）的原因，日志处理需要分布式，而 Flume 似乎是显而易见的选择。
然而，我们认为我们一定错过了一些明显的东西，因为我们没有看到 Flume 如何提供自动可扩展性和故障转移。
我想定义一个流程，对每个日志行执行 A 件事，然后传递它并执行 B 件事，然后传递它并执行 C 件事，依此类推，这似乎与 Flume 非常匹配。然而，我希望能够用纯粹的逻辑术语来定义这个流，然后基本上说，“嘿 Flume，这里是服务器，这里是流定义，开始工作吧！”。服务器将会死亡（并且操作将重新启动它们），我们将向集群添加服务器，并淘汰其他服务器，而 Flume 只会将工作定向到具有可用容量的任何节点。
这个描述是 Hadoop Map-Reduce 如何实现可扩展性和故障转移的，我假设 Flume 也是一样的。但是，文档似乎暗示我需要手动配置每个逻辑节点在哪些物理服务器上运行，并为每个节点配置特定的故障转移场景。
我是对的吗？Flume 不符合我们的目的，还是我错过了什么？
感谢您的帮助。
My company is considering using flume for some fairly high volume log processing. We believe that the log processing needs to be distributed, both for volume (scalability) and failover (reliability) reasons, and Flume seems the obvious choice.
However, we think we must be missing something obvious, because we don't see how Flume provides automatic scalability and failover.
I want to define a flow that says for each log line, do thing A, then pass it along and do thing B, then pass it along and do thing C, and so on, which seems to match well with Flume. However, I want to be able to define this flow in purely logical terms, and then basically say, "Hey Flume, here are the servers, here is the flow definition, go to work!". Servers will die, (and ops will restart them), we will add servers to the cluster, and retire others, and flume will just direct the work to whatever nodes have available capacity.
This description is how Hadoop map-reduce implements scalability and failover, and I assumed that Flume would be the same. However, the documentation sees to imply that I need to manually configure which physical servers each logical node runs on, and configure specific failover scenarios for each node.
Am I right, and Flume does not serve our purpose, or did I miss something?
Thanks for your help.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
根据您是否使用多个主服务器，您可以对配置进行编码以遵循故障转移模式。
指南中对此进行了相当详细的说明： http://archive.cloudera.com/cdh/ 3/flume/UserGuide/index.html#_automatic_failover_chains
回答你的问题，坦率地说，Flume还不具备自动找出故障转移方案的能力。
Depending on whether you are using multiple masters, you can code your configuration to follow a failover pattern.
This is fairly detailed in the guide: http://archive.cloudera.com/cdh/3/flume/UserGuide/index.html#_automatic_failover_chains
To answer your question, bluntly, Flume does not yet have an ability to figure out a failover scheme automatically.