消息队列是高带宽数据馈送的正确策略吗？
我有一个庞大的数据收集服务器网络，可以生成大量实时数据。
过去，我为合作伙伴提供了使用 HTTP GET 近乎实时获取此数据的能力。但出于多种原因，我渴望放弃这个。
所以是的...我渴望构建一个新的分发系统，并且我认为消息队列系统是可行的方法。
我需要能够将数据从我的来源分发给许多不同的合作伙伴。有些合作伙伴会收到全部，而其他合作伙伴则只收到一部分。而且，如果合作伙伴断开连接，他们需要能够重新连接并且不会丢失任何数据。 （尽管，出于磁盘和内存的考虑，我希望它们的排队消息在一小时左右后过期）
最后，我需要系统能够每分钟处理数万个队列。
您认为消息队列是一个合适的方案吗？
我正在考虑使用 RabbitMQ。维护起来困难吗？
非常感谢！
-Z
I have a huge network of data-collection servers which generate a large volume of real-time data.
In the past I've provided partners with the ability to get this data in near-real-time using HTTP GET's. But for many reasons I'm eager to ditch this.
So yeah... I'm eager to build out a new distribution system and I was thinking that a Message Queuing System was the way to go.
I need to be able to distribute data from my sources to a number of different partners. Some partners receive all of it, others just get a portion. And, if a partner gets disconnected, they need to be able to reconnect and not miss any data. (Although, for the sake of disk and memory I'd like their queued messages to expire after hour or so)
Lastly I need the system to be able to handle tens of thousands of enqueue's per minute.
Do you think Message Queuing is an appropriate scheme?
I was looking at using RabbitMQ. Is it difficult to maintain?
Thanks Very Much!
-Z
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
我无法告诉您这对于您的具体情况是否是正确的策略，但消息产品确实每天都在高消息率系统中使用。
投资界的大部分人都使用各种商业产品（Tibco ）和开源（ZeroMQ）仅举两例，用于处理来自交易所和其他来源的市场数据。这些可能至少与您的数据传感器一样活跃。
发布/订阅模型（其中一些接收者想要一些消息，一些接收者想要所有消息）以及延迟加入或其他所谓的保证消息传递确实是大多数设备的标准功能这些产品。
所以请继续研究产品，我自己没有使用过 RabbitMQ ，所以不能具体评论它，但是一个最小的抽象层，您应该能够将自己与太多特定于平台的调用隔离开来，因此允许您在需要时交换消息总线实现者。 （您甚至可能想要构建这样的垫片作为概念验证的一部分，以测试用于您的特定目的的多个产品。您可以获得多种产品的经验，充实外观层，并加快速度产品）
祝你好运
I cannot tell you if it is the right strategy in your specific case, but message products are indeed used in high message rate systems every day.
Much of the investment world uses various products, both commercial (Tibco) and Open source (ZeroMQ) to name just two, to handle market data from exchanges and other sources. These are likely at least as active as your data sensors are.
The publish/subscribe model, where some receivers want some messages and some receivers want all, along with late-join or other so-called guaranteed messaging are indeed standard features on most of these products.
So do go ahead and investigate products, I have not used RabbitMQ myself, so cannot comment on it specifically, however with a minimal abstraction layer, you should be able to insulate yourself from too many platform specific calls, and therefore allow you to swap message-bus implementers if the need arises. (You may even want to build such a shim as part of a proof-of-concept to test out more than one product for your specific purpose. You get experience in multiple products, flesh out the facade layer, and get up to speed on the products)
Good Luck