如何实施消息队列解决方案
我有一个场景，大约 10 条不同的消息需要排队然后出队/处理。一个订阅者需要全部 10 条消息，而另一个订阅者只需要 10 条消息中的 8 条。我试图了解设置这种类型的架构的最佳方法是什么。您是否为每种消息类型创建一个队列，以便订阅者可以订阅相关队列，或者是否将它们全部转储到同一队列并忽略与该订阅者不相关的消息？我想确保解决方案灵活/可扩展等。
流程：
- 10 条不同的 xml 消息将排队到 IBM WebSphere MQ 服务器。
- 我们将使用 .Net（最有可能是 WCF，因为 WebSphere MQ 7.1 添加了 WCF 支持）。
- 我们将使消息出队并将它们加载到另一个后端数据库（最有可能是 SQL Server）。
- 解决方案需要很好地扩展，因为我们将处理大量消息，并且消息数量可能会增长（可能为 40-50,000 条/小时）。至少对我们来说数额很大。
一如既往地非常感谢这些信息。
--S
I have a scenario where about 10 different messages will need to be enqueued and then dequeued / processed. One subscriber will need all 10 messages, but another will only need 8 of the 10 messages. I am trying to understand what the best way is to setup this type of architecture. Do you create a queue for each message type so the subscriber(s) can just subscribe to the relevant queues or do you dump them all to the same queue and ignore the messages that are not relevant to that subscriber? I want to ensure the solution is flexible / scalable, etc.
Process:
- 10 different xml messages will be enqueued to an IBM WebSphere MQ server.
- We will use .Net (Most likely WCF since WebSphere MQ 7.1 has added in WCF support)
- We will dequeue the messages and load them into another backend DB (Most likely SQL Server).
- Solution needs to scale well because we will be processing a very large number of messages and this could grow (Probably 40-50,000 / hr). At least large amount for us.
As always greatly appreciate the info.
--S
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（2）
从资源角度来看，创建队列相对“便宜”，而且是的，最好为每个特定目的使用队列，因此在这种情况下，如果可能的话，最好按目标客户端将它们分开。使用队列根据某些条件（相关 ID 或其他一些东西）有选择地提取消息通常是一个坏主意。消息传递中性能最佳的场景是最简单的一种：只需在消息到达时从队列中提取消息，而不是有选择地查看和接收。
至于扩展性，我不能说 Websphere MQ 或其他 IBM 产品，但每小时 40-50K 消息对于 Windows Server 上的 MSMQ 来说并不是特别难处理，所以我认为 IBM 也可以做到这一点。通常瓶颈不是排队平台本身，而是出队和处理单个消息的过程。
Creating queues is relatively 'cheap' from a resource perspective, plus yes, it's better to use a queue for each specific purpose, so it's probably better in this case to separate them by target client if possible. Using a queue to pull messages selectively based on some criteria (correlation ID or some other thing) is usually a bad idea. The best performing scenario in messaging is the most straightforward one: simply pull messages from the queue as they arrive, rather than peeking and receiving selectively.
As to scaling, I can't speak for Websphere MQ or other IBM products, but 40-50K messages per hour isn't particularly hard for MSMQ on Windows Server to handle, so I'd assume IBM can do that as well. Usually the bottleneck isn't the queuing platform itself but rather the process of dequeuing and processing individual messages.
好的，根据评论，这里有一个可扩展的建议，不需要对应用程序进行太多更改。
在生产者方面，我将消息选择标准复制到消息属性，然后将消息发布到主题。应用程序所需的唯一更改是消息属性。如果由于某种原因您不想使用本机功能发布它，您可以为主题定义别名。该应用程序认为它正在发送消息，但它们实际上是出版物。
在消费者方面，你有几个选择。一种是为每个应用程序创建管理订阅并在订阅中使用选择器。然后，根据选择标准，消息将被输送到每个消费者的专用队列。这些应用程序认为它们只是在消费消息。
或者，应用程序可以简单地订阅该主题。这使您可以选择动态订阅，当应用程序断开连接时不接收消息（如果您实际上想要这样做）或持久订阅，其功能相当于管理订阅。
该解决方案可以轻松扩展到您引用的数量。另一种选择是生产者不使用属性。在这里，消费者应用程序使用所有消息，打开每个消息的消息负载，并决定是处理还是忽略该消息。在此解决方案中，生产者仍然发布到某个主题。任何涉及直接排队的解决方案都会迫使生产者知道所有目的地。添加另一个消费者，更改生产者。此外，每个目的地都有一个 PUT。
最糟糕的情况是生产者放置多条消息，而消费者必须读取每条消息来决定是否将其忽略。该选项可能会出现扩展问题，具体取决于选择标准字段在有效负载中的深度。非常长的 XPath 表达式 = 性能很差，并且无法调整 WMQ 来弥补它，因为此时延迟全部在应用程序中。
最好的情况是，生产者设置消息属性并发布。消费者可以在订阅中选择房产，或者管理订阅可以为他们完成此操作。就可伸缩性而言，该解决方案使用应用程序订阅还是管理订阅都没有任何区别。
OK, based on the comments, here's a suggestion that will scale and doesn't require much change on the apps.
On the producer side, I'd copy the message selection criteria to a message property and then publish the message to a topic. The only change that is required here to the app is the message property. If for some reason you don't want to make it publish using the native functionality, you can define an alias over a topic. The app thinks it is sending messages but they are really publications.
On the consumer side you have a couple of choices. One is to create administrative subscriptions for each app and use a selector in the subscription. The messages are then funneled to a dedicated queue per consumer, based on the selection criteria. The apps think that they are simply consuming messages.
Alternatively the app can simply subscribe to the topic. This gives you the option of a dynamic subscription that doesn't receive messages when the app is disconnected (if in fact you wanted that) or a durable subscription that is functionally equivalent to the administrative subscription.
This solution will easily scale to the volumes you cited. Another option is that the producer doesn't use properties. Here, the consumer application consumes all messages, breaks open the message payload on each and decides whether to process or ignore the message. In this solution the producer is still publishing to a topic. Any solution involving straight queueing forces the producer to know all the destinations. Add another consumer, change the producer. Also, there's a PUT for each destination.
The worst case is a producer putting multiple messages and a consumer having to read each one to decide if it's going to be ignored. That option might have problems scaling, depending on how deep in the payload the selection criteria field lies. Really long XPath expression = poor performance and no way to tune WMQ to make up for it since the latency is all in the application at that point.
Best case, producer sets a message property and publishes. Consumers select on property in their subscription or an administrative subscription does this for them. Whether this solution uses application subscriptions or administrative subscriptions doesn't make any difference as far as scalability is concerned.