多个消费者&连接到消息队列的生产者,这在 AMQP 中可能吗?

发布于 2024-08-20 06:04:58 字数 259 浏览 8 评论 0原文

我想创建一个能够 OCR 文本的进程群。 我考虑过使用由多个 OCR 进程读取的单个消息队列。

我想确保:

  • 队列中的每条消息最终都会得到处理
  • 工作或多或少均匀分布
  • 图像将仅由一个 OCR 进程解析
  • 一个 OCR 进程不会一次获取多条消息(因此任何其他免费 OCR进程可以处理该消息)。

使用 AMQP 可以做到吗?

我打算使用 python 和rabbitmq

I'd like to create a farm of processes that are able to OCR text.
I've thought about using a single queue of messages which is read by multiple OCR processes.

I would like to ensure that:

  • each message in queue is eventually processed
  • the work is more or less equally distributed
  • an image will be parsed only by one OCR process
  • An OCR process won't get multiple messages at once (so that any other free OCR process can handle the message).

Is that possible to do using AMQP?

I'm planning to use python and rabbitmq

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。

评论(2

御弟哥哥 2024-08-27 06:04:58

是的,正如@nailxx 指出的那样。 AMQP 编程模型与 JMS 略有不同,因为您只有队列,队列可以在工作程序之间共享,或者由单个工作程序私有使用。您还可以轻松设置 RabbitMQ 来执行 PubSub 用例或 JMS 中所谓的主题。请访问 入门 页面。 rabbitmq.com/" rel="nofollow noreferrer">RabbitMQ 网站 查找大量有关此内容的有用信息。

现在,特别是对于您的用例,已经有很多可用的工具。 Celery 是人们经常使用的一种。这是一篇关于它的博客文章,我认为这会帮助您开始:

如果您有任何疑问,请给我们发送电子邮件或发布到rabbitmq-discuss 邮件列表。

Yes, as @nailxx points out. The AMQP programming model is slightly different from JMS in that you only have queues, which can be shared between workers, or used privately by a single worker. You can also easily set up RabbitMQ to do PubSub use cases or what in JMS are called topics. Please go to our Getting Started page on the RabbitMQ web site to find a ton of helpful info about this.

Now, for your use case in particular, there are already plenty of tools available. One that people are using a lot, and that is well supported, is Celery. Here is a blog post about it, that I think will help you get started:

If you have any questions please email us or post to the rabbitmq-discuss mailing list.

踏雪无痕 2024-08-27 06:04:58

是的,这是可能的。我正在开发的实时MMO游戏的服务器集群就是这样操作的。我们使用 ActiveMQ,但我认为这一切也可以通过 RabbitMQ 实现。

除了最后一件之外,您提到的所有物品都可以开箱即用。

  • 队列中的每条消息最终都会得到处理 - 这是消息代理的主要职责之一
  • 工作或多或少是平均分配的 - 这是另一个职责 :)
  • an图像将仅由一个 OCR 进程解析 - 为此存在 /topic 和 /queue 的区别。主题就像广播信号,队列就是任务。您的场景中需要一个 /queue

为了使最后一个消息以所需的方式工作,消费者在订阅队列时发送特定于 AMQ 的参数:

activemq.prefetchSize: 1

此设置保证消费者在接收一条消息后不会再接收任何消息,直到它发送 < code>ack 到 AMQ。我相信 RabbitMQ 中也存在类似的东西。

Yes, that's possible. Server cluster for a real-time MMO game I'm working on operate this way. We use ActiveMQ, but I think all this possible with RabbitMQ as well.

All items that you mentioned you get out of the box, except last one.

  • each message in queue is eventually processed - this is one of main responsibilities of message brokers
  • the work is more or less equally distributed - this is another one :)
  • an image will be parsed only by one OCR process - the distinction of /topic and /queue exists for this. Topics are like broadcast signals, queues are tasks. You need a /queue in your scenario

To make last one work in desired way, consumers send AMQ-specific argument when subscribing to the queue:

activemq.prefetchSize: 1

This setting guarantees that consumer will not take any more messages after it took one and until it send an ack to AMQ. I believe something similar exists in RabbitMQ.

~没有更多了~
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文