使用 Celery 动态创建队列
我正在使用 Django、Celery 和 RabbitMQ 编写一个邮件列表管理器。当收到消息时，将为每个收件人执行一项任务。所有任务都进入一个队列，一个或多个工作人员使用队列中的任务，构建电子邮件消息并发送它们。
单个队列会导致公平问题：如果邮件进入大型邮件列表，则会将大量任务添加到队列中，并且其他邮件无法到达其他较小的邮件列表，直到所有邮件都到达大型列表已发送。我怎样才能找到解决这个问题的方法？
从概念上讲，解决方案是为每个邮件列表创建一个队列，并让工作人员从各个队列循环中消耗任务。鉴于我需要能够动态创建新的邮件列表，这在 Celery 中可能吗？我还没有看到动态创建队列或使工作人员从新队列消费的功能。
I am writing a mailing-list manager using Django, Celery, and RabbitMQ. When a message comes in, a task is executed for each recipient. All tasks go to a single queue, and one or more workers consume tasks from the queue, constructing the email messages and sending them.
The single queue causes a fairness problem: if a message comes in to a large mailing list, a large number of tasks are added to the queue, and other messages cannot get through to other, smaller mailing lists until all the messages to the large list have been sent. How can I find a way around this?
Conceptually, a solution would be to create a queue for each mailing list and have the worker(s) consume tasks from the various queue round robin. Is this possible in Celery, given that I need to be able to create new mailing lists dynamically? I have not seen functionality for creating queues dynamically or for making the worker(s) consume from new queues.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
如下图所示，考虑一个使用主题交换而不是直接交换的系统。
与直接交换不同，主题交换允许我们将不同的消息路由到不同的队列。这是通过为每个消息设置routing_key并绑定某些队列以仅接受具有特定路由键的消息来完成的。
可以建立一种系统，将高优先级消息发送到专用队列和消费者，类似地，普通或低优先级消息由一个或多个队列处理。
Celery 支持多个工作人员之间的负载平衡（工作人员数量取决于硬件）。 Celery 配置设置（如 BROKER_POOL_LIMIT、PREFETCH_LIMIT 等）有助于更好的负载平衡并减少拥塞。
As shown in the image below consider a system which makes use of a topic exchange instead of a direct exchange.
Unlike a direct exchange, Topic exchange allows us to route different messages to different queues. this is done by setting routing_key for each message and binding certain queues to accept only messages with a particular routing keys.
A system can be set up that high priority go to a dedicated queue and consumer, similarly normal or low priority messages are handled by one or more queues.
Celery supports load balancing across multiple workers(number of workers are hardware dependent). Celery config setting like BROKER_POOL_LIMIT, PREFETCH_LIMIT etc help better load balance and reduce congestion.