RabbitMQ 上 celery 任务优先级的解决方法?

发布于 2024-09-12 11:00:21 字数 742 浏览 7 评论 0 原文

我在 RabbitMQ 之上运行 Django 和 Celery 作为队列来处理一些数据处理任务。当用户首次注册时,我会启动 celery 任务,并定期更新他们的数据。但是,我当然想优先处理当前在线用户运行的任务。我注意到celery中的任务有一个优先级设置,但rabbitmq似乎不支持这个。此帖子http: //groups.google.com/group/celery-users/browse_thread/thread/ac3b6123d63421e5/b7740def1389e87e?lnk=gst&q=priority#b7740def1389e87e 建议有两个不同的队列,一个高优先级队列和一个低优先级队列,或为较低优先级任务设置速率限制。

有人有一个好的解决方法来实现优先级吗?提前致谢!

I am running Django with Celery on top of RabbitMQ as a queue to handle some data processing tasks. I am kicking off celery tasks when a user first signs up, as well as periodically to update their data. However, I'd like to of course give priority to the tasks running users who are currently online. I noticed there was a priority setting for tasks in celery, but it seems that rabbitmq does not support this. This thread http://groups.google.com/group/celery-users/browse_thread/thread/ac3b6123d63421e5/b7740def1389e87e?lnk=gst&q=priority#b7740def1389e87e suggests have two different queues, a high priority one and a low priority one, or setting a rate limit for lower priority tasks.

Does anyone have a good workaround to implement priority? Thanks in advance!

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。

评论(2

生死何惧 2024-09-19 11:00:21

除此之外,您还可以将紧急任务推送到某个队列(例如紧急队列)并设置消费者优先级,即让所有消费者从紧急队列中获取高优先级的任务。

https://github.com/celery/celery/issues/3098

在消费者端,您可以在队列中定义 x-priority 参数来消费。在下面的示例中,消费者从优先级为 0 的 celery 队列和优先级为 10 的 hipri 中获取任务。

示例:

CELERY_QUEUES = (
    Queue('celery', Exchange('celery', type='direct'), routing_key='celery',
          consumer_arguments={'x-priority': 0}),
    Queue('hipri', Exchange('hipri', type='direct'), routing_key='hipri',
          consumer_arguments={'x-priority': 10}),
)

Apart from this, you can push urgent tasks to some queue (let's say urgent-queue) and set consumer priorities, i.e, let all consumers pick up task from urgent-queue with high priority.

https://github.com/celery/celery/issues/3098

At consumer end, you can define x-priority argument in queues to consume from. In the below example, consumer picks up tasks from celery queue with priority 0 and from hipri with priority 10.

Example:

CELERY_QUEUES = (
    Queue('celery', Exchange('celery', type='direct'), routing_key='celery',
          consumer_arguments={'x-priority': 0}),
    Queue('hipri', Exchange('hipri', type='direct'), routing_key='hipri',
          consumer_arguments={'x-priority': 10}),
)
~没有更多了~
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文