限制芹菜中活动任务的数量

Question

有任务

from celery import Celery
celery = Celery(__name__, **kwargs) # Some setting here probably?

@celery.task(name="task_runner", bind=True)
def generic_task(self, function_type):
    task_id = self.request.id
    # Do whatever
    if function_type == 1:
        high_ram_usage_1() # Really a high memory function
    if function_type == 2:
        high_ram_usage_2() # Another high memory function
    # Other functions

如何将主动任务计数限制为整个芹菜应用的两个？我的内存中只有两个活动功能最终将服务器崩溃在我的开发计算机上。

编辑：

celery = Celery(__name__, concurrency=2) # Does not work
celery.conf.update(concurrency=2) # Does not work
@celery.task(concurrency=2) # Does not work
def dummy_task(self):
    time.sleep(5)

工人日志

<redacted>:Task dummy_task[b62052bf-c893-4ad4-bc30-cffaa39bcbb1] succeeded in 5.110082127997885s: True
<redacted>:Task dummy_task[a2def96d-6306-422a-be79-14f43886aa7f] succeeded in 5.104002231993945s: True
<redacted>:Task dummy_task[4b377742-c22f-432c-a675-9c0dbdd2cb41] succeeded in 5.119215640006587s: True
<redacted>:Task dummy_task[5735de81-79f7-43ac-b28c-42cb071011ca] succeeded in 5.139429216011195s: True
<redacted>:Task dummy_task[5b37c19f-693d-45d4-8580-6b493632c5ab] succeeded in 5.142507184995338s: True
<redacted>:Task dummy_task[d8b00cfc-0773-43cd-bc40-d5a55a1dfda0] succeeded in 5.158245797007112s: True
<redacted>:Task dummy_task[0d93b652-b4f9-4bff-97b5-0660cdc05586] succeeded in 5.168131768004969s: True
<redacted>:Task dummy_task[b7946f28-7df6-42d8-9a9d-1e7b5efc3a4a] succeeded in 5.173699700011639s: True
<redacted>:Task dummy_task[4f1f9389-040a-45dd-b252-4342ba1d3445] succeeded in 5.187171609024517s: True
<redacted>:Task dummy_task[6651040f-bffa-46b6-ab4f-950d8f518b46] succeeded in 5.192372458986938s: True
<redacted>:Task dummy_task[b65d2de1-5029-47f7-8048-73c4c38065ad] succeeded in 5.279150495975045s: True
<redacted>:Task dummy_task[3211729b-e8b9-4c72-9354-caf16e7a6970] succeeded in 6.117392421991099s: True

Answer 1

您无法在任务上指定并发性，因此@celery.task（consurrency = 2）不起作用。
@tharunk指向的线程已经回答了您的问题，因为您遇到了同样的问题。但是，看来您没有完全理解它。

该线程的答案基本上说是您需要配置芹菜工人以产生不超过n个工作过程。在您的情况下n = 2。

有几种方法可以做到，其中之一是 - 并发芹菜工人的命令行选项。

您已经非常接近使用配置对象设置它，但是您使用了错误的密钥。正确的键是 worker_concurrency .conf.update（worker_concurrency = 2）应该完成工作。