如何热身芹菜工人
我需要我的芹菜工人在开始处理任务之前预先加载大量数据。 任务涉及使用大量磁盘的数据，这需要几秒钟的时间，即使没有缓存，这太长了。因此，我希望加载在承担第一项任务之前发生。在Gunicorn-Land中，这很容易带有前叉或叉后钩，但是Afaict芹菜不提供类似的东西。 （我真的想使用gevent或eventlet，因为我的任务使用gpu，所以他们不能分叉。）浏览芹菜代码，有很多地方可能会起作用，但是每当我尝试它们时t行为正确。例如，这似乎应该有效：
if __name__ == "__main__":
logger.info("Pre-loading data")
warmup_data()
logger.info("CeleryQ worker starting")
worker = app.worker_main(
argv = [
'worker',
'--loglevel=DEBUG',
'--pool=gevent',
'--concurrency=4',
]
)
而且它几乎有效。问题是任务被接受，但响应永远不会回头。我猜这些任务由于某种原因而默默崩溃，这本身就是奇怪的。我已经尝试了很多类似的变体，但它们并不完全相同，但没有什么可用。
I need my celeryq workers to pre-load a bunch of data before they start processing tasks.
Tasks involve using lots of data from disk, which takes several seconds if not cached, which is too long. So I want the loading to happen before they take their first task. In gunicorn-land this is easy with a pre-fork or post-fork hook, but AFAICT celery doesn't offer anything like this. (I really want to use gevent or eventlet because my tasks use GPU and so they can't fork.) Looking through the celery code there are a bunch of places where something like this might work, but whenever I try them, the workers don't behave correctly. For example, this seems like it should work:
if __name__ == "__main__":
logger.info("Pre-loading data")
warmup_data()
logger.info("CeleryQ worker starting")
worker = app.worker_main(
argv = [
'worker',
'--loglevel=DEBUG',
'--pool=gevent',
'--concurrency=4',
]
)
and it almost works. The problem is tasks get accepted but responses never go back. I'm guessing the tasks are crashing silently for some reason, which is odd in its own right. I've tried a bunch of similar variations on this, and they're not all the same, but nothing quite works.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
查找适当的 worker Signals您需要的逻辑吗？从您在问题中写的内容来看，您应该实现
worker_init或celeryd_init处理程序。我会首先选择worker_init，然后查看它的发展。Find appropriate worker signal and implement handler for it that does the logic you need. Judging by what you wrote in your question you should either implement
worker_initor perhapsceleryd_inithandler. I would go forworker_initfirst, and then see how it goes.