Celery 任务分析
正如我在 top 实用程序 celery 进程中看到的那样，消耗了大量的 CPU 时间。所以我想介绍一下它。
我可以在开发人员机器上手动执行此操作，如下所示：
python -m cProfile -o test-`date +%Y-%m-%d-%T`.prof ./manage.py celeryd -B
但为了获得准确的计时，我需要在生产机器上对其进行分析。在该机器（Fedora 14）上，celery 由 init 脚本启动。例如，
service celeryd start
我发现这些脚本最终会调用 manage.py celeryd_multi 。所以我的问题是如何告诉 celeryd_multi 在启用分析的情况下启动 celery？就我而言，这意味着将 -m cProfile -o out.prof 选项添加到 python 中。
非常感谢任何帮助。
As I can see in top utility celery procecess consume a lot of CPU time. So I want to profile it.
I can do it manually on developer machine like so:
python -m cProfile -o test-`date +%Y-%m-%d-%T`.prof ./manage.py celeryd -B
But to have accurate timings I need to profile it on production machine. On that machine (Fedora 14) celery is launched by init scripts. E.g.
service celeryd start
I have figured out these scripts eventually call manage.py celeryd_multi eventually. So my question is how can I tell celeryd_multi to start celery with profiling enabled? In my case this means add -m cProfile -o out.prof options to python.
Any help is much appreciated.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
我认为你混淆了两个不同的问题。您可能正在处理太多的单独任务，或者单个任务可能效率低下。
您可能知道其中哪个是问题所在，但从您的问题中并不清楚问题所在。
要跟踪正在处理的任务数量，我建议您查看 celerymon。如果某个特定任务出现的频率比您预期的要高，那么您可以调查它是从哪里调用的。
对整个 celery 进行分析可能没有帮助，因为您将获得大量无法控制的代码。正如您所说，这也意味着您在生产中运行它时遇到问题。我建议您考虑将分析代码直接添加到您的任务定义中。
您可以使用
cProfile.run('func()')作为 celery 和代码之间的间接层，以便对任务的每次运行进行分析。如果您生成唯一的文件名并将其作为第二个参数传递给run，您将拥有一个充满配置文件数据的目录，您可以逐个任务地检查这些数据，或者使用 pstats.add 将多个任务运行组合在一起。最后，按任务分析意味着您还可以使用项目代码中的设置全局或按任务打开或关闭分析，而无需修改服务器上的初始化脚本。
I think you're confusing two separate issues. You could be processing too many individual tasks or an individual task could be inefficient.
You may know which of these is the problem, but it's not clear from your question which it is.
To track how many tasks are being processed I suggest you look at celerymon. If a particular task appears more often that you would expect then you can investigate where it is getting called from.
Profiling the whole of celery is probably not helpful as you'll get lots of code that you have no control over. As you say it also means you have a problem running it in production. I suggest you look at adding the profiling code directly into your task definition.
You can use
cProfile.run('func()')as a layer of indirection between celery and your code so each run of the task is profiled. If you generate a unique filename and pass it as the second parameter torunyou'll have a directory full of profile data that you can inspect on a task-by-task basis, or use pstats.add to combine multiple task runs together.Finally, per-task profiling means you can also turn profiling on or off using a setting in your project code either globally or by task, rather than needing to modify the init scripts on your server.