如何调查cherrypy崩溃?
我们有一个与多个后端 Web 服务集成的 Cherrypy 服务。在负载测试期间,cherrypy 进程会在一段时间(45 分钟)后定期崩溃。我们知道瓶颈是我们正在使用的后端 Web 服务。在崩溃之前我们在访问后端服务时看到500和503错误,但我不明白为什么cherrypy本身会崩溃(整个进程被杀死)。您能给我一些如何调查问题所在的想法吗?是否有可能 thread_poll (50) 排队的请求过多?
We have a cherrypy service that integrates with several backend web services. During load testing cherrypy process is regularly crashed after a while (45 minutes). We know the bottleneck is the backend web services we are using. Before crashing we see 500 and 503 errors when accessing the backend services, but I can't figure why cherrypy itself will crash (the whole process was killed). Can you give me ideas how to investigate where the problem is? Is it possible that the thread_poll (50) is queueing up too many requests?
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论
评论(1)
在我早期的 CherryPy 时代,我曾经崩溃过一次。我的意思是由段错误引起的Python进程崩溃。当我调查它时,我发现我搞乱了 MySQLdb 连接,将它们缓存在由 CherryPy 线程可互换访问的对象中。因为 MySQLdb 连接不是线程安全的,所以只能从创建的线程进行访问。此外,由于涉及并发性,崩溃似乎是不确定的,并且仅出现在负载测试中。因此,负载测试可以用作调试工具 - 尝试 Apache JMeter 或 Locust(Pythonic)。
当进程崩溃时,您可以指示 Linux 编写一个核心转储,其中包含堆栈跟踪(例如在我的示例中的 MySQLdb C 代码端)。然而,陌生的低级 C 环境对你(对我来说)来说是陌生的,堆栈跟踪可以帮助找到导致崩溃的库,或者至少缩小嫌疑范围。这是一篇关于它的文章 。
我还想指出,CherryPy 中不太可能出现问题。它实际上非常稳定。
In my early CherryPy days I had it crashing once. I mean a Python process crash caused by a segfault. When I investigated it I found that I messed with MySQLdb connections, caching them in objects which were accessed by CherryPy threads interchangeably. Because a MySQLdb connection is not thread-safe it should be accessed only from the thread in was created in. Also because of concurrency involved the crashes seemed nondeterministic, and only appeared in load-testing. So load-testing can work as a debugging tool here -- try Apache JMeter or Locust (Pythonic).
When a process crashes you can instruct Linux to write a core dump which will have a stack trace (e.g. on MySQLdb C-code side in my example). However alien low-level C environment is to you (it is to me), the stack trace can help find what library is causing the crash or at least narrow a circle of suspects. Here is an article about it.
Also I want to note that unlikely problem is in CherryPy. It is actually very stable.