监控作业中的 gevent 异常
我正在使用 gevent 构建一个应用程序。我的应用程序现在变得相当大,因为有很多工作被催生和销毁。现在我注意到,当其中一个作业崩溃时,我的整个应用程序会继续运行(如果异常来自非主 greenlet),这很好。但问题是我必须查看控制台才能看到错误。因此,我的应用程序的某些部分可能会“死亡”,但我并没有立即意识到这一点,并且应用程序仍在运行。
使用 try catch 来抖动我的应用程序似乎不是一个干净的解决方案。 也许是一个自定义的生成函数,它会报告一些错误?
监控 gevent 作业/greenlet 的正确方法是什么?捕获异常?
就我而言,我监听几个不同来源的事件,并且我应该处理每个不同的来源。 有大约 5 个工作非常重要。网络服务器greenlet、websocket greenlet、 数据库 greenlet、警报 greenlet 和 zmq greenlet。如果其中任何一个“死亡”,我的应用程序应该完全死亡。其他死去的工作并不那么重要。例如,websocket greenlet 可能会由于引发一些异常而死掉,而其余应用程序则保持正常运行,就像什么都没发生一样。它现在完全没用而且危险,应该会严重崩溃。
I'm building an application using gevent. My app is getting rather big now as there are a lot of jobs being spawned and destroyed. Now I've noticed that when one of these jobs crashes my entire application just keeps running (if the exception came from a non main greenlet) which is fine. But the problem is that I have to look at my console to see the error. So some part of my application can "die" and I'm not instantly aware of that and the app keeps running.
Jittering my app with try catch stuff does not seem to be a clean solution.
Maybe a custom spawn function which does some error reporting?
What is the proper way to monitor gevent jobs/greenlets? catch exceptions?
In my case I listen for events of a few different sources and I should deal with each different.
There are like 5 jobs extremely important. The webserver greenlet, websocket greenlet,
database greenlet, alarms greenlet, and zmq greenlet. If any of those 'dies' my application should completely die. Other jobs which die are not that important. For example, It is possible that websocket greenlet dies due to some exception raised and the rest of the applications keeps running fine like nothing happened. It is completely useless and dangerous now and should just crash hard.
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论
评论(4)
我认为最干净的方法是捕获您认为致命的异常并执行
sys.exit()
(您需要 gevent 1.0 因为之前SystemExit
没有退出该进程)。另一种方法是使用 link_exception,如果 greenlet 因异常而终止,则会调用该方法。
请注意,您还需要 gevent 1.0 才能正常工作。
如果在 0.13.6 上,请执行以下操作来终止进程:
I think the cleanest way would be to catch the exception you consider fatal and do
sys.exit()
(you'll need gevent 1.0 since before thatSystemExit
did not exit the process).Another way is to use link_exception, which would be called if the greenlet died with an exception.
Note, that you also need gevent 1.0 for this to work.
If on 0.13.6, do something like this to kill the process:
您想要
greenlet.link_exception()
所有你的 greenlet 具有看门人的功能。janitor 函数将传递任何死亡的 greenlet,从中它可以检查其
greenlet .exception
查看发生了什么,并在必要时采取措施。You want to
greenlet.link_exception()
all of your greenlets to a to janitor function.The janitor function will be passed any greenlet that dies, from which it can inspect its
greenlet.exception
to see what happened, and if necessary do something about it.正如 @Denis 和 @lvo 所说,
link_exception
是可以的,但我认为会有更好的方法,无需更改当前代码来生成 greenlet。一般来说,每当 greenlet 中抛出异常时,都会为该 greenlet 调用
_report_error
方法(在gevent.greenlet.Greenlet
中)。它将执行一些操作,例如调用所有链接函数,最后使用当前堆栈中的 exc_info 调用 self.parent.handle_error 。这里的 self.parent 是全局的 Hub 对象,这意味着每个 greenlet 中发生的所有异常都会集中到一个方法来处理。默认情况下,Hub.handle_error
区分异常类型,忽略某些类型并打印其他类型(这就是我们总是在控制台中看到的)。通过修补
Hub.handle_error
方法,我们可以轻松注册自己的错误处理程序,并且再也不会丢失错误。我写了一个辅助函数来实现它:要使用它,只需在初始化事件循环之前调用它即可:
该解决方案已在 gevent 1.0.2 和 1.1b3 下进行了测试,我们使用它向哨兵发送 greenlet 错误信息(a异常跟踪系统),到目前为止它运行得很好。
As @Denis and @lvo said,
link_exception
is OK, but I think there would be a better way for that, without change your current code to spawn greenlet.Generally, whenever an exception is thrown in a greenlet,
_report_error
method (ingevent.greenlet.Greenlet
) will be called for that greenlet. It will do some stuff like call all the link functions and finally, callself.parent.handle_error
with exc_info from current stack. Theself.parent
here is the globalHub
object, this means, all the exceptions happened in each greenlet will always be centralize to one method for handling. By defaultHub.handle_error
distinguish the exception type, ignore some type and print the others (which is what we always saw in the console).By patching
Hub.handle_error
method, we can easily register our own error handlers and never lose an error anymore. I wrote a helper function to make it happen:To use it, just call it before the event loop is initialized:
This solution has been tested under gevent 1.0.2 and 1.1b3, we use it to send greenlet error information to sentry (a exception tracking system), it works pretty well so far.
greenlet.link_exception() 的主要问题是它没有提供任何关于回溯的信息,而这对于记录来说非常重要。
对于使用回溯进行日志记录,我使用装饰器来 spwan 作业,这些作业间接调用到一个简单的日志记录函数中:
当然,您可以添加
link_exception
调用来管理作业(我不需要)The main issue with
greenlet.link_exception()
is that it does not give any information on traceback which can be really important to log.For logging with traceback, I use a decorator to spwan jobs which indirect job call into a simple logging function:
Of course, you can add the
link_exception
call to manage jobs (which I did not need)