监控作业中的 gevent 异常

发布于 2025-01-04 06:25:27 字数 547 浏览 4 评论 0原文

我正在使用 gevent 构建一个应用程序。我的应用程序现在变得相当大,因为有很多工作被催生和销毁。现在我注意到,当其中一个作业崩溃时,我的整个应用程序会继续运行(如果异常来自非主 greenlet),这很好。但问题是我必须查看控制台才能看到错误。因此,我的应用程序的某些部分可能会“死亡”,但我并没有立即意识到这一点,并且应用程序仍在运行。

使用 try catch 来抖动我的应用程序似乎不是一个干净的解决方案。 也许是一个自定义的生成函数,它会报告一些错误?

监控 gevent 作业/greenlet 的正确方法是什么?捕获异常?

就我而言,我监听几个不同来源的事件,并且我应该处理每个不同的来源。 有大约 5 个工作非常重要。网络服务器greenlet、websocket greenlet、 数据库 greenlet、警报 greenlet 和 zmq greenlet。如果其中任何一个“死亡”,我的应用程序应该完全死亡。其他死去的工作并不那么重要。例如,websocket greenlet 可能会由于引发一些异常而死掉,而其余应用程序则保持正常运行,就像什么都没发生一样。它现在完全没用而且危险,应该会严重崩溃。

I'm building an application using gevent. My app is getting rather big now as there are a lot of jobs being spawned and destroyed. Now I've noticed that when one of these jobs crashes my entire application just keeps running (if the exception came from a non main greenlet) which is fine. But the problem is that I have to look at my console to see the error. So some part of my application can "die" and I'm not instantly aware of that and the app keeps running.

Jittering my app with try catch stuff does not seem to be a clean solution.
Maybe a custom spawn function which does some error reporting?

What is the proper way to monitor gevent jobs/greenlets? catch exceptions?

In my case I listen for events of a few different sources and I should deal with each different.
There are like 5 jobs extremely important. The webserver greenlet, websocket greenlet,
database greenlet, alarms greenlet, and zmq greenlet. If any of those 'dies' my application should completely die. Other jobs which die are not that important. For example, It is possible that websocket greenlet dies due to some exception raised and the rest of the applications keeps running fine like nothing happened. It is completely useless and dangerous now and should just crash hard.

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。

评论(4

顾铮苏瑾 2025-01-11 06:25:27

我认为最干净的方法是捕获您认为致命的异常并执行 sys.exit() (您需要 gevent 1.0 因为之前 SystemExit 没有退出该进程)。

另一种方法是使用 link_exception,如果 greenlet 因异常而终止,则会调用该方法。

spawn(important_greenlet).link_exception(lambda *args: sys.exit("important_greenlet died"))

请注意,您还需要 gevent 1.0 才能正常工作。

如果在 0.13.6 上,请执行以下操作来终止进程:

gevent.get_hub().parent.throw(SystemExit())

I think the cleanest way would be to catch the exception you consider fatal and do sys.exit() (you'll need gevent 1.0 since before that SystemExit did not exit the process).

Another way is to use link_exception, which would be called if the greenlet died with an exception.

spawn(important_greenlet).link_exception(lambda *args: sys.exit("important_greenlet died"))

Note, that you also need gevent 1.0 for this to work.

If on 0.13.6, do something like this to kill the process:

gevent.get_hub().parent.throw(SystemExit())
十二 2025-01-11 06:25:27

您想要greenlet.link_exception()所有你的 greenlet 具有看门人的功能。

janitor 函数将传递任何死亡的 greenlet,从中它可以检查其 greenlet .exception 查看发生了什么,并在必要时采取措施。

You want to greenlet.link_exception() all of your greenlets to a to janitor function.

The janitor function will be passed any greenlet that dies, from which it can inspect its greenlet.exception to see what happened, and if necessary do something about it.

中二柚 2025-01-11 06:25:27

正如 @Denis 和 @lvo 所说,link_exception 是可以的,但我认为会有更好的方法,无需更改当前代码来生成 greenlet。

一般来说,每当 greenlet 中抛出异常时,都会为该 greenlet 调用 _report_error 方法(在 gevent.greenlet.Greenlet 中)。它将执行一些操作,例如调用所有链接函数,最后使用当前堆栈中的 exc_info 调用 self.parent.handle_error 。这里的 self.parent 是全局的 Hub 对象,这意味着每个 greenlet 中发生的所有异常都会集中到一个方法来处理。默认情况下,Hub.handle_error区分异常类型,忽略某些类型并打印其他类型(这就是我们总是在控制台中看到的)。

通过修补 Hub.handle_error 方法,我们可以轻松注册自己的错误处理程序,并且再也不会丢失错误。我写了一个辅助函数来实现它:

from gevent.hub import Hub


IGNORE_ERROR = Hub.SYSTEM_ERROR + Hub.NOT_ERROR


def register_error_handler(error_handler):

    Hub._origin_handle_error = Hub.handle_error

    def custom_handle_error(self, context, type, value, tb):
        if not issubclass(type, IGNORE_ERROR):
            # print 'Got error from greenlet:', context, type, value, tb
            error_handler(context, (type, value, tb))

        self._origin_handle_error(context, type, value, tb)

    Hub.handle_error = custom_handle_error

要使用它,只需在初始化事件循环之前调用它即可:

def gevent_error_handler(context, exc_info):
    """Here goes your custom error handling logics"""
    e = exc_info[1]
    if isinstance(e, SomeError):
        # do some notify things
        pass
    sentry_client.captureException(exc_info=exc_info)

register_error_handler(gevent_error_handler)

该解决方案已在 gevent 1.0.2 和 1.1b3 下进行了测试,我们使用它向哨兵发送 greenlet 错误信息(a异常跟踪系统),到目前为止它运行得很好。

As @Denis and @lvo said, link_exception is OK, but I think there would be a better way for that, without change your current code to spawn greenlet.

Generally, whenever an exception is thrown in a greenlet, _report_error method (in gevent.greenlet.Greenlet) will be called for that greenlet. It will do some stuff like call all the link functions and finally, call self.parent.handle_error with exc_info from current stack. The self.parent here is the global Hub object, this means, all the exceptions happened in each greenlet will always be centralize to one method for handling. By default Hub.handle_error distinguish the exception type, ignore some type and print the others (which is what we always saw in the console).

By patching Hub.handle_error method, we can easily register our own error handlers and never lose an error anymore. I wrote a helper function to make it happen:

from gevent.hub import Hub


IGNORE_ERROR = Hub.SYSTEM_ERROR + Hub.NOT_ERROR


def register_error_handler(error_handler):

    Hub._origin_handle_error = Hub.handle_error

    def custom_handle_error(self, context, type, value, tb):
        if not issubclass(type, IGNORE_ERROR):
            # print 'Got error from greenlet:', context, type, value, tb
            error_handler(context, (type, value, tb))

        self._origin_handle_error(context, type, value, tb)

    Hub.handle_error = custom_handle_error

To use it, just call it before the event loop is initialized:

def gevent_error_handler(context, exc_info):
    """Here goes your custom error handling logics"""
    e = exc_info[1]
    if isinstance(e, SomeError):
        # do some notify things
        pass
    sentry_client.captureException(exc_info=exc_info)

register_error_handler(gevent_error_handler)

This solution has been tested under gevent 1.0.2 and 1.1b3, we use it to send greenlet error information to sentry (a exception tracking system), it works pretty well so far.

征﹌骨岁月お 2025-01-11 06:25:27

greenlet.link_exception() 的主要问题是它没有提供任何关于回溯的信息,而这对于记录来说非常重要。

对于使用回溯进行日志记录,我使用装饰器来 spwan 作业,这些作业间接调用到一个简单的日志记录函数中:

from functools import wraps    

import gevent

def async(wrapped):

    def log_exc(func):

        @wraps(wrapped)
        def wrapper(*args, **kwargs):
            try:
                func(*args, **kwargs)
            except Exception:
                log.exception('%s', func)
        return wrapper

    @wraps(wrapped)
    def wrapper(*args, **kwargs):
        greenlet = gevent.spawn(log_exc(wrapped), *args, **kwargs)

    return wrapper

当然,您可以添加 link_exception 调用来管理作业(我不需要)

The main issue with greenlet.link_exception() is that it does not give any information on traceback which can be really important to log.

For logging with traceback, I use a decorator to spwan jobs which indirect job call into a simple logging function:

from functools import wraps    

import gevent

def async(wrapped):

    def log_exc(func):

        @wraps(wrapped)
        def wrapper(*args, **kwargs):
            try:
                func(*args, **kwargs)
            except Exception:
                log.exception('%s', func)
        return wrapper

    @wraps(wrapped)
    def wrapper(*args, **kwargs):
        greenlet = gevent.spawn(log_exc(wrapped), *args, **kwargs)

    return wrapper

Of course, you can add the link_exception call to manage jobs (which I did not need)

~没有更多了~
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文