Web 服务中扭曲的延迟与阻塞

发布于 2024-11-05 12:19:05 字数 2274 浏览 3 评论 0原文

我正在努力在使用延迟对象的 Web 服务代码中产生与不使用延迟对象的代码相同的行为。我的目标是编写一个装饰器，它将任何方法（与 Twisted 解耦）的处理委托给 Twisted 线程池，以便反应器不会被阻塞，而不改变该方法的任何语义。

当下面的类 echo 的实例作为 Web 服务公开时，此代码：

from twisted.web import server, resource
from twisted.internet import defer, threads
from cgi import escape
from itertools import count

class echo(resource.Resource):
  isLeaf = True
  def errback(self, failure): return failure
  def callback1(self, request, value):
    #raise ValueError  # E1
    lines = ['<html><body>\n',
             '<p>Page view #%s in this session</p>\n' % (value,),
             '</body></html>\n']
    return ''.join(lines)
  def callback2(self, request, encoding):
    def execute(message):
      #raise ValueError  # E2
      request.write(message.encode(encoding))
      #raise ValueError  # E3
      request.finish()
      #raise ValueError  # E4
      return server.NOT_DONE_YET
    return execute
  def render_GET(self, request):
    content_type, encoding = 'text/html', 'UTF-8'
    request.setHeader('Content-Type', '%s; charset=%s' %
        tuple(map(str, (content_type, encoding))))
    s = request.getSession()
    if not hasattr(s, 'counter'):
      s.counter = count(1)
    d = threads.deferToThread(self.callback1, request, s.counter.next())
    d.addCallback(self.callback2(request, encoding))
    d.addErrback(self.errback)
    #raise ValueError  # E5
    return server.NOT_DONE_YET

当所有 raise 语句都被注释掉时，将向浏览器显示 HTML 文档，并在以下情况下显示格式良好的堆栈跟踪（Twisted 为我做的）包含标记为“E5”的 raise 语句。这就是我想要的。同样，如果我根本不使用 Deferred 对象并将回调 1 和回调 2 中的所有行为都放在 render_GET() 中，则 render_GET 中任何地方引发的异常都将产生所需的堆栈跟踪。

我正在尝试编写能够立即响应浏览器的代码，不会导致 Twisted 内的资源泄漏，并且在其中包含任何 raise 语句“E1”到“E3”的情况下也会导致浏览器堆栈跟踪显示延迟的代码——当然我知道堆栈跟踪本身会有所不同。（我不太关心“E4”案例。）在阅读了 Twisted 文档和本网站上的其他问题后，我不确定如何实现这一目标。我本以为添加一个 errback 应该会促进这一点，但显然不是。肯定有一些关于延迟对象和twisted.web 堆栈的东西我不理解。

我在此处记录的对日志记录的影响可能会受到我使用 PythonLoggingObserver 将 Twisted 日志记录桥接到标准日志记录模块的影响。

当包含“E1”时，浏览器将等待直到反应器关闭，此时会记录带有堆栈跟踪的 ValueError 异常，并且浏览器会收到一个空文档。

当包含“E2”时，会立即记录带有堆栈跟踪的 ValueError 异常，但浏览器会等到反应器关闭，此时它会收到一个空文档。

当包含“E3”时，立即记录带有堆栈跟踪的 ValueError 异常，浏览器等待直到反应器关闭，然后收到预期的文档。

当包含 raise 语句“E4”时，预期文档将立即返回到浏览器，并立即记录带有堆栈跟踪的 ValueError 异常。（这种情况下有没有资源泄漏的可能？）

原文

I'm struggling to produce the same behavior in web service code that uses Deferred objects as in code that does not. My objective is to write a decorator that will delegate processing of any method (which is decoupled from Twisted) to the Twisted thread pool, so that the reactor is not blocked, without changing any of that method's semantics.

When an instance of class echo below is exposed as a web service, this code:

from twisted.web import server, resource
from twisted.internet import defer, threads
from cgi import escape
from itertools import count

class echo(resource.Resource):
  isLeaf = True
  def errback(self, failure): return failure
  def callback1(self, request, value):
    #raise ValueError  # E1
    lines = ['<html><body>\n',
             '<p>Page view #%s in this session</p>\n' % (value,),
             '</body></html>\n']
    return ''.join(lines)
  def callback2(self, request, encoding):
    def execute(message):
      #raise ValueError  # E2
      request.write(message.encode(encoding))
      #raise ValueError  # E3
      request.finish()
      #raise ValueError  # E4
      return server.NOT_DONE_YET
    return execute
  def render_GET(self, request):
    content_type, encoding = 'text/html', 'UTF-8'
    request.setHeader('Content-Type', '%s; charset=%s' %
        tuple(map(str, (content_type, encoding))))
    s = request.getSession()
    if not hasattr(s, 'counter'):
      s.counter = count(1)
    d = threads.deferToThread(self.callback1, request, s.counter.next())
    d.addCallback(self.callback2(request, encoding))
    d.addErrback(self.errback)
    #raise ValueError  # E5
    return server.NOT_DONE_YET

will display an HTML document to the browser when all the raise statements are commented out, and display a nicely formatted stack trace (which Twisted does for me) when the raise statement labelled "E5" is included. That is what I want. Likewise, if I do not use Deferred objects at all and place all the behavior from callback1 and callback2 within render_GET(), an exception raised anywhere within render_GET will produce the desired stack trace.

I am trying to write code that will respond to the browser immediately, not cause resource leaks within Twisted, and cause the browser stack trace to also be displayed in the cases where any of the raise statements "E1" to "E3" is included in the deferred code--though of course I understand that the stack traces themselves will be different. (The "E4" case I don't care about as much.) After reading the Twisted documentation and other questions on this site I am unsure how to achieve this. I would have thought that adding an errback should facilitate this, but evidently not. There must be something about Deferred objects and the twisted.web stack that I'm not understanding.

The effects on logging I document here may be affected by my use of the PythonLoggingObserver to bridge Twisted logging to the standard logging module.

When "E1" is included, the browser waits until the reactor is shut down, at which point the ValueError exception with stack trace is logged and the browser receives an empty document.

When "E2" is included, the ValueError exception with stack trace is logged immediately, but the browser waits until the reactor shuts down at which point it receives an empty document.

When "E3" is included, the ValueError exception with stack trace is logged immediately, the browser waits until the reactor shuts down, and at that point receives the intended document.

When raise statement "E4" is included, the intended document is returned to the browser immediately, and the ValueError exception with stack trace is logged immediately. (Is there any possibility of a resource leak in this case?)

分享到QQ

分享到微博

如果你对这篇内容有疑问，欢迎到本站社区发帖提问参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

发布评论

需要登录才能够评论，你可以免费注册一个本站的账号。

樱花落人离去 2024-11-12 12:19:05

好吧，读了几遍你的问题后，我想我明白你在问什么了。我还修改了您的代码，使其比您原来的答案更好一些。这个新答案应该展示 deferred 的所有功能。

from twisted.web import server, resource
from twisted.internet import defer, threads
from itertools import count

class echo(resource.Resource):
  isLeaf = True
  def errback(self, failure, request):
    failure.printTraceback() # This will print the trace back in a way that looks like a python exception.
    # log.err(failure) # This will use the twisted logger. This is the best method, but
    # you need to import twisted log.

    request.processingFailed(failure) # This will send a trace to the browser and close the request.
    return None #  We have dealt with the failure. Clean it out now.

  def final(self, message, request, encoding): 
    # Message will contain the message returned by callback1
    request.write(message.encode(encoding)) # This will write the message and return it to the browser.

    request.finish() # Done

  def callback1(self, value):
    #raise ValueError  # E1
    lines = ['<html><body>\n',
             '<p>Page view #%s in this session</p>\n' % (value,),
             '</body></html>\n']
    return ''.join(lines)

    #raise ValueError  # E4

  def render_GET(self, request):
    content_type, encoding = 'text/html', 'UTF-8'
    request.setHeader('Content-Type', '%s; charset=%s' %
        tuple(map(str, (content_type, encoding))))
    s = request.getSession()
    if not hasattr(s, 'counter'):
      s.counter = count(1)
    d = threads.deferToThread(self.callback1, s.counter.next())
    d.addCallback(self.final, request, encoding)
    d.addErrback(self.errback, request) # We put this here in case the encoding raised an exception.
    #raise ValueError  # E5
    return server.NOT_DONE_YET

另外，我建议您阅读 krondo 教程。它将教您关于延迟需要了解的一切。

编辑：

修改了上面的代码以修复一些愚蠢的错误。也对其进行了改进。如果异常发生在任何地方（除了 self.errback 中，但我们需要一定程度的信任），那么它将被传递给 self.errback ，它将记录或打印错误扭曲，然后将跟踪发送到浏览器并关闭请求。这应该可以阻止任何资源泄漏。

Ok, after reading your question several times, I think I understand what your asking. I have also reworked you code to make a little better than your original answer. This new answer should show off all the powers of deferred's.

from twisted.web import server, resource
from twisted.internet import defer, threads
from itertools import count

class echo(resource.Resource):
  isLeaf = True
  def errback(self, failure, request):
    failure.printTraceback() # This will print the trace back in a way that looks like a python exception.
    # log.err(failure) # This will use the twisted logger. This is the best method, but
    # you need to import twisted log.

    request.processingFailed(failure) # This will send a trace to the browser and close the request.
    return None #  We have dealt with the failure. Clean it out now.

  def final(self, message, request, encoding): 
    # Message will contain the message returned by callback1
    request.write(message.encode(encoding)) # This will write the message and return it to the browser.

    request.finish() # Done

  def callback1(self, value):
    #raise ValueError  # E1
    lines = ['<html><body>\n',
             '<p>Page view #%s in this session</p>\n' % (value,),
             '</body></html>\n']
    return ''.join(lines)

    #raise ValueError  # E4

  def render_GET(self, request):
    content_type, encoding = 'text/html', 'UTF-8'
    request.setHeader('Content-Type', '%s; charset=%s' %
        tuple(map(str, (content_type, encoding))))
    s = request.getSession()
    if not hasattr(s, 'counter'):
      s.counter = count(1)
    d = threads.deferToThread(self.callback1, s.counter.next())
    d.addCallback(self.final, request, encoding)
    d.addErrback(self.errback, request) # We put this here in case the encoding raised an exception.
    #raise ValueError  # E5
    return server.NOT_DONE_YET

Also I recommend that you read the krondo tutorial. It will teach you everything you need to know about deferred.

Edit:

Have modified the code above to fix some silly bugs. Also improved it. If an exception happens anywhere (except in self.errback, but we need some level of trust) then it will be passed to self.errback which will log or print the error in twisted and then send the trace to the browser and close the request. This should stop any resource leaks.

回复收藏 0 原文

时光倒影 2024-11-12 12:19:05

我通过挖掘 Twisted 源代码发现了这一点。必要的见解是，反应器和延迟回调/错误返回链逻辑与请求对象分离，这就是数据返回浏览器的方式。 errback 是必要的，但不能像我发布的原始代码那样仅仅沿着链传播 Failure 对象。 errback 必须向浏览器报告错误。

下面的代码满足我的要求（永远不会让浏览器等待，总是提供堆栈跟踪，不需要重新启动反应器才能让事情再次进行），并且允许我装饰阻塞方法，从而将它们委托给线程以保持反应器响应到其他事件（此类方法本质上将取代这里的callback1）。但是，我确实发现在下面的代码中，取消注释“E4”raise 语句会在后续浏览器请求中产生非常奇怪的行为（先前请求的部分数据返回到浏览器；死锁）。

希望其他人会发现这是一个有用的延迟示例。

from twisted.web import server, resource
from twisted.internet import defer, threads
from itertools import count

class echo(resource.Resource):
  isLeaf = True
  def errback(self, request):
    def execute(failure):
      request.processingFailed(failure)
      return failure
    return execute
  def callback1(self, value):
    #raise ValueError  # E1
    lines = ['<html><body>\n',
             '<p>Page view #%s in this session</p>\n' % (value,),
             '</body></html>\n']
    return ''.join(lines)
  def callback2(self, request, encoding):
    def execute(message):
      #raise ValueError  # E2
      request.write(message.encode(encoding))
      #raise ValueError  # E3
      request.finish()
      #raise ValueError  # E4
      return server.NOT_DONE_YET
    return execute
  def render_GET(self, request):
    content_type, encoding = 'text/html', 'UTF-8'
    request.setHeader('Content-Type', '%s; charset=%s' %
        tuple(map(str, (content_type, encoding))))
    s = request.getSession()
    if not hasattr(s, 'counter'):
      s.counter = count(1)
    d = threads.deferToThread(self.callback1, s.counter.next())
    eback = self.errback(request)
    d.addErrback(eback)
    d.addCallback(self.callback2(request, encoding))
    d.addErrback(eback)
    #raise ValueError  # E5
    return server.NOT_DONE_YET

I figured it out by digging through the Twisted source. The necessary insight is that the reactor and Deferred callback/errback chain logic is decoupled from the request object, which is how data gets back to the browser. The errback is necessary, but cannot merely propagate the Failure object down the chain as in the original code I posted. The errback must report the error to the browser.

The below code meets my requirements (never keeps the browser waiting, always gives the stack trace, does not require a reactor restart to get things going again) and will allow me to decorate blocking methods and thereby delegate them to threads to keep the reactor responsive to other events (such methods will essentially take the place of callback1 here). However, I did find that in the below code, uncommenting the "E4" raise statement produces very strange behavior on subsequent browser requests (partial data from previous requests returned to the browser; deadlock).

Hopefully others will find this to be a useful Deferred example.

from twisted.web import server, resource
from twisted.internet import defer, threads
from itertools import count

class echo(resource.Resource):
  isLeaf = True
  def errback(self, request):
    def execute(failure):
      request.processingFailed(failure)
      return failure
    return execute
  def callback1(self, value):
    #raise ValueError  # E1
    lines = ['<html><body>\n',
             '<p>Page view #%s in this session</p>\n' % (value,),
             '</body></html>\n']
    return ''.join(lines)
  def callback2(self, request, encoding):
    def execute(message):
      #raise ValueError  # E2
      request.write(message.encode(encoding))
      #raise ValueError  # E3
      request.finish()
      #raise ValueError  # E4
      return server.NOT_DONE_YET
    return execute
  def render_GET(self, request):
    content_type, encoding = 'text/html', 'UTF-8'
    request.setHeader('Content-Type', '%s; charset=%s' %
        tuple(map(str, (content_type, encoding))))
    s = request.getSession()
    if not hasattr(s, 'counter'):
      s.counter = count(1)
    d = threads.deferToThread(self.callback1, s.counter.next())
    eback = self.errback(request)
    d.addErrback(eback)
    d.addCallback(self.callback2(request, encoding))
    d.addErrback(eback)
    #raise ValueError  # E5
    return server.NOT_DONE_YET

回复收藏 0 原文

~没有更多了~