由于 UnicodeDecodeError,Twisted FileSender 无法发送二进制文件

发布于 2024-12-29 01:58:59 字数 2859 浏览 0 评论 0原文

当尝试在 Twisted 中通过 HTTP 发送文件时,我遇到了一个相当烦人的问题。我对 Python 相当陌生(因为这是我第一次真正尝试做除 Hello World 等之外的任何事情)。

现在,我相当确定当 FileSender 发送不兼容 UTF-8 的二进制数据时会出现问题。在通过网络移动图像、视频、可执行二进制文件等文件时,这似乎是相当常见的事情。

在我看来,初始化时,Twisted 缓冲区被初始化为 "",它(我可能在这里犯了很大的错误)是一个 UTF-8 (或者你的 Python 的默认字符编码是什么)字符string?...然后,当新数据添加到缓冲区时,旧缓冲区(UTF-8)与新缓冲区连接(呃...其他一些二进制编码?),这会导致:

Unhandled Error
Traceback (most recent call last):
Failure: exceptions.RuntimeError: Producer was not unregistered for /vfs/videos/thevid.avi
Unhandled Error
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 84, in callWithLogger
    return callWithContext({"system": lp}, func, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 69, in callWithContext
    return context.call({ILogContext: newCtx}, func, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
--- <exception caught here> ---
  File "/usr/lib/python2.7/dist-packages/twisted/internet/selectreactor.py", line 146, in _doReadOrWrite
    why = getattr(selectable, method)()
  File "/usr/lib/python2.7/dist-packages/twisted/internet/tcp.py", line 428, in doWrite
    result = abstract.FileDescriptor.doWrite(self)
  File "/usr/lib/python2.7/dist-packages/twisted/internet/abstract.py", line 199, in doWrite
    self.dataBuffer = buffer(self.dataBuffer, self.offset) + "".join(self._tempDataBuffer)
  File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
exceptions.UnicodeDecodeError: 'utf8' codec can't decode byte 0xd3 in position 6: invalid continuation byte
Unhandled Error
Traceback (most recent call last):
Failure: exceptions.RuntimeError: Producer was not unregistered for /vfs/videos/thevid.avi

当然必须有一种可靠地设置二进制数据传输缓冲区的方法?我的文件发送功能的代码如下。我使用 TxJSONRPC 来创建和管理服务器(除了 render 方法,该方法被重写以处理不仅仅是 RPC 的情况)。如果需要更多信息(例如关于 Request 对象或 Reactor,我可以去挖掘)。

注意:我很清楚我没有错误处理程序,但目前这对我来说并不重要。一旦我可靠地传输文件,就会添加错误处理!

def writeFile(self, request, location):
    # Setup headers (mime type, filename)
    request.setHeader('Content-Type', mimetypes.guess_type("file://" + location))
    request.setHeader('Content-Disposition', "attachment; filename=" + path.basename(location))

    # Open file
    handle = open(location, "rb")

    # Setup transfer (then cleanup etc)
    d = FileSender().beginFileTransfer(handle, request)
    def fileFinished(ignored):
        handle.close()
        request.finish()
    d.addCallback(fileFinished).addErrback(fileFinished)

I have a rather annoying problem when trying to send a file via HTTP in Twisted. I'm reasonably new to Python (being that this is my first real attempt to do anything other than Hello World etc).

Now, I'm fairly sure that the problem occurs when the FileSender sends binary data that is not UTF-8 compatible. That seems like a fairly common thing to do when moving files like images, videos, executable binaries around over the network.

It would seem to me that when initialized, the Twisted buffer is initialized to "", which (I may be very mistaken here) is a UTF-8 (or whatever your Python's default character encoding is) character string?... Then, when new data is added to the buffer, the old buffer (UTF-8) is joined with the new buffer (er... some other binary encoding?) which results in:

Unhandled Error
Traceback (most recent call last):
Failure: exceptions.RuntimeError: Producer was not unregistered for /vfs/videos/thevid.avi
Unhandled Error
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 84, in callWithLogger
    return callWithContext({"system": lp}, func, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/python/log.py", line 69, in callWithContext
    return context.call({ILogContext: newCtx}, func, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 118, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, in callWithContext
    return func(*args,**kw)
--- <exception caught here> ---
  File "/usr/lib/python2.7/dist-packages/twisted/internet/selectreactor.py", line 146, in _doReadOrWrite
    why = getattr(selectable, method)()
  File "/usr/lib/python2.7/dist-packages/twisted/internet/tcp.py", line 428, in doWrite
    result = abstract.FileDescriptor.doWrite(self)
  File "/usr/lib/python2.7/dist-packages/twisted/internet/abstract.py", line 199, in doWrite
    self.dataBuffer = buffer(self.dataBuffer, self.offset) + "".join(self._tempDataBuffer)
  File "/usr/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
exceptions.UnicodeDecodeError: 'utf8' codec can't decode byte 0xd3 in position 6: invalid continuation byte
Unhandled Error
Traceback (most recent call last):
Failure: exceptions.RuntimeError: Producer was not unregistered for /vfs/videos/thevid.avi

Surely there has to be a way to reliably set the buffer for binary data transfer? My code for the file sending function is below. I'm using TxJSONRPC to create and manage the server (except the render method, which is overridden to handle more than just RPC). If more information is needed (eg about the Request object or the Reactor, I can go digging).

Note: I'm well aware that I have no error handler, but at the moment that is of little importance to me. Error handling will be added once I get files transferring reliably!

def writeFile(self, request, location):
    # Setup headers (mime type, filename)
    request.setHeader('Content-Type', mimetypes.guess_type("file://" + location))
    request.setHeader('Content-Disposition', "attachment; filename=" + path.basename(location))

    # Open file
    handle = open(location, "rb")

    # Setup transfer (then cleanup etc)
    d = FileSender().beginFileTransfer(handle, request)
    def fileFinished(ignored):
        handle.close()
        request.finish()
    d.addCallback(fileFinished).addErrback(fileFinished)

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。
列表为空,暂无数据
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文