Python 2.x 与 Python 3.x 中的 BufferedReader

发布于 2024-10-09 03:32:08 字数 831 浏览 5 评论 0原文

我有一个在 Python 2 和 Python 3 中运行的程序，但速度有很大差异。据我所知，交换机中进行了许多内部更改，但 io.BufferedReader 的差异确实很大。在这两个版本中，我都使用 io.BufferedReader，因为主程序循环一次只需要一个字节的数据。以下是脚本的 cProfile 输出的摘录（请参阅 cumtime，而不是 tottime）：

Python 2:
 ncalls  tottime  percall  cumtime  percall filename:lineno(function)
 36984   0.188    0.000    0.545    0.000   io.py:929(read)

Python 3:
 36996    0.063   0.000    0.063    0.000   {method 'read' of '_io.BufferedReader' objects}

当我打印对象时，两者都会返回类似 io.BufferedReader 的内容，因此我确定他们都使用 BufferedReader。

这里是有问题的代码。参见第 28 行。调用者负责设置 bufstream。我使用了 bufstream = io.open('testfile', 'rb')

为什么 BufferedReader 读取文件中单个字节的速度存在如此巨大的差异，以及如何“修复” Python 2.x 的问题？我正在运行 Python 2.6 和 Python 3.1。

原文

I have a program that runs in Python 2 and Python 3, but there is a drastic difference in speed. I understand a number of internal changes were made in the switch, but the difference in io.BufferedReader are really high. In both versions, I use io.BufferedReader because the main program loop only needs data one byte at a time. Here is an excerpt from the cProfile output for the script (see cumtime, not tottime):

Python 2:
 ncalls  tottime  percall  cumtime  percall filename:lineno(function)
 36984   0.188    0.000    0.545    0.000   io.py:929(read)

Python 3:
 36996    0.063   0.000    0.063    0.000   {method 'read' of '_io.BufferedReader' objects}

When I print the object, both return something like io.BufferedReader so I am certain they are both using BufferedReader.

Here is the code in question. See line 28. The caller is responsible for setting up bufstream. I used bufstream = io.open('testfile', 'rb')

Why is there such a drastic difference in speed of BufferedReader for reading single bytes in the files, and how can I "fix" the issue for Python 2.x? I am running Python 2.6 and Python 3.1.

分享到QQ

分享到微博

如果你对这篇内容有疑问，欢迎到本站社区发帖提问参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

发布评论

需要登录才能够评论，你可以免费注册一个本站的账号。

别低头，皇冠会掉 2024-10-16 03:32:08

为了给您更完整的答案，人们需要查看您的代码（或者更好的是，查看代码的可执行摘要）。

然而，可以从您的配置文件输出中收集到部分答案：io.py表明“Python 2”（为避免疑问，给出实际版本号）正在Python中实现BufferedReader，而_io.BufferedReader 建议“Python3”正在 C 中实现它。

最新消息：Python 2.6 的 io.py 超过 64Kb，并在前面包含以下注释：

# This is a prototype; hopefully eventually some of this will be
# reimplemented in C.

Python 2.7 的 io.py 大约 4Kb，似乎是 _io 模块的薄包装。

如果您需要 2.6 解决方法的真正帮助，请显示您的代码。

Python 2.6 的可能解决方法

而不是：

test = io.open('test.bmp', 'rb')

这样做：

test = open('test.bmp', 'rb')

一些粗略的计时数字，包括缺失的链接（Python 2.7）：

Windows 7 Pro，32位，大约5 Mb文件，代码的核心是：

while 1:
    c = f.read(1)
    if not c: break

2.6: io.open 20.4s, open 5.1s
2.7: io.open  3.3s, open 4.8s # io.open is better
3.1: io.open  3.6s, open 3.6s # effectively same code is used

所以一个更好的故事似乎是这样的：一般来说，不要迷恋io。除非您有充分的理由，例如您希望 2.7 运行得更快，否则请打开。

To give you a fuller answer, one would need to see your code (or, better, an executable precis of your code).

However a partial answer can be gleaned from your profile output: io.py suggests that "Python 2" (for avoidance of doubt, give the actual version numbers) is implementing BufferedReader in Python, whereas _io.BufferedReader suggests that "Python3" is implementing it in C.

Late-breaking news: Python 2.6's io.py is over 64Kb and includes the following comment up the front :

# This is a prototype; hopefully eventually some of this will be
# reimplemented in C.

Python 2.7's io.py is about 4Kb and appears to be a thin wrapper of an _io module.

If you want real assistance with a workaround for 2.6, show your code.

Probable workaround for Python 2.6

Instead of:

test = io.open('test.bmp', 'rb')

do this:

test = open('test.bmp', 'rb')

Some rough timing figures, including the missing link (Python 2.7):

Windows 7 Pro, 32-bit, approx 5 Mb file, guts of code is:

while 1:
    c = f.read(1)
    if not c: break

2.6: io.open 20.4s, open 5.1s
2.7: io.open  3.3s, open 4.8s # io.open is better
3.1: io.open  3.6s, open 3.6s # effectively same code is used

So a better story seems to be this: In general, don't faff about with io.open unless you have good reason to e.g. you want 2.7 to go faster.

回复收藏 0 原文