PyPy 明显慢于 CPython

发布于 2024-11-29 17:45:51 字数 7387 浏览 5 评论 0原文

我一直在测试我制作的缓存系统。其目的是加速 Django Web 应用程序。它将所有内容存储在内存中。根据 cProfile，我的测试中的大部分时间都花在 QuerySet._clone() 内，结果证明效率非常低（考虑到实现，这实际上并不奇怪）。

我对使用 PyPy 来加快速度寄予厚望。我有一台 64 位机器。然而，在安装所有必需的库后，发现 PyPy 编译的代码运行速度比常规 Python 代码慢约 2.5 倍，而且我不知道如何利用它。该代码受 CPU 限制（绝对没有数据库查询，因此 IO 限制不是一个选项）。单个测试运行大约 10 秒，所以我想这应该足以让 JIT 启动。我使用的是 PyPy 1.5。需要注意的是 - 我自己没有编译源代码，只是下载了 64 位 Linux 版本。

我想知道 CPU 密集型代码在 PyPy 下实际运行速度变慢的频率有多少。我是否可能做错了什么，导致 PyPy 无法以最佳状态运行。

编辑

精确的cPython输出：

PyPy 1.5：

    3439146 function calls (3218654 primitive calls) in 19.094 seconds

    Ordered by: cumulative time

    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       2/1    0.000    0.000   18.956   18.956 <string>:1(<module>)
       2/1    0.000    0.000   18.956   18.956 /path/to/my/project/common/integrity/models/transactions.py:200(newfn)
       2/1    0.000    0.000   18.956   18.956 /path/to/my/project/common/integrity/models/transactions.py:134(recur)
       2/1    0.000    0.000   18.956   18.956 /usr/local/pypy/site-packages/django/db/transaction.py:210(inner)
       2/1    0.172    0.086   18.899   18.899 /path/to/my/project/common/integrity/tests/optimization.py:369(func_cached)
      9990    0.122    0.000   18.632    0.002 /usr/local/pypy/site-packages/django/db/models/manager.py:131(get)
      9990    0.127    0.000   16.638    0.002 /path/to/my/project/common/integrity/models/cache.py:1068(get)
      9990    0.073    0.000   12.478    0.001 /usr/local/pypy/site-packages/django/db/models/query.py:547(filter)
      9990    0.263    0.000   12.405    0.001 /path/to/my/project/common/integrity/models/cache.py:1047(_filter_or_exclude)
      9990    0.226    0.000   12.096    0.001 /usr/local/pypy/site-packages/django/db/models/query.py:561(_filter_or_exclude)
      9990    0.187    0.000    8.383    0.001 /path/to/my/project/common/integrity/models/cache.py:765(_clone)
      9990    0.212    0.000    7.662    0.001 /usr/local/pypy/site-packages/django/db/models/query.py:772(_clone)
      9990    1.025    0.000    7.125    0.001 /usr/local/pypy/site-packages/django/db/models/sql/query.py:226(clone)
129942/49972  1.674    0.000    6.021    0.000 /usr/local/pypy/lib-python/2.7/copy.py:145(deepcopy)
140575/110605 0.120    0.000    4.066    0.000 {len}
      9990    0.182    0.000    3.972    0.000 /usr/local/pypy/site-packages/django/db/models/query.py:74(__len__)
     19980    0.260    0.000    3.777    0.000 /path/to/my/project/common/integrity/models/cache.py:1062(iterator)
      9990    0.255    0.000    3.154    0.000 /usr/local/pypy/site-packages/django/db/models/sql/query.py:1149(add_q)
      9990    0.210    0.000    3.073    0.000 /path/to/my/project/common/integrity/models/cache.py:973(_query)
      9990    0.371    0.000    2.316    0.000 /usr/local/pypy/site-packages/django/db/models/sql/query.py:997(add_filter)
      9990    0.364    0.000    2.168    0.000 /path/to/my/project/common/integrity/models/cache.py:892(_deduct)
29974/9994    0.448    0.000    2.078    0.000 /usr/local/pypy/lib-python/2.7/copy.py:234(_deepcopy_tuple)
     19990    0.362    0.000    2.065    0.000 /path/to/my/project/common/integrity/models/cache.py:566(__init__)
     10000    0.086    0.000    1.874    0.000 /path/to/my/project/common/integrity/models/cache.py:1090(get_query_set)
     19990    0.269    0.000    1.703    0.000 /usr/local/pypy/site-packages/django/db/models/query.py:31(__init__)
      9990    0.122    0.000    1.643    0.000 /path/to/my/project/common/integrity/models/cache.py:836(_deduct_recur)
     19980    0.274    0.000    1.636    0.000 /usr/local/pypy/site-packages/django/utils/tree.py:55(__deepcopy__)
      9990    0.607    0.000    1.458    0.000 /path/to/my/project/common/integrity/models/cache.py:789(_deduct_local)
     10020    0.633    0.000    1.437    0.000 /usr/local/pypy/site-packages/django/db/models/sql/query.py:99(__init__)
    129942    0.841    0.000    1.191    0.000 /usr/local/pypy/lib-python/2.7/copy.py:267(_keep_alive)
 9994/9992    0.201    0.000    1.019    0.000 /usr/local/pypy/lib-python/2.7/copy.py:306(_reconstruct)

Python 2.7：

   3326403 function calls (3206359 primitive calls) in 12.430 CPU seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000   12.457   12.457 <string>:1(<module>)
        1    0.000    0.000   12.457   12.457 /path/to/my/project/common/integrity/models/transactions.py:200(newfn)
        1    0.000    0.000   12.457   12.457 /path/to/my/project/common/integrity/models/transactions.py:134(recur)
        1    0.000    0.000   12.457   12.457 /usr/local/lib/python2.7/dist-packages/django/db/transaction.py:210(inner)
        1    0.000    0.000   12.457   12.457 /path/to/my/project/common/integrity/models/transactions.py:165(recur2)
        1    0.089    0.089   12.450   12.450 /path/to/my/project/common/integrity/tests/optimization.py:369(func_cached)
     9990    0.198    0.000   12.269    0.001 /usr/local/lib/python2.7/dist-packages/django/db/models/manager.py:131(get)
     9990    0.087    0.000   11.281    0.001 /path/to/my/project/common/integrity/models/cache.py:1068(get)
     9990    0.040    0.000    8.161    0.001 /usr/local/lib/python2.7/dist-packages/django/db/models/query.py:547(filter)
     9990    0.110    0.000    8.121    0.001 /path/to/my/project/common/integrity/models/cache.py:1047(_filter_or_exclude)
     9990    0.127    0.000    7.983    0.001 /usr/local/lib/python2.7/dist-packages/django/db/models/query.py:561(_filter_or_exclude)
     9990    0.100    0.000    5.593    0.001 /path/to/my/project/common/integrity/models/cache.py:765(_clone)
     9990    0.122    0.000    5.125    0.001 /usr/local/lib/python2.7/dist-packages/django/db/models/query.py:772(_clone)
     9990    0.405    0.000    4.899    0.000 /usr/local/lib/python2.7/dist-packages/django/db/models/sql/query.py:226(clone)
129942/49972 1.456    0.000    4.505    0.000 /usr/lib/python2.7/copy.py:145(deepcopy)
129899/99929 0.191    0.000    3.117    0.000 {len}
     9990    0.111    0.000    2.968    0.000 /usr/local/lib/python2.7/dist-packages/django/db/models/query.py:74(__len__)
    19980    0.070    0.000    2.843    0.000 /path/to/my/project/common/integrity/models/cache.py:1062(iterator)
     9990    0.208    0.000    2.190    0.000 /path/to/my/project/common/integrity/models/cache.py:973(_query)
     9990    0.182    0.000    2.114    0.000 /usr/local/lib/python2.7/dist-packages/django/db/models/sql/query.py:1149(add_q)
19984/9994   0.291    0.000    1.644    0.000 /usr/lib/python2.7/copy.py:234(_deepcopy_tuple)
     9990    0.288    0.000    1.599    0.000 /usr/local/lib/python2.7/dist-packages/django/db/models/sql/query.py:997(add_filter)
     9990    0.171    0.000    1.454    0.000 /path/to/my/project/common/integrity/models/cache.py:892(_deduct)
    19980    0.177    0.000    1.208    0.000 /usr/local/lib/python2.7/dist-packages/django/utils/tree.py:55(__deepcopy__)
     9990    0.099    0.000    1.199    0.000 /path/to/my/project/common/integrity/models/cache.py:836(_deduct_recur)
     9990    0.349    0.000    1.040    0.000 /path/to/my/project/common/integrity/models/cache.py:789(_deduct_local)

原文

I've been testing a cacheing system of my making. Its purpose is to speed up a Django web application. It stores everything in-memory. According to cProfile most of the time in my tests is spent inside QuerySet._clone() which turns out to be terribly inefficient (it's actually not that strange given the implementation).

I was having high hopes for using PyPy to speed things up. I've got a 64-bit machine. However after installing all the required libraries it turns out that PyPy compiled code runs about 2.5x slower than regular Python code, and I don't know what to make out of it. The code is CPU bound (there are absolutely no database queries, so IO-bounding is not an option). A single test runs for about 10 seconds, so I guess it should be enough for JIT to kick in. I'm using PyPy 1.5. One note - I didn't compile the sources myself, just downloaded a 64-bit linux version.

I'd like to know how frequent it is for a CPU intensive code to actually run slower under PyPy. Is there hopefully something wrong I could have done that would prevent PyPy from running at its best.

EDIT

Exact cPython output:

PyPy 1.5:

    3439146 function calls (3218654 primitive calls) in 19.094 seconds

    Ordered by: cumulative time

    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       2/1    0.000    0.000   18.956   18.956 <string>:1(<module>)
       2/1    0.000    0.000   18.956   18.956 /path/to/my/project/common/integrity/models/transactions.py:200(newfn)
       2/1    0.000    0.000   18.956   18.956 /path/to/my/project/common/integrity/models/transactions.py:134(recur)
       2/1    0.000    0.000   18.956   18.956 /usr/local/pypy/site-packages/django/db/transaction.py:210(inner)
       2/1    0.172    0.086   18.899   18.899 /path/to/my/project/common/integrity/tests/optimization.py:369(func_cached)
      9990    0.122    0.000   18.632    0.002 /usr/local/pypy/site-packages/django/db/models/manager.py:131(get)
      9990    0.127    0.000   16.638    0.002 /path/to/my/project/common/integrity/models/cache.py:1068(get)
      9990    0.073    0.000   12.478    0.001 /usr/local/pypy/site-packages/django/db/models/query.py:547(filter)
      9990    0.263    0.000   12.405    0.001 /path/to/my/project/common/integrity/models/cache.py:1047(_filter_or_exclude)
      9990    0.226    0.000   12.096    0.001 /usr/local/pypy/site-packages/django/db/models/query.py:561(_filter_or_exclude)
      9990    0.187    0.000    8.383    0.001 /path/to/my/project/common/integrity/models/cache.py:765(_clone)
      9990    0.212    0.000    7.662    0.001 /usr/local/pypy/site-packages/django/db/models/query.py:772(_clone)
      9990    1.025    0.000    7.125    0.001 /usr/local/pypy/site-packages/django/db/models/sql/query.py:226(clone)
129942/49972  1.674    0.000    6.021    0.000 /usr/local/pypy/lib-python/2.7/copy.py:145(deepcopy)
140575/110605 0.120    0.000    4.066    0.000 {len}
      9990    0.182    0.000    3.972    0.000 /usr/local/pypy/site-packages/django/db/models/query.py:74(__len__)
     19980    0.260    0.000    3.777    0.000 /path/to/my/project/common/integrity/models/cache.py:1062(iterator)
      9990    0.255    0.000    3.154    0.000 /usr/local/pypy/site-packages/django/db/models/sql/query.py:1149(add_q)
      9990    0.210    0.000    3.073    0.000 /path/to/my/project/common/integrity/models/cache.py:973(_query)
      9990    0.371    0.000    2.316    0.000 /usr/local/pypy/site-packages/django/db/models/sql/query.py:997(add_filter)
      9990    0.364    0.000    2.168    0.000 /path/to/my/project/common/integrity/models/cache.py:892(_deduct)
29974/9994    0.448    0.000    2.078    0.000 /usr/local/pypy/lib-python/2.7/copy.py:234(_deepcopy_tuple)
     19990    0.362    0.000    2.065    0.000 /path/to/my/project/common/integrity/models/cache.py:566(__init__)
     10000    0.086    0.000    1.874    0.000 /path/to/my/project/common/integrity/models/cache.py:1090(get_query_set)
     19990    0.269    0.000    1.703    0.000 /usr/local/pypy/site-packages/django/db/models/query.py:31(__init__)
      9990    0.122    0.000    1.643    0.000 /path/to/my/project/common/integrity/models/cache.py:836(_deduct_recur)
     19980    0.274    0.000    1.636    0.000 /usr/local/pypy/site-packages/django/utils/tree.py:55(__deepcopy__)
      9990    0.607    0.000    1.458    0.000 /path/to/my/project/common/integrity/models/cache.py:789(_deduct_local)
     10020    0.633    0.000    1.437    0.000 /usr/local/pypy/site-packages/django/db/models/sql/query.py:99(__init__)
    129942    0.841    0.000    1.191    0.000 /usr/local/pypy/lib-python/2.7/copy.py:267(_keep_alive)
 9994/9992    0.201    0.000    1.019    0.000 /usr/local/pypy/lib-python/2.7/copy.py:306(_reconstruct)

Python 2.7:

   3326403 function calls (3206359 primitive calls) in 12.430 CPU seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        1    0.000    0.000   12.457   12.457 <string>:1(<module>)
        1    0.000    0.000   12.457   12.457 /path/to/my/project/common/integrity/models/transactions.py:200(newfn)
        1    0.000    0.000   12.457   12.457 /path/to/my/project/common/integrity/models/transactions.py:134(recur)
        1    0.000    0.000   12.457   12.457 /usr/local/lib/python2.7/dist-packages/django/db/transaction.py:210(inner)
        1    0.000    0.000   12.457   12.457 /path/to/my/project/common/integrity/models/transactions.py:165(recur2)
        1    0.089    0.089   12.450   12.450 /path/to/my/project/common/integrity/tests/optimization.py:369(func_cached)
     9990    0.198    0.000   12.269    0.001 /usr/local/lib/python2.7/dist-packages/django/db/models/manager.py:131(get)
     9990    0.087    0.000   11.281    0.001 /path/to/my/project/common/integrity/models/cache.py:1068(get)
     9990    0.040    0.000    8.161    0.001 /usr/local/lib/python2.7/dist-packages/django/db/models/query.py:547(filter)
     9990    0.110    0.000    8.121    0.001 /path/to/my/project/common/integrity/models/cache.py:1047(_filter_or_exclude)
     9990    0.127    0.000    7.983    0.001 /usr/local/lib/python2.7/dist-packages/django/db/models/query.py:561(_filter_or_exclude)
     9990    0.100    0.000    5.593    0.001 /path/to/my/project/common/integrity/models/cache.py:765(_clone)
     9990    0.122    0.000    5.125    0.001 /usr/local/lib/python2.7/dist-packages/django/db/models/query.py:772(_clone)
     9990    0.405    0.000    4.899    0.000 /usr/local/lib/python2.7/dist-packages/django/db/models/sql/query.py:226(clone)
129942/49972 1.456    0.000    4.505    0.000 /usr/lib/python2.7/copy.py:145(deepcopy)
129899/99929 0.191    0.000    3.117    0.000 {len}
     9990    0.111    0.000    2.968    0.000 /usr/local/lib/python2.7/dist-packages/django/db/models/query.py:74(__len__)
    19980    0.070    0.000    2.843    0.000 /path/to/my/project/common/integrity/models/cache.py:1062(iterator)
     9990    0.208    0.000    2.190    0.000 /path/to/my/project/common/integrity/models/cache.py:973(_query)
     9990    0.182    0.000    2.114    0.000 /usr/local/lib/python2.7/dist-packages/django/db/models/sql/query.py:1149(add_q)
19984/9994   0.291    0.000    1.644    0.000 /usr/lib/python2.7/copy.py:234(_deepcopy_tuple)
     9990    0.288    0.000    1.599    0.000 /usr/local/lib/python2.7/dist-packages/django/db/models/sql/query.py:997(add_filter)
     9990    0.171    0.000    1.454    0.000 /path/to/my/project/common/integrity/models/cache.py:892(_deduct)
    19980    0.177    0.000    1.208    0.000 /usr/local/lib/python2.7/dist-packages/django/utils/tree.py:55(__deepcopy__)
     9990    0.099    0.000    1.199    0.000 /path/to/my/project/common/integrity/models/cache.py:836(_deduct_recur)
     9990    0.349    0.000    1.040    0.000 /path/to/my/project/common/integrity/models/cache.py:789(_deduct_local)

分享到QQ

分享到微博

如果你对这篇内容有疑问，欢迎到本站社区发帖提问参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

发布评论

需要登录才能够评论，你可以免费注册一个本站的账号。

向地狱狂奔 2024-12-06 17:45:51

抛开 PyPy 对于您的情况来说本质上可能会较慢的事实不谈，有一些因素可能会使其不必要地变慢：

众所周知，分析使 PyPy 的速度比 CPython 慢得多。
一些调试/日志记录代码可以禁用优化（例如通过强制帧）。
您使用的服务器可能是性能的主要因素（想想经典的 CGI 使用 JIT 会是多么糟糕：它永远不会预热）。它还可以简单地影响结果（不同的 WSGI 服务器显示出不同的加速效果）。
旧式课程比新式课程慢。
即使所有内容都在内存中，您也可能会遇到 PyPy 的 SQLite 中的慢速路径。

您还可以查看 JIT Friendiness wiki 页面，了解有关哪些因素会使 PyPy 变慢的更多提示。每夜构建可能也会更快，因为相对于 1.5 有许多改进。

对您的堆栈（服务器、操作系统、数据库）和设置（您如何进行基准测试？有多少查询？）的更详细描述将使我们能够给出更好的答案。