为什么 Jython 在使用 PyStone 测试时表现不一致？

发布于 2024-07-13 22:23:38 字数 2021 浏览 6 评论 0原文

我最近一直在使用 Jython，并决定使用 pystone 进行一些快速而肮脏的基准测试。为了获得参考，我首先测试了 cPython 2.6，循环数量不断增加（我认为这可能是相关的，因为 Jython 应该在一段时间后才开始从 JIT 中获利）。

(richard garibaldi):/usr/local/src/pybench% python ~/tmp/pystone.py 
Pystone(1.1) time for 50000 passes = 1.04
This machine benchmarks at 48076.9 pystones/second

(richard garibaldi):/usr/local/src/pybench% python ~/tmp/pystone.py 500000 
Pystone(1.1) time for 500000 passes = 10.33
This machine benchmarks at 48402.7 pystones/second

(richard garibaldi):/usr/local/src/pybench% python ~/tmp/pystone.py 1000000 
Pystone(1.1) time for 1000000 passes = 19.6
This machine benchmarks at 51020.4 pystones/second

正如您所看到的，cPython 的行为一致：完成测试所需的时间随循环数量线性增加。知道这一点后，我开始测试 Jython。

(richard garibaldi):/usr/local/src/pybench% jython ~/tmp/pystone.py 
Pystone(1.1) time for 50000 passes = 2.29807
This machine benchmarks at 21757.4 pystones/second

(richard garibaldi):/usr/local/src/pybench% jython ~/tmp/pystone.py 500000 
Pystone(1.1) time for 500000 passes = 10.931
This machine benchmarks at 45741.4 pystones/second


(richard garibaldi):/usr/local/src/pybench% jython ~/tmp/pystone.py 1000000
Pystone(1.1) time for 1000000 passes = 107.183
This machine benchmarks at 9329.86 pystones/second

在第一次运行期间，Jython 与其 C 兄弟相比运行得相当糟糕。当增加循环数量时，它开始感觉更好，接近 cPython，就像我最初的假设预测的那样。请注意，循环数量增加了 10 倍，但 Jython 完成循环所需的时间只长了大约 5 倍。因此，正如您想象的那样，我期待 Jython 在最终测试中真正表现出色。然而，令我非常失望的是，它的表现非常糟糕：比最初运行时慢了两倍多。

您的假设是什么：为什么 Jython 的行为方式如此不一致？ GC 是否会在某个时刻启动并花费大量时间？我查看了 PyStone 的代码，垃圾收集似乎没有被关闭，但我希望 Java 的 GC 至少和 Python 的一样好......你认为这种减慢是永久性的，还是会消失在增加循环次数后的某个时刻？ Jython 在真正长时间运行的进程中应如何表现？

编辑：不幸的是，如果我将循环数增加到 200 万，我会得到 java.lang.OutOfMemoryError...

（当然，Jython 仍处于测试阶段，因此在最终版本中应该会变得更好。）

我正在使用 Jython 2.5b1（主干：5903：5905，2009 年 1 月 9 日，16:01:29）、Java(TM) SE 运行时环境（build 1.6.0_07-b06-153）和 Java HotSpot(TM) 64- MacOS X 10.5 上的 Bit Server VM（内部版本 1.6.0_07-b06-57，混合模式）。

感谢您的回答。

原文

I've been playing recently with Jython and decided to do some quick and dirty benchmarking with pystone. In order to have a reference, I first tested cPython 2.6, with an increasing numbers of loops (I thought this may be relevant as Jython should start to profit from the JIT only after some time).

(richard garibaldi):/usr/local/src/pybench% python ~/tmp/pystone.py 
Pystone(1.1) time for 50000 passes = 1.04
This machine benchmarks at 48076.9 pystones/second

(richard garibaldi):/usr/local/src/pybench% python ~/tmp/pystone.py 500000 
Pystone(1.1) time for 500000 passes = 10.33
This machine benchmarks at 48402.7 pystones/second

(richard garibaldi):/usr/local/src/pybench% python ~/tmp/pystone.py 1000000 
Pystone(1.1) time for 1000000 passes = 19.6
This machine benchmarks at 51020.4 pystones/second

As you can see, cPython behaves consistently: the time it takes to complete the test increases linearly to the number of loops. Knowing this, I started testing Jython.

(richard garibaldi):/usr/local/src/pybench% jython ~/tmp/pystone.py 
Pystone(1.1) time for 50000 passes = 2.29807
This machine benchmarks at 21757.4 pystones/second

(richard garibaldi):/usr/local/src/pybench% jython ~/tmp/pystone.py 500000 
Pystone(1.1) time for 500000 passes = 10.931
This machine benchmarks at 45741.4 pystones/second


(richard garibaldi):/usr/local/src/pybench% jython ~/tmp/pystone.py 1000000
Pystone(1.1) time for 1000000 passes = 107.183
This machine benchmarks at 9329.86 pystones/second

During the first run Jython runs rather lousily in comparison to its C brother. When increased the number of loops it started feeling better, coming close to cPython, like my initial hypothesis predicted. Note that the number of loops increased 10 times, but it took Jython only about 5 times longer to complete them. So, as you imagine, I was expecting that Jython would really rock in the final test. To my great disappointment, however, it did really bad: more than twice slower than in the initial run.

What are your hypotheses: why does Jython behave such an inconsistent manner? Could it be that GC is kicking in at some moment, and taking a lot of time? I've looked at PyStone's code and garbage collection doesn't seem to be turned off, but I would expect Java's GC to be at least as good as Python's... Do you think this slowing down is permanent, or it will go away at some point after increasing the number of loops? How shall Jython behave in a really long running processes?

EDIT: unfortunately, I get java.lang.OutOfMemoryError if I increase the number of loops to 2 million...

(Of course, Jython is still beta, so it should get better in the final release.)

I am using Jython 2.5b1 (trunk:5903:5905, Jan 9 2009, 16:01:29), Java(TM) SE Runtime Environment (build 1.6.0_07-b06-153) and Java HotSpot(TM) 64-Bit Server VM (build 1.6.0_07-b06-57, mixed mode) on MacOS X 10.5.

Thanks for your answers.

分享到QQ

分享到微博

如果你对这篇内容有疑问，欢迎到本站社区发帖提问参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

发布评论

需要登录才能够评论，你可以免费注册一个本站的账号。

放肆 2024-07-20 22:23:39

对像 JVM 这样复杂的运行时环境进行基准测试非常困难。即使排除 JIT 和 GC，运行之间也会有很大的堆、内存布局和缓存变化。

对 Jython 有帮助的一件事就是在单个 VM 会话中多次运行基准测试：一次用于预热 JIT，一次或多次单独测量。我已经做了很多 Jython 基准测试，不幸的是，通常需要 10-50 次尝试才能达到合理的时间。

您可以使用一些 JVM 标志来观察 GC 和 JIT 行为，以了解预热期应该多长，尽管很明显您不应该在打开调试标志的情况下进行基准测试。例如：

% ./jython -J-XX:+PrintCompilation -J-verbose:gc
  1       java.lang.String::hashCode (60 bytes)
  2       java.lang.String::charAt (33 bytes)
  3       java.lang.String::lastIndexOf (156 bytes)
  4       java.lang.String::indexOf (151 bytes)
[GC 1984K->286K(7616K), 0.0031513 secs]

如果您执行所有这些操作并使用 HotSpot Server VM，您会发现 Jython 比 pystone 上的 CPython 稍快，但这绝不代表 Jython 的总体性能。对于 2.5 版本，Jython 开发人员更加关注正确性而不是性能；未来一年左右随着2.6/2.7/3.0的发布性能会更加强调。通过查看一些微基准，您可以发现一些痛点（最初源自 PyPy）我运行。

Benchmarking a runtime environment as complex as the JVM is hard. Even excluding the JIT and GC, you've got a big heap, memory layout and cache variation between runs.

One thing that helps with Jython is simply running the benchmark more than once in a single VM session: once to warm up the JIT and one or more times you measure individually. I've done a lot of Jython benchmarking, and unfortunately it often takes 10-50 attempts to achieve a reasonable time

You can use some JVM flags to observe GC and JIT behavior to get some idea how long the warmup period should be, though obviously you shouldn't benchmark with the debugging flags turned on. For example:

% ./jython -J-XX:+PrintCompilation -J-verbose:gc
  1       java.lang.String::hashCode (60 bytes)
  2       java.lang.String::charAt (33 bytes)
  3       java.lang.String::lastIndexOf (156 bytes)
  4       java.lang.String::indexOf (151 bytes)
[GC 1984K->286K(7616K), 0.0031513 secs]

If you do all this, and use the HotSpot Server VM, you'll find Jython slightly faster than CPython on pystone, but this is in no way representative of Jython performance in general. The Jython developers are paying much more attention to correctness than performance for the 2.5 release; over the next year or so with a 2.6/2.7/3.0 release performance will be more emphasized. You can see a few of the pain points by looking at some microbenchmarks (originally derived from PyPy) I run.

回复收藏 0 原文

笑，眼淚并存 2024-07-20 22:23:39

这可能是 jython 2.5b1 中的错误。您应该考虑将其报告给 jython 团队。我刚刚在 MacBook 上使用当前稳定版本的 jython (2.2.1) 运行了 pystone 基准测试，得到了缓慢但一致的结果：

mo$ ~/Coding/Jython/jython2.2.1/jython pystone.py 50000
Pystone(1.1) time for 50000 passes = 2.365
This machine benchmarks at 21141.6 pystones/second

mo$ ~/Coding/Jython/jython2.2.1/jython pystone.py 500000
Pystone(1.1) time for 500000 passes = 22.246
This machine benchmarks at 22476 pystones/second

mo$ ~/Coding/Jython/jython2.2.1/jython pystone.py 1000000
Pystone(1.1) time for 1000000 passes = 43.94
This machine benchmarks at 22758.3 pystones/second

mo$ java -version
java version "1.5.0_16"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b06-275)
Java HotSpot(TM) Client VM (build 1.5.0_16-132, mixed mode, sharing)

对我来说，cPython 结果或多或少是相同的。我重新进行了三次测试，每次都得到非常相似的结果。

我还尝试给java一个更大的初始和最大堆（-Xms256m -Xmx512m），但没有值得注意的结果

但是，将JVM设置为-server（启动速度较慢，长时间运行性能更好，对于“交互式”工作不太好）扭转了局面一点：

mo$ ~/Coding/Jython/jython2.2.1/jython pystone.py 50000
Pystone(1.1) time for 50000 passes = 1.848
This machine benchmarks at 27056.3 pystones/second

mo$ ~/Coding/Jython/jython2.2.1/jython pystone.py 500000
Pystone(1.1) time for 500000 passes = 9.998
This machine benchmarks at 50010 pystones/second

mo$ ~/Coding/Jython/jython2.2.1/jython pystone.py 1000000
Pystone(1.1) time for 1000000 passes = 19.9
This machine benchmarks at 50251.3 pystones/second

我用 (-server -Xms256m -Xmx512m) 进行了最后一次运行：

mo$ ~/Coding/Jython/jython2.2.1/jython pystone.py 5000000
Pystone(1.1) time for 5000000 passes = 108.664
This machine benchmarks at 46013.4 pystones/second

我的猜测是，第一次运行缓慢是由于 VM 启动/JIT 尚未真正启动。较长时间运行的结果更多或者不太一致并显示热点/JIT 的效果

也许您可以使用更大的堆重新运行上次测试？要更改 JVM 开关，只需编辑 Jython 安装中的 jython 文件即可。

This might be a bug in jython 2.5b1. You should consider reporting it back to the jython team. I have just run the pystone benchmark on my MacBook with the current stable release of jython (2.2.1) and I get slow but consistent results:

mo$ ~/Coding/Jython/jython2.2.1/jython pystone.py 50000
Pystone(1.1) time for 50000 passes = 2.365
This machine benchmarks at 21141.6 pystones/second

mo$ ~/Coding/Jython/jython2.2.1/jython pystone.py 500000
Pystone(1.1) time for 500000 passes = 22.246
This machine benchmarks at 22476 pystones/second

mo$ ~/Coding/Jython/jython2.2.1/jython pystone.py 1000000
Pystone(1.1) time for 1000000 passes = 43.94
This machine benchmarks at 22758.3 pystones/second

mo$ java -version
java version "1.5.0_16"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b06-275)
Java HotSpot(TM) Client VM (build 1.5.0_16-132, mixed mode, sharing)

The cPython results for me are more or less the same. I reran eacht test three times and got very similar results all the time.

I also tried giving java a bigger initial and maximum Heap (-Xms256m -Xmx512m) without a noteworthy result

However, setting the JVM to -server (slower startup, better long running performance, not so good for "interactive" work) turned the picture a bit:

mo$ ~/Coding/Jython/jython2.2.1/jython pystone.py 50000
Pystone(1.1) time for 50000 passes = 1.848
This machine benchmarks at 27056.3 pystones/second

mo$ ~/Coding/Jython/jython2.2.1/jython pystone.py 500000
Pystone(1.1) time for 500000 passes = 9.998
This machine benchmarks at 50010 pystones/second

mo$ ~/Coding/Jython/jython2.2.1/jython pystone.py 1000000
Pystone(1.1) time for 1000000 passes = 19.9
This machine benchmarks at 50251.3 pystones/second

I made one final run with (-server -Xms256m -Xmx512m):

mo$ ~/Coding/Jython/jython2.2.1/jython pystone.py 5000000
Pystone(1.1) time for 5000000 passes = 108.664
This machine benchmarks at 46013.4 pystones/second

My guess would be, that the slow first run is due to VM startup/JIT not yet having really kicked in. The results of the longer runs are more or less consitent and show the effects of hotspot/JIT

Maybe you could rerun your last test with a bigger heap? To change the JVM switches, just edit the jython file in your Jython installation.

回复收藏 0 原文

高冷爸爸 2024-07-20 22:23:39

我的笔记本电脑运行 Ubuntu Jaunty，使用 JRE 1.6.0_12-b04，结果相同：

nathell@breeze:/usr/lib/python2.5/test$ python pystone.py 500000
Pystone(1.1) time for 500000 passes = 12.98
This machine benchmarks at 38520.8 pystones/second

nathell@breeze:/usr/lib/python2.5/test$ python pystone.py 1000000
Pystone(1.1) time for 1000000 passes = 26.05
This machine benchmarks at 38387.7 pystones/second

nathell@breeze:/usr/lib/python2.5/test$ ~/jython/jython pystone.py
Pystone(1.1) time for 50000 passes = 2.47788
This machine benchmarks at 20178.6 pystones/second

nathell@breeze:/usr/lib/python2.5/test$ ~/jython/jython pystone.py 500000
Pystone(1.1) time for 500000 passes = 19.7294
This machine benchmarks at 25342.9 pystones/second

nathell@breeze:/usr/lib/python2.5/test$ ~/jython/jython pystone.py 1000000
Pystone(1.1) time for 1000000 passes = 38.9272
This machine benchmarks at 25689 pystones/second

所以也许这毕竟与 JRE 而不是 Jython 版本有关。 Armed Bear Common Lisp 项目在 JRE 1.6 早期版本中遇到的问题也可能暗示了这一点。

The same results from my laptop running Ubuntu Jaunty, with JRE 1.6.0_12-b04:

nathell@breeze:/usr/lib/python2.5/test$ python pystone.py 500000
Pystone(1.1) time for 500000 passes = 12.98
This machine benchmarks at 38520.8 pystones/second

nathell@breeze:/usr/lib/python2.5/test$ python pystone.py 1000000
Pystone(1.1) time for 1000000 passes = 26.05
This machine benchmarks at 38387.7 pystones/second

nathell@breeze:/usr/lib/python2.5/test$ ~/jython/jython pystone.py
Pystone(1.1) time for 50000 passes = 2.47788
This machine benchmarks at 20178.6 pystones/second

nathell@breeze:/usr/lib/python2.5/test$ ~/jython/jython pystone.py 500000
Pystone(1.1) time for 500000 passes = 19.7294
This machine benchmarks at 25342.9 pystones/second

nathell@breeze:/usr/lib/python2.5/test$ ~/jython/jython pystone.py 1000000
Pystone(1.1) time for 1000000 passes = 38.9272
This machine benchmarks at 25689 pystones/second

So perhaps this is related to the JRE rather than Jython version, after all. The problems the Armed Bear Common Lisp project has had with early versions of JRE 1.6 might also hint at this.

回复收藏 0 原文

歌枕肩 2024-07-20 22:23:39

我非常确定可以通过调整 JVM 配置来改进结果（JRuby 使用了很多有趣的标志来实现这一点），而且我也非常确定可以调整垃圾收集。
如果您对此基准测试非常感兴趣，这里有一个配置 VM 的好资源：调整垃圾收集。我还会看一下 JRuby 配置。

./亚历克斯

回复收藏 0 原文

兮颜 2024-07-20 22:23:39

我在 XP_Win32_PC 上的工作台：

C:\jython\jython2.5b1>bench "50000"

C:\jython\jython2.5b1>jython Lib\test\pystone.py "50000"
Pystone(1.1) time for 50000 passes = 1.73489
This machine benchmarks at 28820.2 pystones/second

C:\jython\jython2.5b1>bench "100000"

C:\jython\jython2.5b1>jython Lib\test\pystone.py "100000"
Pystone(1.1) time for 100000 passes = 3.36223
This machine benchmarks at 29742.2 pystones/second

C:\jython\jython2.5b1>bench "500000"

C:\jython\jython2.5b1>jython Lib\test\pystone.py "500000"
Pystone(1.1) time for 500000 passes = 15.8116
This machine benchmarks at 31622.3 pystones/second

C:\jython\jython2.5b1>bench "1000000"

C:\jython\jython2.5b1>jython Lib\test\pystone.py "1000000"
Pystone(1.1) time for 1000000 passes = 30.9763
This machine benchmarks at 32282.8 pystones/second

C:\jython\jython2.5b1>jython
Jython 2.5b1 (trunk:5903:5905, Jan 9 2009, 16:01:29)
[Java HotSpot(TM) Client VM (Sun Microsystems Inc.)] on java1.5.0_17

它不是那么快，但是...

没有“特殊效果”

这是 java-vm “问题”吗？

如果您想了解我在这台旧 Win32-PC 上进行基准测试的更多信息，请添加评论

my bench on a XP_Win32_PC :

C:\jython\jython2.5b1>bench "50000"

C:\jython\jython2.5b1>jython Lib\test\pystone.py "50000"
Pystone(1.1) time for 50000 passes = 1.73489
This machine benchmarks at 28820.2 pystones/second

C:\jython\jython2.5b1>bench "100000"

C:\jython\jython2.5b1>jython Lib\test\pystone.py "100000"
Pystone(1.1) time for 100000 passes = 3.36223
This machine benchmarks at 29742.2 pystones/second

C:\jython\jython2.5b1>bench "500000"

C:\jython\jython2.5b1>jython Lib\test\pystone.py "500000"
Pystone(1.1) time for 500000 passes = 15.8116
This machine benchmarks at 31622.3 pystones/second

C:\jython\jython2.5b1>bench "1000000"

C:\jython\jython2.5b1>jython Lib\test\pystone.py "1000000"
Pystone(1.1) time for 1000000 passes = 30.9763
This machine benchmarks at 32282.8 pystones/second

C:\jython\jython2.5b1>jython
Jython 2.5b1 (trunk:5903:5905, Jan 9 2009, 16:01:29)
[Java HotSpot(TM) Client VM (Sun Microsystems Inc.)] on java1.5.0_17

It is not so fast, but ...

no "special effects"

Is it a java-vm 'problem' ?

Add a comment if you want further infos to my benchmarking on this old Win32-PC

回复收藏 0 原文

~没有更多了~