对于LevelDB，如何才能获得与声称的“官方”相同的随机写入性能？绩效报告？
leveldb的官方网站（http://code.google.com/p/leveldb/），有一份性能报告。我粘贴如下。
以下来自官方 leveldb 基准测试
这是包含的 db_bench 程序运行时的性能报告（带解释）。结果有些嘈杂，但应该足以获得大概的性能估计。
设置
我们使用一个包含一百万个条目的数据库。每个条目都有一个 16 字节的密钥和一个 100 字节的值。基准测试使用的值压缩到原始大小的一半左右。 LevelDB：版本 1.1
CPU：4 x Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz
CPUCache：4096 KB
键：每个 16 字节
值：每个 100 字节（压缩后 50 字节）
条目：1000000
原始大小：110.6 MB（估计）
文件大小：62.9 MB（估计）
写入性能
“填充”基准以顺序或随机顺序创建一个全新的数据库。
“fillsync”基准测试在每次操作后将数据从操作系统刷新到磁盘；其他写入操作会将数据保留在操作系统缓冲区高速缓存中一段时间。 “覆盖”基准测试会执行随机写入来更新数据库中的现有密钥。
fillseq：1.765 微秒/操作； 62.7 MB/s
填充同步：268.409 微秒/操作； 0.4 MB/s（10000 次操作）
填充随机：2.460 微秒/操作； 45.0 MB/s
覆盖：2.380 微秒/操作； 46.5 MB/s
上面的每个“操作”对应于单个键/值对的写入。即，随机写入基准测试的速度为每秒大约 400,000 次写入。
下面是我的 leveldb 基准测试，
我对 leveldb 做了一些基准测试，但写入速度比报告低 100 倍。
这是我的实验设置：
- CPU：Intel Core2 Duo T6670 2.20GHz
- 3.0GB内存
- 32位Windows 7，
- 无压缩
- 选项。write_buffer_size = 100MB
- options.block_cache = 640MB
我所做的很简单：我只是放入200万个{key，value并且根本没有读取。键是一个字节数组，有 20 个随机字节，值也是一个字节数组，有 100 个随机字节。我不断地输入新随机的{key, value} 200万次，没有任何其他操作。
在我的实验中，我可以看到书写速度从一开始就下降了。瞬时速度（测量每 1024 次写入的速度）在 50/s 到 10, 000/s 之间波动。我的 200 万对的总体平均写入速度约为 3,000/秒。峰值写入速度为 10, 000/s。
由于报告声称写入速度可以达到 400, 000/s，我的基准测试的写入速度慢了 40 到 130 倍 strong> 我只是想知道我的基准有什么问题。
我不需要在这里粘贴我的测试代码，因为它非常简单，我只有一个 200 万次的 while 循环，并且在循环内部，对于每次迭代，我生成一个 20 字节的键和 100 字节的值，然后将它们放入leveldb数据库中。我还测量了 {key, value} 生成所花费的时间，它花费了 0 毫秒。
谁能帮我解决这个问题吗？如何使用 leveldb 达到 400、000/s 的写入速度？我应该改进哪些设置？
谢谢
此外，
我刚刚在我的机器上运行了官方的 db_bench.cc 。比报告慢了28倍。
我认为，当我使用他们自己的基准测试程序时，我的基准测试和他们的基准测试之间的唯一区别是机器。
One the official site of leveldb(http://code.google.com/p/leveldb/), there is a performance report. I pasted as below.
Below is from official leveldb benchmark
Here is a performance report (with explanations) from the run of the included db_bench program. The results are somewhat noisy, but should be enough to get a ballpark performance estimate.
Setup
We use a database with a million entries. Each entry has a 16 byte key, and a 100 byte value. Values used by the benchmark compress to about half their original size.
LevelDB: version 1.1
CPU: 4 x Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz
CPUCache: 4096 KB
Keys: 16 bytes each
Values: 100 bytes each (50 bytes after compression)
Entries: 1000000
Raw Size: 110.6 MB (estimated)
File Size: 62.9 MB (estimated)
Write performance
The "fill" benchmarks create a brand new database, in either sequential, or random order.
The "fillsync" benchmark flushes data from the operating system to the disk after every operation; the other write operations leave the data sitting in the operating system buffer cache for a while. The "overwrite" benchmark does random writes that update existing keys in the database.
fillseq : 1.765 micros/op; 62.7 MB/s
fillsync : 268.409 micros/op; 0.4 MB/s (10000 ops)
fillrandom : 2.460 micros/op; 45.0 MB/s
overwrite : 2.380 micros/op; 46.5 MB/s
Each "op" above corresponds to a write of a single key/value pair. I.e., a random write benchmark goes at approximately 400,000 writes per second.
Below is from My leveldb benchmark
I did some benchmark for leveldb but got write speed 100 times less than the report.
Here is my experiment settings:
- CPU: Intel Core2 Duo T6670 2.20GHz
- 3.0GB memory
- 32-bit Windows 7
- without compression
- options.write_buffer_size = 100MB
- options.block_cache = 640MB
What I did is very simple: I just put 2 million {key, value} and no reads at all. The key is a byte array which has 20 random bytes and the value is a byte array too with 100 random bytes. I constantly put newly random {key, value} for 2 million times, without any operation else.
In my experiment, I can see that the speed of writing decreases from the very beginning. The instant speed (measuring the speed of every 1024 writes) swings between 50/s to 10, 000/s. And my overall average speed of writes for 2 million pairs is around 3,000/s. The peak speed of writes is 10, 000/s.
As the report claimed that the speed of writes can be 400, 000/s, the write speed of my benchmark is 40 to 130 times slower and I am just wondering what's wrong with my benchmark.
I don't need to paste my testing codes here as it is super easy, I just have a while loop for 2 million times, and inside the loop, for every iteration, I generate a 20 bytes of key, and 100 bytes of value, and then put them to the leveldb database. I also measured the time spent on {key, value} generation, it costs 0 ms.
Can anyone help me with this? How can I achieve 400, 000/s writes speed with leveldb? What settings I should improve to?
Thanks
Moreover
I just ran the official db_bench.cc on my machie. It is 28 times slower than the report.
I think as I used their own benchmark program, the only difference between my benchmark and theirs is the machine.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（2）
您有 200 万个键值对，每个键值对总共 120 字节，因此 200 万 * 120 字节 = 228 MB 数据！您的缓存为 640 MB，因此您的所有数据很可能仍在 RAM 中，并且从未真正到达磁盘。正如 Kitsune 指出的那样：您的硬件远不如 Google 测试的硬件快，如果 Google 具有相同的缓存大小，那么很容易会产生 30 倍的差异。
其他潜在问题：
我们可以继续下去，但有太多的变量需要考虑。如果您发布一些代码来演示您的测试如何运行，那么我们可以建议一些优化，以便您获得更好的性能。
You have 2 million key-value pairs and each key value pair is a total of 120 bytes, so 2 million * 120 bytes = 228 MB of data! Your cache is 640 MB so it's quite possible that all of your data is still in RAM and it never really got to disk. As Kitsune pointed out: your hardware is nowhere near as fast as the one that Google tested with and if Google had the same cache size then that could easily produce 30 times difference.
Other potential issues:
We can keep going on and on and on, but there are just too many variables to consider. If you post some code that demonstrates how your test runs, then we can recommend some optimizations so you can get better performance.
当您在完全不同的硬件上运行相同的基准测试时，您一定会看到一些差异。
无法比较苹果到橙子或苹果到[未知水果]。
When you run the same benchmark on completely different hardware you're bound to see some differences.
Can't compare apples to oranges or apples to [unknown fruit].