Berkeley DB Java 版 - 大量数据的调优
我需要将超过 10 亿个密钥加载到 Berkley DB 中，因此我想提前对其进行调整以获得更好的性能。使用标准配置，我现在需要大约 15 分钟才能加载 1'000'000 个密钥，这太慢了。 是否有适当的方法来调整 Berkley DB 的 B+Tree（节点大小等）？
（作为比较，在调整 tokyo Cabinet 后，它在 25 分钟内加载了 10 亿个密钥）。
聚苯乙烯 我正在寻找作为代码的调整技巧，而不是为正在运行的系统设置的参数（如 jvm 大小等......）
I need to load over 1 billion keys into Berkley DB and therefore I want to tune it in advance to get better performance. With standard configuration it takes me now about 15min to load 1'000'000 keys which is too slow.
Is there a proper way to tune for example the B+Tree of Berkley DB (node size etc...)?
(As an comparision, after tuning tokyo cabinet, it loads 1 billion keys in 25min).
P.S.
I'm looking for tuning tips as a code and not parameters to set for a running system (like jvm size etc...)
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（2）
我很好奇，当 TokyoCabinet 在 25 分钟内加载 1B 个密钥时，存储的键/值的大小是多少？您使用的 I/O 系统和存储系统是什么？您使用术语“加载”来表示 1B 事务提交到永久稳定存储吗？这大约是每秒 666,666 次插入，对于我所知道的任何 I/O 系统来说，这在物理上都是不可能的。将该数字乘以键和值的大小，现在您已经无可救药地超出了物理限制。
请查看 Gustavo Duarte 的博客，了解一些有关 I/O 系统及其工作原理的信息在硬件方面工作，然后检查你的陈述。我非常有兴趣了解东京内阁到底在做什么、没有做什么。如果我不得不猜测，我会说它要么提交到操作系统中的文件系统缓存，但不将这些缓冲区刷新（fdsync（）-ing）到磁盘。
全面披露：我是 Oracle Berkeley DB（TokyoCabinet 的直接竞争对手）的产品经理，我已经使用这些数据库及其最好的硬件大约十年了，所以我既有偏见又持怀疑态度。
Berkeley DB 有可以在事务句柄上设置的标志，这些标志模仿此和其他 用耐久性（ACID 中的“D”）换取速度的类似方法。
至于如何使 Berkeley DB Java Edition (BDB-JE) 更快，您可以尝试以下操作：
到事务日志只要
可能（当缓冲区已满时，它
刷新数据）
B 树（包括我们的）有很多作用
更好地按顺序插入
快速加载时间 -
文件从默认的 10MiB 到
更大的东西，比如 100MiB，这个
降低 I/O 成本 -
明确数据库的性能声明非常重要。它们看起来很简单，但事实证明，要使它们正确，以便它们不会损坏数据或丢失已提交的事务，这是非常非常棘手的。
我希望这对你有一点帮助。
I'm curious, when TokyoCabinet loads 1B keys in 25 minutes what are the sizes of the keys/values being stored? What's the I/O systems and the storage system you're using? Are you using the term "load" to mean 1B transactional commits to permanent stable storage? That would be ~666,666 inserts/second, which is physically impossible given any I/O system I'm aware of. Multiply that number times the key and value size and now you're hopelessly beyond physical limits.
Please take a look at Gustavo Duarte's blog, read a bit about I/O systems and how things work in hardware and then review your statement. I'm very interested in finding out what exactly TokyoCabinet is doing and what it isn't doing. If I had to guess I'd say that either it's committing to file-system cache in the operating system, but not flushing (fdsync()-ing) those buffers to disk.
Full Disclosure: I'm a product manager at Oracle for Oracle Berkeley DB (a direct competitor of TokyoCabinet), I've been playing with these databases and the best hardware around for them for about ten years so I'm both biased and skeptical.
Berkeley DB has flags you can set on the transaction handle which mimic this and other similar methods of trading off durability (the "D" in ACID) for speed.
As far as how to make Berkeley DB Java Edition (BDB-JE) faster you can try the following:
to the transaction log for as long as
possible (when buffers are full, it
flushes the data)
B-Trees (ours included) do much
better with in-order insertions for
fast load times-
files from the default of 10MiB to
something larger, like 100MiB, this
reduces I/O cost-
It's very important to be clear about claims of performance with databases. They seems simple, but it turns out to be very very tricky to get them right so that they don't ever corrupt data or lose committed transactions.
I hope this helps you a bit.
如果将 BDB-JE 上的批量插入分组到单个事务中，则速度会快一个数量级。原因是每次提交都会导致（默认情况下）同步写入磁盘，而事务在提交时同步。在我的应用程序中，将 100'000 个小键写入单次提交需要一分钟多的时间，而在事务中只需要几秒钟。
Bulk inserts on BDB-JE are an order of magnitude faster if you group them into a single transaction. The reason is that each single commit causes (by default) a sync write to disk while a transaction is synchronized on commit. In my application writing 100'000 small keys as single commits tooks more than a minute while in a transaction it takes just a few seconds.