数据结构 内存映射还是数据库? (数百万件商品)

发布于 2024-11-28 16:57:46 字数 208 浏览 4 评论 0原文

我有一个数据结构,本质上是对一些计算的查找,这些计算需要很长时间(100 毫秒)来计算,并且需要反复使用。我有大约 6,000,000 个这样的计算,并希望在我的应用程序启动时将它们加载到内存中(我将预先计算所有这些)。

问题是我可以将其存储为内存映射文件(某物的字典),还是应该将其存储在数据库中,然后在程序启动时将其加载到内存中?二进制序列化有多快?

我有什么选择?

I have a data structure which is essentially a lookup for some calculations that take a really long time (100ms) to calculate and need to be used over and over. I have roughly 6,000,000 of these calculations and want to load them into memory when my application starts (I will precalculate them all).

The question is can I store this as a memory mapped file (dictionary of something) or should I store it in a db and then load it into ram on program start up? How fast would binary serialization be?

What are my options?

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。

评论(2

鲸落 2024-12-05 16:57:46

二进制序列化速度很快,特别是如果您只需加载一次。数据库的加载速度实际上取决于数据的结构。使用数据库的优点是易于管理。如果您想轻松管理、更改、跟踪更改或将这些值与多个客户端一起使用,那么数据库将是您的最佳选择。如果它们永远不会改变,一个文件就足够了。

Binary serialization is fast, specially if you only have to load it once. Load speed from a database really depends on how the data is structured. The advantage to using a database is easy of management. If you want to easily manage, change, track changes, or use these values with multiple clients, then a DB would be the way to go. If they are never going to change, a file would suffice.

吝吻 2024-12-05 16:57:46

您必须尝试不同的方法并自行衡量。没有其他途径可以解决性能问题。请注意,您需要牢记一些具体目标(例如加载 1 秒/查找 1 毫秒)。

选项:

  • 在启动时计算所有值,在某些查找(预分配数组/字典)中计算所有值
  • ,在某些查找(预分配数组/字典)中计算按需计算并在某些查找中
  • 提前计算并加载未压缩的数据(请注意,您拥有的数据量很大,并且会需要明显的 1-3 秒时间来加载)
  • 提前计算并按需加载
  • 提前计算并加载压缩数据

我建议尝试计算加载的所有值并查看它是否工作得足够快 - 最有可能最简单的方法。

You have to try different approaches and measure for yourself. There is no other road to solve performance problems. Note that you need to have some concreate goal in mind (like 1 second for load/1ms for lookup).

Options:

  • calculate all values on startup and sotre in some lookup (preallocated array/dictionary)
  • calculate on demand and sotre in some lookup (preallocated array/dictionary)
  • calculate in advance and load uncompressed (note that amount of data you have is large and will take noticable 1-3 seconds time to load)
  • calculate in advance and load on demand
  • calculate in advance and load compressed data

I'd recommned trying to compute all values on load and see if it works fast enough - most likely easiest way to go.

~没有更多了~
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文