当前位置：文江博客话题详情

Mnesia 数据库的存储容量是多少？

发布于 2024-07-12 09:44:46 字数 37 浏览 8 评论 0原文

有些地方规定2GB期限。有些地方指出这取决于节点的数量。

原文

分享到QQ

分享到微博

如果你对这篇内容有疑问，欢迎到本站社区发帖提问参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

发布评论

需要登录才能够评论，你可以免费注册一个本站的账号。

成熟的代价 2024-07-19 09:44:46

如果您的问题是“由大量 disc_only_copies 表组成的 mnesia 数据库的存储容量是多少”，那么这个容量就相当大了 - 您很大程度上受到可用磁盘空间的限制。

一个更容易回答的问题是不同类型的单个 mnesia 表的最大容量是多少。 ram_copies 表受可用内存的限制。 disc_copies 表受到其 dets 后端的限制 (Hakan Mattsson 谈 Mnesia) - 目前数据限制为 4Gb。

因此，简单的答案是，简单的 disc_copies 表在遇到问题之前最多可以存储 4Gb 的数据。（如果超过磁盘大小限制，Mnesia 实际上不会崩溃 - 表的 ram_copies 部分继续运行，因此您可以通过删除数据或在运行时进行其他安排来修复此问题）

但是，如果如果你考虑其他记忆功能，那么答案就更复杂了。

local_content 表。如果
表是一个 local_content 表，
那么它可以有不同的内容
在 mnesia 集群的每个节点上，
所以桌子的容量是
4Gb * <节点数量>
碎片表。 Mnesia 支持用户可配置的表分区或使用表片段的分片。在这种情况下，您可以有效地将表中的数据分布和重新分布到多个原始表上。这些原始表都可以有自己的配置 - 比如说一个 ram_copies 表和其余的 Disc_only_copies 表。这些原始表具有与前面提到的相同的大小限制，现在分段表的有效容量为4Gb * <分段数>。（遗憾的是，如果您对表进行分段，则必须修改表访问代码以使用 mnesia:activity/4 而不是 mnesia:write 等，但是如果您提前计划，这是可以管理的）
外部副本如果您喜欢生活在最前沿的是，您可以将 mnesiaex 补丁应用于 mnesia 并将表数据存储在外部系统，例如 Amazon S3 或东京内阁。在这种情况下，表的容量受到后端存储的限制。

Quite large if your question is "what's the storage capacity of an mnesia database made up of a huge number of disc_only_copies tables" - you're largely limited by available disk space.

An easier question to answer is what's the maximum capacity of a single mnesia table of different types. ram_copies tables are limited by available memory. disc_copies tables are limited by their dets backend (Hakan Mattsson on Mnesia) - this limit is 4Gb of data at the moment.

So the simple answer is that simple disc_copies table can store up to 4Gb of data before they run into problems. (Mnesia doesn't actually crash if you exceed the on-disk size limit - the ram_copies portion of the table continues running, so you can repair this by deleting data or making other arrangements at runtime)

However if you consider other mnesia features, then the answer is more complicated.

local_content tables. If the
table is a local_content table,
then it can have different contents
on each node in the mnesia cluster,
so the capacity of the table is
4Gb * <number of nodes>
fragmented tables. Mnesia supports user configurable table partitioning or sharding using table fragments. In this case you can effectively distribute and redistribute the data in your table over a number of primitive tables. These primitive tables can each have their own configuration - say one ram_copies table and the rest disc_only_copies tables. These primitive tables have the same size limits as mentioned earlier and now the effective capacity of the fragmented table is 4Gb * <number of fragments>. (Sadly if you fragment your table, you then have to modify your table access code to use mnesia:activity/4 instead of mnesia:write and friends, but if you plan this in advance it's managable)
external copies If you like living on the extreme bleeding edge, you could apply the mnesiaex patches to mnesia and store your table data in an external system such as Amazon S3 or Tokyo Cabinet. In this case the capacity of the table is limited by the backend storage.

回复收藏 0 原文

烂人 2024-07-19 09:44:46

TL;DR：Mnesia 数据库的存储容量仅受^*可用 RAM 的限制。

^* 假设您使用表类型 ram_copies 或 disc_copies。此外，如果您在 disc_copies 表中存储大量数据，则需要在启动时从磁盘读取数据，这可能会增加启动时间，超出可接受的范围。

当涉及 disc_copies 类型的表时，这个答案与两个现有答案相矛盾。首先让我澄清一些一般性的观点：

ram_copies 类型的 mnesia 表仅受可用 RAM 的限制（除非您使用的是 32 位机器）。数据存储在 ETS 表中。
disc_only_copies 类型的 mnesia 表存储在 Dets 表中。由于文件格式的限制，Dets 表的大小限制为 2 GB。
规避该限制的明显方法是创建更多表，可能通过表碎片。
该架构也存储在 Dets 表中，因此描述所有现有表的信息也限制为 2 GB。不过，在达到这一限制之前，您可能会遇到其他限制。
disc_copies 类型的 mnesia 表既存储在 RAM 中，又存储在磁盘上，因此它受到可用 RAM 的限制 - 也许还有其他原因？

我将在下面尝试说明 Mnesia 对 disc_copies 表的大小没有施加具体限制。但请注意，许多 Erlang 程序员认为 disc_copies 表的大小限制为 2 GB。这个问题的已接受的答案中对此进行了说明，在撰写本文时，该答案的得分是该答案的 7 倍。

2001 年，disc_copies 从 dets 移至 disk_log

人们普遍认为 disc_copies 表由 Dets 表支持。据我所知，这种情况一直持续到 Erlang/OTP R7B-4（2001 年 9 月 30 日发布）为止。来自自述文件：

  -- mnesia -----------------------------------------------------------------

        OTP-3712 - Speed/load improvements disc_copies tables are not 
                   implemented with dets anymore.

查看差异了解更多详细信息，特别是 mnesia_lib.erl和 mnesia_loader。呃。

支持 dets 和 2 / 4 GB 限制的来源

archelaus 的答案来自 http://erlang.org/~hakan/mnesia_conspiration.txt，它解释了 disc_copies 表驻留在 ets 和dets 表。然而，查看目录索引，我们发现该文档的日期为 1999 年：

[TXT] mnesia_consumption.txt  26-Oct-1999 10:57    10k

这是有道理的它会这样说，因为它是在改变之前两年写的。

Ray Boosen 的回答来自 Erlang 常见问题解答：

11.5 Mnesia 可以存储多少数据？
Dets 使用 32 位整数作为文件偏移量，因此最大可能的 mnesia 表（目前）为 4Gb。
实际上，在达到此限制之前，您的机器会慢得像爬行一样。

FAQ 至少从 2001 年 1 月起就一直这么说（参见 Wayback Machine 中最早的副本）。这意味着此常见问题解答条目的日期是在切换到 disk_log 之前，并且已经很长时间没有更新了。（无论如何，Dets 表大小限制是 2 GB，而不是 4 GB。）我提交了 a常见问题解答的拉取请求。

支持更高限制的来源

Mnesia 的 Learn You Some Erlang 章节说：

ram_copies
此选项使得所有数据都专门存储在 ETS 中，因此仅存储在内存中。对于在 32 位上编译的虚拟机，内存应限制在理论 4GB（实际上约为 3GB），但在 64 位虚拟机上，假设有超过 4GB 的可用内存，则此限制会进一步推高。
disc_only_copies
此选项意味着数据仅存储在 DETS 中。仅限光盘，因此存储空间仅限于 DETS 的 2GB 限制。
光盘副本
此选项意味着数据既存储在 ETS 中又存储在磁盘上，即内存和硬盘上。 Disc_copies 表不受 DETS 限制，因为 Mnesia 使用复杂的事务日志和检查点系统，允许在内存中创建基于磁盘的表备份。

我不确定这是什么时候写的，但是上面的文本存在于最早的 Wayback Machine 副本，日期为 2012 年 4 月。

在一篇关于 erlang 问题的帖子，题为“击败 mnesia 至死（是 RE：在 Erlang VM 中使用 4Gb 内存）”，日期为 2005 年 11 月 7 日，Ulf Wiger 写道：

在 16 GB 计算机上，您可以：
同时运行 600 万个进程
（通过使用 erlang:hibernate，我实际上是
能够运行 2000 万次 - 生成时间：6.3 us，
消息传递时间：5.3 us，我有
1.8 GB 剩余空间。）
使用至少 12 GB 的数据填充 mnesia，但是
考虑一下你想如何表示它，因为
64 位字长让事情变得有点复杂。
在 mnesia 中保留一个 10 GB 以上的 disk_copy 表。这
加载时间和日志转储成本似乎可以接受
（加载需要10分钟，卸载需要一段时间，但是
在后台运行得很好。）

限制

混乱似乎源于官方来源的信息缺失或过时：

Mnesia 文档没有提到任何表大小限制
Erlang FAQ 说 Mnesia 受 4 GB Dets 大小的限制，但这个答案是在 dets 更改为 disk_log 之前编写的。erlang.org
域上唯一的其他文档是 Håkan Mattsson 的文档，其日期可以追溯到 dets 更改为 disk_log 之前

LYSE 似乎是第一个提到 disc_copies 表不受 Dets 表大小限制。

TL;DR: the storage capacity of a Mnesia database is limited only^* by available RAM.

^* Assuming you use table types ram_copies or disc_copies. Also, if you store a lot of data in a disc_copies table, it needs to be read from disk at startup, which might increase startup time beyond what's acceptable.

This answer contradicts the two existing answers when it comes to tables of type disc_copies. Let me first get a few general points out of the way:

A mnesia table of type ram_copies is only limited by available RAM (except if you're on a 32-bit machine). Data is stored in an ETS table.
A mnesia table of type disc_only_copies is stored in a Dets table. Dets tables are limited to 2 GB, because of limits in the file format.
The obvious way to circumvent that limit is to create more tables, possibly through table fragmentation.
The schema is also stored in a Dets table, so the information describing all existing tables is also limited to 2 GB. You are likely to run into other limits before you hit that one, though.
A mnesia table of type disc_copies is stored both in RAM and on disk, so it is limited by available RAM - and perhaps something else?

I'm going to try to show below that there is no specific limit imposed by Mnesia on the size of a disc_copies table. Note however that many Erlang programmers believe that disc_copies tables are limited to 2 GB. That is stated in the accepted answer to this question, which at the time of writing outscores this answer by a factor of 7.

disc_copies moved from dets to disk_log in 2001

It is commonly believed that disc_copies tables are backed by Dets tables. As far as I can tell, this was the case until Erlang/OTP R7B-4 (released on 30th September 2001). From the README:

  -- mnesia -----------------------------------------------------------------

        OTP-3712 - Speed/load improvements disc_copies tables are not 
                   implemented with dets anymore.

Look at the diff for more details, in particular mnesia_lib.erl and mnesia_loader.erl.

Sources supporting dets and a 2 / 4 GB limit

archelaus's answer draws from http://erlang.org/~hakan/mnesia_consumption.txt, which explains that disc_copies tables reside in ets and dets tables. However, looking at the index for the directory, we see that this document is dated 1999:

[TXT] mnesia_consumption.txt  26-Oct-1999 10:57    10k

It makes sense that it would say this, as it was written two years before the change.

Ray Boosen's answer draws from the Erlang FAQ:

11.5 How much data can be stored in Mnesia?
Dets uses 32 bit integers for file offsets, so the largest possible mnesia table (for now) is 4Gb.
In practice your machine will slow to a crawl way before you reach this limit.

The FAQ has been saying that since at least January 2001 (see the earliest copy in the Wayback Machine). That means that this FAQ entry dates from before the switch to disk_log, and hasn't been updated for a long time. (Anyway, the Dets table size limit is 2 GB, not 4 GB.) I submitted a pull request for the FAQ.

Sources supporting higher limits

The Learn You Some Erlang chapter on Mnesia says:

ram_copies
This option makes it so all data is stored exclusively in ETS, so memory only. Memory should be limited to a theoretical 4GB (and practically around 3GB) for virtual machines compiled on 32 bits, but this limit is pushed further away on 64 bits virtual machines, assuming there is more than 4GB of memory available.
disc_only_copies
This option means that the data is stored only in DETS. Disc only, and as such the storage is limited to DETS' 2GB limit.
disc_copies
This option means that the data is stored both in ETS and on disk, so both memory and the hard disk. disc_copies tables are not limited by DETS limits, as Mnesia uses a complex system of transaction logs and checkpoints that allow to create a disk-based backup of the table in memory.

I'm not sure when this was written, but the text above exists in the earliest Wayback Machine copy, dated April 2012.

In a post on erlang-questions titled "beating mnesia to death (was RE: Using 4Gb of ram with Erlang VM)", dated 7th November 2005, Ulf Wiger writes:

On a 16 GB machine, you can:
run 6 million simultaneous processes
(through use of erlang:hibernate, I was actually
able to run 20 million - spawn time: 6.3 us,
message passing time: 5.3 us, and I had
1.8 GB to spare.)
populate mnesia with at least 12 GB of data, but
think through how you want to represent it, since
the 64-bit word size blows things up a bit.
keep a 10 GB+ disc_copy table in mnesia. The
load times and log dump cost seem acceptable
(10 minutes to load, dumping takes a while but
runs in the background quite nicely.)

Conclusions

The confusion seems to stem from missing or out-dated information from official sources:

The Mnesia documentation doesn't mention any table size limits
The Erlang FAQ says that Mnesia is subject to a 4 GB Dets size limit, but this answer was written before the dets to disk_log change
The only other document on the erlang.org domain is Håkan Mattsson's document, dating from before the dets to disk_log change

LYSE seems to be the first "authoritative" source that mentions disc_copies tables not being subject to the Dets table size limit.

回复收藏 0 原文