内存映射文件可以有多大？
什么限制了内存映射文件的大小？ 我知道它不能大于未分配地址空间的最大连续块，并且应该有足够的可用磁盘空间。 但还有其他限制吗？
What limits the size of a memory-mapped file? I know it can't be bigger than the largest continuous chunk of unallocated address space, and that there should be enough free disk space. But are there other limits?
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（6）
您太保守了：内存映射文件可能大于地址空间。 内存映射文件的视图受到操作系统内存限制的限制，但这只是您一次查看的文件的一部分。 （我想从技术上讲，您可以一次映射文件不连续部分的多个视图，因此除了开销和页面长度限制之外，只有您正在查看的字节总数构成了限制。您可以查看字节数[0 到 1024] 和字节 [240 到 240 + 1024]，具有两个单独的视图。）
在 MS Windows 中，查看 MapViewOfFile 函数。 它实际上采用 64 位文件偏移量和 32 位长度。
You're being too conservative: A memory-mapped file can be larger than the address space. The view of the memory-mapped file is limited by OS memory constraints, but that's only the part of the file you're looking at at one time. (And I guess technically you could map multiple views of discontinuous parts of the file at once, so aside from overhead and page length constraints, it's only the total # of bytes you're looking at that poses a limit. You could look at bytes [0 to 1024] and bytes [240 to 240 + 1024] with two separate views.)
In MS Windows, look at the MapViewOfFile function. It effectively takes a 64-bit file offset and a 32-bit length.
这是我在 Win32 下使用内存映射文件时的经验：
如果将整个文件映射到一个段中，它通常会占用 750 MB 左右的空间，因为它找不到更大的连续内存块。 如果将其分成更小的部分，例如每个 100MB，则可以获得大约 1500MB-1800MB，具体取决于正在运行的其他内容。
如果您使用 /3g 开关，您可以获得超过 2GB 的空间大约 2700MB，但操作系统性能受到影响。
我不确定 64 位，我从未尝试过，但我认为最大文件大小仅受您拥有的物理内存量的限制。
This has been my experience when using memory-mapped files under Win32:
If your map the entire file into one segment, it normally taps out at around 750 MB, because it can't find a bigger contiguous block of memory. If you split it up into smaller segments, say 100MB each, you can get around 1500MB-1800MB depending on what else is running.
If you use the /3g switch you can get more than 2GB up to about 2700MB but OS performance is penalized.
I'm not sure about 64-bit, I've never tried it but I presume the max file size is then limited only by the amount of physical memory you have.
是的，内存映射文件有限制。 最让人震惊的是：
即使在我的 64 位 32GB RAM 系统上，如果我尝试读取一个大的 numpy 内存映射文件而不是使用字节偏移量获取其中的一部分，也会收到以下错误：
溢出错误：内存映射大小必须保持积极态度
使用大数据集确实很痛苦。
Yes, there are limits to memory-mapped files. Most shockingly is:
Even on my 64-bit, 32GB RAM system, I get the following error if I try to read in one big numpy memory-mapped file instead of taking portions of it using byte-offsets:
Overflow Error: memory mapped size must be positiveBig datasets are really a pain to work with.
在 Windows 下：“文件视图的大小仅限于未保留虚拟内存的最大可用连续块。这最多为 2 GB 减去进程已保留的虚拟内存。”
来自 MDSN。
我不确定 LINUX/OSX/Whatever Else，但它可能也与地址空间有关。
Under Windows: "The size of a file view is limited to the largest available contiguous block of unreserved virtual memory. This is at most 2 GB minus the virtual memory already reserved by the process. "
From MDSN.
I'm not sure about LINUX/OSX/Whatever Else, but it's probably also related to address space.
在 64 位 Windows 系统上，虚拟地址空间的限制为 >16 TB。 这里讨论的问题很可能与混合 DWORD 和 SIZE_T 有关。
The limit of virtual address space is >16 Terabyte on 64Bit Windows systems. The issue discussed here is most probably related to mixing DWORD with SIZE_T.
不应有其他限制。 这些还不够吗？ ;-)
There should be no other limits. Aren't those enough? ;-)