I don't know much about the subject, but it might be possible to write an STL-like interface to a memory mapped file?
edit: This approach might be suitable if you're trying to get at a specific part of a huge file. If you're attempting to do something with the entire file, you'll likely generate a huge number of page faults as you read in uncached parts of the file.
I have implemented some thing very similar. Implementing the iterators is the most challenging. I used boost::iterator_facade to implement the iterators. Using boost::iterator_facade you can easy adapt any cached on disk data structures to have a STL container interface.
I've never had to do anything quite like this, but It might be possible to do what you want to do by writing a custom allocator that makes use of a memory mapped files to back your data.
If (as you write) you're not interested in persistence the simplest solution would be to increase your heap size and use your operating system's virtual memory facilities. The part of the heap that will not fit in your computer's physical memory will end up being paged on disk, giving you exactly what you want: normal STL access to data often stored on disk. The operating system will take care of caching the most used pages in the physical memory and evicting to disk those you don't use a lot. Your code will remain the same, and you can increase its performance simply by adding more physical memory.
To increase your heap size check your operating system's parameters, like ulimit(1) on Unix systems and System properties - Advanced - Performance - Advanced - Virtual Memory on Windows XP. If you've hit the 32-bit 4GB limit consider moving to a 64 bit architecture or compiling your program for 64 bits.
发布评论
评论（4）
我对这个主题不太了解，但也许可以为内存映射文件编写一个类似STL的接口？
编辑：如果您尝试获取大文件的特定部分，则此方法可能适合。 如果您尝试对整个文件执行某些操作，则在读取文件的未缓存部分时可能会生成大量页面错误。
I don't know much about the subject, but it might be possible to write an STL-like interface to a memory mapped file?
edit: This approach might be suitable if you're trying to get at a specific part of a huge file. If you're attempting to do something with the entire file, you'll likely generate a huge number of page faults as you read in uncached parts of the file.
我已经实现了一些非常相似的东西。 实现迭代器是最具挑战性的。 我使用 boost::iterator_facade 来实现迭代器。 使用
boost::iterator_facade，您可以轻松调整任何缓存在磁盘数据结构上的内容，以拥有STL容器接口。I have implemented some thing very similar. Implementing the iterators is the most challenging. I used boost::iterator_facade to implement the iterators. Using
boost::iterator_facadeyou can easy adapt any cached on disk data structures to have a STL container interface.我从来没有做过类似的事情，但是通过编写一个自定义分配器，利用内存映射文件来支持您的数据，也许可以做您想做的事情。
请参阅 boost::interprocesses 有关其易于使用的内存映射文件实现的文档，这篇 Dobbs 博士文章有关编写分配器的详细讨论，以及此 IEEE 软件专栏 的问题描述和 < a href="http://www.spinellis.gr/blog/20101030/smap.cpp" rel="noreferrer">示例代码。
I've never had to do anything quite like this, but It might be possible to do what you want to do by writing a custom allocator that makes use of a memory mapped files to back your data.
See boost::interprocesses for docs on their easy to use implementation of memory mapped files, this Dr. Dobbs article for a detailed discussion on writing allocators, and this IEEE Software column for a description of the problem and example code.
如果（如您所写）您对持久性不感兴趣，最简单的解决方案是增加堆大小并使用操作系统的虚拟内存设施。 堆中不适合计算机物理内存的部分最终将在磁盘上进行分页，这正是您想要的：对通常存储在磁盘上的数据进行正常的 STL 访问。 操作系统将负责在物理内存中缓存最常用的页面，并将不经常使用的页面逐出到磁盘。 您的代码将保持不变，并且只需添加更多物理内存即可提高其性能。
要增加堆大小，请检查操作系统的参数，例如 Unix 系统上的 ulimit(1) 和 Windows XP 上的系统属性 - 高级 - 性能 - 高级 - 虚拟内存。 如果您已达到 32 位 4GB 限制，请考虑迁移到 64 位架构或编译 64 位程序。
If (as you write) you're not interested in persistence the simplest solution would be to increase your heap size and use your operating system's virtual memory facilities. The part of the heap that will not fit in your computer's physical memory will end up being paged on disk, giving you exactly what you want: normal STL access to data often stored on disk. The operating system will take care of caching the most used pages in the physical memory and evicting to disk those you don't use a lot. Your code will remain the same, and you can increase its performance simply by adding more physical memory.
To increase your heap size check your operating system's parameters, like ulimit(1) on Unix systems and System properties - Advanced - Performance - Advanced - Virtual Memory on Windows XP. If you've hit the 32-bit 4GB limit consider moving to a 64 bit architecture or compiling your program for 64 bits.