写入 FileStream 时 C# 中的 OutOfMemoryException
C# 中的 OutOfMemoryExcetpion ...我正在尝试创建一个主要基于 BPlus 树的新索引。该索引应该用于大量数据（超过 50 000 000 项）。为了使索引更快，我将内部树节点（每个节点有约 4KB 的信息，256 个对子节点的引用）保留在 RAM 中（它们被频繁访问）。叶子（可变大小）被写入磁盘，写入最大 200MB 的文件。
为了从文件中写入或读取数据，我使用以下方法：
FileStream _FS = new FileStream(FullPath, FileMode.OpenOrCreate);
...
byte[] Data = new byte[size];
_FS.Read(Data, 0, size);
...
_FS.Write(Data, 0, _BlockSize);
问题是插入的项目数量较多（19 000 000）时，我收到 OutOfMemoryException。我理解树的内部结构（内部节点）可能会占用大量内存，但是，我相信这不是内存已满的原因。
当我使用以下构造函数时：
_FS = new FileStream(FullPath, FileMode.OpenOrCreate, FileAccess.ReadWrite,
FileShare.None, 1024, FileOptions.WriteThrough | Settings.FILE_FLAG_NO_BUFFERING);
不会引发 OutOfMemoryException（至少在 20 000 000 个项目时不会引发），但是，插入项目的速度较小。 我相信，当使用第一个 FileStream 构造函数时，.net 使用一些内部缓冲区（在 RAM 中），这会填充内存并导致引发 OutOfMemoryException。
问题是，如何使用 FileStream（或其他东西）以便能够高速插入项目（写入磁盘），而不引发 OutOfMemoryException？
OutOfMemoryExcetpion in c# ... I am trying to create a new index that is mainly based on BPlus trees. The index is supposed to be used on huge amounts of data (more than 50 000 000 items). In order to make the index fast I am keeping the internal tree nodes (each node has ~ 4KB of information, 256 references to child nodes) in RAM (they are frequently accessed). The leaves (variable size) are written to the disk, into files of maximum 200MB.
In order to write or read the data from a file I am using the following:
FileStream _FS = new FileStream(FullPath, FileMode.OpenOrCreate);
...
byte[] Data = new byte[size];
_FS.Read(Data, 0, size);
...
_FS.Write(Data, 0, _BlockSize);
The problem is that at a higher number of inserted items (19 000 000) I get the OutOfMemoryException. I understand that the internal structure of the tree (internal nodes) may occupy a huge amount of memory, however, I believe that this is not the reason why the memory is full.
When I am using the following constructor:
_FS = new FileStream(FullPath, FileMode.OpenOrCreate, FileAccess.ReadWrite,
FileShare.None, 1024, FileOptions.WriteThrough | Settings.FILE_FLAG_NO_BUFFERING);
the OutOfMemoryException is not raised (at least not at 20 000 000 items), however, the speed at which items are inserted is smaller.
I believe that when using the first FileStream constructor .net uses some internal buffer (in RAM), which fills the memory and results in raising the OutOfMemoryException.
The question is, how to use the FileStream (or something else) in order to be able to insert items (write to disk) at a high speed, withou raising the OutOfMemoryException?
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
如果您将 _FS.Flush() 放入第一个示例的循环中（假设每 10000 条记录），会发生什么情况？
What happens if you put a _FS.Flush() in the loop for your first example, say every 10000 records?