大对象堆碎片：CLR 有什么解决方案吗？
如果您的应用程序必须对大尺寸对象（> 85000 字节）进行大量分配/取消分配，它最终将导致内存碎片，并且您的应用程序将抛出内存不足异常。
这个问题有什么解决办法或者是CLR内存管理的限制吗？
If you application is such that it has to do lot of allocation/de-allocation of large size objects (>85000 Bytes), its eventually will cause memory fragmentation and you application will throw an Out of memory exception.
Is there any solution to this problem or is it a limitation of CLR memory management?
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（6）
不幸的是，我见过的所有信息都只建议您自己管理风险因素：重用大型对象，在开始时分配它们，确保它们的大小是彼此的倍数，使用替代数据结构（列表，树）而不是数组。这给了我另一个想法，即创建一个非碎片列表，而不是一个大数组，而是分成更小的数组。数组/列表似乎是 IME 最常见的罪魁祸首。
这是一篇关于它的 MSDN 杂志文章：
http://msdn.microsoft.com/en-us/magazine/cc534993.aspx ，但它并没有多大用处。
Unfortunately, all the info I've ever seen only suggests managing risk factors yourself: reuse large objects, allocate them at the beginning, make sure they're of sizes that are multiples of each other, use alternative data structures (lists, trees) instead of arrays. That just gave me an another idea of creating a non-fragmenting List that instead of one large array, splits into smaller ones. Arrays / Lists seem to be the most frequent culprits IME.
Here's an MSDN magazine article about it:
http://msdn.microsoft.com/en-us/magazine/cc534993.aspx, but there isn't that much useful in it.
CLR 垃圾收集器中的大对象的特点是它们在不同的堆中进行管理。
垃圾收集器使用一种称为“压缩”的机制，该机制基本上是对常规堆中的对象进行碎片化和重新链接。
问题是，由于“压缩”大型对象（复制并重新链接它们）是一个昂贵的过程，GC 为它们提供了一个不同的堆，永远不会被压缩。
还要注意内存分配是连续的。这意味着如果您先分配对象 #1，然后分配对象 #2，则对象 #2 将始终放置在对象 #1 之后。
这可能是导致您出现 OutOfMemoryExceptions 的原因。
我建议看看像享元（Flyweight）、延迟初始化（Lazy Initialization）和对象池（Object Pool）这样的设计模式。
如果您怀疑其中一些大型对象已经死亡并且由于控制流中的缺陷而尚未被收集，导致它们在准备好收集之前到达更高的代，您也可以强制进行 GC 收集。
The thing about large objects in the CLR's Garbage Collector is that they are managed in a different heap.
The garbage collector uses a mechanism called "Compacting", which is basically fragmentation and re-linkage of objects in the regular heap.
The thing is, since "compacting" large objects (copying and re-linking them) is an expensive procedure, the GC provides a different heap for them, which is never being compacted.
Note also that memory allocation is contiguous. Meaning if you allocate Object #1 and then Object #2, Object #2 will always be placed after Object #1.
This is probably what's causing you to get OutOfMemoryExceptions.
I would suggest having a look at design patterns like Flyweight, Lazy Initialization and Object Pool.
You could also force GC collection, if you're suspecting that some of those large objects are already dead and have not been collected due to flaws in your flow of control, causing them to reach higher generations just before being ready for collection.
程序总是在 OOM 上崩溃，因为它请求的内存块太大，而不是因为它完全耗尽了所有虚拟内存地址空间。您可能会认为这是 LOH 碎片化的问题，也很容易认为程序使用了过多的虚拟内存。
一旦程序超出了分配一半可寻址虚拟内存（一千兆字节）的范围，就真的是时候考虑使其代码更智能了，这样它就不会消耗太多内存。或者将 64 位操作系统作为先决条件。后者总是更便宜。它也不会从你的口袋里出来。
A program always bombs on OOM because it is asking for a chunk of memory that's too large, never because it completely exhausted all virtual memory address space. You could argue that's a problem with the LOH getting fragmented, it is just as easy to argue that the program is using too much virtual memory.
Once a program goes beyond allocating half the addressable virtual memory (a gigabyte), it is really time to either consider making its code smarter so it doesn't gobble so much memory. Or making a 64-bit operating system a prerequisite. The latter is always cheaper. It doesn't come out of your pocket either.
除了重新考虑您的设计之外，没有其他解决方案。这不是 CLR 的问题。请注意，非托管应用程序的问题是相同的。事实证明，应用程序同时使用了太多的内存，并且在内存中分段放置了“不利”的内存。如果仍然必须指出某些外部罪魁祸首，我宁愿指出操作系统内存管理器，它（当然）不会压缩其虚拟机地址空间。
CLR 在空闲列表中管理 LOH 的空闲区域。在大多数情况下，这是防止碎片的最佳方法。但由于对于非常大的对象，每个 LOH 段的对象数量会减少 - 最终每个段只有一个对象。这些对象在虚拟机空间中的位置完全取决于操作系统的内存管理器。这意味着，碎片主要发生在操作系统级别，而不是 CLR。这是堆碎片经常被忽视的一个方面，这不能归咎于 .NET。 （但这也是事实，碎片也可能发生在托管端，就像 那篇文章。）
常见的解决方案已经被命名：重用你的大对象。到目前为止，我还没有遇到过任何无法通过适当设计来完成的情况。然而，有时这可能很棘手，因此可能很昂贵。
There is no solution besides reconsidering your design. And it is not a problem of the CLR. Note, the problem is the same for unmanaged applications. It is given by the fact, that too much memory is used by the application at the same time and in segments laying 'disadvantageous' out in memory. If some external culprit has to be pointed at nevertheless, I would rather point at the OS memory manager, which (of course) does not compact its vm address space.
The CLR manages free regions of the LOH in a free list. This in most cases is the best what can be done against fragmentation. But since for really large objects, the number of objects per LOH segment decreases - we eventually end up having only one object per segment. And where those objects are positioned in the vm space is completely up to the memory manager of the OS. This means, the fragmentation mostly happens on the OS level - not on the CLR. This is an often overseen aspect of heap fragmentation and it is not .NET to blame for it. (But it is also true, fragmentation can also occour on the managed side like nicely demonstrated in that article.)
Common solutions have been named already: reuse your large objects. I up to now was not confronted with any situation, where this could not be done by proper design. However, it can be tricky sometimes and therefore may be expensive though.
我们在多个线程中处理图像。当图像足够大时，这也会由于内存碎片而导致 OutOfMemory 异常。我们尝试通过使用不安全内存和为每个线程预分配堆来解决该问题。不幸的是，这并没有完全帮助，因为我们依赖于几个库：我们能够解决代码中的问题，但不能解决第三方的问题。
最终我们用进程取代了线程，让操作系统来完成繁重的工作。操作系统很早就构建了内存碎片的解决方案，因此忽略它是不明智的。
We were precessing images in multiple threads. With images being large enough, this also caused OutOfMemory exceptions due to memory fragmentation. We tried to solve the problem by using unsafe memory and pre-allocating heap for every thread. Unfortunately, this didn't help completely since we relied on several libraries: we were able to solve the problem in our code, but not 3rd party.
Eventually we replaced threads with processes and let operating system do the hard work. Operating systems have long ago built a solution for memory fragmentation, so it's unwise to ignore it.
我在另一个答案中看到 LOH 的大小可以缩小：
大型数组和 LOH 碎片。公认的约定是什么？
”
...
现在，话虽如此，如果 LOH 末端的区域完全没有活动对象，那么 LOH 的大小可能会缩小，因此唯一的问题是，如果您将对象留在那里很长时间（例如应用程序的持续时间）。
...
。
除此之外，您还可以让您的程序在 32 位系统上使用高达 3GB 的扩展内存运行，在 64 位系统上使用高达 4GB 的扩展内存运行
只需在链接器或此构建后事件中添加标志 /LARGEADDRESSAWARE：
调用“$(DevEnvDir)..\tools\vsvars32.bat”
editbin /LARGEADDRESSAWARE "$(TargetPath)"
最后，如果您计划长时间运行带有大量大对象的程序，您将必须优化内存使用，甚至可能必须重用分配的对象以避免垃圾收集器这在概念上类似于使用实时系统。
I have seen in a different answer that the LOH can shrink in size:
Large Arrays, and LOH Fragmentation. What is the accepted convention?
"
...
Now, having said that, the LOH can shrink in size if the area at its end is completely free of live objects, so the only problem is if you leave objects in there for a long time (e.g. the duration of the application).
...
"
Other then that you can make your program run with extended memory up to 3GB on 32bit system and up to 4 GB on 64bit system.
Just add the flag /LARGEADDRESSAWARE in your linker or this post build event:
call "$(DevEnvDir)..\tools\vsvars32.bat"
editbin /LARGEADDRESSAWARE "$(TargetPath)"
In the end if you are planning to run the program for a long time with lots of large objects you will have to optimize the memory usage and you might even have to reuse allocated objects to avoid garbage collector which is similar in concept, to working with real time systems.