如何最大化大对象堆中最大的连续内存块
情况是我正在对远程服务器进行 WCF 调用，该服务器以字符串形式返回 XML 文档。
大多数时候这个返回值是几K，有时是几十K，极少数情况下是几百K，但很少有可能是几兆字节（第一个问题是我无法知道）。
正是这些罕见的情况引起了悲伤。我得到一个开始的堆栈跟踪：
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.Xml.BufferBuilder.AddBuffer()
at System.Xml.BufferBuilder.AppendHelper(Char* pSource, Int32 count)
at System.Xml.BufferBuilder.Append(Char[] value, Int32 start, Int32 count)
at System.Xml.XmlTextReaderImpl.ParseText()
at System.Xml.XmlTextReaderImpl.ParseElementContent()
at System.Xml.XmlTextReaderImpl.Read()
at System.Xml.XmlTextReader.Read()
at System.Xml.XmlReader.ReadElementString()
at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderMDRQuery.Read2_getMarketDataResponse()
at Microsoft.Xml.Serialization.GeneratedAssembly.ArrayOfObjectSerializer2.Deserialize(XmlSerializationReader reader)
at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle, XmlDeserializationEvents events)
at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle)
at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)
at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)
我已经阅读了周围的内容，这是因为大对象堆变得太碎片化，因此即使在调用之前快速检查 StringBuilder.EnsureCapacity 也会导致 OutOfMemoryException 提前抛出（并且因为我猜测需要什么，实际上可能不需要那么多，所以我的检查造成的问题比它解决的问题还要多）。一些意见是我对此无能为力。
我问自己的一些问题：
- 使用更少的内存 - 您检查过泄漏吗？是的。内存使用量会上下波动，但没有根本性的增长来保证这种情况的发生。有时它失败了，但之前那个阶段却成功了。
- 转移较小金额不是一个选项，这是我无法控制的第三方网络服务（或者至少需要很长时间才能解决，同时我仍然有问题）
- < strong>你能对 LOH 做点什么来降低它失败的可能性吗？...现在这是最富有成效的过程。这是一个 32 位进程（必须出于各种政治、技术和无聊的原因），但通常有数百兆空闲（是我们见过的失败的最大数量的倍数）。
- 我们可以监控 LOH 吗？ 使用 perfmon 我可以跟踪堆的大小，但我认为没有办法监控最大的可用连续内存块。
问题是：对于尝试的事情有什么建议或建议吗？
The situation is that I am making a WCF call to a remote server which is returns an XML document as a string.
Most of the time this return value is a few K, sometimes a few dozen K, very occasionally a few hundred K, but very rarely it could be several megabytes (first problem is that there is no way for me to know).
It's these rare occasions that are causing grief. I get a stack trace that starts:
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.Xml.BufferBuilder.AddBuffer()
at System.Xml.BufferBuilder.AppendHelper(Char* pSource, Int32 count)
at System.Xml.BufferBuilder.Append(Char[] value, Int32 start, Int32 count)
at System.Xml.XmlTextReaderImpl.ParseText()
at System.Xml.XmlTextReaderImpl.ParseElementContent()
at System.Xml.XmlTextReaderImpl.Read()
at System.Xml.XmlTextReader.Read()
at System.Xml.XmlReader.ReadElementString()
at Microsoft.Xml.Serialization.GeneratedAssembly.XmlSerializationReaderMDRQuery.Read2_getMarketDataResponse()
at Microsoft.Xml.Serialization.GeneratedAssembly.ArrayOfObjectSerializer2.Deserialize(XmlSerializationReader reader)
at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle, XmlDeserializationEvents events)
at System.Xml.Serialization.XmlSerializer.Deserialize(XmlReader xmlReader, String encodingStyle)
at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)
at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)
I've read around and it is because the Large Object Heap is just getting too fragmented, so even preceding the call with a quick check to StringBuilder.EnsureCapacity just causes the OutOfMemoryException to be thrown earlier (and because I'm guessing at what's needed, it might not actually need that much so my check is causing more problems than it is solving). Some opinions are that there's not much I can do about it.
Some of the questions I've asked myself:
- Use less memory - have you checked for leaks? Yes. The memory usage goes up and down, but there's no fundamental growth that guarantees this to happen. Some of the times it fails, it succeeded at that stage previously.
- Transfer smaller amounts Not an option, this is a third party web service over which I have no control (or at least it would take a long time to resolve, in the meantime I still have a problem)
- Can you do something to the LOH to make it less likely to fail? ... now this is most fruitful course. It's a 32-bit process (it has to be for various political, technical and boring reasons) but there's normally hundreds of meg free (multiples of the largest amount for which we've seen failures).
- Can we monitor the LOH? Using perfmon I can track the size of the heaps, but I don't think there's a way to monitor the largest available contiguous block of memory.
Question is: any advice or suggestions for things to try?
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（4）
您可以检查绑定的 TransferMode 属性，看看您是否满足将其从默认值“Buffered”更改为“Streamed”或“流式响应”。
另外，请检查 maxBufferPoolSize 和 maxBufferSize 的值。增加所使用的内部缓冲区的大小有助于提高内存利用率，尤其是在处理大型消息时。
如果您收到大消息，maxReceivedMessageSize 也可能已经设置，但我也会检查该值。
我已经看到上面的值之一，如果超过阈值，则会失败并显示一条模糊的、与内存相关的消息。最初的异常实际上被我的应用程序显示的消息隐藏了。启用WCF 跟踪有助于诊断问题并查看真正的错误 - 我需要增加上面一个或多个绑定属性的值。
我没有从您的帖子中感受到您使用的绑定，但我相信这些设置在主要设置中是常见的。查看关于 basicHttpBinding 的 MSDN 文档例子。
如果确实是 LOH 碎片，那么一旦调优工作耗尽，就没有任何办法了。可能需要滚动回收应用程序来缓解它（我讨厌推荐这样做），但如果您已经用尽了其他努力，您可能会留下这样的结果。
You might review the TransferMode property of your binding to see if you meet the requirements to change it from its default value of "Buffered", to "Streamed" or "StreamedResponse".
Also, review the values for maxBufferPoolSize and maxBufferSize. Increasing the size of the internal buffers used can help with memory utilization, especially with processing large messages.
maxReceivedMessageSize is also likely already set if your receiving large messages, but I would review that value as well.
I've seen one of the values above, if your exceeding the threshold, fail with an obscure, memory related message. The original exception was actually hidden by the message that was surfaced to my application. Enabling WCF Tracing helped diagnose the problem and see the real error - I needed to increase the value of one, or more, of the binding properties above.
I didn't get the feel for the binding your using from your post, but I believe that these settings are common across major ones. Check out MSDN documentation on basicHttpBinding for example.
If it is truly LOH fragmentation there isn't anything to be done about it once tuning efforts have been exhausted. Rolling recycle of the application might be required to mitigate it (I hate recommending that) but if you've exhausted other efforts you might be left with that.
我无法解决任何 WCF 特定问题，但如果您需要最大化 32 位进程的 LOH 空间，您应该使应用程序 大地址感知并在 64 位上运行。当在 64 位 Windows 上运行时，大型地址感知 32 位进程将能够寻址整个 4 GB 地址空间。这将为您提供进程通常使用的地址空间之上的相当大的内存块。
I cannot address any of the WCF specific issues, but if you need to maximize LOH space for a 32 bit process, you should make the application large address aware and run it on 64 bit. A large address aware 32 bit process will be able to address the entire 4 GB address space when run on 64 bit Windows. This will give you a sizable chunk of memory above the address space normally used by the process.
如果可能的话，我会采用基于流的方法并结合使用仅向前的 Xml 解析器，这也会为您提供更好的性能。
如果您不是绝对必须使用 WCF，您可以编写自己的 HttpRequest，然后将响应传递给 XmlDeserializer，然后像这样解析响应。它可能会让您更好地控制和深入了解问题实际发生的位置。您还可以尝试模拟服务，该服务返回您正在查找的类型的非常大的文档。
LOH 碎片问题也让我们很头疼，所以我真的能感受到你们的痛苦。
我在构建缓冲区时注意到的一个问题是，每次缓冲区填满时，.NET 都会将容量加倍，这会导致内存碎片，因为对于大小为 10mb 的文档，需要分多个步骤分配内存。如果您事先知道所需的缓冲区大小，那么立即分配它会更有效。因此，如果您知道传入文档有多大，则可以创建一个具有该大小的 StringBuilder。
If possible I would go for a stream based approach and use a forward only Xml parser in combination, which should give you better performance as well.
If you don't absolutely have to use WCF, you can write your own HttpRequest and then pass the response to the XmlDeserializer and then parse the response like that. It might give you more control and insight into where the problem actually occurs. You can also experiment with a mock service that returns very large documents of the type you are looking for.
We had a lot of headaches with the LOH fragmentation as well so I really feel your pain.
A problem I noticed when building buffers .NET tends to double capacity every time the buffer is filled up, which causes memory fragmentation since for a document of size 10mb, memory needs to be allocated in many steps. If you know the needed buffer size in advance, it is more efficient to allocate it at once. So if you know how big the incoming document will be, you can create a StringBuilder with exactly that size.
我认为您的问题可能是由于使用 XmlSerializer 而不是使用此 MSDN 文章：
不错，呵呵。答案是缓存您的 XmlSerializer（假设您甚至创建了它）。
要真正弄清楚你需要做什么 Tess 告诉你做什么。她真是个天才。
I think your problem Might be an Assembly Leak caused by using XmlSerializer and not using one of two constructors as indicated in this MSDN article:
Nice, huh. The answer is to cache your XmlSerializer (assuming you even create it).
To really figure it out you need to do what Tess tells you to do. She's a freakin genius.