处理大文件时出现 OutOfMemoryException
我们正在将一个大的平面文件加载到 BizTalk Server 2006（原始版本，不是 R2）中 - 大约 125 MB。我们针对它运行一个映射，然后获取每一行并调用一个存储过程。
我们在编排处理期间收到 OutOfMemoryException，Windows 服务重新启动，使用完整的 2 GB 内存，然后再次崩溃。
服务器是 32 位并设置为使用 /3GB 开关。
此外，我还将流程分成 3 个主机 - 一台用于接收，另一台用于编排，第三台用于发送。
有人有任何建议可以让这个文件在没有错误的情况下进行处理吗？
谢谢， 克里普
We are loading a large flat file into BizTalk Server 2006 (Original release, not R2) - about 125 MB. We run a map against it and then take each row and make a call out to a stored procedure.
We receive the OutOfMemoryException during orchestration processing, the Windows Service restarts, uses full 2 GB memory, and crashes again.
The server is 32-bit and set to use the /3GB switch.
Also I've separated the flow into 3 hosts - one for receive, the other for orchestration, and the third for sends.
Anyone have any suggestions for getting this file to process wihout error?
Thanks,
Krip
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（2）
如果这是通过地图发送的平面文件，您是否会将其转换为 XML？尺寸的增加可能是巨大的。 XML 可以轻松地比平面文件增加 5-10 倍。特别是如果您使用描述性或长 xml 标记名称（通常您会这样做）。
您可以尝试的简单方法是将 xml 节点重命名为较短的名称，具体取决于记录数量（听起来很多），它实际上可能会对您的内存占用产生相当大的影响。
也许一种更企业化的方法是将其在自定义管道中细分为单独的消息包，这些消息包可以通过系统以更易于管理的块形式馈送（类似于克里斯的建议）。然后系统节流和内存指标就可以接管。如果不了解有关数据的更多信息，很难说如何最好地做到这一点，但对于 125 MB 的文件，我猜测您可能有大量不需要按顺序处理的重复行。
If this is a flat file being sent through a map you are converting it to XML right? The increase in size could be huge. XML can easily add a factor of 5-10 times over a flat file. Especially if you use descriptive or long xml tag names (which normally you would).
Something simple you could try is to rename the xml nodes to shorter names, depending on the number of records (sounds like a lot) it might actually have a pretty significant impact on your memory footprint.
Perhaps a more enterprise approach, would be to subdivide this in a custom pipeline into separate message packets that can be fed through the system in more manageable chunks (similar to what Chris suggests). Then the system throttling and memory metrics could take over. Without knowing more about your data it would be hard to say how to best do this, but with a 125 MB file I am guessing that you probably have a ton of repeating rows that do not need to be processed sequentially.
它在哪里崩溃？它是否能够通过变换形状？另一个尝试的建议是在接收端口中运行转换。为了更有效地处理，您甚至可以分批消息并让多个同步编排实例调用存储过程。这肯定会减少内存配置并提高性能。
Where does it crash? Does it make it past the Transform shape? Another suggestion to try is to run the transform in the Receive Port. For more efficient processing, you could even debatch the message and have multiple simultaneous orchestration instances be calling the stored procs. This would definately reduce the memory profile and increase performance.