如何优先考虑 Java G1 内存垃圾收集而不是速度？
我有一个 Java/Spring 数据传输服务，它从 .csv 文件读取记录，解析它们，在内存中整理它们，然后将它们加载到数据库中。
每次运行都会解析一个包含约 800k 记录的文件。
该服务部署到 Kubernetes 容器中。生产环境和应用程序设计都不适合此应用程序，但我必须在一些限制范围内进行操作。
运行两三次后，服务因内存错误而崩溃。我们看到垃圾收集时间很长，并假设垃圾收集没有跟上。
我们正在使用 G1 垃圾收集器，我想调整收集以优先考虑内存而不是速度。我不关心服务的效率或速度有多快，它只需要执行几次数据传输。
哪些设置可以实现此目的？
I have a Java/Spring data transfer service that reads records in from a .csv file, parses them, collates them in memory, and then loads them into a database.
Each run of parses a file that contains ~800k records.
The service is deployed to a Kubernetes container. Neither the production environment nor the application design are ideal for this application, but I have to operate within some restrictions.
After two or three runs, the service crashes due to a memory error. We are seeing long garbage collection times, and assume that garbage collection is not keeping pace.
We are using the G1 garbage collector, and I want to tune the collection to prioritize memory over speed. I don't care how efficient or fast the service is, it only has to perform this data transfer a few times.
What settings will accomplish this?
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
GC 时间过长只是问题的症状，而不是问题的根本原因。如果 GC 根本跟不上，则不应导致 OOME。
（大量使用终结器、
Reference对象或类似对象可能会使 GC 更难以跟上，但这仍然是一个症状。这似乎与您的用例相关。 ）我的理论是，收集时间长的真正原因是你的堆太小了。当堆接近满时，GC 必须运行越来越频繁，并且能够回收越来越少的空间。这导致收集时间很长。最后，您会收到 OOME，因为您完全用完了堆空间，或者因为您达到了 GC 开销阈值。
另一种可能性是，您的堆对于可用 RAM 来说太大了……并且您正在遭受虚拟内存抖动
。无论哪种情况，简单地调整 GC 设置都没有帮助。您需要先确定问题的根本原因，然后才能解决问题。
我的看法是，要么存在内存泄漏，要么没有足够的 RAM，要么您的应用程序设计存在问题。
在设计方面，不要将整个文件读取/解析为内存中的数据结构，而是使用流式/基于事件的解析器。一次读取一个记录，处理它们，然后丢弃它们……在内存中保留尽可能少的有关它们的信息。换句话说，减少应用程序对内存的占用。
Long GC times are a symptom of the problem rather than the root cause of the problem. If the GC is simply not keeping up, that should not cause OOMEs.
(It possible that heavy use of finalizers,
Referenceobjects or similar make it harder for the GC to keep up, but that is still a symptom. It seems likely that this is relevant in your use-case.)My theory is that the real cause of the long collection times is that your heap is too small. When your heap is nearly full, the GC has to run more and more often and is able to reclaim less and less space. That leads to long collection times. Then finally, you get an OOME because either you run out of heap space entirely, or because you hit the GC overhead threshold.
Another possibility is that your heap is too big for the available RAM ... and you are getting virtual memory thrashing
In either case, simply tweaking the GC settings is not going to help. You need to identify the root cause of the problem before you can fix it.
My take is that either you have a memory leak, or not enough RAM, or there is a problem with your application's design.
On the design side, rather than reading / parsing the entire file as an in-memory data structure, use a streaming / event-based parser. Read records one at a time, process them and then discard them ... keeping as little information about them in memory as you can get away with. In other words, make the application less memory hungry.