让 JVM 根据需要增加内存需求,直至达到 VM 限制的大小?
我们发布了一个 Java 应用程序,其内存需求可能会根据其正在处理的数据大小而变化很大。 如果您不设置最大 VM(虚拟内存)大小,通常情况下 JVM 由于大数据上的 GC 失败而退出。
我们希望看到的是 JVM 请求更多内存,因为 GC 无法提供足够的内存,直到可用的 VM 总量耗尽。 例如,从 128Mb 开始,并在 GC 失败时以几何级数(或其他步骤)增加。
JVM(“Java”)命令行允许显式设置最大 VM 大小(各种 -Xm* 命令),您可能认为这已经足够了。 我们尝试在应用程序附带的 .cmd 文件中执行此操作。 但如果你选择任何特定的数字 你会得到两种不良行为之一:1)如果你的数量足够小,可以处理大多数情况 目标系统(例如,1Gb),它对于大数据来说不够大,或者2)如果你把它设置得非常大,JVM将拒绝在那些实际VM小于指定值的系统上运行。
如何设置 Java 在需要时使用可用的 VM,而无需提前知道该数字,并且无需在启动时获取所有 VM?
We ship a Java application whose memory demand can vary quite a lot depending on the size of the data it is processing. If you don't set the max VM (virtual memory) size, quite often
the JVM quits with an GC failure on big data.
What we'd like to see, is the JVM requesting more memory, as GC fails to provide enough, until the total available VM is exhausted. e.g., start with 128Mb, and increase geometrically (or some other step) whenever the GC failed.
The JVM ("Java") command line allows explicit setting of max VM sizes (various -Xm* commands), and you'd think that would be designed to be adequate. We try to do this in a .cmd file that we ship with the application. But if you pick any specific number,
you get one of two bad behaviors: 1) if your number is small enough to work on most
target systems (e.g., 1Gb), it isn't big enough for big data, or 2) if you make it very large, the JVM refuses to run on those systems whose actual VM is smaller than specified.
How does one set up Java to use the available VM when needed, without knowing that number in advance, and without grabbing it all on startup?
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论
评论(14)
您还可以使用以下选项:
-XX:+AggressiveHeap
根据[文档][1]:
[1]: http://java.sun.com/docs/热点/gc1.4.2/#4.2.2。 AggressiveHeap|大纲
You can also use the option:
-XX:+AggressiveHeap
This according to the [documentation][1]:
[1]: http://java.sun.com/docs/hotspot/gc1.4.2/#4.2.2. AggressiveHeap|outline
我们有一个小型 C 应用程序,用于通过 JNI 启动所有 Java 应用程序。 我们能够:
这使 在应用程序中,我们只是硬编码堆限制,但您可以根据可用内存轻松动态配置最大堆大小。
这种小应用程序实际上很容易做(这是使用 JNI 做的最简单的事情之一)。 一个好的起点是 JDK 的源代码(您可以使用 java.exe 本身的子文件夹 - 这就是我们所做的)。 大多数人都非常惊讶地发现 java.exe 是一个很小的应用程序(不到 200 行代码),它仅调用 JNI 并传递命令行参数(哎呀,甚至使用名为 main() 的方法也是相当可选的一旦你开始自己启动东西)。
下面的代码不仅启动 JVM 等...而且还根据计算机的可用 RAM 确定最大堆空间。 这是一个 SO 帖子的大量代码,而且一点也不漂亮 - 但这是久经沙场的代码 - 它已经使用了近十年,超过数百次安装,等等......享受:
We have a small C application that we use for launching all of our Java applications via JNI. This allows us to:
For our apps, we just hard code the heap limit, but you could easily dynamically configure max heap size based on available memory.
This sort of little app is actually pretty easy to do (it's one of the easiest things to do with JNI). A good starting point would be the source for the JDK (there's a sub-folder for java.exe itself that you can use - that's what we did). Most folks are quite surprised to find that java.exe is a little tiny application (< 200 lines of code) that just invokes JNI and passes command line arguments in (heck, even the use of a method called main() is pretty optional once you start launching things yourself).
Here's code that not only starts up the JVM, etc... but also determines the maximum heap space based on available RAM of the computer. This is a lot of code for an SO post, and it's not at all pretty - but this is battle hardened code - it's been used for almost a decade over many hundreds of installs, etc... Enjoy :
最大虚拟机大小确实满足了这种需求(它设置了最大值,但虚拟机将只采取必要的步骤),但是如果您需要多种配置,除了提供不同的“cmd”文件之外,我真的看不到一种方式(尽管我会多搜索一点)
[编辑]
如何使用第一个程序/脚本(甚至另一个java程序),它会检查系统的可用资源,然后根据从系统检索到的内容,仅使用适当的-Xm调用您的程序?
这样它就能适应机器,即使你以前不了解它们。 可能是一个想法......
[第二次编辑]
好的,这已经由 skaffman,我的错。
The max VM sizes indeed answer to that need (it sets the max value, but the VM will take only necessary, step by step), but if you need several configurations, besides supplying different "cmd" files, I don't really see a way (though i'll search a bit more)
[edit]
How about using a first program/script (or even another java program), which would check the available resources for the system, and then only call your program with the appropriate -Xm, according to what it retrieved from system ?
That way it would adapt to machines, even if you don't know them before. Could be an idea...
[second edit]
Ok, this has been proposed already by skaffman, my bad.
还有一个选择...我正在开发一个名为 WinRun4J 的启动器,它允许您指定最大堆大小占其运行所在计算机上可用内存的百分比(即,它检查可用内存量并在启动时动态设置 -Xmx 参数)。
INI 选项是“vm.heapsize.max.percent”。 还有另一个选项“vm.heapsize.preferred”,它将 -Xmx 参数设置为机器上最大可用内存量。
我相信其他一些启动器(例如 Launch4J、Janel)提供相同的功能。
One more option... I work on a launcher called WinRun4J, which allows you to specify a max heap size as a percentage of the available memory on the machine its running on (ie. it does a check for the amount of memory available and sets the -Xmx parameter dynamically on startup).
The INI option is "vm.heapsize.max.percent". There is also another option "vm.heapsize.preferred", which sets the -Xmx parameter as the maximum available memory on the machine up to this amount.
I believe some of the other launchers (eg. Launch4J, Janel) offer the same functionality.
我认为你运气不好:-(
-Xms
和-Xmx
选项不提供这种灵活性。所以我认为你需要包装你的 JVM 调用使用可以确定最大内存量的脚本,然后适当地设置
-Xmx
(可能是使用 WMI(Windows 上))。或者它可能会在第一次运行时询问用户?我担心有点痛苦。
I think you're out of luck :-( The
-Xms
and-Xmx
options don't provide that flexibility.So I think you will need to wrap your JVM invocation with a script that can determine the maximum amount of memory, and then set
-Xmx
appropriately (probably a .vbs script using WMI on Windows). Or perhaps it asks the users the first time it's run ?A bit of a pain, I fear.
我认为最简单的方法是通过一些包装应用程序启动 JVM,该应用程序将检查系统资源以确定内存可用性,然后使用适当的 -Xmx 参数启动 JVM。
接下来的问题就变成了如何编写该包装器。 包装器应用程序本身甚至有可能是 JVM,尽管我不认为 API 或系统属性会公开必要的信息。 也许 shell 脚本或您的选择可以获取该信息。
I think the easiest way to do this would be to launch the JVM via some wrapper application, which would check system resources to determine memory availability, and then launch the JVM with the appropriate -Xmx parameter.
The question then becomes how that wrapper would be written. It may even be possible for the wrapper app to itself be a JVM, although I don't think the API or system properties would expose the necessary information. Maybe a shell script or your choice could get the information.
如果您有很多时间,您可以尝试以下操作:
尝试获取所需的内存与输入数据集。 这样,您可以将处理拆分到一组不同的类中,并创建一个新的 JVM 进程来实际处理数据。 基本上是一个经理和一个工人。 Manager 将对所需的数据集进行基本分析,并生成具有适当内存要求的 Worker。 您还可以将管理器设置为了解环境,并在用户尝试操作其计算机无法处理的数据集时发出警告。
这几乎是 skaffman 提供的答案的扩展,但就用户而言,这一切都将在同一个应用程序中发生。
if you have a lot of time on your hand you could try the following :
Try to obtain what is the needed memory vs input dataset. With this you can split processing in a different set of classes and create a new JVM process to actually process the data. Basically a Manager and a Worker. The Manager would do a basic analysis on the demanded dataset and spawn a Worker with the appropriate memory requirements. You could probably also set your Manager to be aware of the environment and warn the user when they are trying to operate on a dataset their machine cannot handle.
This is pretty much an extension on the answer provided by skaffman but will happen all within the same app as far as the user is concerned.
虚拟机参数中有两个选项可以使用:-Xms 用于设置启动时的内存大小,-Xmx 用于设置最大内存大小...
您可以设置较低的启动内存和较大的最大内存,因此VM 仅在需要时才会分配新内存。
There is two options in the virtual machine arguments that can be used : -Xms to set the memory size at startup and -Xmx to set the maximum memory size...
You can set a low startup memory and a big maximum one, so the VM will allocate new memory only if needed.
我认为 Sun 或 IBM JVM 都无法做到这一点(我知道 AS/400 可以,但这很可能与您无关)。
我建议使用 Java WebStart(在您放弃它之前,请注意它已使用 Java 6 u 10 进行更新,并且更适合启动“本地”应用程序和小程序),因为它允许您提供“小型实例” 、“更大的实例”、“巨大的实例”作为链接/图标。
您很可能会查看“在 Webstart 缓存中注入应用程序”和“离线”选项。
I do not think either the Sun or IBM JVM can do this (I know that the AS/400 one can, but that is most likely not relevant to you).
I would suggest using Java WebStart (and before you discard this, then notice that it has been updated with Java 6 u 10 and is much better suited for launching "local" applications and applet) since it allows you to provide a "small instance", "larger instance", "gigantic instance" as links/icons.
You will most likely look into the "inject application in webstart cache" and "offline"options.
我不认为你能做你想做的事; 相反,您必须针对您的客户、他们的系统以及他们如何修改您的
.cmd
文件以允许更多内存的需求发布说明。当然,如果您的产品面向非技术用户,您可能希望将其隐藏在一些更用户友好的配置文件后面。 例如,
或者可能根据用户在最初订购产品时指定的产品选项来部署不同的配置文件。
I don't think you can do what you are trying to do; instead you'll have to ship instructions specific to your customers, their systems and their demands of how they can modify your
.cmd
file to allow for more memory.Of course, if your product is aimed at very non-technical users, you may wish to hide this behind some more user-friendly config file. E.g.
or possibly deploy different config files depending on which product option a user specifies when they order the product in the first place.
在评论中,您说应用程序的内存量实际上取决于用户提供的输入数据集大小。 这表明,您应该在启动 JVM 之前查看输入数据集大小,并使用它来估计应用程序所需的内存量,而不是尝试获取所有可用的虚拟内存(这可能会给用户的其他应用程序带来问题) 。
假设用户的机器配置了适度的物理内存和巨大的交换空间。 如果您启动具有巨大 VM 大小的 JVM,则可能会导致严重的“抖动”,因为 JVM 会尝试访问非常驻页面中的数据。 相比之下,如果您为 JVM 提供的内存多于应用程序所需且少于可用物理内存,则您应该能够舒适地运行而不会出现抖动。
In comments you say that the amount of memory that your application actually depends on the input dataset size provided by the user. This suggests that instead of trying to grab all available virtual memory (which may cause problems for the user's other applications) you should be looking at the input dataset size before you start the JVM and using that to estimate the amount of memory the application will need.
Suppose that the user's machine is configured with modest physical memory and a huge swap space. If you launch the JVM with a huge VM size, it could cause severe "thrashing" as the JVM tries to access data in non-resident pages. By contrast, if you give the JVM something more than the application needs and less than the available physical memory, you should be able to run comfortably without thrashing.
我通读了这些线程,但没有看到任何表明该应用程序已经过某种分析的内容。 通常,我会在某些条件下分析应用程序,以查找性能或内存使用的热点。 在大多数情况下,可能还有一些可以改进的地方。
如果您可以建立限制并了解应用程序的行为,您就可以更好地告诉客户他们可以或不能使用该应用程序做什么,从而减少支持电话的数量,并让您更好地了解最小或不可以做什么。运送产品的最大堆大小。
也许你可以从这个开始: http://www.eclipse.org/mat/
I read through the threads but didn't see anything which indicated that the application had undergone some sort of profiling. Normally I'd profile the apps under certain conditions to find hot spots in performance or memory usage. There's probably things that could be improved in most cases.
If you could establish the limits and understand the behavior of the application you could be in a position to better tell your customers what they can or cannot do with the application thereby reducing the amount of support calls and giving you a better idea of what minimum or maximum heap size to ship the product with.
Maybe you could start with this: http://www.eclipse.org/mat/
您是否考虑过运行 jps 来为您的进程提供 PID,然后调用 jinfo 来更改 mx 选项? 不确定这是否有效,但可能会。
[编辑]这意味着,当您认为自己有一个大数据集时,您会以某种方式读取内存总量(我认为取决于操作系统。请参阅 http://forums.sun.com/thread.jspa?messageID=10306570)或者您只是增加大小,直到您认为它不再低(如果它首先爆炸,请尝试捕获并显示一条有用的消息,例如“您的机器不够用,是时候跑到弗赖斯那里了”)。
Have you looked at running jps to give you the PID for your process and then calling jinfo to change the mx option? Not sure if this will work but it may.
[Edit] This would mean that when you think you have a big dataset, you read the total amount of ram somehow (OS dependent I think. See http://forums.sun.com/thread.jspa?messageID=10306570) or you just increase the size until you don't think it is low anymore (if it blows up first, try to capture and display a helpful message such as "your machine is inadequate, time to make a run to Frys").
如果您认为您的客户可以在其 32 位计算机上要求 2-3GB RAM,则此讨论毫无意义。 操作系统和其他应用程序也将充分运行。
听起来您的应用程序已经达到了需要 64 位操作系统和更多 RAM 的程度。
This discussion is moot if you think that your clients can ask for 2-3GB of RAM on their 32-bit machine. The OS and other apps will be taking their pound of flesh to run as well.
Sounds like your app is reaching the point where it needs a 64-bit operating system and lots more RAM.