负载测试内存问题
我们正在 Linux 盒子中测试在 IBM 门户服务器上运行的应用程序，但发现 vmstat 的“空闲”值即使在测试之后也在稳步下降。从上面看，“VIRT”值也在稳步增加。通过监控Java应用程序堆的使用情况，初始堆大小（1.5G）从未达到，并且使用量缓慢而稳定地上升（测试期间有小幅上升/下降），从6xxm到大约1g，直到测试结束。测试刚结束时，就大幅回落至6XXm左右。我的问题是： 1、结果是否正常、OK？ 2. 应用程序堆使用行为是否正常？ 3、测试后，vmstat的“free”值稳步下降，top的“VIRT”值稳步上升，没有下降，是否正常？
以下是 top 和 vmstat 输出：
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14157 user01 17 0 2508m 1.2g 47m S 16.0 23.3 11:38.94 java
14157 user01 17 0 2508m 1.2g 47m S 16.9 23.3 11:49.08 java
14157 user01 17 0 2508m 1.2g 47m S 15.8 23.4 11:58.58 java
14157 user01 17 0 2509m 1.2g 47m S 13.0 23.5 12:06.36 java
14157 user01 17 0 2509m 1.2g 47m S 17.6 23.5 12:16.92 java
14157 user01 17 0 2509m 1.2g 47m S 16.9 23.6 12:27.09 java
14157 user01 17 0 2510m 1.2g 47m S 16.1 23.6 12:36.73 java
14157 user01 17 0 2510m 1.2g 47m S 14.5 23.7 12:45.43 java
...
14157 user01 17 0 2514m 1.2g 47m S 15.9 24.6 15:20.18 java
14157 user01 17 0 2514m 1.2g 47m S 16.2 24.6 15:29.88 java
14157 user01 17 0 2514m 1.2g 47m S 16.1 24.7 15:39.56 java
14157 user01 17 0 2515m 1.2g 47m S 19.5 24.7 15:51.28 java
14157 user01 17 0 2516m 1.2g 47m S 11.4 24.8 15:58.11 java
14157 user01 17 0 2515m 1.2g 47m S 14.7 24.8 16:06.91 java
14157 user01 17 0 2515m 1.2g 47m S 16.0 24.9 16:16.51 java
14157 user01 17 0 2515m 1.2g 47m S 16.1 24.9 16:26.15 java
14157 user01 17 0 2515m 1.2g 47m S 14.7 25.0 16:34.96 java
14157 user01 17 0 2516m 1.2g 47m S 11.8 25.0 16:42.03 java
...
14157 user01 17 0 2517m 1.3g 47m S 13.1 25.6 18:18.04 java
14157 user01 17 0 2517m 1.3g 47m S 17.8 25.6 18:28.75 java
14157 user01 17 0 2516m 1.3g 47m S 15.2 25.7 18:37.85 java
14157 user01 17 0 2517m 1.3g 47m S 13.5 25.7 18:45.93 java
14157 user01 17 0 2516m 1.3g 47m S 14.6 25.8 18:54.70 java
14157 user01 17 0 2517m 1.3g 47m S 14.6 25.8 19:03.47 java
14157 user01 17 0 2517m 1.3g 47m S 15.3 25.9 19:12.67 java
14157 user01 17 0 2517m 1.3g 47m S 16.6 25.9 19:22.64 java
14157 user01 17 0 2517m 1.3g 47m S 15.0 26.0 19:31.65 java
14157 user01 17 0 2517m 1.3g 47m S 12.4 26.0 19:39.09 java
...
14157 user01 17 0 2530m 1.4g 47m S 0.0 27.5 23:23.91 java
procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 2004 702352 571508 1928436 0 0 0 54 287 413 1 1 98 0 0
0 0 2004 702368 571528 1928416 0 0 0 12 280 379 0 0 100 0 0
...
24 0 2004 673988 572504 1948000 0 0 0 440 760 751 16 6 78 0 0
0 0 2004 671352 572540 1951048 0 0 0 477 1180 830 19 7 74 0 0
0 0 2004 674756 572572 1946904 0 0 0 380 604 650 13 3 84 0 0
1 0 2004 694208 572612 1928360 0 0 0 222 518 599 7 2 91 0 0
16 0 2004 692068 572640 1929360 0 0 0 539 1075 850 24 7 69 0 0
0 0 2004 689036 572680 1931376 0 0 0 292 978 781 14 6 81 0 0
...
0 0 2004 530432 579120 2007176 0 0 0 453 511 712 18 4 78 0 0
0 0 2004 528440 579152 2008172 0 0 0 200 436 652 10 2 87 0 0
0 0 2004 524352 579192 2010188 0 0 0 401 514 779 17 6 76 0 0
0 0 2004 524964 578208 2012200 0 0 0 514 475 696 15 3 82 0 0
0 0 2004 522484 578260 2013176 0 0 0 416 488 699 15 3 82 0 0
2 0 2004 521264 578300 2015192 0 0 0 368 501 728 14 5 80 0 0
0 0 2004 518400 578340 2016180 0 0 0 404 452 647 14 3 84 0 0
25 0 2004 517064 578368 2018208 0 0 0 414 497 752 15 3 82 0 0
...
0 0 2004 499312 578820 2029064 0 0 0 351 459 660 13 3 84 0 0
0 0 2004 496228 578872 2031068 0 0 0 260 473 701 15 5 80 0 0
0 0 2004 501360 578912 2026916 0 0 0 500 398 622 9 3 88 0 0
1 0 2004 499260 578948 2027908 0 0 0 262 436 638 13 2 85 0 0
1 0 2004 497964 578984 2028900 0 0 0 276 452 628 15 3 82 0 0
0 0 2004 497492 579024 2029888 0 0 0 200 384 548 7 2 91 0 0
0 0 2004 496620 579044 2030896 0 0 0 172 393 586 9 2 89 0 0
...
1 0 2004 357876 566000 2104592 0 0 0 374 510 736 18 6 76 0 0
23 0 2004 358544 566032 2105588 0 0 0 362 456 644 12 3 85 0 0
0 0 2004 376332 566084 2087032 0 0 0 353 441 614 13 3 84 0 0
0 0 2004 375888 566120 2088024 0 0 0 220 411 620 10 2 88 0 0
0 0 2004 375280 566156 2087988 0 0 0 224 408 586 7 2 91 0 0
16 0 2004 373092 566188 2090012 0 0 0 233 494 723 12 3 85 0 0
2 0 2004 369564 566236 2090992 0 0 0 455 475 714 14 5 80 1 0
...
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 2004 235156 572776 2155384 0 0 0 8 282 396 0 0 100 0 0
0 0 2004 235132 572796 2155364 0 0 0 24 291 435 0 0 100 0 0
1 0 2004 234780 572828 2155332 0 0 0 101 292 474 1 5 94 0 0
0 0 2004 234804 572844 2155316 0 0 0 45 288 451 0 1 99 0 0
0 0 2004 234852 572856 2155304 0 0 0 12 283 409 0 0 100 0 0
堆使用情况： 
We are testing our app running on IBM portal server in the Linux box but found that the "free" values of vmstat are decreasing steadily, even after the test. By looking at top, the "VIRT" values are also increasing steadily. By monitoring Java app heap usage, the initial heap size (1.5G) was never reached, and the usage was rising slowly and steadily (with minor rises/drops within the test period) from 6xxm to about 1g until the test ended. When the test just ended, it dropped by a large amount back to about 6XXm. My questions are:
1. Is the result normal and OK?
2. Is the app heap usage behaviour OK?
3. Is it normal that the "free" values of vmstat are decreasing steadily and "VIRT" values of top are increasing steadily without drop after the test?
Below are top and vmstat output:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14157 user01 17 0 2508m 1.2g 47m S 16.0 23.3 11:38.94 java
14157 user01 17 0 2508m 1.2g 47m S 16.9 23.3 11:49.08 java
14157 user01 17 0 2508m 1.2g 47m S 15.8 23.4 11:58.58 java
14157 user01 17 0 2509m 1.2g 47m S 13.0 23.5 12:06.36 java
14157 user01 17 0 2509m 1.2g 47m S 17.6 23.5 12:16.92 java
14157 user01 17 0 2509m 1.2g 47m S 16.9 23.6 12:27.09 java
14157 user01 17 0 2510m 1.2g 47m S 16.1 23.6 12:36.73 java
14157 user01 17 0 2510m 1.2g 47m S 14.5 23.7 12:45.43 java
...
14157 user01 17 0 2514m 1.2g 47m S 15.9 24.6 15:20.18 java
14157 user01 17 0 2514m 1.2g 47m S 16.2 24.6 15:29.88 java
14157 user01 17 0 2514m 1.2g 47m S 16.1 24.7 15:39.56 java
14157 user01 17 0 2515m 1.2g 47m S 19.5 24.7 15:51.28 java
14157 user01 17 0 2516m 1.2g 47m S 11.4 24.8 15:58.11 java
14157 user01 17 0 2515m 1.2g 47m S 14.7 24.8 16:06.91 java
14157 user01 17 0 2515m 1.2g 47m S 16.0 24.9 16:16.51 java
14157 user01 17 0 2515m 1.2g 47m S 16.1 24.9 16:26.15 java
14157 user01 17 0 2515m 1.2g 47m S 14.7 25.0 16:34.96 java
14157 user01 17 0 2516m 1.2g 47m S 11.8 25.0 16:42.03 java
...
14157 user01 17 0 2517m 1.3g 47m S 13.1 25.6 18:18.04 java
14157 user01 17 0 2517m 1.3g 47m S 17.8 25.6 18:28.75 java
14157 user01 17 0 2516m 1.3g 47m S 15.2 25.7 18:37.85 java
14157 user01 17 0 2517m 1.3g 47m S 13.5 25.7 18:45.93 java
14157 user01 17 0 2516m 1.3g 47m S 14.6 25.8 18:54.70 java
14157 user01 17 0 2517m 1.3g 47m S 14.6 25.8 19:03.47 java
14157 user01 17 0 2517m 1.3g 47m S 15.3 25.9 19:12.67 java
14157 user01 17 0 2517m 1.3g 47m S 16.6 25.9 19:22.64 java
14157 user01 17 0 2517m 1.3g 47m S 15.0 26.0 19:31.65 java
14157 user01 17 0 2517m 1.3g 47m S 12.4 26.0 19:39.09 java
...
14157 user01 17 0 2530m 1.4g 47m S 0.0 27.5 23:23.91 java
procs -----------memory---------- ---swap-- -----io---- -system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 2004 702352 571508 1928436 0 0 0 54 287 413 1 1 98 0 0
0 0 2004 702368 571528 1928416 0 0 0 12 280 379 0 0 100 0 0
...
24 0 2004 673988 572504 1948000 0 0 0 440 760 751 16 6 78 0 0
0 0 2004 671352 572540 1951048 0 0 0 477 1180 830 19 7 74 0 0
0 0 2004 674756 572572 1946904 0 0 0 380 604 650 13 3 84 0 0
1 0 2004 694208 572612 1928360 0 0 0 222 518 599 7 2 91 0 0
16 0 2004 692068 572640 1929360 0 0 0 539 1075 850 24 7 69 0 0
0 0 2004 689036 572680 1931376 0 0 0 292 978 781 14 6 81 0 0
...
0 0 2004 530432 579120 2007176 0 0 0 453 511 712 18 4 78 0 0
0 0 2004 528440 579152 2008172 0 0 0 200 436 652 10 2 87 0 0
0 0 2004 524352 579192 2010188 0 0 0 401 514 779 17 6 76 0 0
0 0 2004 524964 578208 2012200 0 0 0 514 475 696 15 3 82 0 0
0 0 2004 522484 578260 2013176 0 0 0 416 488 699 15 3 82 0 0
2 0 2004 521264 578300 2015192 0 0 0 368 501 728 14 5 80 0 0
0 0 2004 518400 578340 2016180 0 0 0 404 452 647 14 3 84 0 0
25 0 2004 517064 578368 2018208 0 0 0 414 497 752 15 3 82 0 0
...
0 0 2004 499312 578820 2029064 0 0 0 351 459 660 13 3 84 0 0
0 0 2004 496228 578872 2031068 0 0 0 260 473 701 15 5 80 0 0
0 0 2004 501360 578912 2026916 0 0 0 500 398 622 9 3 88 0 0
1 0 2004 499260 578948 2027908 0 0 0 262 436 638 13 2 85 0 0
1 0 2004 497964 578984 2028900 0 0 0 276 452 628 15 3 82 0 0
0 0 2004 497492 579024 2029888 0 0 0 200 384 548 7 2 91 0 0
0 0 2004 496620 579044 2030896 0 0 0 172 393 586 9 2 89 0 0
...
1 0 2004 357876 566000 2104592 0 0 0 374 510 736 18 6 76 0 0
23 0 2004 358544 566032 2105588 0 0 0 362 456 644 12 3 85 0 0
0 0 2004 376332 566084 2087032 0 0 0 353 441 614 13 3 84 0 0
0 0 2004 375888 566120 2088024 0 0 0 220 411 620 10 2 88 0 0
0 0 2004 375280 566156 2087988 0 0 0 224 408 586 7 2 91 0 0
16 0 2004 373092 566188 2090012 0 0 0 233 494 723 12 3 85 0 0
2 0 2004 369564 566236 2090992 0 0 0 455 475 714 14 5 80 1 0
...
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 2004 235156 572776 2155384 0 0 0 8 282 396 0 0 100 0 0
0 0 2004 235132 572796 2155364 0 0 0 24 291 435 0 0 100 0 0
1 0 2004 234780 572828 2155332 0 0 0 101 292 474 1 5 94 0 0
0 0 2004 234804 572844 2155316 0 0 0 45 288 451 0 1 99 0 0
0 0 2004 234852 572856 2155304 0 0 0 12 283 409 0 0 100 0 0
Heap Usage:
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
您看到的最可能的原因是 JAVA_OPTS 中的 -Xms 和 -Xmx 值不同。这里发生的情况是操作系统在 JVM 需要时分配内存。在启动时将所需的所有堆分配给 JVM 从来都不是坏习惯。将 2 个值设置为其上限。通过将两个值设置为相等，JVM 永远不必向操作系统请求内存，并且可以在自己的内存空间内自由地完成所需的工作。
您所看到的行为并不罕见，JVM 将继续向操作系统请求更多内存，直到达到上限 (-Xmx)。如果您不熟悉调整堆大小或调整 JVM 的其他技术，请查看 本指南。
另一方面，top 和 vmstat 只能向您展示 JVM 内存所发生情况的大部分情况。您所看到的是操作系统分配给它的内容。您将需要使用其他工具，例如 jmap和jvisualvm看看里面的内存怎么样JVM 正在响应。这些工具将为您的应用程序提供更好的基准。他们将向您展示的是新生代和老一代、垃圾收集和其他非常重要的统计数据。
The most likely cause that you are seeing is that the -Xms and -Xmx values differ in your JAVA_OPTS. What appears to be happening here is that the OS is allocating memory when needed by the JVM. It is never bad practice to allocate all of the heap you will need to the JVM on start-up. Set the 2 values to their upper limit. By setting the two values equal, the JVM never has to request memory from the OS and is free to do the work it needs to within its own memory space.
It is not uncommon to see the behavior you have seen, the JVM will continue to request more memory from the OS until the upper limits (-Xmx) are reached. If you are new to Sizing the heap, or other techniques around tuning the JVM, have a look at this guide.
On another note, top and vmstat will only show you so much of the picture into what is happening with the JVM's memory. What you are seeing is what the operating system is allocating to it. You will want to use other tools such as jmap and jvisualvm to see how the memory inside the JVM is responding. These tools will be a better bench mark for your application. What they will show you is the New and Old generations, Garbage collections and other stats which are really important.