容器无法启动:用于继续的Java运行时环境的内存不足
我们有一个在Java 8上运行的企业应用程序。部署环境是构建&通过BitBucket管道更新。我有一个图形显示环境的高级体系结构。除了某些特定于应用程序的环境变量外,我们还有两个运行相同配置的应用程序服务器。
直到一周前,成功的管道运行后,这一切都很好,其中一台服务器上的两个应用程序实例停止处理以下错误:
There is insufficient memory for the Java Runtime Environment to continue.
Cannot create GC thread. Out of system resources.
两个实例在另一台服务器上都可以正常工作。相反,容器无法在此服务器上启动。
解决方案
尝试了以下信息:
可能的原因: 该系统不含物理RAM或交换空间 该过程启用了压缩机运行,Java堆可能会阻止本机堆的生长。
可能的解决方案:
- 减少系统上的内存负载
- 增加物理内存或交换空间
- 检查交换备份商店是否完全
- 降低Java堆大小(-XMX/-XMS)
- 减少Java线程的数量
- 减小Java线程堆栈尺寸(-XSS)
- 设置较大的代码带有-xx的缓存:保留codecachesize =
我们尝试过:
- 添加更多交换内存。当我们尝试从4GB到9GB的交易时,该服务器具有8GB RAM。
- 播放了堆尺寸XMS& XMX从1.28m到4096m。
- 将此服务器上的RAM增加到16GB,而另一个工作人员仍在8GB上使用。
这是记忆的方式&交换消费看起来像:
free -mh
total used free shared buff/cache available
Mem: 15Gi 378Mi 12Gi 1.0Mi 2.9Gi 14Gi
Swap: 9Gi 0B 9Gi
我有几个相关文物的链接。其中包括完成docker logs
输出和docker信息的输出
on 失败的服务器和操作服务器。
这就是docker ps -a
吸引我们的原因:
:~$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d29747bf2ad3 :a7608a838625ae945bd0a06fea9451f8bf11ebe4 "catalina.sh run" 10 hours ago Exited (1) 10 hours ago jbbatch
0951b6eb5d42 :a7608a838625ae945bd0a06fea9451f8bf11ebe4 "catalina.sh run" 10 hours ago Exited (1) 10 hours ago jbapp
我们现在已经尝试了几乎所有解决方案堆栈溢出的解决方案。我们缺少什么?
We have an enterprise application running on Java 8. The deployment environment is built & updated through Bitbucket pipelines. I have a graphic showing the high-level architecture of the environment. We have two app servers running identical configurations apart from some application specific environment variables.
It was all working well until a week ago when after a successful pipeline run, the 2 app instances on one of the servers stopped working with the following error:
There is insufficient memory for the Java Runtime Environment to continue.
Cannot create GC thread. Out of system resources.
Both the instances are working fine on the other server. In contrast, the containers fail to start on this server.
Solutions Tried
The error accompanies the following information:
Possible reasons:
The system is out of physical RAM or swap space
The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap.
Possible solutions:
- Reduce memory load on the system
- Increase physical memory or swap space
- Check if swap backing store is full
- Decrease Java heap size (-Xmx/-Xms)
- Decrease number of Java threads
- Decrease Java thread stack sizes (-Xss)
- Set larger code cache with -XX:ReservedCodeCacheSize=
We have tried:
- Adding more swap memory. The server has 8GB of RAM while we have tried the swap from 4GB to 9GB.
- Played with the heap sizes Xms & Xmx from 128m to 4096m.
- Increased the RAM on this server to 16GB while the other server that works still does on 8GB.
Here is how the memory & swap consumption looks like:
free -mh
total used free shared buff/cache available
Mem: 15Gi 378Mi 12Gi 1.0Mi 2.9Gi 14Gi
Swap: 9Gi 0B 9Gi
I have links to several related artifacts. These include the complete docker logs
output and the output of docker info
on the failing server and the operational server.
This is what docker ps -a
gets us:
:~$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d29747bf2ad3 :a7608a838625ae945bd0a06fea9451f8bf11ebe4 "catalina.sh run" 10 hours ago Exited (1) 10 hours ago jbbatch
0951b6eb5d42 :a7608a838625ae945bd0a06fea9451f8bf11ebe4 "catalina.sh run" 10 hours ago Exited (1) 10 hours ago jbapp
We are out of ideas right now as we have tried almost all the solutions on stack overflow. What are we missing?
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论
评论(2)
我看到您的Docker图像使用Ubuntu 22.04 LTS作为其基础。最近,基本的Java图像在此LTS版本之上进行了重建,这在较旧的Docker Runtimes上引起了很多问题。这很可能是您所经历的。它与内存无关,而是与docker不兼容与用作基本图像的较新的Linux版本。
您的操作服务器具有Docker Server版本20.10.10,而失败的服务器具有20.10.09版本。不兼容的问题完全解决了Docker 20.10.10。有关不兼容问题的更多技术详细信息,请在这里。
解决方案将是将故障服务器升级到至少Docker 20.10.10。
I see that your Docker image uses Ubuntu 22.04 LTS as its base. Recently base Java images were rebuilt on top of this LTS version, which caused a lot of issues on older Docker runtimes. Most likely this is what you're experiencing. It has nothing to do with memory, but rather with Docker incompatibility with a newer Linux version used as a base image.
Your operational server has Docker server version 20.10.10, while the failing server has version 20.10.09. The incompatibility issue was fixed exactly in Docker 20.10.10. Some more technical details on the incompatibility issue are available here.
The solution would be to upgrade the failing server to at least Docker 20.10.10.
我有同样的错误。
WAS的输出
:
解决了问题,也许可以在使用/ETC/Docker/seccompp.jsonn进行调整 - 我尝试并失败了。
通过将docker -compose.yml放入服务以及删除和重新创建容器的
情况下
I had had the same error.
Output of
was:
The issue was resolved by putting
in the docker-compose.yml for the service and removing and recreating the container
Maybe the same result could be achieved having /etc/docker/seccomp.jsonn tweaked - I tried and failed.