docker 突发故障, ` aufs: device or resource busy` , 无法启动容器

Question

开始

使用docker-compose 部署的 binux/pysider 集群 ,使用了没搞明白的 dockercloud/haproxy:latest,每个容器之间都有 link

发现问题

稳定运行了几天, 突然爬虫任务全部失败, 检测判断可能是容器之间通信异常

第一次尝试

尝试 移除所有容器

$ docker-cpmpose down
Removing pyspider_phantomjs_3 ... error
...
ERROR: for pyspider_processor_2  b'driver "aufs" failed to remove root filesystem for 50e4c88bf6f91c2697f302cc4d124114bfc50d74dc0ee246fdb86bf2aa158f6e: aufs: unmount error after retries: /var/lib/docker/aufs/mnt/288328abe7a32dcc04f8702fa5296d5417f412e14969b0c674760def626105a6: device or resource busy'
...

$ docker rm pyspider_phantomjs_2
Error response from daemon: driver "aufs" failed to remove root filesystem for 8613dcee7d20f927120a441102c10743615e7fcc0a35a42a2b4474cc70b7ec77: aufs: unmount error after retries: /var/lib/docker/aufs/mnt/adf6c5a7e9416f0b866daa8ea5bcf4ac15d6b8b0bf0181fd9d9c773441c5e343: device or resource busy

失败
判断可能是资源占用

尝试解除磁盘占用

$ sudo unmount /var/lib/docker/aufs/mnt/adf6c5a7e9416f0b866daa8ea5bcf4ac15d6b8b0bf0181fd9d9c773441c5e343
umount: /var/lib/docker/aufs/mnt/adf6c5a7e9416f0b866daa8ea5bcf4ac15d6b8b0bf0181fd9d9c773441c5e343: target is busy

# 失败
# 使用 umount -l 解除占用
docker-compose down 移除dead容器

又出现问题

使用docker-compose 启动新的 容器

$ $ docker-compose -f mysql.yml start
...
compose.cli.verbose_proxy.proxy_callable: docker start <- ('3c1a225d10d5385f024649d8252fd94ff74941b0ba9c504b65aab37de8afe248')
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
...

# 一直重复 最后异常

ERROR: for mysql  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

重启 docker

service docker restart
问题依旧

其他尝试

• 查看 docker log
  dockerd -D 开启调试

启动 docker run -it --rm mysql /bin/bash

**dockerd 输出**
DEBU[0033] Assigning addresses for endpoint upbeat_kirch's interface on network bridge
DEBU[0034] Programming external connectivity on endpoint upbeat_kirch (7224793b5e81325574724c072f6c8dc42f71b5cae09b74a013f57f5f27c03b70)
DEBU[0034] EnableService d2f705c568792c10ab06f3a054b9b94088a457f19ff5857234649bd01cbb6560 START
DEBU[0034] EnableService d2f705c568792c10ab06f3a054b9b94088a457f19ff5857234649bd01cbb6560 DONE
ERRO[0154] containerd: start container                   error="containerd: container did not start before the specified timeout" id=d2f705c568792c10ab06f3a054b9b94088a457f19ff5857234649bd01cbb6560
ERRO[0154] Create container failed with error: containerd: container did not start before the specified timeout
DEBU[0154] attach: stdout: end
DEBU[0154] attach: stderr: end
DEBU[0154] attach: stdin: end
DEBU[0154] Closing buffered stdin pipe
DEBU[0154] Revoking external connectivity on endpoint upbeat_kirch (7224793b5e81325574724c072f6c8dc42f71b5cae09b74a013f57f5f27c03b70)
DEBU[0154] DeleteConntrackEntries purged ipv4:0, ipv6:0
DEBU[0154] Releasing addresses for endpoint upbeat_kirch's interface on network bridge
DEBU[0154] ReleaseAddress(LocalDefault/172.17.0.0/16, 172.17.0.2)
DEBU[0154] Failed to unmount e6c100ba212bed7b2a01db39699242905e2a96833776dfcf33ded530e24caf99 aufs: device or resource busy
ERRO[0154] Error unmounting container d2f705c568792c10ab06f3a054b9b94088a457f19ff5857234649bd01cbb6560: device or resource busy
DEBU[0154] GetMountID id: d2f705c568792c10ab06f3a054b9b94088a457f19ff5857234649bd01cbb6560 -> mountID: e6c100ba212bed7b2a01db39699242905e2a96833776dfcf33ded530e24caf99
DEBU[0154] Cleaning up old mountid e6c100ba212bed7b2a01db39699242905e2a96833776dfcf33ded530e24caf99: start.
DEBU[0155] Cleaning up old mountid e6c100ba212bed7b2a01db39699242905e2a96833776dfcf33ded530e24caf99: done.
DEBU[0155] Removing volume reference: driver local, name d96daf284f326a2ca342f519dc3b4a5571a2620c8720ff4fba55d5aac8b8ad63
ERRO[0155] Handler for POST /v1.30/containers/d2f705c568792c10ab06f3a054b9b94088a457f19ff5857234649bd01cbb6560/start returned error: containerd: container did not start before the specified timeout

**docker run 输出**
$ docker run -it --rm mysql /bin/bash                                               │                                                                                                                                                              │64 bytes from 192.168.1.2
 docker: Error response from daemon: containerd: container did not start before the specified tim│                                                                                                                                                              │21: icmp_seq=867 ttl=62 t
eout.

• 网上有说是系统负载太大造成, 这个没有解决我遇到的问题

last

已经两天遇到同一个问题了,第一次是重启机器才解决,今天是主力服务器,不能重启,求指导是哪里的问题?

Answer 1

我把 docker 停掉之后,发现还留有 phantomjs 进程,怀疑是 phantomjs 逃出了 docker 的控制 (内存溢出?), 最后还是重启解决