Question

---

有时候删除 rbd image 会提示当前 rbd image 正在使用中，无法删除：

rbd rm foo
2016-11-09 20:16:14.018332 7f81877a07c0 -1 librbd: image has watchers - not removing
Removing image: 0% complete...failed.
rbd: error: image still has watchers
This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout.

所以希望能查看到底是谁在使用 rbd image。

对于 rbd image 的使用，目前主要有两种形式：内核模块 map 后再 mount ；通过 libvirt 等供给虚拟机使用。都是利用 rados listwatchers 去查看，只是两种形式下需要读取的文件不一样。

内核模块 map

查看方法如下：

# 查看 rbd pool 中的 image
root@ceph1:~# rbd ls
foo
ltest
test
# 查看 foo 的使用者
root@ceph1:~# rados -p rbd listwatchers foo.rbd
watcher=10.202.0.82:0/1760414582 client.1332795 cookie=1
# 去对应主机上检查
root@ceph2:~#rbd showmapped |grep foo
0  rbd  foo   -    /dev/rbd0 

虚拟机通过 libvirt 使用 rbd image

查看方法如下：

# 查看该虚拟机卷的信息
root@ceph1:~# rbd info volumes/volume-ee0c4077-a607-4bc9-a8cf-e893837361f3
rbd image 'volume-ee0c4077-a607-4bc9-a8cf-e893837361f3':
        size 1024 MB in 256 objects
        order 22 (4096 kB objects)
        block_name_prefix: rbd_data.13c3277850f538
        format: 2
        features: layering, striping
        flags: 
        parent: images/3601459f-060b-460f-b73b-db74237d922e@snap
        overlap: 40162 kB
        stripe unit: 4096 kB
        stripe count: 1
# 查看该 image 的使用者
root@ceph1:~# rados -p volumes listwatchers rbd_header.13c3277850f538
watcher=10.202.0.21:0/1043073 client.1298745 cookie=140256077850496

我们登录控制节点，查看对应的 cinder 卷：

root@controller:~# cinder list | grep ee0c4077-a607-4bc9-a8cf-e893837361f3
| ee0c4077-a607-4bc9-a8cf-e893837361f3 | in-use |      |  1   |      -      |   true   | 96ee1aad-af27-4c9d-968f-291dbb2766a1 |

该卷挂载在 ID 为 96ee1aad-af27-4c9d-968f-291dbb2766a1 的虚拟机上。通过 nova show 命令验证该虚拟机是否在物理机 10.202.0.21 ( computer21 )上：

root@controller:~# nova show 96ee1aad-af27-4c9d-968f-291dbb2766a1
+--------------------------------------+---------------------------------------------------------------------------------+
| Property                             | Value                                                                           |
+--------------------------------------+---------------------------------------------------------------------------------+
| OS-DCF:diskConfig                    | AUTO                                                                            |
| OS-EXT-AZ:availability_zone          | az_ip                                                                           |
| OS-EXT-SRV-ATTR:host                 | computer21                                                                      |
| OS-EXT-SRV-ATTR:hostname             | byp-volume                                                                      |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | computer21                                                                      |
| OS-EXT-SRV-ATTR:instance_name        | instance-00000989                                                               |
| OS-EXT-SRV-ATTR:kernel_id            |                                                                                 |
| OS-EXT-SRV-ATTR:launch_index         | 0                                                                               |
| OS-EXT-SRV-ATTR:ramdisk_id           |                                                                                 |
| OS-EXT-SRV-ATTR:reservation_id       | r-hwiyx15c                                                                      |
| OS-EXT-SRV-ATTR:root_device_name     | /dev/vda                                                                        |
| OS-EXT-SRV-ATTR:user_data            | -                                                                               |
| OS-EXT-STS:power_state               | 1                                                                               |
| OS-EXT-STS:task_state                | -                                                                               |
| OS-EXT-STS:vm_state                  | active                                                                          |
| OS-SRV-USG:launched_at               | 2016-11-09T08:19:41.000000                                                      |
| OS-SRV-USG:terminated_at             | -                                                                               |
| accessIPv4                           |                                                                                 |
| accessIPv6                           |                                                                                 |
| blue-net network                     | 192.168.50.27                                                                   |
| config_drive                         |                                                                                 |
| created                              | 2016-11-09T07:53:59Z                                                            |
| description                          | byp-volume                                                                      |
| flavor                               | m1.small (2)                                                                    |
| hostId                               | 5104ee1a0538048d6ef80b14563a0cbac461f86523c5c81f5d18069e                        |
| host_status                          | UP                                                                              |
| id                                   | 96ee1aad-af27-4c9d-968f-291dbb2766a1                                            |
| image                                | Attempt to boot from volume - no image supplied                                 |
| key_name                             | octavia_ssh_key                                                                 |
| locked                               | False                                                                           |
| metadata                             | {}                                                                              |
| name                                 | byp-volume                                                                      |
| os-extended-volumes:volumes_attached | [{"id": "ee0c4077-a607-4bc9-a8cf-e893837361f3", "delete_on_termination": true}] |
| progress                             | 0                                                                               |
| security_groups                      | default                                                                         |
| status                               | ACTIVE                                                                          |
| tenant_id                            | f21a9c86d7114bf99c711f4874d80474                                                |
| updated                              | 2016-11-09T08:19:41Z                                                            |
| user_id                              | 142d8663efce464c89811c63e45bd82e                                                |
| zq-test network                      | 192.168.6.6                                                                     |
+--------------------------------------+---------------------------------------------------------------------------------+