记一次rbd 删除snap 镜像失败问题SnapshotUnprotectReq rbd: listing children failed uest

admin

查看有多少镜像：
' F  K) X3 |- C* r0 e( \, U[root@ceph1 ~]# rbd ls -p images -l |grep d25d8b59-dfff-4f2b-b63e-aa22dc533778
d25d8b59-dfff-4f2b-b63e-aa22dc533778       2 GiB          2           
; a$ X0 o& L- k* J: z( h, zd25d8b59-dfff-4f2b-b63e-aa22dc533778@snap  2 GiB          2 yes      
, S# ~+ x& L0 F; j0 _
解除父进程关系：
[root@ceph1 ~]# rbd children images/d25d8b59-dfff-4f2b-b63e-aa22dc533778@snap
rbd: listing children failed: 2021-12-29 15:45:01.087 7f6ad90a1b00 -1 librbd::api::Image: list_descendants: error looking up name for image id 9753a327e2e1b4 in pool volumes-hdd(2) No such file or directory

失败，再次尝试
[root@ceph1 ~]# rbd children images/d25d8b59-dfff-4f2b-b63e-aa22dc533778@snap
rbd: listing children failed: 2021-12-29 15:45:31.459 7fad7a5d1b00 -1 librbd::api::Image: list_descendants: error looking up name for image id 9753a327e2e1b4 in pool volumes-hdd(2) No such file or directory
1 ^. O/ M# O) q) Y5 u7 K5 X$ N3 c
[root@ceph1 ~]# rbd children images/d25d8b59-dfff-4f2b-b63e-aa22dc533778
4 ^% j$ H5 |- Q$ w2 k# h0 U7 jrbd: listing children failed: 2021-12-29 15:45:59.067 7fbb3d475b00 -1 librbd::api::Image: list_descendants: error looking up name for image id 9753a327e2e1b4 in pool volumes-hdd(2) No such file or directory
/ a" H2 o& {) L3 o6 _
到报错的pool池中查找，也无相关的记录：
[root@ceph1 ~]# rbd ls -p volumes-hdd |grep 9753a327e2e1b4
3 X7 H2 c1 q+ K* k% z  W[root@ceph1 ~]# rbd ls -p volumes-hdd |grep 9753a327e2e1b4
. H: K# ], L2 J) I, Z! R3 `[root@ceph1 ~]# rbd ls -p volumes-hdd |grep 9753a32
2 V: C. F+ m. p! m& z; k4 \. j[root@ceph1 ~]# rbd ls -p volumes-hdd
[root@ceph1 ~]# rbd ls -p volumes-hdd
[root@ceph1 ~]# rbd ls -p volumes-hdd
[root@ceph1 ~]#
6 F8 `- m1 P" ?+ M, P5 C[root@ceph1 ~]#
2 g. r) z; |6 G7 ]* e[root@ceph1 ~]#
[root@ceph1 ~]#

查看所有的镜像：
# B0 B) |6 L, C8 ]( a7 m' u1 u[root@ceph1 ~]# rbd ls -l images
5 w! A2 i% ~+ `2 r' @( N/ hNAME                                      SIZE   PARENT FMT PROT LOCK
, D+ g6 A" @# L6 S33652a4d-3ff4-4b40-bada-6f7514f23345      12 GiB          2           
33652a4d-3ff4-4b40-bada-6f7514f23345@snap 12 GiB          2 yes      
46e08623-2616-429e-9b0c-008a1749d1a5      12 GiB          2           
2 V: b  T/ L) W: u+ p$ h/ B46e08623-2616-429e-9b0c-008a1749d1a5@snap 12 GiB          2 yes      
3 G1 b& Z& ^% m- Qbda91b25-d146-4084-923f-834019eae8db       3 GiB          2           
3 A: ]7 W4 r; i8 Ybda91b25-d146-4084-923f-834019eae8db@snap  3 GiB          2 yes      
& {* x& j7 g# B- U* s3 z1 dc47afb15-a38a-46b6-8aa7-9dfba4243f5d       2 GiB          2           
c47afb15-a38a-46b6-8aa7-9dfba4243f5d@snap  2 GiB          2 yes      
d25d8b59-dfff-4f2b-b63e-aa22dc533778       2 GiB          2           
# r, r7 s9 M5 Md25d8b59-dfff-4f2b-b63e-aa22dc533778@snap  2 GiB          2 yes      
d6d6c4cd-00ad-44ce-994e-8a5fa8022139       3 GiB          2           
5 h' y) r( @1 w1 o; v  f$ }d6d6c4cd-00ad-44ce-994e-8a5fa8022139@snap  3 GiB          2 yes      
e6ec5576-03e3-425c-86bc-871acd6d13eb      12 GiB          2           
6 _) o  k. `) t3 X& Le6ec5576-03e3-425c-86bc-871acd6d13eb@snap 12 GiB          2 yes      
- i0 }& {% x" R+ he828dc6f-b4ad-4ee0-b866-36b2df342553       3 GiB          2           
e828dc6f-b4ad-4ee0-b866-36b2df342553@snap  3 GiB          2 yes      

删除下镜像：
. U9 h2 G. D2 u6 N# p[root@ceph1 ~]# rbd rm d25d8b59-dfff-4f2b-b63e-aa22dc533778 -p images
2021-12-29 15:51:50.536 7faff2c6ab00 -1 librbd::image::PreRemoveRequest: 0x558372b2d180 check_image_snaps: image has snapshots - not removing
9 Y: w7 J0 J( O1 X1 v3 n4 fRemoving image: 0% complete...failed.
# w2 o: U+ Z# J! C7 urbd: image has snapshots - these must be deleted with 'rbd snap purge' before the image can be removed.

删除失败，按提示执行rbd snap purge
9 O/ v. H3 _# R  X8 u6 Y
9 p# p/ E- N9 M- u2 ~1 H1 ]6 t& e[root@ceph1 ~]# rbd snap purge d25d8b59-dfff-4f2b-b63e-aa22dc533778 -p images
rbd: error removing snapshot(s) 'snap', which is protected - these must be unprotected with `rbd snap unprotect`.
6 a& Z4 C/ I* Q) E6 Y# tRemoving all snapshots: 0% complete...failed.

还是报错：
+ {: Q9 s9 @+ M, [) e) f5 s[root@ceph1 ~]# rbd snap unprotect d25d8b59-dfff-4f2b-b63e-aa22dc533778 -p images
rbd: snapshot name was not specified
" ]9 j& s* m0 w' t: G[root@ceph1 ~]# rbd snap unprotect d25d8b59-dfff-4f2b-b63e-aa22dc533778@snap -p images
/ c; ~- v9 G: q. J* V; t2021-12-29 15:52:30.984 7f4bdb7fe700 -1 librbd::SnapshotUnprotectRequest: cannot unprotect: at least 1 child(ren) [9753a327e2e1b4] in pool 'volumes-hdd'
! G' z3 f3 V0 f( _7 S2021-12-29 15:52:30.985 7f4bdb7fe700 -1 librbd::SnapshotUnprotectRequest: encountered error: (16) Device or resource busy
2021-12-29 15:52:30.985 7f4bdb7fe700 -1 librbd::SnapshotUnprotectRequest: 0x55f981533360 should_complete_error: ret_val=-16
. X, h' ~& @) R( ?3 i2021-12-29 15:52:30.987 7f4bdb7fe700 -1 librbd::SnapshotUnprotectRequest: 0x55f981533360 should_complete_error: ret_val=-16
rbd: unprotecting snap failed: (16) Device or resource busy
删除失败：
2 }  Y: t' [/ w8 O[root@ceph1 ~]# rbd snap rm d25d8b59-dfff-4f2b-b63e-aa22dc533778@snap -p images
Removing snap: 0% complete...failed.
4 k- M3 }+ c1 k8 E1 Hrbd: snapshot 'snap' is protected from removal.
2021-12-29 15:52:42.995 7f2908879b00 -1 librbd::Operations: snapshot is protected
[root@ceph1 ~]# rbd snap protect d25d8b59-dfff-4f2b-b63e-aa22dc533778@snap -p images
rbd: snap is already protected
[root@ceph1 ~]# rbd snap unprotect d25d8b59-dfff-4f2b-b63e-aa22dc533778@snap -p images
2021-12-29 15:53:02.360 7ff8f6ffd700 -1 librbd::SnapshotUnprotectRequest: cannot unprotect: at least 1 child(ren) [9753a327e2e1b4] in pool 'volumes-hdd'
$ }8 y/ C% F6 p3 D$ p) I, q) |4 l2021-12-29 15:53:02.361 7ff8f6ffd700 -1 librbd::SnapshotUnprotectRequest: encountered error: (16) Device or resource busy
2021-12-29 15:53:02.361 7ff8f6ffd700 -1 librbd::SnapshotUnprotectRequest: 0x5619a74f4230 should_complete_error: ret_val=-16
2021-12-29 15:53:02.363 7ff8f6ffd700 -1 librbd::SnapshotUnprotectRequest: 0x5619a74f4230 should_complete_error: ret_val=-16rbd: unprotecting snap failed:
(16) Device or resource busy
[root@ceph1 ~]# rbd snap unprotect d25d8b59-dfff-4f2b-b63e-aa22dc533778@snap -p images
2021-12-29 15:53:15.412 7fe7e3de8700 -1 librbd::SnapshotUnprotectRequest: cannot unprotect: at least 1 child(ren) [9753a327e2e1b4] in pool 'volumes-hdd'
2021-12-29 15:53:15.413 7fe7e3de8700 -1 librbd::SnapshotUnprotectRequest: encountered error: (16) Device or resource busy
$ h; m: X! }* d4 p1 i2021-12-29 15:53:15.413 7fe7e3de8700 -1 librbd::SnapshotUnprotectRequest: 0x557a25e9b330 should_complete_error: ret_val=-16
2021-12-29 15:53:15.414 7fe7e3de8700 -1 librbd::SnapshotUnprotectRequest: 0x557a25e9b330 should_complete_error: ret_val=-16
rbd: unprotecting snap failed: (16) Device or resource busy
% R8 F& k  m  A( s2 ^0 K- x, f& k9 ?
* C' G4 y. U: B7 N$ p1 I: t

- L( }& @& [! d( u

8 j7 U, j+ d! Y7 x7 K! X( K; U实在无法删除，只好通过更新数据库信息来完成删除的任务：
MariaDB [glance]> select id,name,deleted,protected,size  from images  ;
+--------------------------------------+----------------------+---------+-----------+-------------+
4 G, a* o, _- H+ V| id                                   | name                 | deleted | protected | size        |
+--------------------------------------+----------------------+---------+-----------+-------------+

9 g+ ~' |4 Q5 I$ {* C| c47afb15-a38a-46b6-8aa7-9dfba4243f5d | Centos7.9-origin-ssd |       0 |         0 |  2147483648 |
4 f9 R5 Z$ e/ b| d25d8b59-dfff-4f2b-b63e-aa22dc533778 | centos7.9-origin-hdd |       1 |         0 |  2147483648 |
|
+--------------------------------------+----------------------+---------+-----------+-------------+
; i7 }' d) |$ z5 s29 rows in set (0.00 sec)
% P4 n4 x! j; f- w
MariaDB [glance]> update glance.images set deleted='1' where id='c47afb15-a38a-46b6-8aa7-9dfba4243f5d';
; `7 q5 I! d- ^( M& }0 Z* y- b2 ^Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0
* T7 i- X' r$ M. q$ D+ t0 P
9 g2 k3 f  g* `+ H. y. G$ J