openstack上创建卷，底层ceph 删除卷，然后rbd导入卷，创建新的虚机即可使用

admin

查看导入的ceph镜像的大小：
[root@controller1 src]# rbd info hdd-8T-volumes/volume-7339c495-25dd-44a3-b0cb-3b00cd59e070
% k: R" E2 y: [# D8 Erbd image 'volume-7339c495-25dd-44a3-b0cb-3b00cd59e070':
        size 2.8 GiB in 717 objects
        order 22 (4 MiB objects)
        snapshot_count: 0
0 v& S" |8 A! X; _- B! E        id: 9eb6a0f364b72b
) k' p$ c- a& {/ W# o) K" C% }        block_name_prefix: rbd_data.9eb6a0f364b72b
9 [0 o; n. c" E        format: 2
0 j5 C! y3 N/ W% s; _        features: layering
        op_features:
        flags:
* E9 g1 `. O' Z        create_timestamp: Fri Jun 17 17:06:55 2022
0 R) W4 V7 z5 n, P        access_timestamp: Fri Jun 17 17:06:55 2022
  l3 U2 j* R- H        modify_timestamp: Fri Jun 17 17:06:55 2022
9 ~! z8 _" O5 w# @9 ]3 Z8 G& y

admin

3、 快照导出的文件导入为新的RBD image
8 o7 P6 q' A8 c$ [; z快照的恢复过程使用的是刚刚上面提到的备份到本地的那些文件. 现在假如想恢复到v3那个快照的时间点，那么可以用两个方法 方法1：直接基于v3的时间点的快照做恢复 方法2：直接基于v1的时间点的数据，和后面的增量的v1-v3数据(要按顺序导入)
3.1增量导入：rbd import-diff
#方法1：直接基于v3的时间点的快照做恢复
1 w- H7 R0 C# i7 R#首先随便创建一个image,名称大小都不限制，因为后面恢复的时候会覆盖掉大小的信息
* ?# d# m/ A# Q3 [6 n& p[root@rook-ceph-tools-6f5694c655-hzpj5 /]# rbd create replicapool/resume-v1 --size 1   
[root@rook-ceph-tools-6f5694c655-hzpj5 /]# rbd import-diff csz-busybox-v3 replicapool/resume-v1                                          Importing image diff: 100% complete...done.
[root@rook-ceph-tools-6f5694c655-hzpj5 /]#
3 Z! a( L$ t8 L" u8 O& i#方法2： 基于v1+ v1_v3数据恢复
" {3 J; s" Q- Q5 d- I" v0 S#首先随便创建一个image,名称大小都不限制，因为后面恢复的时候会覆盖掉大小的信息
5 X* J. l4 q. |0 Y  X3 @& d1 B3 [[root@rook-ceph-tools-6f5694c655-hzpj5 /]# rbd create replicapool/resume-v3 --size 1
7 }' I/ C3 J# e- {, y# D[root@rook-ceph-tools-6f5694c655-hzpj5 /]# rbd import-diff csz-busybox-v1 replicapool/resume-v3
1 R( o) o- Q9 d6 E4 k6 \Importing image diff: 100% complete...done.
[root@rook-ceph-tools-6f5694c655-hzpj5 /]# rbd import-diff csz-busybox-v1-v3 replicapool/resume-v3
1234567891011
​ 实际项目当中就是，定期做快照，然后导出某个时间点快照的数据，然后导出增量的快照的数据，就可以了
0 Y' I7 C; V* _  a3.2全量导入：rbd import
0 m' P" |9 n& r+ E- ^7 b" t( ?注：不用提前创建目标image，否则rbd import会报错
- \- U1 y0 V) H+ @4 _[root@rook-ceph-tools-6f5694c655-hzpj5 /]# rbd create replicapool/resume-full --size 1
[root@rook-ceph-tools-6f5694c655-hzpj5 /]# rbd import-diff csz-busybox-full replicapool/resume-full
; N/ x/ ^& l1 G5 w6 \rbd: invalid or unexpected diff banner
$ H" k# R- i$ @$ }9 w9 a' ~! s% G* Qrbd: import-diff failed: (22) Invalid argument
[root@rook-ceph-tools-6f5694c655-hzpj5 /]#
[root@rook-ceph-tools-6f5694c655-hzpj5 /]#
[root@rook-ceph-tools-6f5694c655-hzpj5 /]# rbd import csz-busybox-full replicapool/resume-full
rbd: image creation failed
Importing image: 0% complete...failed.
/ Y0 c& X; E2 o% P0 Orbd: import failed: 2021-01-29T02:11:56.790+0000 7fdfdcb4b500 -1 librbd: rbd image resume-full already exists
! i1 t7 J, A" W: ]2 d(17) File exists
[root@rook-ceph-tools-6f5694c655-hzpj5 /]# rbd import csz-busybox-full replicapool/resume-full1
Importing image: 100% complete...done.
8 j7 N- T. C$ Z+ y6 w7 B[root@rook-ceph-tools-6f5694c655-hzpj5 /]#
. v" R+ [( D8 b3 k" Z! y1234567891011121314
四、如何利用这个 实际项目当中就是，定期做快照，然后导出某个时间点快照的数据，然后导出增量的快照的数据，就可以了，例如： 今天对所有的rbd的image做一个基础快照，然后导出这个快照的数据，然后从今天开始，每天晚上做一个快照，然后导出快照时间点之间的数据，这样每天导出来的就是一个增量的数据了，在做恢复的时候，就从第一个快照导入，然后按顺序导入增量的快照即可，也可以定期做一个快照，导出完整的快照数据，以防中间的增量快照漏了，然后就是要注意可以定期清理快照，如果是做备份的模式，在导入了快照数据后，也可以清理一些本地的数据，本地数据做异地机房复制的时候也可以做一下数据的压缩，来减少数据量的传输
客户端验证：将RBD image恢复到pod
) k4 Z  g1 @/ f: p/ p) n- t思路：将新的image name重命名rename成原来的image name
删除旧的image
, X; k+ G% ^: \2 y! [% q" p# Step1：停掉pod对image的使用
修改pod的yaml文件，将spec: replicas: 副本数改为0
  d1 Q- M: v2 Q" k" [* o. l# Step2：清空旧image的snapshots
% G; {; U3 M6 R. M/ `# u[root@rook-ceph-tools-6f5694c655-hzpj5 /]# rbd snap purge replicapool/csi-vol-387b2608-61f7-11eb-b9c5-0e28ddb26a3c
Removing all snapshots: 100% complete...done.
$ d$ [% v* p, T0 I; X# Step3：删除旧的image
[root@rook-ceph-tools-6f5694c655-hzpj5 /]# rbd rm replicapool/csi-vol-387b2608-61f7-11eb-b9c5-0e28ddb26a3c
  o  f; T  d+ t/ }/ J% b  h; K/ DRemoving image: 100% complete...done.
$ W0 w* s* @, B/ C8 z12345678
/ A# R# r: N- E/ i/ Z/ l# x将new image name重命名为old image name
9 n* v  ?! l3 G" H0 X* [[root@rook-ceph-tools-6f5694c655-hzpj5 /]# rbd rename replicapool/test-v3 replicapool/csi-vol-387b2608-61f7-11eb-b9c5-0e28ddb26a3c
  g# |7 L8 D  e/ I* k( z: f4 w1
启动pod
. C5 c7 |# D6 J& p8 N9 b修改pod的yaml文件，将spec: replicas: 副本数由0改回原值
1
FAQ:
1 u3 S( I7 l! n3 r6 crbd 删除image时，报错“rbd: image has snapshots - these must be deleted with ‘rbd snap purge’ before the image can be removed”
& ?) c. h# k% c% P3 f( I9 d[root@rook-ceph-tools-6f5694c655-hzpj5 /]# rbd rm replicapool/csi-vol-387b2608-61f7-11eb-b9c5-0e28ddb26a3c
Removing image: 0% complete...failed.
+ B) i4 c; N& Y0 X: @rbd: image has snapshots - these must be deleted with 'rbd snap purge' before the image can be removed.
+ M' W+ W5 Z4 z, n5 n4 w5 X- b123
+ M7 [: \. X( Y# ?. \3 H1 K  \' a  u& `解决方法：执行’rbd snap purge’删掉image相关snapshots
( T! N1 v: h  m& N  C' {; prbd 删除image时，报错“rbd: error: image still has watchers”
4 y: L- M! @" B" {[root@rook-ceph-tools-6f5694c655-hzpj5 /]# rbd rm replicapool/csi-vol-6ee1b63a-6060-11eb-b9c5-0e28ddb26a3c
2021-01-29T06:22:09.698+0000 7fe4677fe700 -1 librbd::image::PreRemoveRequest: 0x55a645137470 check_image_watchers: image has watchers - not removing
Removing image: 0% complete...failed.
7 X0 V& T4 `* Z3 r: }; lrbd: error: image still has watchers
) G/ s# S7 |& K' ]( l# j& mThis means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout.
$ n2 \# i9 W4 z5 T12345
0 R) u. y$ |3 k! y8 `9 ]解决方法：删除使用image的pod（修改pod的yaml文件，将spec: replicas: 副本数改为0）
0 l* X0 p; e- y/ a0 n  _7 h7 V

admin

[root@controller1 src]# rbd resize --size 50G hdd-8T-volumes/volume-7339c495-25dd-44a3-b0cb-3b00cd59e070
" X. `* w7 b% b8 O# ?Resizing image: 100% complete...done.
[root@controller1 src]# rbd info hdd-8T-volumes/volume-7339c495-25dd-44a3-b0cb-3b00cd59e070
' Z) A9 a3 w$ W( Xrbd image 'volume-7339c495-25dd-44a3-b0cb-3b00cd59e070':
        size 50 GiB in 12800 objects
        order 22 (4 MiB objects)
        snapshot_count: 0
- p7 l2 v; f, s0 d, i        id: 9eb6a0f364b72b
        block_name_prefix: rbd_data.9eb6a0f364b72b
        format: 2
        features: layering
        op_features:
        flags:
4 s1 X3 Z# j6 t4 T7 Y" \        create_timestamp: Fri Jun 17 17:06:55 2022
, G0 [% Y8 m$ f        access_timestamp: Fri Jun 17 17:06:55 2022
7 v$ x( L" q" b6 A( x        modify_timestamp: Fri Jun 17 17:06:55 2022

admin

在openstack上扩展卷后，底层执行命令：
3 F' t* |$ R  B8 r[root@controller1 src]# rbd info hdd-8T-volumes/volume-7339c495-25dd-44a3-b0cb-3b00cd59e070
9 W2 Y4 A2 `& D2 N. C+ jrbd image 'volume-7339c495-25dd-44a3-b0cb-3b00cd59e070':
! G: M4 ?) Y; W9 |7 S/ x( ~* G        size 50 GiB in 12800 objects
' s% O+ O- U  F, S3 Z2 U9 D/ E4 |        order 22 (4 MiB objects)
        snapshot_count: 0
        id: 9eb6a0f364b72b
        block_name_prefix: rbd_data.9eb6a0f364b72b
        format: 2
: w6 {2 n/ f% ^% w% Q! s# `        features: layering
        op_features:
        flags:
# j1 M. e) a, }. X( \7 T        create_timestamp: Fri Jun 17 17:06:55 2022
        access_timestamp: Fri Jun 17 17:06:55 2022
        modify_timestamp: Fri Jun 17 17:06:55 2022
, V$ T' r# X/ V3 j[root@controller1 src]# rbd info hdd-8T-volumes/volume-7339c495-25dd-44a3-b0cb-3b00cd59e070
rbd image 'volume-7339c495-25dd-44a3-b0cb-3b00cd59e070':
        size 51 GiB in 13056 objects
        order 22 (4 MiB objects)
        snapshot_count: 0
8 G8 |1 d* u* @        id: 9eb6a0f364b72b
; d  T$ b4 ^* X3 z- p        block_name_prefix: rbd_data.9eb6a0f364b72b
9 ?# k1 g5 T9 l; Y3 R( \0 A" r        format: 2
0 s1 |' W* B$ C: A8 M$ l/ y$ y5 W        features: layering
8 G8 w5 o' z1 m        op_features:
' [3 d* ]4 K! e8 M! h# [/ s" U        flags:
- K5 C! d; i$ ~; M6 [        create_timestamp: Fri Jun 17 17:06:55 2022
        access_timestamp: Fri Jun 17 17:06:55 2022
* Q  k0 _2 d6 d) T! o& {/ V        modify_timestamp: Fri Jun 17 17:06:55 2022
[root@controller1 src]# rbd info hdd-8T-volumes/volume-7339c495-25dd-44a3-b0cb-3b00cd59e070
rbd image 'volume-7339c495-25dd-44a3-b0cb-3b00cd59e070':
$ \. P( b0 {' y" I, o& Z        size 51 GiB in 13056 objects
        order 22 (4 MiB objects)
        snapshot_count: 0
3 Z: v' _, S! z7 y        id: 9eb6a0f364b72b
7 R" q' L  g, i! o" [        block_name_prefix: rbd_data.9eb6a0f364b72b
        format: 2
" U8 N4 C# y, E1 d$ ?, P0 r        features: layering
        op_features:
        flags:
        create_timestamp: Fri Jun 17 17:06:55 2022
        access_timestamp: Fri Jun 17 17:06:55 2022
$ i. I" h; ^0 L/ A! q) B        modify_timestamp: Fri Jun 17 17:06:55 2022
! |2 E: D( ~* f* f) J# I* H* m) M% L[root@controller1 src]#

admin

一套完整的流程搞定。openstack上不创建镜像，即可完成镜像使用。

admin

[root@host09 images]# rbd -p volumes-hdd ls |grep 98e50ac1-3b42-4393-b53f-6c6832fec4c4
. J/ N, Y; Z7 \* W; W+ T3 c; C: nvolume-98e50ac1-3b42-4393-b53f-6c6832fec4c4
[root@host09 images]# rbd rename volumes-hdd/volume-98e50ac1-3b42-4393-b53f-6c6832fec4c4 volumes-hdd/volume-98e50ac1-3b42-4393-b53f-6c6832fec4c4-bak
8 G0 ]5 M9 l/ Z5 v[root@host09 images]# rbd -p volumes-hdd ls |grep 98e50ac1-3b42-4393-b53f-6c6832fec4c4
4 D/ c/ O0 L$ _' m; g, Xvolume-98e50ac1-3b42-4393-b53f-6c6832fec4c4-bak
[root@host09 images]# rbd import volume-98e50ac1-3b42-4393-b53f-6c6832fec4c4.raw volumes-hdd/volume-98e50ac1-3b42-4393-b53f-6c6832fec4c4
* _+ m5 B2 C5 U' }5 @2 VImporting image: 100% complete...done.
[root@host09 images]#
( _  G% O: h+ i. \1 u