cinder 对接多个 ceph 存储

admin

环境说明当前 openstack环境正常使用由于后端 ceph 存储容量已经超过 85%不想直接进行扩容, 因为会有大量的数据迁移新创建一个独立的ceph 集群, 并计划用于 openstack 现有环境成为一个新的 ceph后端旧的 ceph 集群称为 ceph-A,  使用中的 pool 为 volumes新的 ceph 集群称为 ceph-B,  使用中的 pool 为 new_volumes目标在 openstack 中,  同时连接到两个不同的 ceph backendcinder server 配置1. ceph 连接配置2. cinder 配置ceph 连接配置

1.同时把两个 ceph 集群中的配置复制到 cinder 服务器 /etc/ceph 目录下, 定义成不同命名

[root@hh-yun-db-129041 ceph]# tree `pwd`/etc/ceph├── ceph.client.admin-develop.keyring      <- ceph-B 集群中的 admin key├── ceph.client.admin-volumes.keyring      <- ceph-A 集群中的 admin key├── ceph.client.developcinder.keyring      <- ceph-B 集群中的用户 developcinder key├── ceph.client.cinder.keyring             <- ceph-A 集群中的 cinder key├── ceph.client.mon-develop.keyring        <- ceph-B 集群中的 mon key├── ceph.client.mon-volumes.keyring        <- ceph-A 集群中的 mon key├── ceph-develop.conf                      <- ceph-B 集群配置文件(包含了 mon 地址等集群信息)└── ceph-volumes.conf                      <- ceph-B 集群配置文件(包含了 mon 地址等集群信息)

这里需要注意, clinet.client.(username).keyring 必须要与连接 ceph 的合法用户命名一致, 否则 cinder server 端, 无法正确获得权限

2.命令行下, 测试连接不同的 ceph 后端测试

ceph-A 连接测试

[root@hh-yun-db-129041 ceph]# ceph -c ceph-volumes.conf -k ceph.client.admin-volumes.keyring -s cluster xxx-xxx-xxxx-xxxx-xxxx

8 ~  v: o) g* j3 @1 e
; G  d4 H+ A9 T% Q, J9 `, r       health HEALTH_OK

3 S- G6 d+ c- u6 |1 ]9 X
       monmap e3: 5 mons at {hh-yun-ceph-cinder015-128055=240.30.128.55:6789/0,hh-yun-ceph-cinder017-128057=240.30.128.57:6789/0,hh-yun-ceph-cinder024-128074=240.30.128.74:6789/0,hh-yun-ceph-cinder025-128075=240.30.128.75:6789/0,hh-yun-ceph-cinder026-128076=240.30.128.76:6789/0}, election epoch 452, quorum 0,1,2,3,4 hh-yun-ceph-cinder015-128055,hh-yun-ceph-cinder017-128057,hh-yun-ceph-cinder024-128074,hh-yun-ceph-cinder025-128075,hh-yun-ceph-cinder026-128076

6 z- e* W7 Q" A1 `$ _, Q3 F% ]
! f2 S& |: b/ r# E; R. c      osdmap e170088: 226 osds: 226 up, 226 in

! q% y) [3 r9 L  U' T/ n8 X) |; c6 N
     pgmap v50751302: 20544 pgs, 2 pools, 157 TB data, 40687 kobjects 474 TB used, 376 TB / 850 TB avail 20537 active+clean 7 active+clean+scrubbing+deep client io 19972 kB/s rd, 73591 kB/s wr, 3250 op/s
+ q. d1 k( ^$ d& Q5 |

ceph-B 连接测试

[root@hh-yun-db-129041 ceph]# ceph -c ceph-develop.conf -k ceph.client.admin-develop.keyring -s cluster 4bf07d3e-a289-456d-9bd9-5a89832b413b
  
* c6 `. C7 `: x  T  o1 v    health HEALTH_OK monmap e1: 5 mons at {240.30.128.214=240.30.128.214:6789/0,240.30.128.215=240.30.128.215:6789/0,240.30.128.39=240.30.128.39:6789/0,240.30.128.40=240.30.128.40:6789/0,240.30.128.58=240.30.128.58:6789/0} election epoch 6, quorum 0,1,2,3,4 240.30.128.39,240.30.128.40,240.30.128.58,240.30.128.214,240.30.128.215


   osdmap e559: 264 osds: 264 up, 264 in flags sortbitwise


   pgmap v116751: 12400 pgs, 9 pools, 1636 bytes data, 171 objects 25091 MB used, 1440 TB / 1440 TB avail 12400 active+clean
cinder 配置

对 cinder 服务端进行配置

/etc/cinder/cinder.conf

! v9 q) i* B* ^7 C- @% x

enabled_backends=CEPH_SATA,CEPH_DEVELOP...
[CEPH_SATA]
glance_api_version=2

volume_backend_name=ceph_sata

8 t8 _* e7 M. d, q; j( M) \rbd_ceph_conf=/etc/ceph/ceph-volumes.conf

rbd_user=cinder
7 \# Z0 Y/ N2 h- [5 f3 u! L
4 R% @6 Q* L- K+ F$ ]# e* ?; xrbd_flatten_volume_from_snapshot=False

0 C8 W3 g4 f- arados_connect_timeout=-1

rbd_max_clone_depth=5
) u# u# E. q$ {' L
volume_driver=cinder.volume.drivers.rbd.RBDDriver

rbd_store_chunk_size=4

rbd_secret_uuid=dc4f91c1-8792-4948-b68f-2fcea75f53b

8 d1 s5 `0 _* n) Irbd_pool=volumeshost=cinder.vclound.com
6 U( l; G+ q3 O6 `* }5 |8 A
[CEPH-new_volumes]

glance_api_version=2
% O/ E- d. q* G( T# c1 q
* n6 @( Z2 S( ^5 h( k1 ~. Pvolume_backend_name=ceph-new_volumes
9 y, y1 t8 ]1 i2 N8 {  g
9 j& N  b5 }( N8 q; |rbd_ceph_conf=/etc/ceph/ceph-new_volumes.conf

rbd_user=cinder
6 y) E( W8 e, i, `  F
rbd_flatten_volume_from_snapshot=False
7 W# k6 b3 U& I  y
rados_connect_timeout=-1
2 N( m2 ~* P5 I* G. W
rbd_max_clone_depth=5

3 n, y7 F- a* w; B; mvolume_driver=cinder.volume.drivers.rbd.RBDDriver

rbd_store_chunk_size=4
4 I: i) v' E2 [, [& N
rbd_secret_uuid=4bf07d3e-a289-456d-9bd9-5a89832b413
' V, X" ]7 E3 D. |
rbd_pool=new_volumes
' n6 R2 S( T2 ~: f, @% g
/ N$ T6 S2 y6 Y7 u# M! i8 e3 G/ W; Uhost=cinder.vclound.com
1 S$ n' f  J/ e: O3 o2 f
( j; y* U3 Q  P7 }6 N2 l4 [

admin

在ceph监视器上执行
* M/ g) S8 N& ^( I! e1 }* mCINDER_PASSWD='cinder1234!'
2 m7 D) ?" I" r( zcontrollerHost='controller'
RABBIT_PASSWD='0penstackRMQ'

7 _8 c2 h3 r! y$ u1.创建pool池
为cinder-volume服务创建pool池(因为我只有一个OSD节点，所以要将副本数设置为1)
2 g- }9 T- O/ X6 aceph osd pool create cinder-volumes 32
ceph osd pool set cinder-volumes size 1
ceph osd pool application enable  cinder-volumes rbd
7 J9 I& u% }" \' d/ |7 k& ]ceph osd lspools

2.查看pool池的使用情况
ceph df

9 ^( \( a9 i5 f( t9 A0 V4 S2 Q* W7 q3.创建账号
ceph auth get-or-create client.cinder-volumes mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=cinder-volumes, allow rwx pool=glance-images' -o /etc/ceph/ceph.client.cinder-volumes.keyring
, v) q- H& o& y+ Y2 D! e( n" ?8 x#查看
ceph auth ls | grep -EA3 'client.(cinder-volumes)'

' g0 @! ~1 |* D7 |* n7 `/ u; r4.修改ceph.conf配置文件并同步到所有的监视器节点（这步一定要操作）
' F3 y9 t8 x2 ssu - cephd
cd ~/ceph-cluster/
cat <<EOF>> ceph.conf
[client.cinder-volumes]
6 j8 V9 S8 K5 Kkeyring = /etc/ceph/ceph.client.cinder-volumes.keyring
" d) d8 G! Q5 I0 D9 s0 R( y# K8 mEOF
: E7 T2 c# N' S* O2 F2 V1 Y7 {ceph-deploy --overwrite-conf admin ceph-mon01
exit

* _! I# Q: v7 w9 Q+ l7 d; R5.安装cinder-volume组件和ceph客户端(如果ceph监视器是在控制节点上不需要执行这一步)
yum -y install openstack-cinder python-keystone ceph-common
. b2 \( b* m$ P3 I
+ L5 k* j8 J( I9 |; m' K" S6.使用uuidgen生成一个uuid(确保cinder和libvirt中的UUID一致)
) B; d% I4 H, Kuuidgen
运行uuidgen命令可以得到下面的UUID值：

086037e4-ad59-4c61-82c9-86edc31b0bc0
6 A+ m  a+ f3 i, l7.配置cinder-volume服务与cinder-api服务进行交互
openstack-config --set  /etc/cinder/cinder.conf DEFAULT transport_url rabbit://openstack:${RABBIT_PASSWD}@${controllerHost}:5672
+ _1 ~7 K0 t, W0 h& Q  m- Qopenstack-config --set /etc/cinder/cinder.conf cache backend  oslo_cache.memcache_pool
( C7 Y9 v8 J' e! nopenstack-config --set /etc/cinder/cinder.conf cache enabled  true
' f- b! H  O; ?. K6 p; uopenstack-config --set /etc/cinder/cinder.conf cache memcache_servers  ${controllerHost}:11211
) V, E' o3 D+ A$ F, B' u4 d' }openstack-config --set  /etc/cinder/cinder.conf DEFAULT auth_strategy  keystone
2 ]' V% ~% ?+ }) Fopenstack-config --set  /etc/cinder/cinder.conf keystone_authtoken  auth_uri  http://${controllerHost}:5000
openstack-config --set  /etc/cinder/cinder.conf keystone_authtoken  auth_url  http://${controllerHost}:5000
openstack-config --set  /etc/cinder/cinder.conf keystone_authtoken  auth_type password
" ?* U, I+ B) O; ^7 Eopenstack-config --set  /etc/cinder/cinder.conf keystone_authtoken  project_domain_id  default
openstack-config --set  /etc/cinder/cinder.conf keystone_authtoken  user_domain_id  default
0 ?3 n" z' |& |, c- Nopenstack-config --set  /etc/cinder/cinder.conf keystone_authtoken  project_name  service
openstack-config --set  /etc/cinder/cinder.conf keystone_authtoken  username  cinder
openstack-config --set  /etc/cinder/cinder.conf keystone_authtoken  password  ${CINDER_PASSWD}
openstack-config --set  /etc/cinder/cinder.conf oslo_concurrency lock_path  /var/lib/cinder/tmp
+ @3 G' W/ [; h& ~
& s) z, c1 m3 j: J8.配置cinder-volume服务使用的后端存储为ceph
' W( F& b% [- A& q- B# H0 v& n- P) Q- }openstack-config --set /etc/cinder/cinder.conf  DEFAULT  enabled_backends  ceph
* m" ~4 `5 S, M% J1 g' i4 x
9.配置cinder-volume服务驱动ceph
) v2 l9 F2 n+ F! mopenstack-config --set /etc/cinder/cinder.conf  ceph volume_driver  cinder.volume.drivers.rbd.RBDDriver
openstack-config --set /etc/cinder/cinder.conf  ceph rbd_pool  cinder-volumes
openstack-config --set /etc/cinder/cinder.conf  ceph rbd_user cinder-volumes
openstack-config --set /etc/cinder/cinder.conf  ceph rbd_ceph_conf  /etc/ceph/ceph.conf
$ h/ ~, s/ N- m" F, G4 G0 Popenstack-config --set /etc/cinder/cinder.conf  ceph rbd_flatten_volume_from_snapshot  false
openstack-config --set /etc/cinder/cinder.conf  ceph bd_max_clone_depth  5
# s0 p9 |' U4 A. J# @openstack-config --set /etc/cinder/cinder.conf  ceph rbd_store_chunk_size  4
openstack-config --set /etc/cinder/cinder.conf  ceph rados_connect_timeout  -1
openstack-config --set /etc/cinder/cinder.conf  ceph glance_api_version 2
$ B$ H9 F2 C" {8 ]% qopenstack-config --set /etc/cinder/cinder.conf  ceph rbd_secret_uuid  086037e4-ad59-4c61-82c9-86edc31b0bc0

  m+ e* n: V5 z  S- @/ O" i1 D' H# e6 ?10.启动cinder-volume服务
systemctl enable openstack-cinder-volume.service
systemctl start openstack-cinder-volume.service
3 y7 P% S6 P3 W* N2 c+ B3 usystemctl status openstack-cinder-volume.service
4 k% Q7 f7 H" h: e/ e8 \
在需要挂载ceph卷的所有计算节点上执行
1.创建secret文件(UUID需要与cinder服务中一致)
cat << EOF > ~/secret.xml
8 T" Z0 c3 b7 r<secret ephemeral='no' private='no'>
     <uuid>086037e4-ad59-4c61-82c9-86edc31b0bc0</uuid>
     <usage type='ceph'>
         <name>client.cinder-volumes secret</name>
     </usage>
3 ]* F/ W) n9 w$ G/ D/ |</secret>
1 h+ r& e% ~$ K# P2 h; I4 l5 iEOF
7 o9 E& }. Q0 G* u; X4 e8 O2 c
( A- A+ J! a9 g, Y2.从ceph监视器上获取cinder-volumes账户的密钥环
' j7 r5 V2 S6 q* _+ f2 ^# |  Dceph auth get-key client.cinder-volumes
得到如下的结果：
AQCxfDFdgp2qKRAAUY/vep29N39Qv7xWKYqMUw==

- x! ~; u. Y# r# c3.在libvirt中注册UUID
virsh secret-define --file ~/secret.xml
+ R( I: F) n! S$ b3 G' k
( K: m2 h3 ~% \% Z1 c4.在libvirt中添加UUID和cinder-volumes密钥环
virsh secret-set-value --secret 086037e4-ad59-4c61-82c9-86edc31b0bc0 --base64 AQCxfDFdgp2qKRAAUY/vep29N39Qv7xWKYqMUw==

5.查看libvirt中添加的UUID
& D6 y) M+ O% T  Zvirsh secret-list
' Q$ I: F( y3 G  m9 H
, X3 G8 v* `* s8 u; m5 Y! `6.重启libvirt
# S: Y& Q; t( [7 vsystemctl restart libvirtd.service
5 h- j+ I* i$ o2 {systemctl status libvirtd.service
2 Y+ ]2 z& S. b" d$ Z
出错回滚的方案
% G; n9 d6 J7 \$ H; o1.删除pool池
$ V% c. X8 A: C9 @先在所有的监视器节点上开启删除pool的权限，然后才可以删除。
3 {4 a; D, V4 Z: q3 o& O( Z删除pool时ceph要求必须输入两次pool名称，同时加上--yes-i-really-really-mean-it选项。
, E# u9 B3 }: O/ T) Q% b6 techo '
mon_allow_pool_delete = true
[mon]
. G) V- d" i1 b6 m) z1 l$ Cmon allow pool delete = true
' >> /etc/ceph/ceph.conf
0 Z6 M5 u. z% D: W  zsystemctl restart ceph-mon.target
ceph osd pool delete cinder-volumes cinder-volumes  --yes-i-really-really-mean-it
2 V5 w$ V8 T2 {/ x
2.删除账号
ceph auth del client.cinder-volumes

. J% D7 k' G: x  k0 L3.删除libvirt中注册的UUID和cinder-volumes密钥环
查看：
virsh secret-list
+ j( |. n, c2 e' t' h5 q0 C, x删除(secret-undefine后跟uuid值)：
1 \" [( H1 ?* D% h* I8 @5 Jvirsh secret-undefine  086037e4-ad59-4c61-82c9-86edc31b0bc0