cinder 对接多个 ceph 存储

admin

环境说明当前 openstack环境正常使用由于后端 ceph 存储容量已经超过 85%不想直接进行扩容, 因为会有大量的数据迁移新创建一个独立的ceph 集群, 并计划用于 openstack 现有环境成为一个新的 ceph后端旧的 ceph 集群称为 ceph-A,  使用中的 pool 为 volumes新的 ceph 集群称为 ceph-B,  使用中的 pool 为 new_volumes目标在 openstack 中,  同时连接到两个不同的 ceph backendcinder server 配置1. ceph 连接配置2. cinder 配置ceph 连接配置

1.同时把两个 ceph 集群中的配置复制到 cinder 服务器 /etc/ceph 目录下, 定义成不同命名

[root@hh-yun-db-129041 ceph]# tree `pwd`/etc/ceph├── ceph.client.admin-develop.keyring      <- ceph-B 集群中的 admin key├── ceph.client.admin-volumes.keyring      <- ceph-A 集群中的 admin key├── ceph.client.developcinder.keyring      <- ceph-B 集群中的用户 developcinder key├── ceph.client.cinder.keyring             <- ceph-A 集群中的 cinder key├── ceph.client.mon-develop.keyring        <- ceph-B 集群中的 mon key├── ceph.client.mon-volumes.keyring        <- ceph-A 集群中的 mon key├── ceph-develop.conf                      <- ceph-B 集群配置文件(包含了 mon 地址等集群信息)└── ceph-volumes.conf                      <- ceph-B 集群配置文件(包含了 mon 地址等集群信息)

这里需要注意, clinet.client.(username).keyring 必须要与连接 ceph 的合法用户命名一致, 否则 cinder server 端, 无法正确获得权限

2.命令行下, 测试连接不同的 ceph 后端测试

ceph-A 连接测试

[root@hh-yun-db-129041 ceph]# ceph -c ceph-volumes.conf -k ceph.client.admin-volumes.keyring -s cluster xxx-xxx-xxxx-xxxx-xxxx

8 m# u! J% D7 m8 h
       health HEALTH_OK
8 _! c4 R) n6 T# }; N

       monmap e3: 5 mons at {hh-yun-ceph-cinder015-128055=240.30.128.55:6789/0,hh-yun-ceph-cinder017-128057=240.30.128.57:6789/0,hh-yun-ceph-cinder024-128074=240.30.128.74:6789/0,hh-yun-ceph-cinder025-128075=240.30.128.75:6789/0,hh-yun-ceph-cinder026-128076=240.30.128.76:6789/0}, election epoch 452, quorum 0,1,2,3,4 hh-yun-ceph-cinder015-128055,hh-yun-ceph-cinder017-128057,hh-yun-ceph-cinder024-128074,hh-yun-ceph-cinder025-128075,hh-yun-ceph-cinder026-128076

3 v- n! q, u7 i- ~; |
/ T) W+ F3 [; ~7 p& A' N      osdmap e170088: 226 osds: 226 up, 226 in
" [& u. H& B- v5 @* L
$ w& I  t9 r6 V  b1 J
     pgmap v50751302: 20544 pgs, 2 pools, 157 TB data, 40687 kobjects 474 TB used, 376 TB / 850 TB avail 20537 active+clean 7 active+clean+scrubbing+deep client io 19972 kB/s rd, 73591 kB/s wr, 3250 op/s

ceph-B 连接测试

[root@hh-yun-db-129041 ceph]# ceph -c ceph-develop.conf -k ceph.client.admin-develop.keyring -s cluster 4bf07d3e-a289-456d-9bd9-5a89832b413b
  
% N. \( b' [/ l! o( `+ e5 m    health HEALTH_OK monmap e1: 5 mons at {240.30.128.214=240.30.128.214:6789/0,240.30.128.215=240.30.128.215:6789/0,240.30.128.39=240.30.128.39:6789/0,240.30.128.40=240.30.128.40:6789/0,240.30.128.58=240.30.128.58:6789/0} election epoch 6, quorum 0,1,2,3,4 240.30.128.39,240.30.128.40,240.30.128.58,240.30.128.214,240.30.128.215
$ A% u& Q) K* ]8 c) B/ o( b

0 k0 h3 A) X) v4 \& S   osdmap e559: 264 osds: 264 up, 264 in flags sortbitwise


   pgmap v116751: 12400 pgs, 9 pools, 1636 bytes data, 171 objects 25091 MB used, 1440 TB / 1440 TB avail 12400 active+clean
; [6 y( M' c: n7 O& j- `; Ocinder 配置

对 cinder 服务端进行配置

/etc/cinder/cinder.conf

enabled_backends=CEPH_SATA,CEPH_DEVELOP...
[CEPH_SATA]
. d& |  C0 j1 O% Y: x* c9 ^: Aglance_api_version=2

( R! P$ k# I, _% r( bvolume_backend_name=ceph_sata

rbd_ceph_conf=/etc/ceph/ceph-volumes.conf

rbd_user=cinder
, e  D2 b& g0 W
rbd_flatten_volume_from_snapshot=False
* S2 a$ z" i& l
" i: a/ \3 y* S- y" m; `, Qrados_connect_timeout=-1

rbd_max_clone_depth=5
6 I# u! {3 {* x, [( m) E6 L
volume_driver=cinder.volume.drivers.rbd.RBDDriver

' O" M; q. a! X* Mrbd_store_chunk_size=4

, S- u; K  {( drbd_secret_uuid=dc4f91c1-8792-4948-b68f-2fcea75f53b
& `. ]0 U! W( U( F
" M# @. @' v5 T: X: e1 x/ frbd_pool=volumeshost=cinder.vclound.com
/ A1 ^$ l6 q5 p0 n
: Y0 U% Q  }  ^  ]) {[CEPH-new_volumes]
# Q! V9 u" r6 v: p7 E' f
8 X% }- k5 T- _, N, P: nglance_api_version=2

% a  E" ?/ q9 H7 Q4 z3 @7 q' E; evolume_backend_name=ceph-new_volumes

/ m0 x2 ?- N3 T- t+ z  l" qrbd_ceph_conf=/etc/ceph/ceph-new_volumes.conf
3 c+ Z4 }" h3 H' t8 p+ o
# P- |! c& ?) e" `% Q. Mrbd_user=cinder

) n$ g1 B1 S. Z% K' C3 S8 Crbd_flatten_volume_from_snapshot=False
( {! R7 W5 W- X  A& @% D& W* s
rados_connect_timeout=-1
1 p6 }5 F& K9 R) D0 A# g
$ h& Q6 s1 I2 \4 Grbd_max_clone_depth=5

volume_driver=cinder.volume.drivers.rbd.RBDDriver

! `2 G  I$ P& M5 }  ~, Y1 Trbd_store_chunk_size=4

: U- M$ N: X" Q6 j8 a& ^) erbd_secret_uuid=4bf07d3e-a289-456d-9bd9-5a89832b413

rbd_pool=new_volumes
: d; ?* u% ~4 w3 P/ ]. R9 w
6 w5 _/ O( k& Y+ H# H1 R- r7 hhost=cinder.vclound.com

1 o! ]( A6 p$ d. f; P1 Q

admin

在ceph监视器上执行
CINDER_PASSWD='cinder1234!'
controllerHost='controller'
0 b1 ~5 Q- |0 j7 h" XRABBIT_PASSWD='0penstackRMQ'

' ]  g* k0 ~, e( h3 W1.创建pool池
) o( g, F) F/ A为cinder-volume服务创建pool池(因为我只有一个OSD节点，所以要将副本数设置为1)
6 l  \& P! c& nceph osd pool create cinder-volumes 32
; X& E. a' s) ?/ y5 k5 Vceph osd pool set cinder-volumes size 1
ceph osd pool application enable  cinder-volumes rbd
- x* \7 w' z8 r, R8 [+ i# dceph osd lspools

. O# w) M0 `* }2.查看pool池的使用情况
* i# E- p5 W. ]1 z  A2 rceph df

0 W3 U0 s6 o& G* Z3.创建账号
ceph auth get-or-create client.cinder-volumes mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=cinder-volumes, allow rwx pool=glance-images' -o /etc/ceph/ceph.client.cinder-volumes.keyring
9 D  N$ e0 W  H* z: J& K: F8 W#查看
ceph auth ls | grep -EA3 'client.(cinder-volumes)'
. L0 p& n& R$ o4 M6 H# Z
4.修改ceph.conf配置文件并同步到所有的监视器节点（这步一定要操作）
( r- r* z7 Z% o7 wsu - cephd
2 @9 ~  R# ?1 n# hcd ~/ceph-cluster/
* }7 ^' ?5 c2 X! Hcat <<EOF>> ceph.conf
6 J/ U6 M2 ]- F, M* s4 J[client.cinder-volumes]
) _9 b, C. I" ^5 U' C3 E& G+ kkeyring = /etc/ceph/ceph.client.cinder-volumes.keyring
EOF
8 {# r: l1 e) D+ [7 vceph-deploy --overwrite-conf admin ceph-mon01
/ E, f' e( J- Q! @3 b2 Texit
2 S) B7 g7 E; F  }0 j7 g) {
+ K# k( F) G9 V8 k( K; a. _+ A# T5.安装cinder-volume组件和ceph客户端(如果ceph监视器是在控制节点上不需要执行这一步)
yum -y install openstack-cinder python-keystone ceph-common
3 q% q: i+ W+ Y# |3 _# B8 k
4 {0 I' U; e! P! D2 W* k6.使用uuidgen生成一个uuid(确保cinder和libvirt中的UUID一致)
5 L% N$ H% g. O3 V6 ?- V1 Uuuidgen
运行uuidgen命令可以得到下面的UUID值：

  E1 P1 q; p! l: h" g) p086037e4-ad59-4c61-82c9-86edc31b0bc0
7.配置cinder-volume服务与cinder-api服务进行交互
5 k1 \+ m$ K6 popenstack-config --set  /etc/cinder/cinder.conf DEFAULT transport_url rabbit://openstack:${RABBIT_PASSWD}@${controllerHost}:5672
0 @3 y& i5 _2 m' |" u1 O6 _openstack-config --set /etc/cinder/cinder.conf cache backend  oslo_cache.memcache_pool
( M- x' g" \3 p/ Aopenstack-config --set /etc/cinder/cinder.conf cache enabled  true
3 Y; i/ H% a5 A# ^$ q, P' bopenstack-config --set /etc/cinder/cinder.conf cache memcache_servers  ${controllerHost}:11211
openstack-config --set  /etc/cinder/cinder.conf DEFAULT auth_strategy  keystone
openstack-config --set  /etc/cinder/cinder.conf keystone_authtoken  auth_uri  http://${controllerHost}:5000
openstack-config --set  /etc/cinder/cinder.conf keystone_authtoken  auth_url  http://${controllerHost}:5000
" P2 N4 f+ |8 G2 `  ?openstack-config --set  /etc/cinder/cinder.conf keystone_authtoken  auth_type password
openstack-config --set  /etc/cinder/cinder.conf keystone_authtoken  project_domain_id  default
9 Z9 [5 c. |5 o% C1 xopenstack-config --set  /etc/cinder/cinder.conf keystone_authtoken  user_domain_id  default
openstack-config --set  /etc/cinder/cinder.conf keystone_authtoken  project_name  service
openstack-config --set  /etc/cinder/cinder.conf keystone_authtoken  username  cinder
openstack-config --set  /etc/cinder/cinder.conf keystone_authtoken  password  ${CINDER_PASSWD}
0 X; t( n9 ?: ?- j/ ]6 i, A+ iopenstack-config --set  /etc/cinder/cinder.conf oslo_concurrency lock_path  /var/lib/cinder/tmp
* E/ J1 |- S" |4 q3 }+ V4 J
8.配置cinder-volume服务使用的后端存储为ceph
* W9 a9 B; |; a& O; X, w! o7 k9 Aopenstack-config --set /etc/cinder/cinder.conf  DEFAULT  enabled_backends  ceph

$ G7 ^$ \  {5 b- L1 @9.配置cinder-volume服务驱动ceph
openstack-config --set /etc/cinder/cinder.conf  ceph volume_driver  cinder.volume.drivers.rbd.RBDDriver
openstack-config --set /etc/cinder/cinder.conf  ceph rbd_pool  cinder-volumes
openstack-config --set /etc/cinder/cinder.conf  ceph rbd_user cinder-volumes
" F  G3 B) ?- y2 [3 }5 d: @openstack-config --set /etc/cinder/cinder.conf  ceph rbd_ceph_conf  /etc/ceph/ceph.conf
openstack-config --set /etc/cinder/cinder.conf  ceph rbd_flatten_volume_from_snapshot  false
# H$ U/ g5 O8 Q3 J# Qopenstack-config --set /etc/cinder/cinder.conf  ceph bd_max_clone_depth  5
. J2 a& t1 p4 A& v# ?( V5 k  Xopenstack-config --set /etc/cinder/cinder.conf  ceph rbd_store_chunk_size  4
openstack-config --set /etc/cinder/cinder.conf  ceph rados_connect_timeout  -1
openstack-config --set /etc/cinder/cinder.conf  ceph glance_api_version 2
: }  c! P9 r$ \& P& `1 b9 N+ q* Copenstack-config --set /etc/cinder/cinder.conf  ceph rbd_secret_uuid  086037e4-ad59-4c61-82c9-86edc31b0bc0

- o% T# I: O) W10.启动cinder-volume服务
systemctl enable openstack-cinder-volume.service
systemctl start openstack-cinder-volume.service
systemctl status openstack-cinder-volume.service
8 q5 T9 P0 E7 w4 m
8 e9 y+ p( K3 q& J在需要挂载ceph卷的所有计算节点上执行
/ Q0 l7 ?1 s5 }. ?1.创建secret文件(UUID需要与cinder服务中一致)
cat << EOF > ~/secret.xml
<secret ephemeral='no' private='no'>
" f6 G: e# t/ k5 P     <uuid>086037e4-ad59-4c61-82c9-86edc31b0bc0</uuid>
; R) p" ^( V8 p3 a$ N     <usage type='ceph'>
1 w# s& J$ |; y& W. |( c2 |3 ~         <name>client.cinder-volumes secret</name>
     </usage>
% N( {  d) O& N7 n. u$ m</secret>
EOF

2.从ceph监视器上获取cinder-volumes账户的密钥环
ceph auth get-key client.cinder-volumes
1 I* x) o  C4 F& r得到如下的结果：
AQCxfDFdgp2qKRAAUY/vep29N39Qv7xWKYqMUw==
% F3 p. r5 ~9 o! a, R
8 T1 I1 Y& o% J5 j# |  u3.在libvirt中注册UUID
5 S: Y: }; x+ `- _virsh secret-define --file ~/secret.xml

4.在libvirt中添加UUID和cinder-volumes密钥环
virsh secret-set-value --secret 086037e4-ad59-4c61-82c9-86edc31b0bc0 --base64 AQCxfDFdgp2qKRAAUY/vep29N39Qv7xWKYqMUw==
/ G; u7 o$ `, m
! H: q2 C" J; d& T! W6 T5.查看libvirt中添加的UUID
virsh secret-list
% g  {; |4 u1 _' v5 x
9 n5 `) [, U$ x5 f: ^9 L3 g6.重启libvirt
+ k. G( R1 m( d2 a9 qsystemctl restart libvirtd.service
2 O1 H6 k: b& X7 j5 w; Ksystemctl status libvirtd.service
' j2 q: q6 V& W+ q
: m1 ~* {+ p$ t) \3 @& C# P出错回滚的方案
1.删除pool池
% t9 ~. F( K6 a$ c/ G' |先在所有的监视器节点上开启删除pool的权限，然后才可以删除。
删除pool时ceph要求必须输入两次pool名称，同时加上--yes-i-really-really-mean-it选项。
' w) ]7 _8 j& _/ mecho '
8 x: i. s0 t6 j8 Lmon_allow_pool_delete = true
[mon]
mon allow pool delete = true
! r3 U9 J1 N* H+ l* o' >> /etc/ceph/ceph.conf
systemctl restart ceph-mon.target
+ G+ q% @5 I4 z3 nceph osd pool delete cinder-volumes cinder-volumes  --yes-i-really-really-mean-it
. E6 S' W: C5 g% x  j
2.删除账号
ceph auth del client.cinder-volumes
) d& u0 O9 c, b2 o0 }
" e2 ~" L% [. \7 ^. p7 A9 H3.删除libvirt中注册的UUID和cinder-volumes密钥环
查看：
virsh secret-list
" x4 \5 O- @2 I& S& a0 I0 T+ Y删除(secret-undefine后跟uuid值)：
1 U+ g. l- m7 L7 h8 tvirsh secret-undefine  086037e4-ad59-4c61-82c9-86edc31b0bc0