cinder 对接多个 ceph 存储

admin

环境说明当前 openstack环境正常使用由于后端 ceph 存储容量已经超过 85%不想直接进行扩容, 因为会有大量的数据迁移新创建一个独立的ceph 集群, 并计划用于 openstack 现有环境成为一个新的 ceph后端旧的 ceph 集群称为 ceph-A,  使用中的 pool 为 volumes新的 ceph 集群称为 ceph-B,  使用中的 pool 为 new_volumes目标在 openstack 中,  同时连接到两个不同的 ceph backendcinder server 配置1. ceph 连接配置2. cinder 配置ceph 连接配置

1.同时把两个 ceph 集群中的配置复制到 cinder 服务器 /etc/ceph 目录下, 定义成不同命名

[root@hh-yun-db-129041 ceph]# tree `pwd`/etc/ceph├── ceph.client.admin-develop.keyring      <- ceph-B 集群中的 admin key├── ceph.client.admin-volumes.keyring      <- ceph-A 集群中的 admin key├── ceph.client.developcinder.keyring      <- ceph-B 集群中的用户 developcinder key├── ceph.client.cinder.keyring             <- ceph-A 集群中的 cinder key├── ceph.client.mon-develop.keyring        <- ceph-B 集群中的 mon key├── ceph.client.mon-volumes.keyring        <- ceph-A 集群中的 mon key├── ceph-develop.conf                      <- ceph-B 集群配置文件(包含了 mon 地址等集群信息)└── ceph-volumes.conf                      <- ceph-B 集群配置文件(包含了 mon 地址等集群信息)

这里需要注意, clinet.client.(username).keyring 必须要与连接 ceph 的合法用户命名一致, 否则 cinder server 端, 无法正确获得权限

2.命令行下, 测试连接不同的 ceph 后端测试

ceph-A 连接测试

[root@hh-yun-db-129041 ceph]# ceph -c ceph-volumes.conf -k ceph.client.admin-volumes.keyring -s cluster xxx-xxx-xxxx-xxxx-xxxx


       health HEALTH_OK
3 D8 g2 m( m: E2 h
9 L# I1 e2 U8 C6 p2 t
; ?* U1 \5 t' X/ i2 r" V  M. Q; L       monmap e3: 5 mons at {hh-yun-ceph-cinder015-128055=240.30.128.55:6789/0,hh-yun-ceph-cinder017-128057=240.30.128.57:6789/0,hh-yun-ceph-cinder024-128074=240.30.128.74:6789/0,hh-yun-ceph-cinder025-128075=240.30.128.75:6789/0,hh-yun-ceph-cinder026-128076=240.30.128.76:6789/0}, election epoch 452, quorum 0,1,2,3,4 hh-yun-ceph-cinder015-128055,hh-yun-ceph-cinder017-128057,hh-yun-ceph-cinder024-128074,hh-yun-ceph-cinder025-128075,hh-yun-ceph-cinder026-128076

1 x: A9 |) D9 ~$ C" s7 e1 a1 e
/ {$ b) l  Z2 A      osdmap e170088: 226 osds: 226 up, 226 in
  n- N' f0 {7 a% n

2 @2 Z4 W& V# u( Q  }' ^  M     pgmap v50751302: 20544 pgs, 2 pools, 157 TB data, 40687 kobjects 474 TB used, 376 TB / 850 TB avail 20537 active+clean 7 active+clean+scrubbing+deep client io 19972 kB/s rd, 73591 kB/s wr, 3250 op/s
1 ?- k. y! N3 n+ w' `6 ~# h5 G% n* ?

ceph-B 连接测试

[root@hh-yun-db-129041 ceph]# ceph -c ceph-develop.conf -k ceph.client.admin-develop.keyring -s cluster 4bf07d3e-a289-456d-9bd9-5a89832b413b
  
    health HEALTH_OK monmap e1: 5 mons at {240.30.128.214=240.30.128.214:6789/0,240.30.128.215=240.30.128.215:6789/0,240.30.128.39=240.30.128.39:6789/0,240.30.128.40=240.30.128.40:6789/0,240.30.128.58=240.30.128.58:6789/0} election epoch 6, quorum 0,1,2,3,4 240.30.128.39,240.30.128.40,240.30.128.58,240.30.128.214,240.30.128.215
* R. \* i6 n* N. u) V' M. \
# m9 A7 j. J' Q% ]2 h! X
   osdmap e559: 264 osds: 264 up, 264 in flags sortbitwise

: M* `+ q% v  I# k% A
   pgmap v116751: 12400 pgs, 9 pools, 1636 bytes data, 171 objects 25091 MB used, 1440 TB / 1440 TB avail 12400 active+clean
+ V5 n5 L6 E7 z- jcinder 配置

对 cinder 服务端进行配置

/etc/cinder/cinder.conf

+ f" o6 g9 e' ]2 f

enabled_backends=CEPH_SATA,CEPH_DEVELOP...
  w  V( `: U3 x( r$ c; `  y% |[CEPH_SATA]
glance_api_version=2
8 t) J+ X. Z* V9 G" x$ u
volume_backend_name=ceph_sata
% T0 \2 H# A% @$ S, R) V
5 T4 s' {! x+ frbd_ceph_conf=/etc/ceph/ceph-volumes.conf

, O% W4 N6 d& ^& c1 i  r6 w7 jrbd_user=cinder
$ d8 @  y( T5 f5 T
rbd_flatten_volume_from_snapshot=False
% }1 L6 T0 q; z0 V  q2 O! j, S3 ]
rados_connect_timeout=-1

  `5 p( e/ L, e8 e" a3 ~rbd_max_clone_depth=5
  `! a/ z/ i3 V5 x
* u7 D6 t* r0 K0 Mvolume_driver=cinder.volume.drivers.rbd.RBDDriver

% v- u0 f- ]9 t% ?) K! T. `/ yrbd_store_chunk_size=4

rbd_secret_uuid=dc4f91c1-8792-4948-b68f-2fcea75f53b
, F7 F" [6 c8 l9 h6 o2 S' m
4 D& P: f; f8 b- W% Krbd_pool=volumeshost=cinder.vclound.com

[CEPH-new_volumes]

glance_api_version=2
1 m( c# p% W2 h6 R
volume_backend_name=ceph-new_volumes
& p$ n4 g7 T) H* y5 r& Y) c$ F* j& T
rbd_ceph_conf=/etc/ceph/ceph-new_volumes.conf

rbd_user=cinder

rbd_flatten_volume_from_snapshot=False

rados_connect_timeout=-1

rbd_max_clone_depth=5

3 M5 _" A- o" m" Z2 K0 T6 d) kvolume_driver=cinder.volume.drivers.rbd.RBDDriver
4 `2 h% U2 @7 F! n2 @1 C
rbd_store_chunk_size=4
5 b5 ~3 U. P8 y% [/ P
* S  m# L9 c* n3 N& @8 S, srbd_secret_uuid=4bf07d3e-a289-456d-9bd9-5a89832b413

rbd_pool=new_volumes

+ \! W7 f7 w5 S1 y9 ]* Xhost=cinder.vclound.com
. C& C5 A2 J1 V" V9 r
3 V1 p  o6 N6 d9 a# L

admin

在ceph监视器上执行
CINDER_PASSWD='cinder1234!'
' V6 \( t& H$ \9 [1 ?/ scontrollerHost='controller'
RABBIT_PASSWD='0penstackRMQ'

! u* E9 J$ M7 i- i7 I( |: G1.创建pool池
7 w. `( d$ q0 w( T7 Z' ]  x/ s# m为cinder-volume服务创建pool池(因为我只有一个OSD节点，所以要将副本数设置为1)
ceph osd pool create cinder-volumes 32
" l5 \" J7 a8 D' Z% w) xceph osd pool set cinder-volumes size 1
$ j7 q0 T0 z/ f: |8 q- sceph osd pool application enable  cinder-volumes rbd
- K; U0 J) m5 M* Q2 B9 uceph osd lspools

2.查看pool池的使用情况
' M3 a3 _' f. |8 N8 l; q; Tceph df
% T& B$ q( j. f6 G1 j
+ y$ u7 b) w$ e( W% F* m2 r3.创建账号
ceph auth get-or-create client.cinder-volumes mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=cinder-volumes, allow rwx pool=glance-images' -o /etc/ceph/ceph.client.cinder-volumes.keyring
#查看
# ^& g6 A4 Y5 Pceph auth ls | grep -EA3 'client.(cinder-volumes)'
0 L2 y2 S2 H; n% r# J+ E! A
4.修改ceph.conf配置文件并同步到所有的监视器节点（这步一定要操作）
su - cephd
cd ~/ceph-cluster/
cat <<EOF>> ceph.conf
+ e* U# ?. R* {' A# D9 M[client.cinder-volumes]
; R. ?  n: t5 a" O. i# Z0 V0 A# pkeyring = /etc/ceph/ceph.client.cinder-volumes.keyring
EOF
0 R: K; ~  D+ f' Z. nceph-deploy --overwrite-conf admin ceph-mon01
3 V' u" q9 x  aexit

5.安装cinder-volume组件和ceph客户端(如果ceph监视器是在控制节点上不需要执行这一步)
! ]) e+ y* |* m  r# a/ m8 Oyum -y install openstack-cinder python-keystone ceph-common
* |' M: O/ P4 |5 F% R* t4 e$ R
2 w* _. @4 L- m* w; _! G7 d( V$ H" L6.使用uuidgen生成一个uuid(确保cinder和libvirt中的UUID一致)
uuidgen
9 q  }' W6 R1 l2 s0 y# ]% E运行uuidgen命令可以得到下面的UUID值：
& n# Y8 E5 X" C0 q; f- T, x! o
086037e4-ad59-4c61-82c9-86edc31b0bc0
7 v/ n- N+ k" F8 g1 l0 u' I0 K7.配置cinder-volume服务与cinder-api服务进行交互
openstack-config --set  /etc/cinder/cinder.conf DEFAULT transport_url rabbit://openstack:${RABBIT_PASSWD}@${controllerHost}:5672
openstack-config --set /etc/cinder/cinder.conf cache backend  oslo_cache.memcache_pool
openstack-config --set /etc/cinder/cinder.conf cache enabled  true
openstack-config --set /etc/cinder/cinder.conf cache memcache_servers  ${controllerHost}:11211
openstack-config --set  /etc/cinder/cinder.conf DEFAULT auth_strategy  keystone
5 b' {; ~% g" e2 ^* S) K- uopenstack-config --set  /etc/cinder/cinder.conf keystone_authtoken  auth_uri  http://${controllerHost}:5000
openstack-config --set  /etc/cinder/cinder.conf keystone_authtoken  auth_url  http://${controllerHost}:5000
openstack-config --set  /etc/cinder/cinder.conf keystone_authtoken  auth_type password
openstack-config --set  /etc/cinder/cinder.conf keystone_authtoken  project_domain_id  default
openstack-config --set  /etc/cinder/cinder.conf keystone_authtoken  user_domain_id  default
: n! C, ^! h7 I% f$ ?+ Dopenstack-config --set  /etc/cinder/cinder.conf keystone_authtoken  project_name  service
openstack-config --set  /etc/cinder/cinder.conf keystone_authtoken  username  cinder
openstack-config --set  /etc/cinder/cinder.conf keystone_authtoken  password  ${CINDER_PASSWD}
/ V  E6 E& x) nopenstack-config --set  /etc/cinder/cinder.conf oslo_concurrency lock_path  /var/lib/cinder/tmp
+ {, ]: b( u. F5 V$ }* l0 r
/ h1 Y& t. d  Y" D8.配置cinder-volume服务使用的后端存储为ceph
  }/ E4 `  a* i# I/ h1 eopenstack-config --set /etc/cinder/cinder.conf  DEFAULT  enabled_backends  ceph
; l6 O: s/ g, F  n8 s( r  |
9.配置cinder-volume服务驱动ceph
9 ^' D7 P. F& N* z2 P" u8 }: r! kopenstack-config --set /etc/cinder/cinder.conf  ceph volume_driver  cinder.volume.drivers.rbd.RBDDriver
openstack-config --set /etc/cinder/cinder.conf  ceph rbd_pool  cinder-volumes
openstack-config --set /etc/cinder/cinder.conf  ceph rbd_user cinder-volumes
2 r0 N" n3 O- F$ V' sopenstack-config --set /etc/cinder/cinder.conf  ceph rbd_ceph_conf  /etc/ceph/ceph.conf
openstack-config --set /etc/cinder/cinder.conf  ceph rbd_flatten_volume_from_snapshot  false
openstack-config --set /etc/cinder/cinder.conf  ceph bd_max_clone_depth  5
openstack-config --set /etc/cinder/cinder.conf  ceph rbd_store_chunk_size  4
openstack-config --set /etc/cinder/cinder.conf  ceph rados_connect_timeout  -1
openstack-config --set /etc/cinder/cinder.conf  ceph glance_api_version 2
openstack-config --set /etc/cinder/cinder.conf  ceph rbd_secret_uuid  086037e4-ad59-4c61-82c9-86edc31b0bc0
" f) C) C7 h: n+ o1 N6 T* {  p0 G) m
$ a7 D. C! ^- Z5 {  \2 V8 Q; H4 Z10.启动cinder-volume服务
systemctl enable openstack-cinder-volume.service
systemctl start openstack-cinder-volume.service
systemctl status openstack-cinder-volume.service
% g' D. z! V) t  k( y
6 }1 Z6 }) n7 }- g% Y0 _3 O在需要挂载ceph卷的所有计算节点上执行
1.创建secret文件(UUID需要与cinder服务中一致)
! W$ X: h. F7 Ycat << EOF > ~/secret.xml
, w4 \4 B, Y" o% i4 E& n: J. c<secret ephemeral='no' private='no'>
     <uuid>086037e4-ad59-4c61-82c9-86edc31b0bc0</uuid>
- a0 S; Q- N9 j3 m% _6 ^! Y     <usage type='ceph'>
         <name>client.cinder-volumes secret</name>
     </usage>
</secret>
EOF

2.从ceph监视器上获取cinder-volumes账户的密钥环
ceph auth get-key client.cinder-volumes
2 a6 O7 @, \" j& O+ j& p得到如下的结果：
1 L5 Q' r( j/ I9 |5 MAQCxfDFdgp2qKRAAUY/vep29N39Qv7xWKYqMUw==
+ e6 [$ j& d% o: ^7 X2 i0 k5 M
' U: [% W9 S* X. N+ M9 }' @2 P% z) z% \3.在libvirt中注册UUID
virsh secret-define --file ~/secret.xml

4.在libvirt中添加UUID和cinder-volumes密钥环
virsh secret-set-value --secret 086037e4-ad59-4c61-82c9-86edc31b0bc0 --base64 AQCxfDFdgp2qKRAAUY/vep29N39Qv7xWKYqMUw==

/ C# V+ a; u% T2 S( P9 a5.查看libvirt中添加的UUID
8 i/ F# S( W1 A* r" g% K- avirsh secret-list

6.重启libvirt
* c/ Q9 p1 I( ]; P) `: I2 q* Hsystemctl restart libvirtd.service
systemctl status libvirtd.service
7 U; ]- b" T( X& F' D
1 Z# E& ^( A7 |$ g! C7 @/ ?$ _; B出错回滚的方案
% s: G5 v- ~. U6 }' Z, `, v1.删除pool池
0 L2 G/ L0 r8 k$ m, D$ f. j, A先在所有的监视器节点上开启删除pool的权限，然后才可以删除。
删除pool时ceph要求必须输入两次pool名称，同时加上--yes-i-really-really-mean-it选项。
3 K) {. L. |# x" hecho '
4 U* S, j( Y5 g+ B4 e- Jmon_allow_pool_delete = true
[mon]
mon allow pool delete = true
' >> /etc/ceph/ceph.conf
1 @7 [8 D/ F" V( I3 Qsystemctl restart ceph-mon.target
ceph osd pool delete cinder-volumes cinder-volumes  --yes-i-really-really-mean-it

2.删除账号
ceph auth del client.cinder-volumes
- @2 E! r( X2 K
0 h) G4 Q6 h% j! g2 W1 j# J8 ]; e3.删除libvirt中注册的UUID和cinder-volumes密钥环
查看：
virsh secret-list
删除(secret-undefine后跟uuid值)：
virsh secret-undefine  086037e4-ad59-4c61-82c9-86edc31b0bc0