cinder 对接多个 ceph 存储

admin

环境说明当前 openstack环境正常使用由于后端 ceph 存储容量已经超过 85%不想直接进行扩容, 因为会有大量的数据迁移新创建一个独立的ceph 集群, 并计划用于 openstack 现有环境成为一个新的 ceph后端旧的 ceph 集群称为 ceph-A,  使用中的 pool 为 volumes新的 ceph 集群称为 ceph-B,  使用中的 pool 为 new_volumes目标在 openstack 中,  同时连接到两个不同的 ceph backendcinder server 配置1. ceph 连接配置2. cinder 配置ceph 连接配置

1.同时把两个 ceph 集群中的配置复制到 cinder 服务器 /etc/ceph 目录下, 定义成不同命名

[root@hh-yun-db-129041 ceph]# tree `pwd`/etc/ceph├── ceph.client.admin-develop.keyring      <- ceph-B 集群中的 admin key├── ceph.client.admin-volumes.keyring      <- ceph-A 集群中的 admin key├── ceph.client.developcinder.keyring      <- ceph-B 集群中的用户 developcinder key├── ceph.client.cinder.keyring             <- ceph-A 集群中的 cinder key├── ceph.client.mon-develop.keyring        <- ceph-B 集群中的 mon key├── ceph.client.mon-volumes.keyring        <- ceph-A 集群中的 mon key├── ceph-develop.conf                      <- ceph-B 集群配置文件(包含了 mon 地址等集群信息)└── ceph-volumes.conf                      <- ceph-B 集群配置文件(包含了 mon 地址等集群信息)

这里需要注意, clinet.client.(username).keyring 必须要与连接 ceph 的合法用户命名一致, 否则 cinder server 端, 无法正确获得权限

2.命令行下, 测试连接不同的 ceph 后端测试

ceph-A 连接测试

[root@hh-yun-db-129041 ceph]# ceph -c ceph-volumes.conf -k ceph.client.admin-volumes.keyring -s cluster xxx-xxx-xxxx-xxxx-xxxx


       health HEALTH_OK
) s3 P# `6 N5 ]0 |: h0 }

4 S4 [- K% W7 R) c" Y       monmap e3: 5 mons at {hh-yun-ceph-cinder015-128055=240.30.128.55:6789/0,hh-yun-ceph-cinder017-128057=240.30.128.57:6789/0,hh-yun-ceph-cinder024-128074=240.30.128.74:6789/0,hh-yun-ceph-cinder025-128075=240.30.128.75:6789/0,hh-yun-ceph-cinder026-128076=240.30.128.76:6789/0}, election epoch 452, quorum 0,1,2,3,4 hh-yun-ceph-cinder015-128055,hh-yun-ceph-cinder017-128057,hh-yun-ceph-cinder024-128074,hh-yun-ceph-cinder025-128075,hh-yun-ceph-cinder026-128076
8 N. `* H4 h9 _5 V# `4 o. P" s
# H& s' k, y- j- G4 X$ c5 g% F
      osdmap e170088: 226 osds: 226 up, 226 in
) n8 [% }1 k' h" {7 e
8 E- e8 d; d9 [  G0 r! Z
2 m: q  B/ {6 N% ~     pgmap v50751302: 20544 pgs, 2 pools, 157 TB data, 40687 kobjects 474 TB used, 376 TB / 850 TB avail 20537 active+clean 7 active+clean+scrubbing+deep client io 19972 kB/s rd, 73591 kB/s wr, 3250 op/s
5 E0 O' k; e6 e) l, g2 F) I

ceph-B 连接测试

[root@hh-yun-db-129041 ceph]# ceph -c ceph-develop.conf -k ceph.client.admin-develop.keyring -s cluster 4bf07d3e-a289-456d-9bd9-5a89832b413b
# s% w( z7 D; C! ]1 ?* h  
    health HEALTH_OK monmap e1: 5 mons at {240.30.128.214=240.30.128.214:6789/0,240.30.128.215=240.30.128.215:6789/0,240.30.128.39=240.30.128.39:6789/0,240.30.128.40=240.30.128.40:6789/0,240.30.128.58=240.30.128.58:6789/0} election epoch 6, quorum 0,1,2,3,4 240.30.128.39,240.30.128.40,240.30.128.58,240.30.128.214,240.30.128.215

$ S! s! W0 z. j& z2 ]
   osdmap e559: 264 osds: 264 up, 264 in flags sortbitwise


$ u) g! l; {2 D$ M1 w   pgmap v116751: 12400 pgs, 9 pools, 1636 bytes data, 171 objects 25091 MB used, 1440 TB / 1440 TB avail 12400 active+clean
9 R- v* O& o4 F5 a& Tcinder 配置

对 cinder 服务端进行配置

/etc/cinder/cinder.conf

( B, b6 P: ~) E3 ^& P1 m7 K$ s! a

enabled_backends=CEPH_SATA,CEPH_DEVELOP...
[CEPH_SATA]
4 b" j+ K5 s4 D8 \8 j5 u& zglance_api_version=2
* [& O# p; j+ r/ N/ G# M4 |
volume_backend_name=ceph_sata
5 G  j, W  y8 w: t1 N4 ~8 Z7 v" }
/ X0 \1 n" O% F0 Erbd_ceph_conf=/etc/ceph/ceph-volumes.conf
4 M2 G0 D2 y: r; s5 E+ U: A5 |8 h
rbd_user=cinder

$ U  I2 R. [, X) W4 Erbd_flatten_volume_from_snapshot=False

rados_connect_timeout=-1

rbd_max_clone_depth=5

volume_driver=cinder.volume.drivers.rbd.RBDDriver
0 C. I! g/ t9 z* z0 H
8 d; J$ B2 B9 J# ~' Lrbd_store_chunk_size=4

# t0 m& p# s5 Crbd_secret_uuid=dc4f91c1-8792-4948-b68f-2fcea75f53b

' N8 F: o) G$ ^9 A, jrbd_pool=volumeshost=cinder.vclound.com
) V, Z8 z: E. B3 W/ V- k
[CEPH-new_volumes]

glance_api_version=2
, R) B! F) K+ s- `) h# A; i
3 ?# `' h! v( Y" X( P4 N) mvolume_backend_name=ceph-new_volumes

0 O& N4 q, v0 A4 \6 x6 k9 Urbd_ceph_conf=/etc/ceph/ceph-new_volumes.conf
# {! i" U3 j! ^$ @; u0 F
# X; @+ r& q7 s# M1 G4 q/ y" Xrbd_user=cinder

rbd_flatten_volume_from_snapshot=False
" v) t: j8 [8 `3 I5 O& E
rados_connect_timeout=-1

9 r' D  i: ^; ]) j' @  Frbd_max_clone_depth=5
" w, x0 ?7 {5 n  ]) |
volume_driver=cinder.volume.drivers.rbd.RBDDriver
/ ?5 z% z6 M* D, Y
0 \0 W$ E* {, V: Z, Q* E# srbd_store_chunk_size=4

rbd_secret_uuid=4bf07d3e-a289-456d-9bd9-5a89832b413

5 X" f& e/ l# p0 t' qrbd_pool=new_volumes
. D+ f7 R6 u9 q8 M! o6 [! U
host=cinder.vclound.com
3 K3 j+ t& X$ r0 a

admin

在ceph监视器上执行
CINDER_PASSWD='cinder1234!'
controllerHost='controller'
; @( O" Y4 U$ N3 j" ^RABBIT_PASSWD='0penstackRMQ'
# X' s& |" O5 O* \2 Q2 L
1.创建pool池
为cinder-volume服务创建pool池(因为我只有一个OSD节点，所以要将副本数设置为1)
& }: A! p$ x6 I) A, X$ O1 yceph osd pool create cinder-volumes 32
ceph osd pool set cinder-volumes size 1
# ~. Q: b) ^3 N( H% V# nceph osd pool application enable  cinder-volumes rbd
5 {( w+ i, ]. }1 @- v( W, b4 Bceph osd lspools

6 R0 x5 k* R. R/ {2.查看pool池的使用情况
ceph df

3.创建账号
4 w8 n1 t6 ]: }/ q, S) uceph auth get-or-create client.cinder-volumes mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=cinder-volumes, allow rwx pool=glance-images' -o /etc/ceph/ceph.client.cinder-volumes.keyring
#查看
) H9 {( |0 i) I3 P: Q+ r# Zceph auth ls | grep -EA3 'client.(cinder-volumes)'
( z) f. c/ D. I! q% _6 B; e: s
3 E+ q* ]6 q) o$ k$ F4.修改ceph.conf配置文件并同步到所有的监视器节点（这步一定要操作）
su - cephd
- E  D) B! {. `7 `% zcd ~/ceph-cluster/
+ B& z8 I( i5 }- |* U/ K5 \5 |3 ucat <<EOF>> ceph.conf
[client.cinder-volumes]
$ l+ q1 h& d. ^. R/ z# `keyring = /etc/ceph/ceph.client.cinder-volumes.keyring
EOF
6 G4 s  X, ~+ L# P0 w! `ceph-deploy --overwrite-conf admin ceph-mon01
0 ?  I: \5 {& O* u! Rexit
5 x! l+ b# z  g, [+ p5 |; K
" o# k: v* A3 [- T# J( x1 i5.安装cinder-volume组件和ceph客户端(如果ceph监视器是在控制节点上不需要执行这一步)
) V$ }% _" F0 |yum -y install openstack-cinder python-keystone ceph-common
' v6 q3 u) p3 _8 v5 P, V, h; o
6.使用uuidgen生成一个uuid(确保cinder和libvirt中的UUID一致)
+ }2 K- z; W+ `1 `, k8 y2 duuidgen
  X1 a" Y' \5 J3 O1 e# ~1 Q5 y运行uuidgen命令可以得到下面的UUID值：

6 l* o! x7 ?+ E6 O4 o  c086037e4-ad59-4c61-82c9-86edc31b0bc0
7.配置cinder-volume服务与cinder-api服务进行交互
9 e# J. c& Y( M- ?openstack-config --set  /etc/cinder/cinder.conf DEFAULT transport_url rabbit://openstack:${RABBIT_PASSWD}@${controllerHost}:5672
openstack-config --set /etc/cinder/cinder.conf cache backend  oslo_cache.memcache_pool
# d! F( k& l9 i( W  P2 |9 F5 Zopenstack-config --set /etc/cinder/cinder.conf cache enabled  true
openstack-config --set /etc/cinder/cinder.conf cache memcache_servers  ${controllerHost}:11211
& |+ V  h' r' O1 Wopenstack-config --set  /etc/cinder/cinder.conf DEFAULT auth_strategy  keystone
openstack-config --set  /etc/cinder/cinder.conf keystone_authtoken  auth_uri  http://${controllerHost}:5000
. Y- b6 C* R& V0 \openstack-config --set  /etc/cinder/cinder.conf keystone_authtoken  auth_url  http://${controllerHost}:5000
openstack-config --set  /etc/cinder/cinder.conf keystone_authtoken  auth_type password
openstack-config --set  /etc/cinder/cinder.conf keystone_authtoken  project_domain_id  default
openstack-config --set  /etc/cinder/cinder.conf keystone_authtoken  user_domain_id  default
openstack-config --set  /etc/cinder/cinder.conf keystone_authtoken  project_name  service
openstack-config --set  /etc/cinder/cinder.conf keystone_authtoken  username  cinder
3 Q! |" n) Y% S  T' Zopenstack-config --set  /etc/cinder/cinder.conf keystone_authtoken  password  ${CINDER_PASSWD}
openstack-config --set  /etc/cinder/cinder.conf oslo_concurrency lock_path  /var/lib/cinder/tmp

) r: u; c) O% P, m' c/ ]8.配置cinder-volume服务使用的后端存储为ceph
openstack-config --set /etc/cinder/cinder.conf  DEFAULT  enabled_backends  ceph

9.配置cinder-volume服务驱动ceph
openstack-config --set /etc/cinder/cinder.conf  ceph volume_driver  cinder.volume.drivers.rbd.RBDDriver
; _9 b# `" r1 E& c7 y, d4 t2 f+ t7 ]openstack-config --set /etc/cinder/cinder.conf  ceph rbd_pool  cinder-volumes
% K$ |/ d1 _6 topenstack-config --set /etc/cinder/cinder.conf  ceph rbd_user cinder-volumes
6 E( s, T# h% }1 S( w% Sopenstack-config --set /etc/cinder/cinder.conf  ceph rbd_ceph_conf  /etc/ceph/ceph.conf
  V' {9 v& b% F# vopenstack-config --set /etc/cinder/cinder.conf  ceph rbd_flatten_volume_from_snapshot  false
" m/ y+ y6 r' Q( t" M& |& aopenstack-config --set /etc/cinder/cinder.conf  ceph bd_max_clone_depth  5
- t$ A1 L+ @% [$ Dopenstack-config --set /etc/cinder/cinder.conf  ceph rbd_store_chunk_size  4
6 x( U) e7 J# j" I7 }# Z" \openstack-config --set /etc/cinder/cinder.conf  ceph rados_connect_timeout  -1
9 F- i, A1 g3 h7 v9 K* F# S. jopenstack-config --set /etc/cinder/cinder.conf  ceph glance_api_version 2
: P5 ^, B, d0 E# C! O4 Copenstack-config --set /etc/cinder/cinder.conf  ceph rbd_secret_uuid  086037e4-ad59-4c61-82c9-86edc31b0bc0

/ F  @: B& l1 |5 `3 v10.启动cinder-volume服务
systemctl enable openstack-cinder-volume.service
" r. c5 `% e9 k7 K% {; j: [& msystemctl start openstack-cinder-volume.service
systemctl status openstack-cinder-volume.service
, Y1 k* q- I$ B6 R, ]% x
在需要挂载ceph卷的所有计算节点上执行
, k* n7 W8 p! v+ B1.创建secret文件(UUID需要与cinder服务中一致)
cat << EOF > ~/secret.xml
. o1 Y. @& ?+ S4 J0 M  N<secret ephemeral='no' private='no'>
1 U, q$ ^# I& p+ b( l# M8 I1 m, z     <uuid>086037e4-ad59-4c61-82c9-86edc31b0bc0</uuid>
* r% k# v4 D+ i# N     <usage type='ceph'>
         <name>client.cinder-volumes secret</name>
* y& W% q3 ?1 x7 v. e, @+ S     </usage>
! x( S' [/ s8 }  _</secret>
# \+ Z4 v$ F3 Z6 _# B8 J* ^EOF

2.从ceph监视器上获取cinder-volumes账户的密钥环
ceph auth get-key client.cinder-volumes
: E; }' P; B; N5 ?2 F得到如下的结果：
AQCxfDFdgp2qKRAAUY/vep29N39Qv7xWKYqMUw==
# s- R* l& W8 l4 g' b$ V. y1 N
2 Y: ^+ S& v, m' M7 M5 j3.在libvirt中注册UUID
8 {$ I1 b# T% C5 B/ E0 S$ `virsh secret-define --file ~/secret.xml

; k* n; d' ~) g" P4 D  A4.在libvirt中添加UUID和cinder-volumes密钥环
virsh secret-set-value --secret 086037e4-ad59-4c61-82c9-86edc31b0bc0 --base64 AQCxfDFdgp2qKRAAUY/vep29N39Qv7xWKYqMUw==

5.查看libvirt中添加的UUID
virsh secret-list

0 q4 I+ o4 r3 @% X+ Z. w5 Q4 m2 U6.重启libvirt
systemctl restart libvirtd.service
( Z# ?4 J$ S/ Tsystemctl status libvirtd.service

$ L0 n: [- `  R* F' |& N出错回滚的方案
1.删除pool池
先在所有的监视器节点上开启删除pool的权限，然后才可以删除。
删除pool时ceph要求必须输入两次pool名称，同时加上--yes-i-really-really-mean-it选项。
1 a% J# \1 }$ z6 T. V- \7 gecho '
mon_allow_pool_delete = true
[mon]
9 ~, _/ O2 L5 B. ]: h3 G3 bmon allow pool delete = true
' >> /etc/ceph/ceph.conf
* m* ]' Q" B) ~: i$ f9 rsystemctl restart ceph-mon.target
  e- q; H( a* `! fceph osd pool delete cinder-volumes cinder-volumes  --yes-i-really-really-mean-it

7 z% c. t7 S5 S7 ^' j2.删除账号
ceph auth del client.cinder-volumes
! S& G9 ^4 L+ E6 |; @5 s2 A
3.删除libvirt中注册的UUID和cinder-volumes密钥环
查看：
& M% G: @! R' kvirsh secret-list
& Y: ?8 Q8 _  _' p% z. H( s删除(secret-undefine后跟uuid值)：
virsh secret-undefine  086037e4-ad59-4c61-82c9-86edc31b0bc0