Openstack对接两套Ceph

admin

环境说明
openpstack-Pike对接cephRBD单集群，配置简单，可参考openstack官网或者ceph官网；
3 B7 n. ?4 p. g8 p' \# @1.Openstack官网参考配置：
) }: g5 I' p' Khttps://docs.openstack.org/cinder/train/configuration/block-storage/drivers/ceph-rbd-volume-driver.html
2.Ceph官网参考配置：
https://docs.ceph.com/docs/master/install/install-ceph-deploy/
由于物理环境和业务需求变更，当前配置云计算环境要求一套openstack对接后台两套不同版本的cephRBD存储集群；
此处以现有以下正常运行环境展开配置；
! h; j/ P- F& _; ?6 S5 C1）openstack-Pike
2）Ceph Luminous 12.2.5
" I/ f. R$ n& j$ p" ^3）Ceph Nautilus 14.2.7
其中，openstack对接ceph Luminous配置完成，且正常运行。现在此套openstack+ceph环境基础上，新增一套ceph Nautilus存储集群，使openstack能够同时调用两套存储资源。

配置步骤
; R/ u! I/ x4 u# c! p  ]1.拷贝配置文件
#拷贝配置文件、cinder账户key到openstack的cinder节点
/etc/ceph/ceph2.conf
/etc/ceph/ceph.client.cinder2.keyring
#此处使用cinder账户，仅拷贝cinder2账户的key即可

2.创建存储池
- D$ c( P3 f1 q! A#OSD添加完成后，创建存储池，指定存储池pg/pgp数，配置其对应功能模式
* l5 w7 q# ]3 t1 d3 F- Fceph osd pool create volumes 512 512
, T4 t( F' ~2 B5 i4 Rceph osd pool create backups 128 128
ceph osd pool create vms 512 512
: Y( J! t. \. w3 \ceph osd pool create images 128 128

ceph osd pool application enable volumes rbd
ceph osd pool application enable backups rbd
! x1 a; V' X0 w/ c' uceph osd pool application enable vms rbd
ceph osd pool application enable images rbd

3.创建集群访问账户
ceph auth get-or-create client.cinder2 mon ‘allow r‘ osd ‘allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=vms, allow rx pool=images‘
$ H! A4 f" o$ o. a$ Nceph auth get-or-create client.cinder2-backup mon ‘allow r‘ osd ‘allow class-read object_prefix rbd_children, allow rwx pool=backups‘
ceph auth get-or-create client.glance mon ‘allow r‘ osd ‘allow class-read object_prefix rbd_children, allow rwx pool=images‘

4.查看进程信息
#查看当前openstack的cinder组件服务进程
source /root/keystonerc.admin
9 Q8 Q& O( m' }  mcinder service-list

5.修改配置文件
#修改cinder配置文件
* u& [# I; e8 _9 g[DEFAULT]
enabled_backends = ceph1,ceph2

[ceph1]
: \# x$ F+ m- h+ L- z7 p. Rvolume_driver = cinder.volume.drivers.rbd.RBDDriver
  C; G0 i# P( a4 ?. ~1 b& M4 kvolume_backend_name = ceph1
rbd_pool = volumes1
rbd_ceph_conf = /etc/ceph1/ceph1.conf
% Y) D2 F* k3 g/ O7 V2 Lrbd_flatten_volume_from_snapshot = false
0 h7 E" p. F' Yrbd_max_clone_depth = 5
' V' s8 d; b1 F  I) l" Irados_connect_timeout = -1
glance_api_version = 2
1 ?# [' w0 W- O8 s# Brbd_user = cinder1
rbd_secret_uuid = **

[ceph2]
volume_driver = cinder.volume.drivers.rbd.RBDDriver
; h' i6 B' S+ i3 z( ?! V7 svolume_backend_name = ceph2
2 G4 f) a9 s! Qrbd_pool = volumes2
rbd_ceph_conf = /etc/ceph/ceph2/ceph2.conf
# Q# N# p* m$ R9 D! ^rbd_flatten_volume_from_snapshot = false
! Z8 S( N5 z( U, Srbd_max_clone_depth = 5
; r4 o8 y9 s7 H* l/ {rados_connect_timeout = -1
7 b* G" h5 x" w4 {( cglance_api_version = 2
3 U0 N9 T- N2 q' W, ~; {8 n8 S2 crbd_user = cinder2
% k4 [5 l& ]& ~5 B" j: e1 c) Brbd_secret_uuid = **

6.重启服务
( W( k- W% S+ Y2 `9 B( y8 ]6 x( V; T#重启cinder-volume服务
* W. g, t( i2 D+ i( lservice openstack-cinder-volume restart Redirecting to /bin/systemctl restart openstack-cinder-volume.service
4 A) q" o& G3 }6 R9 @service openstack-cinder-scheduler restart Redirecting to /bin/systemctl restart openstack-cinder-scheduler.service

7.查看进程
cinder service-list

8.创建卷测试
#卷类型绑定
6 f" O/ R7 }$ q& {3 ^cinder type-create ceph1
. M" e2 Y' ^/ m1 I8 A( Gcinder type-key ceph1 set volume_backend_name=ceph1
cinder type-create ceph2
! _3 U7 @* H( |: b! ~cinder type-key ceph2 set volume_backend_name=ceph2

9.查看绑定结果
9 N2 R* i% y! h( c' ycinder create --volume-type ceph1 --display_name {volume-name}{volume-size}
" d' Z1 W# h  d2 j. _% L0 T0 w1 Kcinder create --volume-type ceph2 --display_name {volume-name}{volume-size}

配置libvirt
1.将第二套ceph的密钥添加到nova-compute节点的libvirt
9 e* u, \- V# s; L& R2 D- L#为了使VM可以访问到第二套cephRBD云盘，需要在nova-compute节点上将第二套ceph的cinder用户的密钥添加到libvirt
ceph -c /etc/ceph2/ceph2/ceph2.conf -k /etc/ceph2/ceph.client.cinder2.keyring auth get-key client.cinder2 |tee client.cinder2.key

#绑定之前cinder.conf中第二个ceph集群的uuid
+ v2 v, e# R5 E/ Mcat > secret2.xml <<EOF
<secret ephemeral=‘no‘ private=‘no‘>
8 {8 K' C) {. O<uuid>***</uuid>
! Y% `1 y3 `# \0 @: a  g5 T<usage type=‘ceph‘>
<name>client.cinder2 secret</name>
/ P' ~: }$ O* k) V/ }# y, u1 G</usage>
</secret>
/ ~* f; x% g( z1 ?0 @. K#以上整段拷贝执行即可，替换uuid值

sudo virsh secret-define --file secret2.xml

sudo virsh secret-set-value --secret ***** --base64 $(cat client.cinder2.key) rm client.cinder2.key secret2.xml
#删除提示信息，输入Y即可

2.验证配置是否生效
#通过之前创建的两个类型的云盘挂载到openstack的VM验证配置
' Z; g! k0 a% Bnova volume-attach {instance-id}{volume1-id}
; P- ?4 t' s9 D9 `nova volume-attach {instance-id}{volume2-id}

参考资料：
  n0 J. h% \( U( }6 `《ceph设计原理与实现》---谢型果
红帽官网
) F7 [6 n* K' A$ }https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/html/ceph_block_device_to_openstack_guide/installing_and_configuring_ceph_clients
6 O( V( y' m; |% i4 a( zceph官网
https://docs.ceph.com/docs/master/install/install-ceph-deploy/

6 d& q, _, E4 v8 c1 ^7 ~9 c- T

admin

The nova-compute, cinder-backup and on the cinder-volume node require both the Python bindings and the client command line tools:

# yum install python-rbd
  ?, l! a2 s* t! m  d; ?0 `# yum install ceph-common
The glance-api node requires the Python bindings for librbd:

# yum install python-rbd
2.1. Copying Ceph Configuration File to OpenStack Nodes
: ~2 L) j! ]- ~* ^5 v: G$ `* |The nodes running glance-api, cinder-volume, nova-compute and cinder-backup act as Ceph clients. Each requires the Ceph configuration file. Copy the Ceph configuration file from the monitor node to the OSP nodes.
$ T% r! C/ M! o7 z
$ Y+ d7 T7 R) A2 X# D0 ~# scp /etc/ceph/ceph.conf osp:/etc/ceph
7 o4 N& Z! u0 m3 o3 I2.2. Setting Up Ceph Client Authentication
From a Ceph monitor node, create new users for Cinder, Cinder Backup and Glance.
) n6 f  T/ `! B* O8 J7 n
# ceph auth get-or-create client.cinder mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=vms, allow rx pool=images'
2 ?. v, B6 j" {5 u" {
. L- }4 j9 g$ ?+ {+ a1 E$ I# ceph auth get-or-create client.cinder-backup mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=backups'

# ceph auth get-or-create client.glance mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=images'
" x8 B9 f$ t$ X$ B( k8 QAdd the keyrings for client.cinder, client.cinder-backup and client.glance to the appropriate nodes and change their ownership:
1 ?0 ?, \! [1 W( b$ ~$ j
# ceph auth get-or-create client.cinder | ssh {your-volume-server} sudo tee /etc/ceph/ceph.client.cinder.keyring
! b- l, B% P. C; @6 ?6 ^" b# ssh {your-cinder-volume-server} chown cinder:cinder /etc/ceph/ceph.client.cinder.keyring

# ceph auth get-or-create client.cinder-backup | ssh {your-cinder-backup-server} tee /etc/ceph/ceph.client.cinder-backup.keyring
" m; ?. j% Q2 S: W# ssh {your-cinder-backup-server} chown cinder:cinder /etc/ceph/ceph.client.cinder-backup.keyring
/ T0 L9 A+ i( @4 \2 n, @
# ceph auth get-or-create client.glance | ssh {your-glance-api-server} sudo tee /etc/ceph/ceph.client.glance.keyring
' d4 V* D0 `$ V1 E! v# ssh {your-glance-api-server} chown glance:glance /etc/ceph/ceph.client.glance.keyring
  Z8 x, ], H5 Q  t2 DNodes running nova-compute need the keyring file for the nova-compute process:

# ceph auth get-or-create client.cinder | ssh {your-nova-compute-server} tee /etc/ceph/ceph.client.cinder.keyring
0 N4 }6 I( s2 T% s3 [- o( LNodes running nova-compute also need to store the secret key of the client.cinder user in libvirt. The libvirt process needs it to access the cluster while attaching a block device from Cinder. Create a temporary copy of the secret key on the nodes running nova-compute:
: R$ ?5 z" e; f* v! ^
# ceph auth get-key client.cinder | ssh {your-compute-node} tee client.cinder.key
If the storage cluster contains Ceph Block Device images that use the exclusive-lock feature, ensure that all Ceph Block Device users have permissions to blacklist clients:
8 v6 d; L% R  q- D( S
# ceph auth caps client.{ID} mon 'allow r, allow command "osd blacklist"' osd '{existing-OSD-user-capabilities}'
" w# p7 e8 y' b( X6 Q( zReturn to the compute node.
/ n( e  M% N; r
# ssh {your-compute-node}
Generate a UUID for the secret, and save the UUID of the secret for configuring nova-compute later.
1 N# a7 c9 I' x0 I' y. y
. p, J  I$ K$ @$ @0 q' G# uuidgen > uuid-secret.txt
0 L! F; Q! S- v+ J2 H3 yNOTE
You don’t necessarily need the UUID on all the compute nodes. However from a platform consistency perspective, it’s better to keep the same UUID.

Then, on the compute nodes, add the secret key to libvirt and remove the temporary copy of the key:
: T5 F, A* t. l  X1 p. F
cat > secret.xml <<EOF
4 y; T* ]2 n0 ?5 g- J8 }<secret ephemeral='no' private='no'>
) Z" P5 K3 e- t% z( s  <uuid>`cat uuid-secret.txt`</uuid>
  <usage type='ceph'>
- P5 ~6 [2 [4 H5 b- _8 Y    <name>client.cinder secret</name>
  </usage>
0 Q# W( U6 t+ b2 v. ^# ^</secret>
EOF
# virsh secret-define --file secret.xml
' o- A! P: Z* ^3 P: K# virsh secret-set-value --secret $(cat uuid-secret.txt) --base64 $(cat client.cinder.key) && rm client.cinder.key secret.xml