Openstack对接两套Ceph

admin

环境说明
openpstack-Pike对接cephRBD单集群，配置简单，可参考openstack官网或者ceph官网；
6 X7 D0 k2 z& p8 g1.Openstack官网参考配置：
7 g+ P/ ?: a9 Mhttps://docs.openstack.org/cinder/train/configuration/block-storage/drivers/ceph-rbd-volume-driver.html
: m+ Q/ a$ T9 g8 K% Y' N1 j2.Ceph官网参考配置：
/ W1 Z( a5 k5 p1 S; n' g  p1 @https://docs.ceph.com/docs/master/install/install-ceph-deploy/
由于物理环境和业务需求变更，当前配置云计算环境要求一套openstack对接后台两套不同版本的cephRBD存储集群；
此处以现有以下正常运行环境展开配置；
1）openstack-Pike
, J9 K2 Z) `. Y* H; K; ]7 _2 C2）Ceph Luminous 12.2.5
4 K  Q; u: Z& }1 \& r5 s9 G: n3）Ceph Nautilus 14.2.7
其中，openstack对接ceph Luminous配置完成，且正常运行。现在此套openstack+ceph环境基础上，新增一套ceph Nautilus存储集群，使openstack能够同时调用两套存储资源。

配置步骤
0 ~0 X/ Y3 B' ~# K4 [; c1.拷贝配置文件
9 U$ f. |, X0 }4 _#拷贝配置文件、cinder账户key到openstack的cinder节点
/etc/ceph/ceph2.conf
/etc/ceph/ceph.client.cinder2.keyring
& K7 {$ P2 l0 r#此处使用cinder账户，仅拷贝cinder2账户的key即可

2.创建存储池
#OSD添加完成后，创建存储池，指定存储池pg/pgp数，配置其对应功能模式
; l& L4 o* Q4 L+ w1 e7 S# \2 c( Z% rceph osd pool create volumes 512 512
; j0 I0 J, l4 a! \; bceph osd pool create backups 128 128
ceph osd pool create vms 512 512
ceph osd pool create images 128 128

ceph osd pool application enable volumes rbd
- Y5 ~1 \, ?( {- L( f! q! U% `7 tceph osd pool application enable backups rbd
ceph osd pool application enable vms rbd
ceph osd pool application enable images rbd

3.创建集群访问账户
$ ^, r9 W) C7 H* Y8 _; Nceph auth get-or-create client.cinder2 mon ‘allow r‘ osd ‘allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=vms, allow rx pool=images‘
ceph auth get-or-create client.cinder2-backup mon ‘allow r‘ osd ‘allow class-read object_prefix rbd_children, allow rwx pool=backups‘
; V1 ?2 m+ Y) T" q) Uceph auth get-or-create client.glance mon ‘allow r‘ osd ‘allow class-read object_prefix rbd_children, allow rwx pool=images‘

4.查看进程信息
! F( M& f' O3 G: E#查看当前openstack的cinder组件服务进程
# ]' S1 z5 m  E! e! X5 G! T/ csource /root/keystonerc.admin
cinder service-list

5.修改配置文件
& y1 a. s9 z8 Z% B8 d$ B#修改cinder配置文件
$ ]3 @0 p( y# q: }% f[DEFAULT]
enabled_backends = ceph1,ceph2

[ceph1]
( H, s- r' M' T+ S( J: Xvolume_driver = cinder.volume.drivers.rbd.RBDDriver
volume_backend_name = ceph1
; V' l- l$ ]. p1 hrbd_pool = volumes1
rbd_ceph_conf = /etc/ceph1/ceph1.conf
# L+ F7 |  z! c- ?4 f* Qrbd_flatten_volume_from_snapshot = false
rbd_max_clone_depth = 5
rados_connect_timeout = -1
glance_api_version = 2
3 `$ \0 a" d& P/ ]rbd_user = cinder1
rbd_secret_uuid = **

[ceph2]
- Y' K+ n& l6 Y% d, m* [) l7 bvolume_driver = cinder.volume.drivers.rbd.RBDDriver
volume_backend_name = ceph2
1 [1 W% y6 c  H5 p, `: ~rbd_pool = volumes2
rbd_ceph_conf = /etc/ceph/ceph2/ceph2.conf
; J1 m5 y/ g# l( orbd_flatten_volume_from_snapshot = false
( Y2 w9 ~. V/ A0 yrbd_max_clone_depth = 5
rados_connect_timeout = -1
glance_api_version = 2
rbd_user = cinder2
rbd_secret_uuid = **

6.重启服务
2 v6 v" |) {2 q/ P7 N2 n#重启cinder-volume服务
service openstack-cinder-volume restart Redirecting to /bin/systemctl restart openstack-cinder-volume.service
1 n) R9 ^: U( n1 S* k( v2 Dservice openstack-cinder-scheduler restart Redirecting to /bin/systemctl restart openstack-cinder-scheduler.service

7.查看进程
! {- T& o- q+ q6 H- S8 Fcinder service-list

8.创建卷测试
#卷类型绑定
0 t: j0 |* ^+ [; W8 i: |1 l  ycinder type-create ceph1
cinder type-key ceph1 set volume_backend_name=ceph1
cinder type-create ceph2
" V. K' h  U! q) ]5 ^/ Ocinder type-key ceph2 set volume_backend_name=ceph2

9.查看绑定结果
cinder create --volume-type ceph1 --display_name {volume-name}{volume-size}
9 p$ C, t, s! B/ W% t& j8 hcinder create --volume-type ceph2 --display_name {volume-name}{volume-size}

配置libvirt
5 K) b8 e& M, }$ p$ F7 A; G, C! D1.将第二套ceph的密钥添加到nova-compute节点的libvirt
#为了使VM可以访问到第二套cephRBD云盘，需要在nova-compute节点上将第二套ceph的cinder用户的密钥添加到libvirt
ceph -c /etc/ceph2/ceph2/ceph2.conf -k /etc/ceph2/ceph.client.cinder2.keyring auth get-key client.cinder2 |tee client.cinder2.key

#绑定之前cinder.conf中第二个ceph集群的uuid
cat > secret2.xml <<EOF
<secret ephemeral=‘no‘ private=‘no‘>
<uuid>***</uuid>
<usage type=‘ceph‘>
<name>client.cinder2 secret</name>
, r) _+ X: ^( Q</usage>
% d5 I# s1 l) o7 X' F2 w</secret>
3 y$ I* r4 B9 l3 R2 f3 ]7 t#以上整段拷贝执行即可，替换uuid值

sudo virsh secret-define --file secret2.xml

sudo virsh secret-set-value --secret ***** --base64 $(cat client.cinder2.key) rm client.cinder2.key secret2.xml
#删除提示信息，输入Y即可

2.验证配置是否生效
#通过之前创建的两个类型的云盘挂载到openstack的VM验证配置
2 ~- _" H) K( w- L1 E" n. T3 S7 |/ \nova volume-attach {instance-id}{volume1-id}
nova volume-attach {instance-id}{volume2-id}

参考资料：
《ceph设计原理与实现》---谢型果
- j- n' N0 H$ N; M红帽官网
' K: F9 u2 z4 k" _https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/html/ceph_block_device_to_openstack_guide/installing_and_configuring_ceph_clients
ceph官网
  v, k7 M$ P4 m% b! whttps://docs.ceph.com/docs/master/install/install-ceph-deploy/

admin

The nova-compute, cinder-backup and on the cinder-volume node require both the Python bindings and the client command line tools:
; s4 e& ^' k  p) Z; \1 X% h
+ i" V1 s+ |) D4 B! `! ]8 @# yum install python-rbd
# yum install ceph-common
3 }5 l& U' l6 d3 rThe glance-api node requires the Python bindings for librbd:

. ~* l) y6 q% n) P8 f8 k: ]# yum install python-rbd
) t. t; X0 [2 s# D2.1. Copying Ceph Configuration File to OpenStack Nodes
The nodes running glance-api, cinder-volume, nova-compute and cinder-backup act as Ceph clients. Each requires the Ceph configuration file. Copy the Ceph configuration file from the monitor node to the OSP nodes.

, b" Y3 ^; l- v5 l# scp /etc/ceph/ceph.conf osp:/etc/ceph
2.2. Setting Up Ceph Client Authentication
# Q% q& ^( V, v: CFrom a Ceph monitor node, create new users for Cinder, Cinder Backup and Glance.

# c" Z* x2 \; s; I( u! L# ceph auth get-or-create client.cinder mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=vms, allow rx pool=images'

# l: i8 f4 L: e* _. Z6 \+ Q6 e# ceph auth get-or-create client.cinder-backup mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=backups'

# ceph auth get-or-create client.glance mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=images'
Add the keyrings for client.cinder, client.cinder-backup and client.glance to the appropriate nodes and change their ownership:
' R4 S2 P: k) n0 F# G$ O
# ceph auth get-or-create client.cinder | ssh {your-volume-server} sudo tee /etc/ceph/ceph.client.cinder.keyring
/ F8 n8 J! i4 G3 v. N( }# ssh {your-cinder-volume-server} chown cinder:cinder /etc/ceph/ceph.client.cinder.keyring

# ceph auth get-or-create client.cinder-backup | ssh {your-cinder-backup-server} tee /etc/ceph/ceph.client.cinder-backup.keyring
0 x3 ^' a2 C& Z5 ]# ssh {your-cinder-backup-server} chown cinder:cinder /etc/ceph/ceph.client.cinder-backup.keyring
  X3 `. R9 ~0 @4 N
# ceph auth get-or-create client.glance | ssh {your-glance-api-server} sudo tee /etc/ceph/ceph.client.glance.keyring
( A/ [" t+ P) w6 Y7 d% d. e' P9 ?# ssh {your-glance-api-server} chown glance:glance /etc/ceph/ceph.client.glance.keyring
Nodes running nova-compute need the keyring file for the nova-compute process:
! Q7 b8 b+ ~! w1 B& E# {) ]
* l" b$ |: Z3 X  v0 E+ b! B2 V# ceph auth get-or-create client.cinder | ssh {your-nova-compute-server} tee /etc/ceph/ceph.client.cinder.keyring
Nodes running nova-compute also need to store the secret key of the client.cinder user in libvirt. The libvirt process needs it to access the cluster while attaching a block device from Cinder. Create a temporary copy of the secret key on the nodes running nova-compute:
+ j" z3 u. Y6 _; h% d; j
) i# r7 U6 d# u) E7 I6 [# ceph auth get-key client.cinder | ssh {your-compute-node} tee client.cinder.key
) a0 W7 t" g: E! s* DIf the storage cluster contains Ceph Block Device images that use the exclusive-lock feature, ensure that all Ceph Block Device users have permissions to blacklist clients:
4 J' `, \% A0 s  k5 S
6 d3 y  v* }/ _$ z# ceph auth caps client.{ID} mon 'allow r, allow command "osd blacklist"' osd '{existing-OSD-user-capabilities}'
Return to the compute node.
' ?1 X! L' d% M3 y: n
" P* N( m! S8 F* |# F# ssh {your-compute-node}
Generate a UUID for the secret, and save the UUID of the secret for configuring nova-compute later.

# uuidgen > uuid-secret.txt
NOTE
You don’t necessarily need the UUID on all the compute nodes. However from a platform consistency perspective, it’s better to keep the same UUID.
" I5 |- B6 a/ t6 |& m7 Z2 r+ \
Then, on the compute nodes, add the secret key to libvirt and remove the temporary copy of the key:

cat > secret.xml <<EOF
<secret ephemeral='no' private='no'>
  <uuid>`cat uuid-secret.txt`</uuid>
  <usage type='ceph'>
    <name>client.cinder secret</name>
  </usage>
3 L, [( z, Y: W3 _9 ]</secret>
EOF
6 L4 ~$ w! t( e6 {# virsh secret-define --file secret.xml
0 i- ?1 r" A5 o. }' d+ g# virsh secret-set-value --secret $(cat uuid-secret.txt) --base64 $(cat client.cinder.key) && rm client.cinder.key secret.xml