Openstack对接两套Ceph

admin

环境说明
openpstack-Pike对接cephRBD单集群，配置简单，可参考openstack官网或者ceph官网；
. P8 l. t# P2 j9 h1.Openstack官网参考配置：
https://docs.openstack.org/cinder/train/configuration/block-storage/drivers/ceph-rbd-volume-driver.html
2.Ceph官网参考配置：
https://docs.ceph.com/docs/master/install/install-ceph-deploy/
由于物理环境和业务需求变更，当前配置云计算环境要求一套openstack对接后台两套不同版本的cephRBD存储集群；
) H# U7 V9 {" |此处以现有以下正常运行环境展开配置；
- i/ n/ Q+ o: B- p9 K8 l1）openstack-Pike
2）Ceph Luminous 12.2.5
3）Ceph Nautilus 14.2.7
其中，openstack对接ceph Luminous配置完成，且正常运行。现在此套openstack+ceph环境基础上，新增一套ceph Nautilus存储集群，使openstack能够同时调用两套存储资源。
+ \5 C$ T# U/ b- C

配置步骤
1.拷贝配置文件
#拷贝配置文件、cinder账户key到openstack的cinder节点
3 C# z9 K+ ]1 E+ A! S2 E/etc/ceph/ceph2.conf
/etc/ceph/ceph.client.cinder2.keyring
- x* d/ o1 G& u4 N#此处使用cinder账户，仅拷贝cinder2账户的key即可

2.创建存储池
7 |3 ]! K' I( T# [) y. x#OSD添加完成后，创建存储池，指定存储池pg/pgp数，配置其对应功能模式
ceph osd pool create volumes 512 512
3 n  a3 {# v- I& L0 }ceph osd pool create backups 128 128
/ I. C' P) e6 Mceph osd pool create vms 512 512
) ^; \# J4 j2 n) ?" g/ j* v/ k- ^) N3 oceph osd pool create images 128 128

ceph osd pool application enable volumes rbd
ceph osd pool application enable backups rbd
, e6 u$ E+ m  h: eceph osd pool application enable vms rbd
ceph osd pool application enable images rbd

3.创建集群访问账户
ceph auth get-or-create client.cinder2 mon ‘allow r‘ osd ‘allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=vms, allow rx pool=images‘
ceph auth get-or-create client.cinder2-backup mon ‘allow r‘ osd ‘allow class-read object_prefix rbd_children, allow rwx pool=backups‘
! H) w( d' C. q* O! u* T/ bceph auth get-or-create client.glance mon ‘allow r‘ osd ‘allow class-read object_prefix rbd_children, allow rwx pool=images‘

4.查看进程信息
" n: S9 d) z' ?& V  s6 `#查看当前openstack的cinder组件服务进程
5 [( T# b- v7 E2 C2 q; P' zsource /root/keystonerc.admin
cinder service-list

5.修改配置文件
#修改cinder配置文件
[DEFAULT]
enabled_backends = ceph1,ceph2

[ceph1]
% X6 ?% l( F  ^- w9 evolume_driver = cinder.volume.drivers.rbd.RBDDriver
volume_backend_name = ceph1
rbd_pool = volumes1
rbd_ceph_conf = /etc/ceph1/ceph1.conf
2 H; }; A1 A# z/ K' m; Z) E3 nrbd_flatten_volume_from_snapshot = false
rbd_max_clone_depth = 5
1 ]9 ?1 T1 c0 T9 l( W, Y: Prados_connect_timeout = -1
glance_api_version = 2
rbd_user = cinder1
rbd_secret_uuid = **

[ceph2]
2 S1 n4 M* G/ m8 f/ rvolume_driver = cinder.volume.drivers.rbd.RBDDriver
volume_backend_name = ceph2
rbd_pool = volumes2
6 z$ J% j8 ~9 h9 R4 q. [+ Q! D) irbd_ceph_conf = /etc/ceph/ceph2/ceph2.conf
rbd_flatten_volume_from_snapshot = false
rbd_max_clone_depth = 5
rados_connect_timeout = -1
# x, o1 R' e  ]& d  Wglance_api_version = 2
+ y3 [* K9 q3 P; b: i* `& Hrbd_user = cinder2
- R* i' m% I: N0 B8 b" Jrbd_secret_uuid = **

6.重启服务
#重启cinder-volume服务
- v7 a. e) G+ k' k  @service openstack-cinder-volume restart Redirecting to /bin/systemctl restart openstack-cinder-volume.service
service openstack-cinder-scheduler restart Redirecting to /bin/systemctl restart openstack-cinder-scheduler.service

7.查看进程
cinder service-list

8.创建卷测试
#卷类型绑定
cinder type-create ceph1
cinder type-key ceph1 set volume_backend_name=ceph1
cinder type-create ceph2
  n  x3 D8 ^, q# B8 ^# S, i* m* ycinder type-key ceph2 set volume_backend_name=ceph2

9.查看绑定结果
, C' ]2 L" a% v4 G) M8 Zcinder create --volume-type ceph1 --display_name {volume-name}{volume-size}
cinder create --volume-type ceph2 --display_name {volume-name}{volume-size}

配置libvirt
1.将第二套ceph的密钥添加到nova-compute节点的libvirt
#为了使VM可以访问到第二套cephRBD云盘，需要在nova-compute节点上将第二套ceph的cinder用户的密钥添加到libvirt
ceph -c /etc/ceph2/ceph2/ceph2.conf -k /etc/ceph2/ceph.client.cinder2.keyring auth get-key client.cinder2 |tee client.cinder2.key

#绑定之前cinder.conf中第二个ceph集群的uuid
! w$ J0 O8 T( e5 E# Qcat > secret2.xml <<EOF
<secret ephemeral=‘no‘ private=‘no‘>
4 C5 {' ^3 W) B; m* M# k<uuid>***</uuid>
<usage type=‘ceph‘>
<name>client.cinder2 secret</name>
</usage>
</secret>
0 T2 @* L2 Q( s: R/ g' }6 p2 }, O4 _#以上整段拷贝执行即可，替换uuid值

sudo virsh secret-define --file secret2.xml

sudo virsh secret-set-value --secret ***** --base64 $(cat client.cinder2.key) rm client.cinder2.key secret2.xml
; s5 ~8 X2 q$ h#删除提示信息，输入Y即可

2.验证配置是否生效
#通过之前创建的两个类型的云盘挂载到openstack的VM验证配置
nova volume-attach {instance-id}{volume1-id}
nova volume-attach {instance-id}{volume2-id}

参考资料：
《ceph设计原理与实现》---谢型果
红帽官网
+ A  _. U4 U2 thttps://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/html/ceph_block_device_to_openstack_guide/installing_and_configuring_ceph_clients
3 Z( {$ E2 ~, s8 n  h# ?ceph官网
' {5 W5 i0 f% }6 k4 T" _, D" ]https://docs.ceph.com/docs/master/install/install-ceph-deploy/

admin

The nova-compute, cinder-backup and on the cinder-volume node require both the Python bindings and the client command line tools:
; n) I0 g' ~9 B0 j% ~# {0 \$ Y( [
# yum install python-rbd
# yum install ceph-common
  t5 ?% ?& u6 U+ V# `6 @" SThe glance-api node requires the Python bindings for librbd:
7 y! T+ s! P: f" k: w0 K- Z
# yum install python-rbd
& u- E9 H9 a) i7 p; ?2.1. Copying Ceph Configuration File to OpenStack Nodes
6 v8 n- _4 |. Z+ B9 @The nodes running glance-api, cinder-volume, nova-compute and cinder-backup act as Ceph clients. Each requires the Ceph configuration file. Copy the Ceph configuration file from the monitor node to the OSP nodes.

$ ~4 w- o  x5 p# w, P# scp /etc/ceph/ceph.conf osp:/etc/ceph
2.2. Setting Up Ceph Client Authentication
4 Y" v. c+ y9 {0 o  {From a Ceph monitor node, create new users for Cinder, Cinder Backup and Glance.

9 b0 z' e$ ^- O# ceph auth get-or-create client.cinder mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=vms, allow rx pool=images'
3 _- l9 ~3 f$ D3 Y- l" h
# ceph auth get-or-create client.cinder-backup mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=backups'
! `- Z0 Z. u+ P- j% p
# j2 D' t5 y* Z& b0 y2 k3 O6 Y# ceph auth get-or-create client.glance mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=images'
Add the keyrings for client.cinder, client.cinder-backup and client.glance to the appropriate nodes and change their ownership:
) E4 A7 x+ ^* c7 j- V0 Y1 X; Z% }
; ?8 X& h' F. N0 r# ceph auth get-or-create client.cinder | ssh {your-volume-server} sudo tee /etc/ceph/ceph.client.cinder.keyring
# ssh {your-cinder-volume-server} chown cinder:cinder /etc/ceph/ceph.client.cinder.keyring

4 a; J% u4 i' j& n6 c' N# ceph auth get-or-create client.cinder-backup | ssh {your-cinder-backup-server} tee /etc/ceph/ceph.client.cinder-backup.keyring
# ssh {your-cinder-backup-server} chown cinder:cinder /etc/ceph/ceph.client.cinder-backup.keyring
5 N& T2 R# \8 Z8 O0 E
9 [( E" O3 T7 n7 S0 y  h5 a% u# ceph auth get-or-create client.glance | ssh {your-glance-api-server} sudo tee /etc/ceph/ceph.client.glance.keyring
: N* Q  U  f  n! E# ssh {your-glance-api-server} chown glance:glance /etc/ceph/ceph.client.glance.keyring
$ l+ U& O$ W, B7 DNodes running nova-compute need the keyring file for the nova-compute process:
' V( r9 v+ e0 U: Q8 y% W  r
# ceph auth get-or-create client.cinder | ssh {your-nova-compute-server} tee /etc/ceph/ceph.client.cinder.keyring
  Z" P) n& x1 t7 i9 U! C, X4 ^Nodes running nova-compute also need to store the secret key of the client.cinder user in libvirt. The libvirt process needs it to access the cluster while attaching a block device from Cinder. Create a temporary copy of the secret key on the nodes running nova-compute:
1 O1 F8 C7 j) x) ^# M; [! A+ W
# ceph auth get-key client.cinder | ssh {your-compute-node} tee client.cinder.key
If the storage cluster contains Ceph Block Device images that use the exclusive-lock feature, ensure that all Ceph Block Device users have permissions to blacklist clients:
' E3 T& H  K$ v: @: S
# ceph auth caps client.{ID} mon 'allow r, allow command "osd blacklist"' osd '{existing-OSD-user-capabilities}'
# W! H/ H4 V8 a1 vReturn to the compute node.
$ e2 K  Y) s4 D
# ssh {your-compute-node}
Generate a UUID for the secret, and save the UUID of the secret for configuring nova-compute later.

$ I: B* `1 g  \# V- s# uuidgen > uuid-secret.txt
% N! v0 }2 T9 hNOTE
0 l" y; ?: f7 w6 W) c6 m) i, hYou don’t necessarily need the UUID on all the compute nodes. However from a platform consistency perspective, it’s better to keep the same UUID.
, B. R% }2 ^1 T$ s' ~4 f) A: s
Then, on the compute nodes, add the secret key to libvirt and remove the temporary copy of the key:
" |0 `4 T: d+ p  l
cat > secret.xml <<EOF
9 Y: {3 B0 O' W/ f" E- {5 s$ N) l<secret ephemeral='no' private='no'>
( P3 H: q1 N  n" {# N! @7 z  <uuid>`cat uuid-secret.txt`</uuid>
0 g+ H$ t" o; v3 V: H1 x5 {5 G  <usage type='ceph'>
* N. L& x! G$ k% i" R. a    <name>client.cinder secret</name>
# p7 S0 \2 l, P5 T  </usage>
</secret>
EOF
; Z% ^4 e) f! }# virsh secret-define --file secret.xml
# virsh secret-set-value --secret $(cat uuid-secret.txt) --base64 $(cat client.cinder.key) && rm client.cinder.key secret.xml