Openstack对接两套Ceph

admin

环境说明
" f7 V8 o3 }8 X& T" Oopenpstack-Pike对接cephRBD单集群，配置简单，可参考openstack官网或者ceph官网；
# |4 [0 x2 Y3 C& h5 ~. z* H/ M1.Openstack官网参考配置：
# u( i0 K6 T9 L6 q: ^6 Vhttps://docs.openstack.org/cinder/train/configuration/block-storage/drivers/ceph-rbd-volume-driver.html
2.Ceph官网参考配置：
8 i7 d8 m" b* H, i, `https://docs.ceph.com/docs/master/install/install-ceph-deploy/
由于物理环境和业务需求变更，当前配置云计算环境要求一套openstack对接后台两套不同版本的cephRBD存储集群；
5 o  }, ]2 K7 i! l( v6 ]% r此处以现有以下正常运行环境展开配置；
1）openstack-Pike
2 G' ]  x; u, y  D2）Ceph Luminous 12.2.5
4 N  q. ~7 P2 f/ \5 [3）Ceph Nautilus 14.2.7
4 d4 K, O8 I( d/ @2 Y其中，openstack对接ceph Luminous配置完成，且正常运行。现在此套openstack+ceph环境基础上，新增一套ceph Nautilus存储集群，使openstack能够同时调用两套存储资源。
& S4 y% \" Y( o# [7 R( f

配置步骤
' I5 d2 G- C0 s2 W2 O9 n1.拷贝配置文件
#拷贝配置文件、cinder账户key到openstack的cinder节点
* C' `1 b! G1 P9 D7 h. _  O* a; I9 x) Y" j% J/etc/ceph/ceph2.conf
/etc/ceph/ceph.client.cinder2.keyring
#此处使用cinder账户，仅拷贝cinder2账户的key即可

2.创建存储池
( H9 {+ _1 o+ ^; S: M#OSD添加完成后，创建存储池，指定存储池pg/pgp数，配置其对应功能模式
ceph osd pool create volumes 512 512
ceph osd pool create backups 128 128
ceph osd pool create vms 512 512
. ?$ U$ T" n* `- V) z) {+ z3 aceph osd pool create images 128 128

ceph osd pool application enable volumes rbd
ceph osd pool application enable backups rbd
" l; L& k, k( L7 s# Mceph osd pool application enable vms rbd
ceph osd pool application enable images rbd

3.创建集群访问账户
ceph auth get-or-create client.cinder2 mon ‘allow r‘ osd ‘allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=vms, allow rx pool=images‘
ceph auth get-or-create client.cinder2-backup mon ‘allow r‘ osd ‘allow class-read object_prefix rbd_children, allow rwx pool=backups‘
& \- ~' ^; y, q1 F( Fceph auth get-or-create client.glance mon ‘allow r‘ osd ‘allow class-read object_prefix rbd_children, allow rwx pool=images‘

4.查看进程信息
  K# C0 O! q4 F" s#查看当前openstack的cinder组件服务进程
) X2 q/ }  X' f3 Q' r5 V& v$ Y' W' Isource /root/keystonerc.admin
cinder service-list

5.修改配置文件
#修改cinder配置文件
1 r3 c6 q; n: S4 E4 ][DEFAULT]
enabled_backends = ceph1,ceph2

[ceph1]
: g0 c! f! A* |# Mvolume_driver = cinder.volume.drivers.rbd.RBDDriver
volume_backend_name = ceph1
  n& x3 U, m7 L+ V5 G4 mrbd_pool = volumes1
rbd_ceph_conf = /etc/ceph1/ceph1.conf
0 L4 I* \% t0 e/ b/ Mrbd_flatten_volume_from_snapshot = false
rbd_max_clone_depth = 5
" s( t# Y8 C# N, u5 qrados_connect_timeout = -1
glance_api_version = 2
rbd_user = cinder1
rbd_secret_uuid = **

[ceph2]
volume_driver = cinder.volume.drivers.rbd.RBDDriver
volume_backend_name = ceph2
rbd_pool = volumes2
rbd_ceph_conf = /etc/ceph/ceph2/ceph2.conf
6 X, g* T! f$ I2 Y8 S+ vrbd_flatten_volume_from_snapshot = false
rbd_max_clone_depth = 5
4 _3 T9 J8 k7 A& g& M- yrados_connect_timeout = -1
+ k9 O/ P3 ?; E3 J$ m+ dglance_api_version = 2
7 Y8 D& T) t9 q9 Brbd_user = cinder2
( S* `' i9 n! f. ?0 |rbd_secret_uuid = **

6.重启服务
# ^. T& ]3 Z& J# U/ G8 B% U: O#重启cinder-volume服务
service openstack-cinder-volume restart Redirecting to /bin/systemctl restart openstack-cinder-volume.service
service openstack-cinder-scheduler restart Redirecting to /bin/systemctl restart openstack-cinder-scheduler.service

7.查看进程
cinder service-list

8.创建卷测试
#卷类型绑定
cinder type-create ceph1
; c) A+ z6 f9 H& Z  l1 a9 Ocinder type-key ceph1 set volume_backend_name=ceph1
9 m0 Z/ T/ z2 E% T* ?3 i, X+ ~cinder type-create ceph2
' h. \: u/ c' e" T9 w* I# qcinder type-key ceph2 set volume_backend_name=ceph2

9.查看绑定结果
% ?' j. Y- I' ^- wcinder create --volume-type ceph1 --display_name {volume-name}{volume-size}
/ l! Q& ?3 }1 K' D. acinder create --volume-type ceph2 --display_name {volume-name}{volume-size}

配置libvirt
& M: E2 k' S+ I2 i1.将第二套ceph的密钥添加到nova-compute节点的libvirt
" H/ _" `; A+ Y2 ?! f7 i' ^: M) D) ^#为了使VM可以访问到第二套cephRBD云盘，需要在nova-compute节点上将第二套ceph的cinder用户的密钥添加到libvirt
7 o. `) K) H  M9 v9 lceph -c /etc/ceph2/ceph2/ceph2.conf -k /etc/ceph2/ceph.client.cinder2.keyring auth get-key client.cinder2 |tee client.cinder2.key

#绑定之前cinder.conf中第二个ceph集群的uuid
" _+ p4 |, u, ~* S5 Y! s* P8 j& rcat > secret2.xml <<EOF
' e) m! g: h; Q5 a, U8 v! ~1 K<secret ephemeral=‘no‘ private=‘no‘>
8 ~" j1 M- R2 S6 S' F<uuid>***</uuid>
8 V, _6 D$ D7 `' g  @<usage type=‘ceph‘>
<name>client.cinder2 secret</name>
7 [/ D2 I; {! h7 }1 _$ z2 ]5 A</usage>
</secret>
3 J! h3 g5 p: ?5 c#以上整段拷贝执行即可，替换uuid值

sudo virsh secret-define --file secret2.xml

sudo virsh secret-set-value --secret ***** --base64 $(cat client.cinder2.key) rm client.cinder2.key secret2.xml
" J+ _3 u  ?+ E6 F& x#删除提示信息，输入Y即可

2.验证配置是否生效
& i" K7 {2 [6 m" ^#通过之前创建的两个类型的云盘挂载到openstack的VM验证配置
nova volume-attach {instance-id}{volume1-id}
8 `6 m/ b) W# }/ Q  t; A- o5 H5 K2 Vnova volume-attach {instance-id}{volume2-id}

参考资料：
: x+ K$ H. I# W& `& `+ J$ j# m6 e《ceph设计原理与实现》---谢型果
红帽官网
https://access.redhat.com/documentation/en-us/red_hat_ceph_storage/3/html/ceph_block_device_to_openstack_guide/installing_and_configuring_ceph_clients
ceph官网
* T6 g) b: h' Mhttps://docs.ceph.com/docs/master/install/install-ceph-deploy/

admin

The nova-compute, cinder-backup and on the cinder-volume node require both the Python bindings and the client command line tools:

9 G9 A1 q" d5 L% G* I# yum install python-rbd
# yum install ceph-common
9 F* N! A0 Q" c- vThe glance-api node requires the Python bindings for librbd:
6 D- A5 C7 f9 _. }2 V
- m$ x2 M9 C4 p0 z) q0 l- q6 {# yum install python-rbd
2.1. Copying Ceph Configuration File to OpenStack Nodes
6 o) G5 d+ q6 EThe nodes running glance-api, cinder-volume, nova-compute and cinder-backup act as Ceph clients. Each requires the Ceph configuration file. Copy the Ceph configuration file from the monitor node to the OSP nodes.
( l1 ]3 M# ?7 s9 n$ ^( ]# J2 g9 ^
# scp /etc/ceph/ceph.conf osp:/etc/ceph
2.2. Setting Up Ceph Client Authentication
# T! X4 b- f* @8 o0 u: wFrom a Ceph monitor node, create new users for Cinder, Cinder Backup and Glance.
0 n  E/ B0 Q1 s
* d' C3 ~6 {* I, D, R! u2 {# ceph auth get-or-create client.cinder mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=vms, allow rx pool=images'

# ceph auth get-or-create client.cinder-backup mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=backups'

! b" J% h. m) ^5 [# ceph auth get-or-create client.glance mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=images'
; d! y- z+ u$ ^$ I/ rAdd the keyrings for client.cinder, client.cinder-backup and client.glance to the appropriate nodes and change their ownership:

# ceph auth get-or-create client.cinder | ssh {your-volume-server} sudo tee /etc/ceph/ceph.client.cinder.keyring
# ssh {your-cinder-volume-server} chown cinder:cinder /etc/ceph/ceph.client.cinder.keyring
, `8 A0 H& g3 c8 K
2 }0 q8 Q5 n7 ?" _# ceph auth get-or-create client.cinder-backup | ssh {your-cinder-backup-server} tee /etc/ceph/ceph.client.cinder-backup.keyring
# ssh {your-cinder-backup-server} chown cinder:cinder /etc/ceph/ceph.client.cinder-backup.keyring

. b1 W* {8 |) [9 Y4 o* E: [# ceph auth get-or-create client.glance | ssh {your-glance-api-server} sudo tee /etc/ceph/ceph.client.glance.keyring
7 b  y$ b8 b, `& H* R! B9 H" z# ssh {your-glance-api-server} chown glance:glance /etc/ceph/ceph.client.glance.keyring
3 Q* g: O* o& T: n' A% u( ]" ENodes running nova-compute need the keyring file for the nova-compute process:
- J& p" C% i# g0 ^0 B
# ceph auth get-or-create client.cinder | ssh {your-nova-compute-server} tee /etc/ceph/ceph.client.cinder.keyring
% D8 }& Z$ u: ?$ L; u: t' @, L; tNodes running nova-compute also need to store the secret key of the client.cinder user in libvirt. The libvirt process needs it to access the cluster while attaching a block device from Cinder. Create a temporary copy of the secret key on the nodes running nova-compute:

5 h0 ^) {6 R. Y5 i# ceph auth get-key client.cinder | ssh {your-compute-node} tee client.cinder.key
' i2 g$ i/ v% ]/ U. L4 ]If the storage cluster contains Ceph Block Device images that use the exclusive-lock feature, ensure that all Ceph Block Device users have permissions to blacklist clients:
+ j7 d% s) w$ G" e
# ceph auth caps client.{ID} mon 'allow r, allow command "osd blacklist"' osd '{existing-OSD-user-capabilities}'
' x/ C9 T3 A! g) N1 rReturn to the compute node.

1 [) v2 t% z, a% ]% l4 T# i, q# ssh {your-compute-node}
Generate a UUID for the secret, and save the UUID of the secret for configuring nova-compute later.

' `) v! ?$ ~* f" Q- ]$ U3 A! o# uuidgen > uuid-secret.txt
8 b( [/ {! n+ ^6 v# [# J3 s( x5 JNOTE
You don’t necessarily need the UUID on all the compute nodes. However from a platform consistency perspective, it’s better to keep the same UUID.

7 K& T! _9 W( ?6 ]Then, on the compute nodes, add the secret key to libvirt and remove the temporary copy of the key:

* G3 g$ j# K! c* A$ x& ]cat > secret.xml <<EOF
& f+ J7 i/ f4 Q3 o" Q6 [5 f% ~<secret ephemeral='no' private='no'>
  <uuid>`cat uuid-secret.txt`</uuid>
  <usage type='ceph'>
  J  E! N3 M3 J( v    <name>client.cinder secret</name>
, F# k+ g  \6 [  </usage>
</secret>
# u3 Q- o0 m7 [, X- XEOF
# virsh secret-define --file secret.xml
# virsh secret-set-value --secret $(cat uuid-secret.txt) --base64 $(cat client.cinder.key) && rm client.cinder.key secret.xml