最新Ceph 版与openstack Pike对接

admin

一、在所有服务器上操作

#使用阿里源

8 b1 o$ r3 u/ Z; R0 B$ Z7 p8 o( S

yum install wget -y

: G8 e4 R2 b/ D: E* P

rm -f /etc/yum.repos.d/*

wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo

4 {( L: T' f4 g

wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo

' e& V3 y0 K3 ]7 ?) k& ?. i% ~9 _) n; u

sed -i '/aliyuncs.com/d' /etc/yum.repos.d/*.repo #删除阿里内网地址

#创建ceph源

echo '#阿里ceph源

[ceph]

1 z* T$ o6 Y& y' p4 O2 G. i

name=ceph

baseurl=http://mirrors.aliyun.com/ceph/rpm-luminous/el7/x86_64/

gpgcheck=0

: E' V' T1 ]  K' k2 a4 d( k

[ceph-noarch]

1 p) I* Q$ }0 y

name=cephnoarch

baseurl=http://mirrors.aliyun.com/ceph/rpm-luminous/el7/noarch/

gpgcheck=0

' T3 o' P* W. n

[ceph-source]

$ A  v6 |: i3 P8 f1 S

name=ceph-source

  d8 \- R! r3 P0 ?- E

baseurl=http://mirrors.aliyun.com/ceph/rpm-luminous/el7/SRPMS/

9 o- i8 {7 e9 O' w% o

gpgcheck=0

#'>/etc/yum.repos.d/ceph.repo

9 m( u* b: i0 h+ L& ]( F

yum clean all && yum makecache #生成缓存

#关闭selinux、防火墙

systemctl stop firewalld.service

) a5 n4 S5 G" F; r/ T; g. p

systemctl disable firewalld.service

6 B) D; j* @# c3 u  {

firewall-cmd --state

* b9 V" X9 ^1 Z: W' H

sed -i '/^SELINUX=.*/c SELINUX=disabled' /etc/selinux/config

4 F: k  S. o6 C/ {5 b

sed -i 's/^SELINUXTYPE=.*/SELINUXTYPE=disabled/g' /etc/selinux/config

0 k. ^) D, W6 X0 R

grep --color=auto '^SELINUX' /etc/selinux/config

setenforce 0

% H6 Q! ^- O+ K' g, }9 T1 {: @4 U
! g4 f3 J; i: M, y

二、只在node1上操作

7 j  e* f1 Y0 \+ i  W) m% |$ ]# G

三台服务器分别为node1 node2 node3   其中我们在node1上面安装ceph官方的自动化部署工具，也就是说我们只需要操作node1即可完成部署

6 V1 ]. B9 s& g) ^- z
6 a9 Z3 ?% q4 l3 G- q. U

#添加服务器主机名

. U; ]5 U* T/ A8 n. R0 y8 s% A

echo '

10.0.0.11    node1

10.0.0.22    node2

10.0.0.33    node3

8 @4 z' H0 t7 k/ ]! \

10.0.0.44    node4

8 L. ^* r/ y9 s

10.0.0.55    node5

10.0.0.66    node6

'>>/etc/hosts

: M) Q" R* Z' g$ J

#创建秘钥

ssh-keygen

* U' M$ @% y! i1 X

ssh-copy node2

( M: m" w; ?! y9 [' t

ssh-copy node3

0 M2 p: @; W# \/ B

#将node1的hosts文件发送到其他服务器上面

scp /etc/hosts node2:/etc

' d+ U1 e) ^" _+ O: |

scp /etc/hosts node3:/etc

$ V: U7 t+ @& z# e+ `
( r, Q1 z/ }0 u! Q

三、开始在node1上安装ceph

#安装ceph-deploy配置工具

" R! c" J) h) u% |8 P, M- U

yum install -y ceph-deploy

yum install python-setuptools

2 G0 Z, o  V  F, i6 o" X

#创建配置目录

6 h1 T6 y0 |6 V4 i( d# c, Q  N* m

mkdir /etc/ceph

cd /etc/ceph/

# U1 [" o1 Y0 W9 g8 y8 I4 @; _

#初始化Mon配置

ceph-deploy new node1

% D+ g! i5 M  @2 |

# #修改冗余份数为2，日志大小2G

- k, |! @1 A( s, F4 |( ]3 v# j

# #配置网络,单网卡忽略

9 z( D# q7 u& r' Q9 I5 i6 A5 p# r

echo '

+ ~7 V6 g6 y& h8 q) i) ~; t& z

public network = 10.0.0.0/24

cluster network = 10.0.0.0/24

7 {( W1 z! b3 @8 F# F( Y
- _. _1 X; p! c2 e; L( D8 b

mon_clock_drift_allowed = 2   

" G2 Y' w+ \4 v. v. k* G

osd_journal_size = 4086

osd_pool_default_pg_num = 128

osd_pool_default_pgp_num = 128

osd pool default size = 2

osd pool default min size = 1

5 C/ {5 e# @- c. s6 U

rbd_default_features = 1

client_quota = true

6 W( p  S/ z; F: b3 B* s

'>>./ceph.conf

#安装Ceph

ceph-deploy install node1 node2 node3

* K) J0 _0 c' z) e- b

#yum install -y ceph ceph-radosgw #实际上是安装这2个rpm

9 W/ ]# i6 o- [+ y1 u7 Y
  m' i2 P; }& U+ I

#初始化monitor和key

& p' N( r" O! V

cd /etc/ceph/

4 n: c2 ?+ M0 U: h' _1 C

ceph-deploy --overwrite-conf mon create-initial

0 t5 ?: |- ^; O: p0 E- k0 Y0 l

#创建osd

: z, M6 t# [/ M1 u* e2 }4 e

ceph-deploy osd create --data /dev/sdb node1

ceph-deploy osd create --data /dev/sdc node1

# w2 F2 s1 j6 Q2 {9 Z5 t

ceph-deploy osd create --data /dev/sdd node1

ceph-deploy osd create --data /dev/sdb node2

ceph-deploy osd create --data /dev/sdc node2

' L  T$ N2 b, B7 e: Q: r6 V. ?$ T7 x( ]* C

ceph-deploy osd create --data /dev/sdd node2

ceph-deploy osd create --data /dev/sdb node3

ceph-deploy osd create --data /dev/sdc node3

ceph-deploy osd create --data /dev/sdd node3

#拷贝配置及密钥

ceph-deploy admin node1 node2 node3

chmod 644 /etc/ceph/ceph.client.admin.keyring

: t- Y* q! E. |" _5 C  r

#创建mon

ceph-deploy --overwrite-conf mon create node1

ceph-deploy --overwrite-conf admin node1

  V3 ]( ^* C+ \4 Q* R* U) p5 u
/ t/ j6 Z: J9 O  j% C9 x! ?/ C' W

#添加mgr

#ceph 12开始，monitor必须添加mgr

ceph-deploy mgr create node1:mon_mgr

/ Y0 H2 k6 E1 K# e& a

#启用dashboard (在node1节点)

ceph mgr module enable dashboard

+ f6 z5 |, c! b+ K* x# v
& j4 K/ M$ w9 l1 V* ]2 m

web访问http://10.0.0.11:7000   即可看到ceph的运行情况

' o: R, V6 P3 V2 }: G3 e* |" w

#查看相关命令

. I. J8 R1 N6 b

ceph health

ceph -s

ceph osd tree

0 O. B0 }4 U4 f; @

ceph df

, G: X1 y4 }0 e8 M: o0 w

ceph mon stat

! `  x) ^& S. U4 i. k

ceph osd stat

" P& a" I; g; g, Z; k% {

ceph pg stat

ceph osd lspools

) C8 V% U- [0 y: f( s. O3 D( P

ceph auth list

0 `+ C! N/ F1 B: m3 X% Q0 F) e. ^
* S/ M2 X5 U! P' V

Ceph L版与openstack Pike对接

& d. E' {. f6 \5 m
8 |2 X" N9 I) |! r# A

环境opestack Pike的controller、computer、cinder三个节点已经安装完毕，并且能够正常的使用

3 b) Z1 \: O! b5 f
( ^8 n, o) ]/ O- `/ V7 ?, C3 q  d: z$ }+ E

node4为controller

; G/ T- C& a; V* y

node5为computer和cinder

; T" O7 c' \9 K) p( b8 g2 X9 D3 r

一、环境准备，所有openstack主机都要操作

#所有openstack节点都添加主机名

#添加服务器主机名

" A0 j8 G- k4 T  e% @

echo '

& ]9 z+ \  {3 s! z$ b- {

10.0.0.11    node1

, s( B/ G+ N$ e: r6 W

10.0.0.22    node2

10.0.0.33    node3

8 y+ ^9 Y6 @! z3 m' g% a: i7 j

10.0.0.44    node4

) z* c+ w5 A" c' J3 A

10.0.0.55    node5

3 O6 a  O! L) x( x( Q1 d2 F2 P

10.0.0.66    node6

'>>/etc/hosts

#创建ceph源

echo '#阿里ceph源

' E3 R$ V" V3 x7 C' ~* ]

[ceph]

1 ^) o& W! \0 ?

name=ceph

baseurl=http://mirrors.aliyun.com/ceph/rpm-luminous/el7/x86_64/

1 H$ X2 @8 `$ J* Y3 ~5 d

gpgcheck=0

) O$ e4 c1 Y+ J; L' j4 l6 }

[ceph-noarch]

. }# G: l9 a5 }1 o$ i1 W

name=cephnoarch

0 \5 o% w# K8 n* e7 H

baseurl=http://mirrors.aliyun.com/ceph/rpm-luminous/el7/noarch/

0 p4 L4 S4 K. P& P, Z) b

gpgcheck=0

. u% V! D/ M6 Q4 Y

[ceph-source]

name=ceph-source

baseurl=http://mirrors.aliyun.com/ceph/rpm-luminous/el7/SRPMS/

gpgcheck=0

3 y5 E; s) N. v: k( d. f" `9 `

#'>/etc/yum.repos.d/ceph.repo

yum clean all && yum makecache #生成缓存

#关闭selinux、防火墙

4 a  O2 B0 M5 U+ t

systemctl stop firewalld.service

  D; r. D- S: e  Z6 O+ z- c

systemctl disable firewalld.service

2 _4 P* A6 {+ E# d

firewall-cmd --state

7 E9 T* y& s9 A+ N+ S

sed -i '/^SELINUX=.*/c SELINUX=disabled' /etc/selinux/config

sed -i 's/^SELINUXTYPE=.*/SELINUXTYPE=disabled/g' /etc/selinux/config

6 J# C) J/ D9 m. I# r/ u) u* d

grep --color=auto '^SELINUX' /etc/selinux/config

setenforce 0

( D( g& O2 {$ m; d( k# o
6 m0 p& s" Y$ ?( h" v. Q

#将glance组件的节点安装ceph客户端

yum install python-rbd

: [5 H" C$ y8 ]% r$ X  z

#将nova和cinder组件的节点安装ceph客户端

yum install ceph-common

! ~! c  @" O8 q2 g  o

#创建POOL

ceph osd pool create volumes 128

: y& N7 M9 b# ^+ L% h

ceph osd pool create images 128

ceph osd pool create vms 128

! |0 w8 _& X2 ]9 z- a# M. m

#初始化POOL

& K$ s1 X- K8 o

rbd pool init volumes

rbd pool init images

0 n% G: @: @* R4 L8 [

rbd pool init vms

  D! W8 T4 l; o8 S+ ~. A

#将ceph的配置文件导入到各个openstack节点上

# t# t" `# x' n2 e' s! c' H

ssh node4 tee /etc/ceph/ceph.conf </etc/ceph/ceph.conf

ssh node5 tee /etc/ceph/ceph.conf </etc/ceph/ceph.conf

: X4 W: V; J$ I! f2 J  C+ x

#创建ceph用户和密钥

# l- Z& z' A0 d  H; Z3 C

ceph auth get-or-create client.glance mon 'profile rbd' osd 'profile rbd pool=images'

- |; m5 N4 n0 Y9 x

ceph auth get-or-create client.cinder mon 'profile rbd' osd 'profile rbd pool=volumes, profile rbd pool=vms, profile rbd-read-only pool=images'

( o) J  j  {- W/ _/ U

ceph auth get-or-create client.cinder-backup mon 'profile rbd' osd 'profile rbd pool=backups'

  J! v4 F/ z# ^& D9 h

#查询用户，写入文件

. [4 y# A2 m3 L

ceph auth get-or-create client.glance | ssh node4  tee /etc/ceph/ceph.client.glance.keyring

ssh node4 chown glance:glance /etc/ceph/ceph.client.glance.keyring

ceph auth get-or-create client.cinder | ssh node5  tee /etc/ceph/ceph.client.cinder.keyring

* n6 I+ d. m5 a. |! J

ssh node5  chown cinder:cinder /etc/ceph/ceph.client.cinder.keyring

ceph auth get-or-create client.cinder-backup | ssh node5  tee /etc/ceph/ceph.client.cinder-backup.keyring

3 D% N" L6 A/ ]5 r. g, ?

ssh node5  chown cinder:cinder /etc/ceph/ceph.client.cinder-backup.keyring

ceph auth get-key client.cinder | ssh node5 tee client.cinder.key

ceph与glance对接

# z1 L( z* k9 k) V: l, |7 o

#更改glance默认存储为ceph

& V$ Y" v( l& Q) @1 b

#修改/etc/glance/glance-api.conf文件，添加一下内容

1 n/ `4 Y% I  S$ [! ]

[DEFAULT]

default_store = rbd

show_image_direct_url = True

7 \6 ?1 Z' f; r+ P0 l  g
7 E' `" i! l- S

[glance_store]

#stores = file,http

. _% }$ [+ {- M' S: r* ]

#default_store = file

#filesystem_store_datadir = /var/lib/glance/images/

- ]  v5 [8 K# B* z$ [' G

stores = rbd

default_store = rbd

rbd_store_pool = images

rbd_store_user = glance

rbd_store_ceph_conf = /etc/ceph/ceph.conf

  v! n; K, }( g+ k; j" b( n2 J3 A

rbd_store_chunk_size = 8

#重启服务

systemctl restart openstack-glance-api openstack-glance-registry

- K$ j4 K) a- {: {6 z; ~8 k

注意：在glance节点你需要查看/etc/ceph目录下是否有glance秘钥

0 M& ?' @# ^( r' K  x6 B

ceph与nova对接，在computer节点上操作

#密钥加进libvirt

# L1 }6 m3 J; v) [' S& r0 x

uuidgen

8 K9 w' a9 C! G* G6 O9 l

cat > secret.xml <<EOF

+ s. N  ?) |$ I! Q: x7 n( s9 J

<secret ephemeral='no' private='no'>

  <uuid>ff2e1190-30f5-4849-9c1c-886b1e1ee181</uuid>

5 Q0 Y' O8 u6 q$ Z' W

  <usage type='ceph'>

: [1 k5 Z, b+ G+ ^8 u

    <name>client.cinder secret</name>

$ T& \2 O; m2 R! C- q( f

  </usage>

</secret>

EOF

% J' L+ \8 C; g/ ~6 q; G) J

virsh secret-define --file secret.xml

/ ?4 {2 X1 j, _' |. c! Q8 _! X: W) w

virsh secret-set-value --secret ff2e1190-30f5-4849-9c1c-886b1e1ee181 --base64 $(cat client.cinder.key) && rm client.cinder.key secret.xml

+ ~. k0 o" U. J

virsh secret-list

5 b/ M8 B4 p+ D% B3 u# Y1 A

UUID                                  用量

" T" f* W. }- i3 }. N

--------------------------------------------------------------------------------

ff2e1190-30f5-4849-9c1c-886b1e1ee181  ceph client.cinder secret

, m9 A& |& X' H5 C: B' {! H" A3 m

在computer节点的ceph配置文件里添加一下内容

ls -l /etc/ceph/

#ceph

1 p( }$ m, L" y7 D+ L( L

echo '

[client]

rbd cache = true

rbd cache writethrough until flush = true

admin socket = /var/run/ceph/guests/$cluster-$type.$id.$pid.$cctid.asok

log file = /var/log/qemu/qemu-guest-$pid.log

rbd concurrent management ops = 20

[client.cinder]

! f- i# c. J3 ?5 [

keyring = /etc/ceph/ceph.client.cinder.keyring

8 Y/ ^( a+ U& |" W# F

'>>/etc/ceph/ceph.conf

" W! W4 f, D0 {- g4 }# }1 A

mkdir -p /var/run/ceph/guests/ /var/log/qemu/

( X1 R8 _# g: D$ T

chown qemu:libvirt /var/run/ceph/guests /var/log/qemu/

#使用ceph存储，在/etc/nova/nova.conf添加以下内容

6 ?0 u9 B" D( D: M

[libvirt]

virt_type = qemu

images_type = rbd

! Q( V, b3 z3 e, V% D! d

images_rbd_pool = vms

images_rbd_ceph_conf = /etc/ceph/ceph.conf

rbd_user = cinder

rbd_secret_uuid = 'ff2e1190-30f5-4849-9c1c-886b1e1ee181'

# I/ i0 D( z4 ?4 `( ~. I

disk_cachemodes="network=writeback"

live_migration_flag="VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE,VIR_MIGRATE_PERSIST_DEST,VIR_MIGRATE_TUNNELLED"

libvirt_inject_password = false

libvirt_inject_key = false

* [8 Y4 i' s& @' G: R

libvirt_inject_partition = -2

6 w, M/ b6 U8 P  ~" b3 i! F$ T
) n' L! l! y0 _4 }  u) [1 C

#重启服务

systemctl restart libvirtd.service openstack-nova-compute.service

3 `4 |+ K$ X, h3 ]/ O, N
+ r% E( ~7 M/ N+ f8 U" L) ~
2 H* x3 X2 T) Q* ^* |

ceph与cinder对接

#修改/etc/cinder/cinder.conf配置文件，注意：将之前的lvm的配置都删掉

[DEFAULT]

+ n' P5 @6 r8 b1 v; K% g

#enabled_backends = lvm

& c( E6 K0 U8 h7 h$ ^

enabled_backends = ceph

[ceph]

' ]  J7 O2 W$ i* U

volume_driver = cinder.volume.drivers.rbd.RBDDriver

rbd_pool = volumes

rbd_ceph_conf = /etc/ceph/ceph.conf

$ O6 M2 U9 K  [* w/ n! S

rbd_flatten_volume_from_snapshot = false

9 W) Z! h/ f8 c6 v0 W6 v

rbd_max_clone_depth = 5

" g4 W8 O9 X* X6 e, D  g

rbd_store_chunk_size = 4

" K& j/ I+ p/ m7 i1 i4 t) U$ G2 _- R

rados_connect_timeout = -1

glance_api_version = 2

2 ^8 P2 C) ?6 f& o" `% k2 R

rbd_user = cinder

rbd_secret_uuid = ff2e1190-30f5-4849-9c1c-886b1e1ee181

. R" B; y9 I$ Y* j* B

#重启服务   

* W2 B! E2 z" S4 h+ Z- n

systemctl restart openstack-cinder-volume.service

: `0 L2 ~1 B, B; X% r4 `

admin

1、创建一个POOL
( i% l$ V5 j/ I8 b3 X* d2 o! CCeph的块设备默认使用“rdb” pool，但建议为Cinder和Glance创建专用的pool。
ceph osd pool create volumes 128
; t- Q" H9 [  b+ b0 s9 Sceph osd pool create images 128
ceph osd pool create backups 128
* M) T2 B7 w- S- w- R9 K0 tceph osd pool create vms 128
) e2 z3 x  y( K  A
2、配置OPENSTACK CEPH CLIENTS
环境的准备，需要事先在ceph管理节点到openstack各服务节点间建立起免密钥登录的关系，且需有使用sudo的权限。

安装ceph客户端软件包：
7 k3 S3 G3 Z; g9 j1 u6 a) _在glance-api节点：sudo yum install python-rbd
在nova-compute, cinder-backup and on the cinder-volume节点：sudo yum install  ceph （both the Python bindings and the client command line tools）

在OpenStack中运行glance-api, cinder-volume, nova-compute ，cinder-backup服务的主机节点，都属于Ceph的客户端。需要配置ceph.conf.
% Q/ z; k3 Y+ U2 z8 _使用以下命令把ceph.conf复制到每个ceph客户端节点：
3 N: M2 G  d* c7 p% J- r- Sssh {your-openstack-server} sudo tee /etc/ceph/ceph.conf </etc/ceph/ceph.conf
& e: Z& y! i9 e! u" [
* `. h3 T: w% ^* H: n  z3、设置Ceph Client Authentication
如果启用了cephx认证，那么就需要为Nova/Cinder创建一个新用户：
* {0 j  }# U- {. I' `0 c. j$ceph auth get-or-create client.cinder mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=volumes, allow rwx pool=vms, allow rx pool=images'
$ ceph auth get-or-create client.glance mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=images'
) r* Z6 X$ T( z  S3 i# f: c1 I$ ceph auth get-or-create client.cinder-backup mon 'allow r' osd 'allow class-read object_prefix rbd_children, allow rwx pool=backups'

. u0 L# f8 s9 o7 s将密钥（client.cinder, client.glance, and client.cinder-backup）分发至对应的主机节点，并调整属主权限：
ceph auth get-or-create client.glance | ssh {your-glance-api-server} sudo tee /etc/ceph/ceph.client.glance.keyring
- n0 \2 g7 @5 m) L' I- p1 ussh {your-glance-api-server} sudo chown glance:glance /etc/ceph/ceph.client.glance.keyring
* y+ d, S' x! H: O3 M; q9 ^ceph auth get-or-create client.cinder | ssh {your-volume-server} sudo tee /etc/ceph/ceph.client.cinder.keyring
ssh {your-cinder-volume-server} sudo chown cinder:cinder /etc/ceph/ceph.client.cinder.keyring
ceph auth get-or-create client.cinder-backup | ssh {your-cinder-backup-server} sudo tee /etc/ceph/ceph.client.cinder-backup.keyring
ssh {your-cinder-backup-server} sudo chown cinder:cinder /etc/ceph/ceph.client.cinder-backup.keyring

运行nova-compute的节点需要使用cinder密钥：
ceph auth get-or-create client.cinder | ssh {your-nova-compute-server} sudo tee /etc/ceph/ceph.client.cinder.keyring

3 \4 D( A2 m, f* I6 Wlibvirt同样也要使用client.cinder密钥：
' E) z" R* F2 A8 H7 ILibvirt进程在从Cinder挂载一个块设备时，需要使用该密钥访问Ceph存储集群。
先要在运行nova-compute的节点上创建一个密钥的临时拷贝：
ceph auth get-key client.cinder | ssh {your-compute-node} tee client.cinder.key
0 K% E2 d$ G( a' a; u7 U4 {
然后登录到compute节点上，将密钥增加到libvirt配置文件中并删除上面的临时文件：
0 Q- s# S. K3 j5 S" [5 }$ uuidgen
22003ebb-0f32-400e-9584-fa90b6efd874
5 }: X9 l+ p- R9 n
5 o! q; j( q0 Y* Bcat > secret.xml <<EOF
+ \( G9 b  X. }& K; |<secret ephemeral='no' private='no'>
  Y. u2 v" X- |! g" s- ]<uuid>22003ebb-0f32-400e-9584-fa90b6efd874</uuid>
<usage type='ceph'>
' i! o" K9 c; K" G5 @7 a<name>client.cinder secret</name>
</usage>
' q: [  U. @6 }+ m8 y3 x) `0 S</secret>
! x2 y' [9 U$ Q" L4 D0 y8 H- o7 z# u/ F/ jEOF
1 M, m8 r9 Q2 _0 L3 |# virsh secret-define --file secret.xml
#Secret 22003ebb-0f32-400e-9584-fa90b6efd874 created
4 y( F7 m2 T( ?+ Z: K, {( b# virsh secret-set-value --secret 22003ebb-0f32-400e-9584-fa90b6efd874 --base64 $(cat client.cinder.key) && rm client.cinder.key secret.xml
Secret value set
! y' @) p9 m- k; D( t! {3 [7 H
9 T: F) w& ~6 h7 ^- S为了便于管理，建议在有多个计算节点时，在上在的操作中使用相同的UUID。
& @) R  s, J. z. [
& R) v( U- O7 R
4、设置Glance集成Ceph
1 K- P5 d; K* n9 Y0 A# uJUNO，编辑/etc/glance/glance-api.conf：
[DEFAULT]
...
8 J3 X5 R3 R. c+ R  O( @default_store = rbd
1 v' b: E  |; i- x* v2 `2 `. H...
( |8 Q( W' b8 X. R0 G[glance_store]
stores = rbd
6 y- \8 j' f3 Y; r0 Nrbd_store_pool = images
rbd_store_user = glance
( |/ e$ M' m8 z* j0 ~rbd_store_ceph_conf = /etc/ceph/ceph.conf
rbd_store_chunk_size = 8

/ Q# Z# n3 [& t* O2 T: P# y- D; w: m: f启用copy-on-write功能：
$ f! x0 M. [- d- P' Z( S在[DEFAULT]中增加以下参数，
show_image_direct_url = True

怎样关闭Glance的镜像缓存：
修改以下参数如以下所示，
3 I* J3 a- w; V  `[paste_deploy]flavor = keystone

- s" {& u* K3 i. Y9 b其它建议设置的Glance参数：
- hw_scsi_model=virtio-scsi: add the virtio-scsi controller and get better performance and support for discard operation
- hw_disk_bus=scsi: connect every cinder block devices to that controller
$ l5 ~0 |* w* y- hw_qemu_guest_agent=yes: enable the QEMU guest agent
- os_require_quiesce=yes: send fs-freeze/thaw calls through the QEMU guest agent

OpenStack官网配置参考：
5 J3 `4 N7 ]- h4 p$ phttp://docs.openstack.org/libert ... -image-service.html
5 S: I4 Y6 T3 n& C& u6 @& q) ?% q
: C4 g- y* z$ N. M5、设置Cinder集成Ceph
编辑/etc/cinder/cinder.conf：
3 {0 H- J& A# `1 i  Z# h7 k[DEFAULT]
...
enabled_backends = ceph
...
! X1 k: b: I# Q, T7 D1 Z4 T2 W[ceph]
0 U5 b& @' ?, w% Q# P5 bvolume_driver = cinder.volume.drivers.rbd.RBDDriver
& B+ d6 g5 }% b4 A' _/ v5 w/ F) X; a4 yrbd_pool = volumes
rbd_ceph_conf = /etc/ceph/ceph.conf
rbd_flatten_volume_from_snapshot = false
rbd_max_clone_depth = 5
  R; |' n. K$ D; `% `rbd_store_chunk_size = 4
rados_connect_timeout = -1
* Q2 J2 Q6 ?4 Cglance_api_version = 2
! Y  V% o" R4 B5 R' P
在启用了cephx认证时，还需要配置认证信息：
[ceph]
...
rbd_user = cinder
" y1 ?  x  \" U' I6 c. a' E; Brbd_secret_uuid = 22003ebb-0f32-400e-9584-fa90b6efd874

注：如果你需要配置多个cinder back ends，一定要在 [DEFAULT] 部分设置glance_api_version = 2 。

下表来自OpenStack官网Liberty的配置文档：
http://docs.openstack.org/libert ... ent/ceph-rados.html
3 Q  ]) p  o2 |Description of Ceph storage configuration options
- l' ~! b/ ?0 K  \, g  o& [4 WConfiguration option = Default value        Description
" R$ ]# ^+ I# Y! V; N; y$ h  @1 K( I[DEFAULT]
rados_connect_timeout = -1        (IntOpt) Timeout value (in seconds) used when connecting to ceph cluster. If value < 0, no timeout is set and default librados value is used.
# u( `. E5 m# a; Z) U4 x; ^rados_connection_interval = 5        (IntOpt) Interval value (in seconds) between connection retries to ceph cluster.
% p0 a3 }8 |( mrados_connection_retries = 3        (IntOpt) Number of retries if connection to ceph cluster failed.
5 g6 P0 d: u5 l+ i9 R9 zrbd_ceph_conf =        (StrOpt) Path to the ceph configuration file
4 L& y0 i( B, Xrbd_cluster_name = ceph        (StrOpt) The name of ceph cluster
rbd_flatten_volume_from_snapshot = False        (BoolOpt) Flatten volumes created from snapshots to remove dependency from volume to snapshot
( \) R1 {  m/ w4 i- [7 Trbd_max_clone_depth = 5        (IntOpt) Maximum number of nested volume clones that are taken before a flatten occurs. Set to 0 to disable cloning.
- n( X$ ]$ ?  a" k; ?( Rrbd_pool = rbd        (StrOpt) The RADOS pool where rbd volumes are stored
$ a- h  v8 N& T* K' ?% M2 l- Prbd_secret_uuid = None        (StrOpt) The libvirt uuid of the secret for the rbd_user volumes
rbd_store_chunk_size = 4        (IntOpt) Volumes will be chunked into objects of this size (in megabytes).
rbd_user = None        (StrOpt) The RADOS client name for accessing rbd volumes - only set when using cephx authentication
5 y" c) o% l8 s* R& wvolume_tmp_dir = None        (StrOpt) Directory where temporary image files are stored when the volume driver does not write them directly to the volume. Warning: this option is now deprecated, please use image_conversion_dir instead.



6、设置Cinder Backup集成Ceph
OpenStack Cinder Backup需要一个专门的进程。在你的Cinder Backup节点上，编辑/etc/cinder/cinder.conf：
$ ~1 h. A, F' K+ r7 ~backup_driver = cinder.backup.drivers.ceph
" ^5 s, W: D: E5 i! a7 tbackup_ceph_conf = /etc/ceph/ceph.conf
backup_ceph_user = cinder-backup
backup_ceph_chunk_size = 134217728
" q2 l/ A# e' p: p3 K0 ~. k  abackup_ceph_pool = backups
backup_ceph_stripe_unit = 0
# V- a' X+ @, o7 q+ t& E3 L" ^& }2 Obackup_ceph_stripe_count = 0
restore_discard_excess_bytes = true

/ q; G2 V) w0 m' K: Q2 O& M" D9 A以下是来自OpenStack官网对Cinder Backup集成Ceph的配置说明：
  H. w( u8 f4 s+ UTo enable the Ceph backup driver, include the following option in the cinder.conf file:
1 X" e( w7 w0 H" k: o1 M! y
backup_driver = cinder.backup.drivers.ceph
The following configuration options are available for the Ceph backup driver.
; b! x7 p' p0 B$ c# K
Table 2.52. Description of Ceph backup driver configuration options
6 ]* G  o: ]8 A. D( DConfiguration option = Default value        Description
[DEFAULT]
backup_ceph_chunk_size = 134217728        (IntOpt) The chunk size, in bytes, that a backup is broken into before transfer to the Ceph object store.
backup_ceph_conf = /etc/ceph/ceph.conf        (StrOpt) Ceph configuration file to use.
, Y7 F! y! [/ D$ {5 _backup_ceph_pool = backups        (StrOpt) The Ceph pool where volume backups are stored.
- V" y4 m! o. O+ F1 [backup_ceph_stripe_count = 0        (IntOpt) RBD stripe count to use when creating a backup image.
; M- C6 V, a4 _8 N/ \! P0 Bbackup_ceph_stripe_unit = 0        (IntOpt) RBD stripe unit to use when creating a backup image.
7 Z, A3 i. y  L: vbackup_ceph_user = cinder        (StrOpt) The Ceph user to connect with. Default here is to use the same user as for Cinder volumes. If not using cephx this should be set to None.
restore_discard_excess_bytes = True        (BoolOpt) If True, always discard excess bytes when restoring volumes i.e. pad with zeroes.
This example shows the default options for the Ceph backup driver.

backup_ceph_conf=/etc/ceph/ceph.conf
backup_ceph_user = cinder
backup_ceph_chunk_size = 134217728
backup_ceph_pool = backups
( y0 h+ C2 d1 Y7 Ybackup_ceph_stripe_unit = 0
backup_ceph_stripe_count = 0

5 v& a" G5 R7 e; g7、设置NOVA集成Ceph
为了直接基于Ceph存储启动虚机，还需要为Nova配置一个临时的存储后端。同时，建议使用RBD缓存，启用admin socket。
admin socket可以通过以下方法访问：
% b; ?, |3 b# l$ @4 E$ e$ C     ceph daemon /var/run/ceph/ceph-client.cinder.19195.32310016.asok help

) @) ^; T. i6 h$ e6 z6 l在你的每个compute节点上，编辑Ceph配置文件：
. m" i5 c' I3 h- g  n) M[client]
rbd cache = true
* e: O1 y1 }8 Y0 Brbd cache writethrough until flush = true
admin socket = /var/run/ceph/guests/$cluster-$type.$id.$pid.$cctid.asok
log file = /var/log/qemu/qemu-guest-$pid.log
& g" y, \! a+ ]+ Rrbd concurrent management ops = 20
1 v# U2 r' e4 A) @7 u+ A
1 e5 p" n* i# @) S9 `调整权限：
mkdir -p /var/run/ceph/guests/ /var/log/qemu/
6 h. k9 D1 F6 F( e- zchown qemu:libvirt /var/run/ceph/guests /var/log/qemu/

注：以上的qemu用户和libvirt组是基于RedHat相关系统的。
4 S: ~; q% ?3 f, D
: i- l$ }2 c( k% m+ Q* p以配置好后，如果虚机已经在运行，则可以重启使上面的配置生效。

. n" x; o( r8 T% p1 `! m& EJUNO
9 {; I/ z! ^. y9 h0 ?在每个compute节点上编辑/etc/nova/nova.conf文件：
! }- @3 x1 n; ?9 A, D  `1 g! }! ?9 v[libvirt]
images_type = rbd
# H, t7 K4 a6 K4 g- u: himages_rbd_pool = vms
5 s' n$ R, K/ I& |images_rbd_ceph_conf = /etc/ceph/ceph.conf
" N0 u9 C: i; _6 I* Y- k9 f2 Nrbd_user = cinder
$ o. B. t  }. v: P+ lrbd_secret_uuid = 22003ebb-0f32-400e-9584-fa90b6efd874
disk_cachemodes="network=writeback"

6 a" h' p% Q$ a2 f4 P/ Q建议关闭nova的密钥注入功能，而是使用基于metadata服务、cloud-init实现类似功能：
% M% p" Z2 J( f5 Y- \& ]# x在每个计算节点上，编辑/etc/nova/nova.conf：
% V0 K! c6 F7 s( g. Kinject_password = false
# i* T, s' |8 Binject_key = false
inject_partition = -2

2 w% Z2 d3 f- b, I启动热迁移支持：
在[libvirt]部分增加：
live_migration_flag="VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE,VIR_MIGRATE_PERSIST_DEST,VIR_MIGRATE_TUNNELLED"

+ a: i4 Y4 Z9 E9 R/ y8、重启OpenStack服务
sudo service openstack-glance-api restart
sudo service openstack-nova-compute restart
+ V; C( n1 m+ n5 P# isudo service openstack-cinder-volume restart
9 [: ^( k* K. i+ k5 x- i/ a& Q8 Csudo service openstack-cinder-backup restart
2 T! S) ^  y/ q9 h7 K# @1 W