|
|
楼主 |
发表于 2018-9-26 09:01:34
|
显示全部楼层
Ceph 分布式存储集群有三大组件组成，分为：Ceph Monitor、Ceph OSD、Ceph MDS，后边使用对象存储和块存储时，MDS 非必须安装，只有当使用 Cephfs 文件存储时，才需要安装。这里我们暂时不安装 MDS。
2 r" P: g( ~6 R1 }9 G3 p3、Ceph 预检' Y4 ~. W( J* d; R6 u$ y, {7 L
3.1 配置节点Host7 s9 N0 E V" K" R8 B+ h
为了方便后边安装，以及 ssh 方式连接各个节点，我们先修改一下各个节点的 Hostname 以及配置 Hosts 如下：2 D( ?# }8 M/ G$ p0 _
admin-node (10.222.77.213)
+ [) B$ m- ]6 k# F
0 k; Y! C* Z, H) F' Z) o7 `; R; y; l$ cat /etc/hostname
6 Y. W, N {' ]admin
0 a9 @4 R" M4 U7 h9 y
8 o- G, f" f" Q/ r* B3 [$ cat /etc/hosts. G, a" T) g V0 f- {2 p! F
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4' _+ L# U+ b3 S$ r
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
, m) Y# G7 I2 I/ ]10.222.77.213 admin+ b& G: L" o5 z( O
10.222.77.242 node0
* f( `" @7 S2 D- U, {10.222.77.253 node1
+ u- D) Y& x/ [7 {. D7 W. U. Z1 _2 K c1 J7 i7 v S$ p
-----------------------------
7 M# B3 W3 r$ ~/ e/ N0 }# q W$ Gnode0（10.222.77.242） 3 h# z9 N2 v- R9 n9 g6 U% K" d) ^7 P
$ cat /etc/hostname * W. D7 p8 v0 Y- W- C9 U
admin h0 }. p0 [4 y8 Y0 S2 Z
' Y% o1 y& v! L$ cat /etc/hosts
+ n! I* [* A7 D- |, e+ v127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
) H5 V' P1 X& @6 P5 F::1 localhost localhost.localdomain localhost6 localhost6.localdomain6; R( E$ m# i1 K X ]
10.222.77.242 node0
% H3 M/ v9 e. u# Z: ^$ R" w10.222.77.213 admin
9 z& T! G4 q' b9 ]4 }/ g10.222.77.253 node1
5 R- u! X) k$ n& E
2 N, S9 s, o+ s-----------------------------
9 W2 c+ ]3 s' P' v; m" |) unode1（10.222.77.253）
" q6 t# m! @9 j2 D% T9 S3 q$ cat /etc/hostname
7 b1 s2 H' y! P0 b3 Radmin
$ E7 T. b7 F9 X3 L. d" h; D, [, T
; g. {! K1 ^. a) X$ cat /etc/hosts
9 w% Q& l/ Q% q. C9 V0 n127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
. O7 {+ N: }, e5 x::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
( L4 a) i1 h6 B* T4 S10.222.77.253 node1
% P ], V, c! ?& n; M: T4 U10.222.77.213 admin+ d: |& _3 x. x" U% r5 |2 y+ {
10.222.77.242 node01 @1 K! S* }6 c, I6 n- `
3.2 安装部署工具 ceph-deploy" _* v ~5 ^' X# w
Ceph 提供了部署工具 ceph-deploy 来方便安装 Ceph 集群，我们只需要在 ceph-deploy 节点上安装即可，这里对应的就是 admin-node 节点。把 Ceph 仓库添加到 ceph-deploy 管理节点，然后安装 ceph-deploy。因为系统是 Centos7 版本，所以配置如下：1 @" a& `! [8 ?/ U1 O
# ceph-deploy (admin-node) 上执行6 W/ @! _: d3 q" h4 I
$ w( b' U R0 M# yum 配置其他依赖包
' e5 S( X) W3 j+ R0 l u) t! x# a# n1 G$ sudo yum install -y yum-utils && sudo yum-config-manager --add-repo https://dl.fedoraproject.org/pub/epel/7/x86_64/ && sudo yum install --nogpgcheck -y epel-release && sudo rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 && sudo rm /etc/yum.repos.d/dl.fedoraproject.org*
! h" p" U8 J9 p( @- d( \! p' V& P4 _
# 添加 Ceph 源
8 ^$ v T1 F3 w3 V$ sudo vim /etc/yum.repos.d/ceph.repo" \2 f3 r5 r0 i
[Ceph-noarch]& I4 ^+ T0 P3 n- b3 d
name=Ceph noarch packages( [# y6 F' p& q5 T! t
baseurl=http://download.ceph.com/rpm-jewel/el7/noarch5 ?. p/ E. Y7 S( x4 }
enabled=1
8 ~1 F1 `! k( ?6 \gpgcheck=1
8 D* t; ~5 C4 I/ ltype=rpm-md
' g, x7 A C* ^& L O9 m5 J' {3 w7 V6 kgpgkey=https://download.ceph.com/keys/release.asc! u; D+ Q8 C. U
priority=18 m8 F, c, ~) ^; i0 d
3 Q3 u# L2 z) d9 `1 V2 Y% v# 安装 ceph-deploy( Q0 Y8 ?" U/ |2 g, X+ Y
$ sudo yum update && sudo yum install ceph-deploy
' ?6 ?5 H+ T* V3.3 安装 NTP 和 Openssh5 x8 C4 R+ \9 C3 s- m" F+ P
官方建议在所有 Ceph 节点上安装 NTP 服务（特别是 Ceph Monitor 节点），以免因时钟漂移导致故障。
+ L! g3 I% B; P( x& K, i# yum 安装 ntp" ]" O: }. k" P2 `3 A: Q
sudo yum install ntp ntpdate ntp-doc
$ a( C. ^7 O, T% k
' `% X* p4 |8 V0 G1 F# 校对系统时钟
7 ^+ j4 z! T! q( T; P& Intpdate 0.cn.pool.ntp.org
$ A. V/ ^, h& q; }% L; J. H后续操作，ceph-deploy 节点需要使用 ssh 方式登录各个节点完成 ceph 安装配置工作，所以要确保各个节点上有可用 SSH 服务。
: f8 Q5 [7 u$ |# r' }6 K# yum 安装 openssh# ?4 g4 u% L0 J& c- z
$ sudo yum install openssh-server- j, U5 j& }2 N3 X1 d( P3 u
5 Y! x4 W' I$ q, \( n# 查看 ssh 版本& i" s4 d, U2 Z5 S! I# k
$ ssh -V( U" U+ Z) H0 L: E+ _
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017
! |3 ^/ i) i0 D$ r. j' l3.4 创建 Ceph 部署用户- k4 g; K; Q; G1 o' T! [8 [
ceph-deploy 工具必须以普通用户登录 Ceph 节点，且此用户拥有无密码使用 sudo 的权限，因为它需要在安装软件及配置文件的过程中，不必输入密码。官方建议所有 Ceph 节点上给 ceph-deploy 创建一个特定的用户，而且不要使用 ceph 这个名字。这里为了方便，我们使用 cephd 这个账户作为特定的用户，而且每个节点上（admin-node、node0、node1）上都需要创建该账户，并且拥有 sudo 权限。4 V4 ?0 q) H' R* g+ B
# 在 Ceph 集群各节点进行如下操作
$ C5 c7 H l# X! f1 ?5 t/ j- g. O/ p- A% O
# 创建 ceph 特定用户
! \' L8 x, b( B$ }$ sudo useradd -d /home/cephd -m cephd* ]7 p: F# Q% e
$ sudo passwd cephd
- b, ?9 y. ^$ F$ q7 k* A9 X/ k2 Y8 W. W I2 {$ y( ?+ M8 {; ~
# 添加 sudo 权限% o2 e3 C/ S9 Q: K7 Y/ @
$ echo "cephd ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/cephd H# v" Y6 d3 i; B; z7 v$ a- T
$ sudo chmod 0440 /etc/sudoers.d/cephd
# E7 i5 |9 ?- p' d( {4 {接下来在 ceph-deploy 节点（admin-node）上，切换到 cephd 用户，生成 SSH 密钥并把其公钥分发到各 Ceph 节点上，注意使用 cephd 账户生成，且提示输入密码时，直接回车，因为它需要免密码登录到各个节点。
~7 `" i) E f x# ]3 Z" P# ceph-deploy (admin-node) 上执行
* y/ y, M% ~/ R$ ?+ S# {
Y# A2 d3 @" H$ F6 \# 生成 ssh 密钥 L0 v: W9 d7 B: w7 H3 J' z, \
$ ssh-keygen; z o$ V0 C3 ~: s/ N# \
Generating public/private rsa key pair.
u) j1 Q; M7 q; ~6 mEnter file in which to save the key (/home/cephd/.ssh/id_rsa): o* e; Q+ k, F$ D0 Y
Created directory '/home/cephd/.ssh'.
; x6 D( A5 C* J" Z' d" x9 F, H/ b' OEnter passphrase (empty for no passphrase):
# J2 @ @' r' F, h9 `9 zEnter same passphrase again:
, ^* E0 n: B" p# A4 v+ V( K X- bYour identification has been saved in /home/cephd/.ssh/id_rsa.6 {: X' e/ F) |$ v4 K
Your public key has been saved in /home/cephd/.ssh/id_rsa.pub.6 u: _* q4 K2 R0 v9 H9 d& }# m5 C5 P
The key fingerprint is:
, P0 O' \0 J$ n& d...
9 y" q1 f8 h' X/ }& s2 p
! G5 J% U$ l& `0 v- g. h# g# 将公钥复制到 node0 节点$ u0 h7 X% Z% H% l( S/ f
$ ssh-copy-id cephd@node0
& |& K# r* S8 j1 E/ Z/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/cephd/.ssh/id_rsa.pub"" A. ?) c( d/ ~4 m3 J& O+ \/ `$ Z3 T
The authenticity of host 'node0 (10.222.77.242)' can't be established.
1 g. H( T: r. A- u5 P7 s0 vECDSA key fingerprint is MD5:3c:e0:a7:a0:e6:3c:dc:c0:df:28:dc:87:16:2d:0f:c6.
2 N0 u: |7 t* I0 G) lAre you sure you want to continue connecting (yes/no)? yes
0 Q( V0 l5 g! ^1 b2 E, U. Y/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
: N3 o/ ]# p( C/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
0 G; R! \& }2 U1 ^) `1 g. ?cephd@node0's password: : G2 d; k8 `, N9 l( i% _. h
; B y+ C2 I5 h. p; P& ]
Number of key(s) added: 1
3 ]% p2 T; p' m/ V# V( b* B
9 S/ L9 ^; |/ ^7 bNow try logging into the machine, with: "ssh 'cephd@node0'"4 N" [2 H- t( @, k' z/ _; h. K- M
and check to make sure that only the key(s) you wanted were added.0 ~0 s c9 R' k. P5 M7 x
0 Q# c6 ^6 r( I8 }* k# 将公钥复制到 node1 节点: V# q9 k5 e$ ]" G& o7 s$ x5 [
$ ssh-copy-id cephd@node1
" ]$ N9 Z( V5 Q/ v, d; m1 {/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/cephd/.ssh/id_rsa.pub"
/ h3 Z+ y/ _( Z. v d' Y# v: zThe authenticity of host 'node1 (10.222.77.253)' can't be established. A& u5 F- G- U* @. c
ECDSA key fingerprint is MD5:3c:e0:a7:a0:e6:3c:dc:c0:df:28:dc:87:16:2d:0f:c6.
9 z' w/ g& x7 m# }' ~" t5 ]$ IAre you sure you want to continue connecting (yes/no)? yes
, b# U9 D' l9 _8 l/ i/ Q/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
) n$ g' ^# O4 A. }: M0 B l/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
) |* c# \; ]* G# kcephd@node1's password: . H9 g; q1 P0 V1 [/ a( d
* s7 |% C: l% y7 l6 L9 @Number of key(s) added: 1
# f. p* v0 \# C; X$ \
5 v9 ]5 p3 D% O+ Z9 X- zNow try logging into the machine, with: "ssh 'cephd@node1'"
4 H. |8 F1 c* R1 y$ h/ k# _and check to make sure that only the key(s) you wanted were added.. ^. W% B2 G, y6 g
复制完毕，测试一下在 ceph-deploy 管理节点免密码登录各个节点。
+ Z$ }; r* ?6 j0 p. `* n$ ssh node0
: b. ^/ O U6 z6 [5 ]Last login: Fri Dec 8 15:50:08 2017 from admin0 i/ p- t c3 \0 T
; n9 q R* a& X
$ ssh node1" j" b: o2 l' [% M
Last login: Fri Dec 8 15:49:27 2017 from admin
' M3 i& x. ]8 s测试没有问题，接下来，修改 ceph-deploy 管理节点上的 ~/.ssh/config 文件，这样无需每次执行 ceph-deploy 都要指定 –username cephd 。这样做同时也简化了 ssh 和 scp 的用法。5 @+ [' t$ w. E( _4 Q) p
$ cat ~/.ssh/config2 w% |6 J$ d0 r
Host node0/ N! b" Z. ?) \, B5 C8 h
Hostname node0
+ w4 { k: d3 U User cephd
" J7 J- L3 i# N p- PHost node1* f% n; O8 D% a: o8 m1 }! Z' n
Hostname node1
7 R0 I# d* ?* k User cephd/ M3 x8 Z+ d0 ], F% B% r
注意，此时再执行 ssh node0 会提示报错 Bad owner or permissions on /home/cephd/.ssh/config。原因是 config 文件权限问题，修改权限 sudo chmod 600 config 即可解决。
$ z/ O2 o) E9 w3 e! q1 x3.5 其他网络配置
3 s8 E$ J& }2 \! ~' s# b/ x6 l官网文档中指定 Ceph 的各 OSD 进程通过网络互联并向 Monitors 上报自己的状态，所以要保证网络为开启状态，不过某些发行版（如 CentOS ）默认关闭网络接口。所以我们需要保证集群各个节点系统网络接口是开启的。. L7 J# k. h/ q! c
# 在 Ceph 集群各节点进行如下操作5 }# W. i" f8 S- L6 E
- ]$ R0 a4 u/ L1 i+ d$ sudo cat /etc/sysconfig/network-scripts/ifcfg-enp0s3
1 J5 _# \ \2 P: S/ U; S% zTYPE="Ethernet"; _. h& H# `( {. @, b3 @
BOOTPROTO="dhcp"
! q, |9 c8 N* V( D( nDEFROUTE="yes"/ n. |7 Z9 K! y3 z- R. W
PEERDNS="yes"
6 b4 k+ T# U6 d- k) @PEERROUTES="yes"' x& R9 G% I' C6 f+ J3 { v
IPV4_FAILURE_FATAL="no"
+ f# x2 ^& ^: k' e4 g5 VIPV6INIT="yes"' e& v" e$ X- N' q5 v6 |. h
IPV6_AUTOCONF="yes"% M1 n, B* X4 z6 }8 X
IPV6_DEFROUTE="yes"4 m/ k+ ~) @& u' M+ U
IPV6_PEERDNS="yes"
2 d3 K k7 n Y" yIPV6_PEERROUTES="yes"
2 I1 T7 c/ `, d+ a. }1 m) J, VIPV6_FAILURE_FATAL="no"
+ l+ a, c- w# kIPV6_ADDR_GEN_MODE="stable-privacy"
, d" I1 B/ {" i0 n6 r0 }6 t. \- l+ t+ VNAME="enp0s3"
% @/ [/ k" V" }: p9 J2 EUUID="3e68d5a3-f9a6-4c83-9969-706f7e3b0bc2"3 y8 E0 d$ _5 _* I
DEVICE="enp0s3"
- F( @/ f# V+ Y6 mONBOOT="yes" # 这里要设置为 yes; J3 Z: p( w8 @3 m
注意：这里因为在我安装的虚拟机集群中网卡为 enp0s3，所以需要修改 /etc/sysconfig/network-scripts/ifcfg-enp0s3 文件，请根据自己系统网卡名去修改对应配置文件。
2 i% t7 G/ L9 Q4 t8 ISELINUX 设置，在 CentOS 系统上， SELinux 默认为 Enforcing 开启状态，为了方便安装，建议把 SELinux 设置为 Permissive 或者 disabled。
- N0 L" f" h( t7 X, a9 G# 在 Ceph 集群各节点进行如下操作( _4 @5 N- q7 p
4 o! b) h3 M/ z; O6 y2 p
# 临时生效设置
: ~* j- [0 I2 b$ w/ E! U0 O$ sudo setenforce 0
- [3 o1 v+ ~* ^- X4 ]# Y: }4 g+ x5 ~" c% b: t- D
# 永久生效设置# [8 q2 ?7 S/ z2 l" `. I8 r
$ sudo cat /etc/selinux/config
4 A" K& G! r$ w( S MSELINUX=disabled # 这里设置为 Permissive | disabled
" \' P- t+ ?. @: g% }SELINUXTYPE=targeted : H' m2 _3 g2 f8 x% N% u: \# P
开放所需端口设置，Ceph Monitors 之间默认使用 6789 端口通信， OSD 之间默认用 6800:7300 这个范围内的端口通信，所以我们需要调整防火墙设置，开放所需端口，允许相应的入站请求。 y! p/ o" p; v# D8 n) h% r5 k6 X
# 防火墙设置
( x" K/ p1 I& {( T1 }! M$ sudo firewall-cmd --zone=public --add-port=6789/tcp --permanent, y' h+ E- F/ B% _; ~7 X, ?2 q; b4 I
7 `5 m6 N$ V3 \8 d/ }( ?( J
# 当然我们也可以关闭防火墙
! B2 [1 f9 }, M* ^+ f& m$ sudo systemctl stop firewalld.service #停止 firewall
; a7 [6 t% B8 |7 |# C$ sudo systemctl disable firewalld.service #禁止 firewall 开机启动+ A8 C- m. Q" s$ o/ H+ a
4、Ceph 存储集群搭建
& ~/ r% {; l1 s) P3 t8 Y! ?* i好了，经过上边一系列的预检设置后，我们就可以开始 Ceph 存储集群搭建了，集群结构为 admin-node (ceph-deploy、Monitor)、node0（osd.0）、node1（osd.1）。首先要提一下的是，如果我们在安装过程中出现了问题，需要重新操作的时候，例如想清理我搭建的这个集群的话，可以使用以下命令。
8 N* r' H$ ^. e' [' X7 |$ f# ceph-deploy (admin-node) 上执行( b `4 t- q# Q) [2 {
2 s8 u7 n" {9 F; W" L) w2 Y# 清理配置
6 S. h) V+ U/ r% |ceph-deploy purgedata admin node0 node1# E3 C/ H% D- |8 e4 p9 P( }+ \" w
ceph-deploy forgetkeys1 m# V2 c `$ y: W
3 E6 K5 F( W% X
# 清理 Ceph 安装包
3 @: X' S @- w( d% J1 Kceph-deploy purge admin node0 node1* {' r( c5 d J6 o% L2 q% D6 u* T' k
好了，现在开始搭建。首先 Cephd 用户创建一个目录 ceph-cluster 并进入到该目录执行一系列操作。因为我们设计的 monitor 节点在 admin-node 节点上，所以，执行如下命令。
' w5 f& d( n. r4 F% [+ E3 M/ N, z# 创建执行目录
- i5 Y5 k; u J) }$ mkdir ~/ceph-cluster && cd ~/ceph-cluster
; d% A: k! `# J$ D' d7 D$ e
' m v- N+ t( ^7 f5 W# U: z2 N# 创建集群/ \7 w' m1 `8 Z0 V" L% ?4 m* w
$ ceph-deploy new admin2 S+ ^8 I0 l* Y0 A
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephd/.cephdeploy.conf
: C% o8 a0 ~' T& R[ceph_deploy.cli][INFO ] Invoked (1.5.39): /bin/ceph-deploy new admin
; D# f0 h; f1 p p[ceph_deploy.cli][INFO ] ceph-deploy options:
" K' r% `, u& c3 i[ceph_deploy.cli][INFO ] username : None
/ q+ m5 B$ h9 f4 w[ceph_deploy.cli][INFO ] func : <function new at 0xf24938>
3 M$ Q) n: z7 Q$ B[ceph_deploy.cli][INFO ] verbose : False
' ~7 H: O' h9 {: S/ b) Q! ^[ceph_deploy.cli][INFO ] overwrite_conf : False5 S* ~8 f: L8 L% q5 Q; Y
[ceph_deploy.cli][INFO ] quiet : False! m6 U# U4 H2 r# {6 F
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0xf1f6c8>
* k, n6 D: `6 C/ _[ceph_deploy.cli][INFO ] cluster : ceph* b8 s7 L+ k" @' w' @" `" Q$ W
[ceph_deploy.cli][INFO ] ssh_copykey : True
3 d$ h' k6 z$ h[ceph_deploy.cli][INFO ] mon : ['admin']
. b1 I" s$ B% |' g+ n, c8 d) y[ceph_deploy.cli][INFO ] public_network : None
4 L2 V3 q% s* t% X9 e$ S$ ~! ]9 ^[ceph_deploy.cli][INFO ] ceph_conf : None! o) ]3 G" m2 F! e4 p
[ceph_deploy.cli][INFO ] cluster_network : None
2 y4 O# _1 r, [9 B: I3 N. [ a. ~[ceph_deploy.cli][INFO ] default_release : False& [- x3 [* J `# D3 T* q1 w* O
[ceph_deploy.cli][INFO ] fsid : None% W& w! K; \7 I8 S, B3 H3 @
[ceph_deploy.new][DEBUG ] Creating new cluster named ceph
0 i( z; `; q$ t) @8 a0 c[ceph_deploy.new][INFO ] making sure passwordless SSH succeeds6 E! n5 ^, Q+ ~8 x, ?. Y
.../ W; v9 O4 u5 [; y/ n
[ceph_deploy.new][DEBUG ] Resolving host admin7 h y- g9 g$ d& `5 O0 D
[ceph_deploy.new][DEBUG ] Monitor admin at 10.222.77.213
8 C# c5 S' f5 Z- i* E- R[ceph_deploy.new][DEBUG ] Monitor initial members are ['admin']# X$ o) Z. c0 P7 p4 T1 p8 D
[ceph_deploy.new][DEBUG ] Monitor addrs are ['10.222.77.213']9 g0 z+ j$ m7 S/ P3 n3 k" r
[ceph_deploy.new][DEBUG ] Creating a random mon key...
& o- H# b0 ~7 i0 H! X0 Z7 D# w8 ][ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring...
! f6 ~7 J& z; O1 j9 }[ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf..." }& B1 g0 p! B0 P
此时，我们会发现 ceph-deploy 会在 ceph-cluster 目录下生成几个文件，ceph.conf 为 ceph 配置文件，ceph-deploy-ceph.log 为 ceph-deploy 日志文件，ceph.mon.keyring 为 ceph monitor 的密钥环。5 z5 g2 Q/ Y ?4 \2 T& A5 r
$ ll ceph-cluster q: y- M6 Z! d# H
-rw-rw-r--. 1 cephd cephd 196 12月 7 14:46 ceph.conf
: }# Y" Y; `% m% H$ C/ |$ J-rw-rw-r--. 1 cephd cephd 3694 12月 7 14:46 ceph-deploy-ceph.log2 M' v9 M6 t1 i: z$ E4 e3 r
-rw-------. 1 cephd cephd 73 12月 7 14:46 ceph.mon.keyring3 y5 L8 ~" t, U6 U! l0 U& `
/ d) w) l$ j9 b8 R( y; s8 M/ k! Y$ cat ceph.conf : C4 G$ r& C# c3 p! z' q: s
[global]
9 s y/ `! r, S7 F; cfsid = 363c3cf4-dba4-4f13-8ec5-ed93416f4e26
# |( h' o7 \0 N4 m% T: z+ h7 S3 w& emon_initial_members = admin
/ |6 S" g7 ~( i9 i3 Emon_host = 10.222.77.2135 r; _# T6 \$ z. Q3 F' g" v
auth_cluster_required = cephx5 t' u8 ?" R. b- F
auth_service_required = cephx1 M# Q I% p5 G& l: J: E, ]# \
auth_client_required = cephx0 m9 I% S {! h+ A6 {
接下来，我们需要修改下 ceph.conf 配置文件，增加副本数为 2，因为我们有两个 osd 节点。
6 p; {" m0 T8 c9 z& b. K$ cat ceph.conf " R' D2 s) V6 I* M L7 j
[global]
$ c1 W- L9 X# v6 a7 m1 P/ f3 l Wfsid = 363c3cf4-dba4-4f13-8ec5-ed93416f4e26# M( C7 F3 G' d; \% n
mon_initial_members = admin; `/ K+ L! U' W+ n6 n; i
mon_host = 10.222.77.2137 y( C/ }/ n; u' W
auth_cluster_required = cephx
- L7 `* \' p g) X4 bauth_service_required = cephx+ x" r3 R$ L& I- G0 r- d
auth_client_required = cephx+ O$ i! L0 ]# [: R: g. Q1 X
osd pool default size = 2 #增加默认副本数为 2
" K; @# L+ Q8 {& x$ r! o8 X然后，我们需要通过 ceph-deploy 在各个节点安装 ceph。5 q* v" ]" t$ {& A1 n# V; T0 E
$ ceph-deploy install admin node0 node1& K3 T- t6 W! z4 f
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephd/.cephdeploy.conf/ _2 M) h" h! H/ Q, R& i1 `) P
[ceph_deploy.cli][INFO ] Invoked (1.5.39): /bin/ceph-deploy install admin node0 node1, X% A& X# C$ |( t# G3 C
[ceph_deploy.cli][INFO ] ceph-deploy options:
B2 D4 s" M- g[ceph_deploy.cli][INFO ] verbose : False
7 z3 ?9 E7 a, x k( O5 U[ceph_deploy.cli][INFO ] testing : None
0 m3 @9 E% T7 u4 k( N[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0xde1c68>% b, I/ Y7 Y; e: h
[ceph_deploy.cli][INFO ] cluster : ceph
# A+ J* G M8 b& U. D- ^" ~3 e[ceph_deploy.cli][INFO ] dev_commit : None
) F3 |' H# E1 U[ceph_deploy.cli][INFO ] install_mds : False2 ~* v0 O; {8 N
[ceph_deploy.cli][INFO ] stable : None
1 J0 `9 _, |( f& k- z[ceph_deploy.cli][INFO ] default_release : False4 `" w; J/ V4 @' Q2 C
[ceph_deploy.cli][INFO ] username : None
3 Z+ T0 y" ^8 D; \) a/ X[ceph_deploy.cli][INFO ] adjust_repos : True
3 i% ~/ i) q' |8 D2 z+ H[ceph_deploy.cli][INFO ] func : <function install at 0xd5b140>
$ @* d1 q4 K1 }# p' Q7 G! ~0 W[ceph_deploy.cli][INFO ] install_mgr : False
L: d) |, {3 Y+ V5 l. [% F$ F[ceph_deploy.cli][INFO ] install_all : False- Q/ L9 D& G* A& T% N
[ceph_deploy.cli][INFO ] repo : False9 I) P: R3 h# B
[ceph_deploy.cli][INFO ] host : ['admin', 'node0', 'node1']8 S5 a$ ~3 {' K* z* x; Z+ e. w
[ceph_deploy.cli][INFO ] install_rgw : False, y0 _+ `( ^% {. c. U
[ceph_deploy.cli][INFO ] install_tests : False
& g1 ]" u9 J1 S9 J[ceph_deploy.cli][INFO ] repo_url : None
) N9 S/ c) R/ n, L$ ]1 C/ z[ceph_deploy.cli][INFO ] ceph_conf : None
$ t) a2 z5 ~! v/ {+ t# H[ceph_deploy.cli][INFO ] install_osd : False
2 {+ k4 U8 b& A2 L# }[ceph_deploy.cli][INFO ] version_kind : stable
" N! v3 _- U" b7 U) `[ceph_deploy.cli][INFO ] install_common : False$ {9 q4 S! j' c
[ceph_deploy.cli][INFO ] overwrite_conf : False6 v- u6 W" |6 Q+ E5 Q5 D
[ceph_deploy.cli][INFO ] quiet : False
& r a0 K4 o) U* e[ceph_deploy.cli][INFO ] dev : master1 q/ m; c$ H: Y0 @# L, X
[ceph_deploy.cli][INFO ] nogpgcheck : False% X9 Z2 `" w+ k$ a
[ceph_deploy.cli][INFO ] local_mirror : None5 U5 }% Y/ c4 a+ p
[ceph_deploy.cli][INFO ] release : None
1 O- u7 c0 i) k0 I0 y0 J[ceph_deploy.cli][INFO ] install_mon : False: `' r- G5 e, a( P9 F) y4 K
[ceph_deploy.cli][INFO ] gpg_url : None, {' u& ~: A8 H# d
[ceph_deploy.install][DEBUG ] Installing stable version jewel on cluster ceph hosts admin node0 node1
" P/ {0 a* y( Y, S$ w+ U, F[ceph_deploy.install][DEBUG ] Detecting platform for host admin .../ n }( U8 A& }5 q* M/ C1 ? K
...: w. r8 f; X2 n) C
此过程需要等待一段时间，因为 ceph-deploy 会 SSH 登录到各 node 上去，依次执行安装 ceph 依赖的组件包。5 h A9 P7 D% U8 S" c4 x; O+ o
漫长的等待安装完毕之后，接下来需要初始化 monitor 节点并收集所有密钥。
6 j9 S% y3 n: F7 c1 m$ k$ ceph-deploy mon create-initial
$ a$ N$ C7 _9 i5 A) t...
" h' ^- O+ V9 O4 r; `ceph_deploy.mon][ERROR ] RuntimeError: config file /etc/ceph/ceph.conf exists with different content; use --overwrite-conf to overwrite
( g7 K$ t l5 h) R1 i[ceph_deploy][ERROR ] GenericError: Failed to create 1 monitors
" k, V' t4 ~6 {8 _* ]+ o$ u...# b$ K2 K3 X* B9 U4 D6 m% v+ o
不过很遗憾，执行过程中报错了。查看原因应该是已经存在了 /etc/ceph/ceph.conf 配置文件了，解决方案就是加上 --overwrite-conf 参数，覆盖已存在的配置。
: f8 g" B* _0 y) w$ k( o+ y! s$ ceph-deploy --overwrite-conf mon create-initial' O4 j i R' b$ M7 r( V9 O; U! {; l
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephd/.cephdeploy.conf
2 }3 m: t- H0 m, O[ceph_deploy.cli][INFO ] Invoked (1.5.39): /bin/ceph-deploy --overwrite-conf mon create-initial
p( E; B$ C. @- }[ceph_deploy.cli][INFO ] ceph-deploy options:$ u1 V5 e) b) L3 D9 p( y
[ceph_deploy.cli][INFO ] username : None D# E% i0 {4 Q
[ceph_deploy.cli][INFO ] verbose : False
* y, a' u+ f0 A: W6 K, c8 R i5 k[ceph_deploy.cli][INFO ] overwrite_conf : True
8 n, Z+ U5 o( U5 e; i! x2 ^[ceph_deploy.cli][INFO ] subcommand : create-initial9 ]) _8 I3 ]$ ~! w6 i( o* U
[ceph_deploy.cli][INFO ] quiet : False
% N, r6 K& W2 P* n9 x% `, n[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x1d9e758>* w0 {' X& H; y9 N/ ~6 P
[ceph_deploy.cli][INFO ] cluster : ceph
* ?" J/ i& [1 U/ M" |+ U[ceph_deploy.cli][INFO ] func : <function mon at 0x1d8ea28>% v ~: z, u u9 M7 i
[ceph_deploy.cli][INFO ] ceph_conf : None
+ p4 c9 D' t" k0 ?5 _8 W[ceph_deploy.cli][INFO ] default_release : False ` }! [$ w& g7 l$ E' m! D
[ceph_deploy.cli][INFO ] keyrings : None
j8 b9 K @" M4 e$ w$ U[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts admin
( l0 Z- n' A( ~( y' L8 R[ceph_deploy.mon][DEBUG ] detecting platform for host admin ...
$ W5 ?( f$ w6 y! ?2 P) \" `...7 {0 l7 k/ n# h% Q
[admin][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --admin-daemon=/var/run/ceph/ceph-mon.admin.asok mon_status7 P- Z0 c+ t, c: _9 H
[admin][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-admin/keyring auth get client.admin
$ A# R3 F2 W* K0 `& F4 c[admin][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-admin/keyring auth get client.bootstrap-mds
6 x2 X: ]( f$ I T6 q. A* x( y, ~[admin][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-admin/keyring auth get client.bootstrap-mgr8 Z' }1 h, c3 Z# c! Y$ N9 n
[admin][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-admin/keyring auth get-or-create client.bootstrap-mgr mon allow profile bootstrap-mgr7 A* P0 E9 K* x8 `) e
[admin][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-admin/keyring auth get client.bootstrap-osd
! J% E4 B4 ]7 o7 N3 O[admin][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-admin/keyring auth get client.bootstrap-rgw
6 f( I7 A& w( i$ ~4 |8 S9 n7 v# x2 r[ceph_deploy.gatherkeys][INFO ] Storing ceph.client.admin.keyring: Q& N4 ~/ D% c. r7 X
[ceph_deploy.gatherkeys][INFO ] Storing ceph.bootstrap-mds.keyring: f: V x9 W) |% X
[ceph_deploy.gatherkeys][INFO ] Storing ceph.bootstrap-mgr.keyring7 L* U. C9 Q4 ]& M2 e
[ceph_deploy.gatherkeys][INFO ] keyring 'ceph.mon.keyring' already exists
) a8 z0 Z4 s& ]% R/ H, ][ceph_deploy.gatherkeys][INFO ] Storing ceph.bootstrap-osd.keyring
+ K2 |/ L- l# @- ?1 I0 u[ceph_deploy.gatherkeys][INFO ] Storing ceph.bootstrap-rgw.keyring
1 V) k; S$ @+ n4 I8 t[ceph_deploy.gatherkeys][INFO ] Destroy temp directory /tmp/tmpLv4mug: w5 I) N! @, ~
执行完毕后，会在当前目录下生成一系列的密钥环，应该是各组件之间访问所需要的认证信息吧。
: A7 k& H2 r! R: {- b7 e2 U8 i$ ll ~/ceph-cluster; {, m1 R' o* \1 @+ L
-rw-------. 1 cephd cephd 113 12月 7 15:13 ceph.bootstrap-mds.keyring
" {/ X$ w" K, _) k: k) v2 {9 e0 Y-rw-------. 1 cephd cephd 71 12月 7 15:13 ceph.bootstrap-mgr.keyring
2 T, H4 ]; m, g-rw-------. 1 cephd cephd 113 12月 7 15:13 ceph.bootstrap-osd.keyring# v; {# i2 _2 K* S1 ]" ^
-rw-------. 1 cephd cephd 113 12月 7 15:13 ceph.bootstrap-rgw.keyring: o, V* ^3 @" [$ O9 {2 J; g
-rw-------. 1 cephd cephd 129 12月 7 15:13 ceph.client.admin.keyring8 A/ u Z5 s. W2 C. t$ }
-rw-rw-r--. 1 cephd cephd 222 12月 7 14:47 ceph.conf
& w6 U6 {8 O4 G# K1 v+ u, o3 b7 \-rw-rw-r--. 1 cephd cephd 120207 12月 7 15:13 ceph-deploy-ceph.log
& Y- d# J9 Z: f, P-rw-------. 1 cephd cephd 73 12月 7 14:46 ceph.mon.keyring
- t" T8 T2 b, }到此，ceph monitor 已经成功启动了。接下来需要创建 OSD 了，OSD 是最终数据存储的地方，这里我们准备了两个 OSD 节点，分别为 osd.0 和 osd.1。官方建议为 OSD 及其日志使用独立硬盘或分区作为存储空间，不过本机虚拟机上不具备条件，但是我们可以在虚拟机本地磁盘上创建目录，来作为 OSD 的存储空间。
1 U% t7 F2 z) I, P4 e$ a) t2 ~& g# ceph-deploy (admin-node) 上执行# J, X- _# \# }
: h* {$ z6 W9 P( D+ g$ ssh node07 i' v( C! u* i: W
$ sudo mkdir /var/local/osd0
# U% z, L0 M2 T' {# K$ x, x0 @# r' t$ sudo chown -R ceph:ceph /var/local/osd0
/ Y! k. {+ s$ M# _$ exit- ~5 X) B. |9 L" T% Y
$ e' E$ T+ y) K/ `, `" T' A$ ssh node1
# \) T! M$ t2 |' ^3 i; f! u8 o3 l$ sudo mkdir /var/local/osd1% i; {* h/ y/ d* e; \9 L
$ sudo chown -R ceph:ceph /var/local/osd10 i. r9 @1 Q3 \" Q9 U$ U6 ^( ]
$ exit! l: l3 p$ _; r4 k
注意：这里执行了 chown -R ceph:ceph 操作，将 osd0 和 osd1 目录的权限赋予 ceph:ceph，否则，接下来执行 ceph-deploy osd activate ... 时会报权限错误。 j& ?+ ]$ X. U
接下来，我们需要 ceph-deploy 节点执行 prepare OSD 操作，目的是分别在各个 OSD 节点上创建一些后边激活 OSD 需要的信息。 9 L7 r1 X. E N2 V" u! }# T4 z- ~/ @
$ ceph-deploy --overwrite-conf osd prepare node0:/var/local/osd0 node1:/var/local/osd1# m+ R( e/ K/ y* M0 i2 |
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephd/.cephdeploy.conf% K1 @/ t4 Y' p" f' h0 Y7 q9 K; \0 J1 P
[ceph_deploy.cli][INFO ] Invoked (1.5.39): /bin/ceph-deploy --overwrite-conf osd prepare node0:/var/local/osd0 node1:/var/local/osd1
3 [5 M, X4 ~2 @. }[ceph_deploy.cli][INFO ] ceph-deploy options:
% v& k# S. V* n9 K, L" D" c" N[ceph_deploy.cli][INFO ] username : None
- p7 t9 S; e' g& C1 V# ^; a[ceph_deploy.cli][INFO ] block_db : None
9 T9 { I& E' I' O+ W9 N, O4 R& o[ceph_deploy.cli][INFO ] disk : [('node0', '/var/local/osd0', None), ('node1', '/var/local/osd1', None)]
* d' T; p- s, V[ceph_deploy.cli][INFO ] dmcrypt : False S9 p4 I/ U+ r, ]' k9 u) C2 |
[ceph_deploy.cli][INFO ] verbose : False; `* _& Z" ~* V$ e( O
[ceph_deploy.cli][INFO ] bluestore : None
0 @% I. ^; a: | g' c$ j4 W[ceph_deploy.cli][INFO ] block_wal : None
; B2 u! H" }+ j' g. ~7 i1 `, m7 q[ceph_deploy.cli][INFO ] overwrite_conf : True/ U' d* A" r, ^& }: t$ d- c
[ceph_deploy.cli][INFO ] subcommand : prepare
* m9 }0 v* }0 S+ j6 w[ceph_deploy.cli][INFO ] dmcrypt_key_dir : /etc/ceph/dmcrypt-keys0 n* A' m+ {! k0 O
[ceph_deploy.cli][INFO ] quiet : False$ L3 K% B! E. D, M7 Y1 D
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x117d950>
) g0 `' J5 R$ x$ s[ceph_deploy.cli][INFO ] cluster : ceph) }+ O* t3 _' L- s* w) l$ l
[ceph_deploy.cli][INFO ] fs_type : xfs2 q1 w, f0 r4 ~2 a, k" a
[ceph_deploy.cli][INFO ] filestore : None& L" ^4 b* w. g2 M/ `
[ceph_deploy.cli][INFO ] func : <function osd at 0x116a320>
; w( _, D2 x( A[ceph_deploy.cli][INFO ] ceph_conf : None* K5 @6 x% x# F) M3 H. r4 x: J
[ceph_deploy.cli][INFO ] default_release : False
: I2 O& e2 A4 i: f3 v, N% b[ceph_deploy.cli][INFO ] zap_disk : False
) E& x8 f, g0 |6 ~! u! \' y[ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks node0:/var/local/osd0: node1:/var/local/osd1:
: r. [, t- N( Z2 |& W[node0][DEBUG ] connection detected need for sudo
( E9 W& V0 L$ N, d2 J0 ^[node0][DEBUG ] connected to host: node0 8 S* w2 E& G5 [: W
[node0][DEBUG ] detect platform information from remote host
0 b5 d6 B. y: \[node0][DEBUG ] detect machine type w! S. R. \# e c+ [8 w% b
[node0][DEBUG ] find the location of an executable: R' E2 A3 |/ y0 L4 p, t( A
[ceph_deploy.osd][INFO ] Distro info: CentOS Linux 7.3.1611 Core6 f& w( L; v: E1 U
[ceph_deploy.osd][DEBUG ] Deploying osd to node0
+ Y4 [! V: _) c...
2 g& `7 A& B1 {% q h0 a+ ?, s[node0][INFO ] checking OSD status...
4 t$ A) q- f2 w1 v. S[node0][DEBUG ] find the location of an executable+ Z0 p2 t3 f* |" Z* S! J5 L6 S4 S
[node0][INFO ] Running command: sudo /bin/ceph --cluster=ceph osd stat --format=json2 t- t1 R0 W- c
[ceph_deploy.osd][DEBUG ] Host node0 is now ready for osd use.' U3 Q( S) a- C: z
[node1][DEBUG ] connection detected need for sudo
) e" n9 E! G/ g* | L# d5 f[node1][DEBUG ] connected to host: node1 o. a: o! V" j: X7 f7 J; E
[node1][DEBUG ] detect platform information from remote host2 G$ ?% K# s6 E4 |" A
[node1][DEBUG ] detect machine type
5 K1 D0 A$ M6 F: I2 {[node1][DEBUG ] find the location of an executable
7 m D' T* s% F[ceph_deploy.osd][INFO ] Distro info: CentOS Linux 7.3.1611 Core2 v9 F7 f% l% c1 V
[ceph_deploy.osd][DEBUG ] Deploying osd to node1
; Q+ |9 m3 F6 t..., {$ p5 |' B+ S U+ U& U# R, y
[node1][INFO ] checking OSD status...
8 |$ R0 `5 f& n, W- `6 O ?[node1][DEBUG ] find the location of an executable
! c! t. I- n% r* {. A[node1][INFO ] Running command: sudo /bin/ceph --cluster=ceph osd stat --format=json, C7 W" Q5 j+ n/ R
[ceph_deploy.osd][DEBUG ] Host node1 is now ready for osd use.
- A; V) {: q5 N) x& S" {OK 接下来，我们需要激活 activate OSD。, }7 Y6 ~* L8 I# T" k
$ ceph-deploy osd activate node0:/var/local/osd0 node1:/var/local/osd1
* P0 a' R% G* W. t3 N8 M' N6 f1 Y[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephd/.cephdeploy.conf& Y* q1 S5 ~% V/ T9 D" l0 n
[ceph_deploy.cli][INFO ] Invoked (1.5.39): /bin/ceph-deploy osd activate node0:/var/local/osd0 node1:/var/local/osd1) y+ n% d& r% W$ D2 [
[ceph_deploy.cli][INFO ] ceph-deploy options:
) @/ w$ l7 G" ?! I4 O[ceph_deploy.cli][INFO ] username : None4 B6 M! {1 G/ u% f$ e z
[ceph_deploy.cli][INFO ] verbose : False
1 F g) \* p; F1 |; G[ceph_deploy.cli][INFO ] overwrite_conf : False7 b4 F3 t m6 R1 l( t
[ceph_deploy.cli][INFO ] subcommand : activate
7 v, f. b1 W0 p/ d) T8 ?- k[ceph_deploy.cli][INFO ] quiet : False" y0 d, e8 O- ?6 Z4 Q
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x2045950>
" S/ a3 [/ T# C6 T" s[ceph_deploy.cli][INFO ] cluster : ceph1 E x; Q U7 u+ }- @9 v7 h/ V
[ceph_deploy.cli][INFO ] func : <function osd at 0x2032320>
% x3 W; [1 h8 u7 @2 e# R" A1 A% q! Q[ceph_deploy.cli][INFO ] ceph_conf : None- [( {- y: H x( X3 u3 R; |! m7 n
[ceph_deploy.cli][INFO ] default_release : False9 ]$ M, h7 x* T3 X+ t! y; t& y
[ceph_deploy.cli][INFO ] disk : [('node0', '/var/local/osd0', None), ('node1', '/var/local/osd1', None)], p8 u3 j! F2 H, s7 e
[ceph_deploy.osd][DEBUG ] Activating cluster ceph disks node0:/var/local/osd0: node1:/var/local/osd1:, P- X9 X0 N6 h. G6 `/ x9 K( I
[node0][DEBUG ] connection detected need for sudo) t: H/ X" S6 X3 g- N c# W4 t3 p
[node0][DEBUG ] connected to host: node0 6 i1 J0 ~( a& g) m) ^9 ^' P# v
[node0][DEBUG ] detect platform information from remote host
# l/ K- Y- t# u. g0 v9 ][node0][DEBUG ] detect machine type
; g6 P, f6 S! l5 k. L[node0][DEBUG ] find the location of an executable7 j5 S1 h8 v. q4 ?7 k
[ceph_deploy.osd][INFO ] Distro info: CentOS Linux 7.3.1611 Core
: {7 U6 c$ L0 Y( s6 [! l[ceph_deploy.osd][DEBUG ] activating host node0 disk /var/local/osd0' ?( n# d# b' X( G/ F a
[ceph_deploy.osd][DEBUG ] will use init type: systemd7 p- j6 {( L; k$ F, x( u
...
% u+ g% e7 b& D6 q[node0][INFO ] checking OSD status...
3 {0 P; ?0 v( l/ f% m0 X[node0][DEBUG ] find the location of an executable6 T. @" V( [5 }2 Q
[node0][INFO ] Running command: sudo /bin/ceph --cluster=ceph osd stat --format=json
& B3 W. ^9 P9 q# H. n: H[node0][INFO ] Running command: sudo systemctl enable ceph.target5 p: W9 I Z; J- Q+ R
[node1][DEBUG ] connection detected need for sudo) C& [( V0 C% J! W6 c# l: {
[node1][DEBUG ] connected to host: node1
: u8 ]3 W$ H6 G: h& |9 p5 W3 z[node1][DEBUG ] detect platform information from remote host) _3 R0 N2 W, w+ g H- A
[node1][DEBUG ] detect machine type% \/ Y! F' h2 y" X& T
[node1][DEBUG ] find the location of an executable2 Z4 O0 m+ G! o5 A5 w5 n
[ceph_deploy.osd][INFO ] Distro info: CentOS Linux 7.3.1611 Core6 f8 M: k# k) O: D( A! M2 S
[ceph_deploy.osd][DEBUG ] activating host node1 disk /var/local/osd1
! L7 r! K* s/ H[ceph_deploy.osd][DEBUG ] will use init type: systemd
# n3 z) |0 K/ E...' e N5 t% d- u6 ], G: p0 F' m
[node1][INFO ] checking OSD status...
# b3 i0 s. A- Z5 X[node1][DEBUG ] find the location of an executable
3 Q: a" S! i7 L k9 J[node1][INFO ] Running command: sudo /bin/ceph --cluster=ceph osd stat --format=json! k0 P% V% c! H9 g0 ~
[node1][INFO ] Running command: sudo systemctl enable ceph.target
+ H" Q% W7 Z& B! g( _看日志，激活也没有问题，最后一步，通过 ceph-deploy admin 将配置文件和 admin 密钥同步到各个节点，以便在各个 Node 上使用 ceph 命令时，无需指定 monitor 地址和 ceph.client.admin.keyring 密钥。8 _1 U7 T# X8 _" @/ P4 n
$ ceph-deploy admin admin node0 node1
8 I/ d5 S8 |. Q# k[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephd/.cephdeploy.conf- X/ h: Y q( t7 C1 F
[ceph_deploy.cli][INFO ] Invoked (1.5.39): /bin/ceph-deploy admin admin node0 node12 q1 m. W( I) {0 c4 e' ^
[ceph_deploy.cli][INFO ] ceph-deploy options:
1 I9 l7 p2 W- S d7 x[ceph_deploy.cli][INFO ] username : None
$ m: e2 l% F3 p, u: H* e% C y[ceph_deploy.cli][INFO ] verbose : False! V) y9 T: o0 a4 N O$ d
[ceph_deploy.cli][INFO ] overwrite_conf : False
5 X/ ]: V/ }/ Z; I, x1 u: y[ceph_deploy.cli][INFO ] quiet : False
# k0 T& R: f# d/ X# j1 C. M[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x15a93b0>
& u% b& \2 T# C[ceph_deploy.cli][INFO ] cluster : ceph
! n3 e, U: J( s1 S[ceph_deploy.cli][INFO ] client : ['admin', 'node0', 'node1']
9 \, T m- q6 X, q$ r, ~- ^3 Y4 o[ceph_deploy.cli][INFO ] func : <function admin at 0x1500cf8>
8 s% M. n8 ]4 Q' e2 Q4 k[ceph_deploy.cli][INFO ] ceph_conf : None$ [9 ~7 T ]" J3 X& [4 w4 e" L& P
[ceph_deploy.cli][INFO ] default_release : False2 }' f0 H! n0 O" F3 [
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to admin) l" Z" a/ O* d% G
[admin][DEBUG ] connection detected need for sudo
7 i4 n2 d# C. k2 H& g6 _# A3 `+ l[admin][DEBUG ] connected to host: admin
- p/ X. t2 f# D1 `' i: L[admin][DEBUG ] detect platform information from remote host w8 l3 J8 l0 d
[admin][DEBUG ] detect machine type9 _# P4 V ^ G' q( e9 C
[admin][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf& d& G6 S! c7 [! ^( B0 [ @
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node0
* C2 H) g6 q* A! f3 H[node0][DEBUG ] connection detected need for sudo
# y- s! Z8 }+ o: n4 E; P- {[node0][DEBUG ] connected to host: node0
- B. x! s/ F6 M8 g8 i, E( M6 j- w[node0][DEBUG ] detect platform information from remote host! e7 c7 T5 f$ E1 V8 Q0 F
[node0][DEBUG ] detect machine type- [! G* K/ v9 J1 i, o5 d( V
[node0][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf4 h; `/ L$ X; l6 f
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node1+ p$ @+ }) A" _5 `
[node1][DEBUG ] connection detected need for sudo i! I4 Y6 G* g# D: e
[node1][DEBUG ] connected to host: node1 * X( @7 d O% C0 `4 \- E" G
[node1][DEBUG ] detect platform information from remote host8 G& f3 i' c& P8 C$ h
[node1][DEBUG ] detect machine type
$ I# j% B7 l0 V' G, K! V- v, l[node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf4 k3 ]* J+ p2 x
同时为了确保对 ceph.client.admin.keyring 有正确的操作权限，所以还需要增加权限设置。
. y% y9 ?+ A. F$ sudo chmod +r /etc/ceph/ceph.client.admin.keyring
1 U7 M6 W8 f+ V$ ?" c至此，Ceph 存储集群已经搭建完毕了，我们可以查看那一下集群是否启动成功！
* P7 {# {! b, ~* I! c# 查看集群状态
8 ]: r8 d2 l( c5 L$ ceph -s
^+ O& d& z+ x: i8 jcluster 363c3cf4-dba4-4f13-8ec5-ed93416f4e26
/ q/ F9 L2 k$ s0 _3 V( N5 w4 O health HEALTH_OK( Q8 N. i3 p2 {" `0 ?* F' A
monmap e1: 1 mons at {admin=10.222.77.213:6789/0}: }* w' O% f0 ?9 J" e4 n
election epoch 6, quorum 0 admin7 O' g4 b, G1 t
osdmap e15: 2 osds: 2 up, 2 in3 M& _# J. F! n; D0 l) \+ @
flags sortbitwise,require_jewel_osds8 ]; C6 O: q( U/ f9 t r- w2 n
pgmap v9276: 64 pgs, 1 pools, 863 MB data, 226 objects
- o( |( L1 B: @8 d/ Z" W0 ~/ M3 B 33440 MB used, 33574 MB / 67015 MB avail) D* f( Q2 j3 ]8 P
64 active+clean
, j/ Y+ A# D: M% `: Y# S+ q2 ^4 Z4 w9 M7 ]6 u- u6 b5 ]* i- j
# 或者查看集群健康状况4 F0 r. r7 r3 B* n
$ ceph health
9 ~" j3 l$ r" e. C# a0 u" P! m7 xHEALTH_OK
: i3 Z; g1 W2 \! {5 N5 M, U6 n6 A1 l0 X; K+ ?. s
# 查看集群 OSD 信息
. e' F+ Y0 S1 p" r6 x$ ceph osd tree' c% O Q; C! |: k- s) s9 }9 h
ID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
5 N W+ O8 ]) j! r) a/ y, n-1 0.06400 root default
; B0 p0 d% d: S" ~9 v& \-2 0.03200 host node0
" l; o; h( q: R. `, ]" i4 E( R 0 0.03200 osd.0 up 1.00000 1.00000 4 y- I% }* v. v
-3 0.03200 host node1 & i4 Z' Z; z$ u' D; }
1 0.03200 osd.1 up 1.00000 1.00000 : ^; y$ n3 t1 T+ Q; Q9 k9 z
好了，通过参考官方文档中的 Ceph 安装和集群搭建说明，一步一步的完成整个存储集群的搭建，文档写的很详细，非常容易上手操作。本次就先记录到这，下一篇我们继续研究学习 Ceph 存储体系中的对象存储、块设备和文件系统。 |
|
发表于 2018-7-18 10:20:46
