|
|
楼主 |
发表于 2018-9-26 09:01:34
|
显示全部楼层
Ceph 分布式存储集群有三大组件组成，分为：Ceph Monitor、Ceph OSD、Ceph MDS，后边使用对象存储和块存储时，MDS 非必须安装，只有当使用 Cephfs 文件存储时，才需要安装。这里我们暂时不安装 MDS。0 T3 J H" J: J, w6 t0 N V
3、Ceph 预检
0 k; `5 T( y& w' m; d3.1 配置节点Host) M8 g K0 t; [# R+ p8 G
为了方便后边安装，以及 ssh 方式连接各个节点，我们先修改一下各个节点的 Hostname 以及配置 Hosts 如下：
" Z9 ~& U+ j2 dadmin-node (10.222.77.213)
}3 O. G& k$ y6 J% r* A A! X: G, j* K- A
+ V$ I/ f# c% f$ cat /etc/hostname . K4 {0 w' I, u: q( ~7 K# }
admin. k3 s, p6 ~4 f9 b& o
" l) z6 a! d t1 r% B$ cat /etc/hosts$ K p( |+ A. ?, z0 A
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4( Q% i$ P$ L+ T) D& X. F/ t& B% a
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6" i n6 c% d0 c" Y; \& q0 r! ]
10.222.77.213 admin
J9 S# D+ _$ u! _) u3 U10.222.77.242 node0
5 @4 Q. X: r' \3 Z: V! `10.222.77.253 node13 N; ]7 g; q' ~* G4 h0 [6 I
, w8 y2 ]% w6 t& N! A4 _, h
-----------------------------9 v0 j( X3 X& E1 O1 b
node0（10.222.77.242） . v1 c0 o' p% R
$ cat /etc/hostname 6 t! Q! c" q% r L3 Z, f
admin: H5 F e; ]: B( u& ], T b
% t7 I' N4 A+ a# U
$ cat /etc/hosts1 W: [' }5 f( ?5 f$ l0 \ K6 Z
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
7 e S6 M* _, I/ d. p7 N::1 localhost localhost.localdomain localhost6 localhost6.localdomain6, T; R4 j# V' _! E& O" W% W
10.222.77.242 node0; l" M' P! P$ ]7 u$ I
10.222.77.213 admin I7 y: T" m7 E9 C9 h
10.222.77.253 node1
0 U$ T. |8 @2 {& c1 v4 A* p, p& [3 G
4 {! J9 \# q& a8 O-----------------------------" G" Q( g$ p$ Z+ l
node1（10.222.77.253）
$ Y$ e) n6 `" _ j7 s6 X1 ?( S$ cat /etc/hostname
0 N0 ]- @3 T* K$ Y! x1 D( Jadmin# ] N) l1 \ R2 A. e' `
" ^5 o/ w( U" x: N2 N9 |
$ cat /etc/hosts
# m! E& _5 ]( s, Y127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
& r3 R' u6 f {9 i0 s::1 localhost localhost.localdomain localhost6 localhost6.localdomain65 W3 ]0 e% Q) W
10.222.77.253 node15 K9 H/ |# @( Z1 p, q# y
10.222.77.213 admin
2 c- w! y+ c, Q" P8 a6 m4 N10.222.77.242 node08 X/ d4 g9 j3 v
3.2 安装部署工具 ceph-deploy
2 \0 D6 _. M' K6 P- o" mCeph 提供了部署工具 ceph-deploy 来方便安装 Ceph 集群，我们只需要在 ceph-deploy 节点上安装即可，这里对应的就是 admin-node 节点。把 Ceph 仓库添加到 ceph-deploy 管理节点，然后安装 ceph-deploy。因为系统是 Centos7 版本，所以配置如下：3 b, O% X9 p8 Z9 Y% R' A1 E( i' @
# ceph-deploy (admin-node) 上执行/ S8 t$ X, L: \) E7 N2 B
3 Z2 x$ N9 V) V- U6 \, V& g8 z
# yum 配置其他依赖包) c. H: i# X1 ?# S$ i
$ sudo yum install -y yum-utils && sudo yum-config-manager --add-repo https://dl.fedoraproject.org/pub/epel/7/x86_64/ && sudo yum install --nogpgcheck -y epel-release && sudo rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 && sudo rm /etc/yum.repos.d/dl.fedoraproject.org*4 }: s& [0 M: E' T. y
; X! w2 {4 m& p. b# {4 T7 y# 添加 Ceph 源. Z/ N# D4 @% q$ J% \+ }# y$ O" j
$ sudo vim /etc/yum.repos.d/ceph.repo
% u+ @8 f4 Q# t* p4 s! O5 Q- I8 t[Ceph-noarch]7 C1 Q; ~' g+ k" m: _( o; f
name=Ceph noarch packages
$ m) ` \% Q) g4 {4 f" Abaseurl=http://download.ceph.com/rpm-jewel/el7/noarch e5 b! E M7 }
enabled=1- T! x! G' s& [7 d% _' u: `! T! D
gpgcheck=11 F. h5 v$ p5 W0 x' b5 {- ]: t
type=rpm-md9 }7 u2 h. {: M3 s
gpgkey=https://download.ceph.com/keys/release.asc& l" @3 ~7 F- x, a6 M
priority=1" ?1 t, w' ]- x c2 o% |' c
. p7 h# {3 y/ i/ V, o( q
# 安装 ceph-deploy$ l$ I0 }: I5 ]" C0 y
$ sudo yum update && sudo yum install ceph-deploy
! y/ ]: v, y* m3.3 安装 NTP 和 Openssh
% Y S4 ^$ Y3 v2 B d! p官方建议在所有 Ceph 节点上安装 NTP 服务（特别是 Ceph Monitor 节点），以免因时钟漂移导致故障。
F6 _6 a! @' C* C% i4 J0 g: M5 m# yum 安装 ntp3 W5 Z) W5 z# O8 o/ {
sudo yum install ntp ntpdate ntp-doc
' W" I- V3 H( g; y$ u: u1 K+ E
' O' D( d( t& W& V. }( k# 校对系统时钟- u2 K6 V5 N0 {
ntpdate 0.cn.pool.ntp.org
; h$ |# Q# g/ s: G5 D- G后续操作，ceph-deploy 节点需要使用 ssh 方式登录各个节点完成 ceph 安装配置工作，所以要确保各个节点上有可用 SSH 服务。
m, W6 J/ u- o# z \' x# yum 安装 openssh9 U) |7 b" a4 `8 B
$ sudo yum install openssh-server
* I, _2 a" q# w
; ?1 i3 W+ j1 X; C' J' p# 查看 ssh 版本
4 H! S! m$ P# P8 }$ ssh -V q2 k# ~6 m1 ~% b7 u% w
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017
' ?7 S" P5 M$ R4 S+ T3.4 创建 Ceph 部署用户
8 l( u* j* x5 Y; z2 p: [ceph-deploy 工具必须以普通用户登录 Ceph 节点，且此用户拥有无密码使用 sudo 的权限，因为它需要在安装软件及配置文件的过程中，不必输入密码。官方建议所有 Ceph 节点上给 ceph-deploy 创建一个特定的用户，而且不要使用 ceph 这个名字。这里为了方便，我们使用 cephd 这个账户作为特定的用户，而且每个节点上（admin-node、node0、node1）上都需要创建该账户，并且拥有 sudo 权限。6 Q% G% ?: _8 Y5 P/ g# c
# 在 Ceph 集群各节点进行如下操作8 b [3 \- [0 K& Y w: B5 Z
6 z, m/ v0 {" o1 [/ {0 R4 I4 J( E
# 创建 ceph 特定用户
8 f- C9 |6 H" m) m) ?$ sudo useradd -d /home/cephd -m cephd
- B& K6 L3 h. B" O$ sudo passwd cephd
" w w$ \: _% [! b7 @% r' Q4 p" k |. b# F5 E! U) J) ^ d
# 添加 sudo 权限3 {) H, j' \' y# l) `
$ echo "cephd ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/cephd5 ?5 R) u3 W2 D& o1 X$ [8 B9 B$ [
$ sudo chmod 0440 /etc/sudoers.d/cephd. h) ]& k, L* |% f4 x
接下来在 ceph-deploy 节点（admin-node）上，切换到 cephd 用户，生成 SSH 密钥并把其公钥分发到各 Ceph 节点上，注意使用 cephd 账户生成，且提示输入密码时，直接回车，因为它需要免密码登录到各个节点。
! y1 R; G# M7 T9 n# ceph-deploy (admin-node) 上执行
' q0 @5 p) _" n
+ I; |) {& \+ v* {6 f- H; _# 生成 ssh 密钥1 }; x( j6 }, @7 R9 J1 M' m
$ ssh-keygen, F$ Y4 L( t7 _8 [
Generating public/private rsa key pair.
( S9 G J9 J) e4 dEnter file in which to save the key (/home/cephd/.ssh/id_rsa):
: T' r' G2 h7 w+ @0 QCreated directory '/home/cephd/.ssh'.; a- K6 j' m2 E0 c! W+ }
Enter passphrase (empty for no passphrase): . T& w$ p* n, l2 b5 o ?6 K. N
Enter same passphrase again:
: x6 j* a, L* \( n! x5 g. hYour identification has been saved in /home/cephd/.ssh/id_rsa.1 a8 e$ |- M# ]3 w& W
Your public key has been saved in /home/cephd/.ssh/id_rsa.pub.
% w8 q, L; e% U# ]6 c1 i/ Z6 p& fThe key fingerprint is:
+ m) A7 J0 I- ^7 q; b...
6 ^9 J0 r0 H. L5 @( s2 h
6 W. H3 S$ O, ?8 t0 x# 将公钥复制到 node0 节点. D v P! G- ~! @1 f6 u
$ ssh-copy-id cephd@node04 b- b1 e; ], n: K6 G
/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/cephd/.ssh/id_rsa.pub"
8 ^0 R' n: ~7 R3 y" D# Q8 o- DThe authenticity of host 'node0 (10.222.77.242)' can't be established.# l9 l. M9 A- E
ECDSA key fingerprint is MD5:3c:e0:a7:a0:e6:3c:dc:c0:df:28:dc:87:16:2d:0f:c6.
2 c, \6 W j1 a, XAre you sure you want to continue connecting (yes/no)? yes
1 G# b# \% x0 H: O2 v: \& p/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed5 v; G. W7 u0 k2 q! B- _
/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys3 O9 x6 W- `9 K' p5 L5 d
cephd@node0's password:
% W5 c% E, |; o' N
8 [4 m7 q) F" T" \; G5 ]3 `Number of key(s) added: 1& i" I8 J, W# m% [
3 ]' m) _9 I' K3 Q) SNow try logging into the machine, with: "ssh 'cephd@node0'" j7 U Z" c/ g* b' @- u" c
and check to make sure that only the key(s) you wanted were added./ { N9 `* R. }( [; L `
4 W `) x% Y x0 e6 M! @# 将公钥复制到 node1 节点. n9 {+ v7 |) V2 {4 C
$ ssh-copy-id cephd@node1
3 r2 H/ B9 \3 {' a1 [1 U5 h" [/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/cephd/.ssh/id_rsa.pub"
) h& {6 a4 [' F/ LThe authenticity of host 'node1 (10.222.77.253)' can't be established.* r# I" v8 [# U; P
ECDSA key fingerprint is MD5:3c:e0:a7:a0:e6:3c:dc:c0:df:28:dc:87:16:2d:0f:c6.# e* C/ a m, X
Are you sure you want to continue connecting (yes/no)? yes- g& q8 N4 \; r: N6 B$ _
/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed" v' t$ g( S( U
/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
1 Q ^ L# v+ b0 _cephd@node1's password: ( y9 H& V+ b! e$ n% {+ c" s
1 x% @7 L- T; J9 E; M
Number of key(s) added: 1
, a z3 m I4 X$ N& b8 o* ]+ O6 ^
: I, d3 N! C) _0 T. v1 L5 NNow try logging into the machine, with: "ssh 'cephd@node1'" M) L3 [! R+ e S7 l0 L: \; p' ~% N. K
and check to make sure that only the key(s) you wanted were added.! V7 h, B6 A% T$ S7 i/ x
复制完毕，测试一下在 ceph-deploy 管理节点免密码登录各个节点。3 J4 c5 Z5 B4 k* t2 F
$ ssh node0& F+ W* Z' Q9 r2 H( M# B9 r6 |
Last login: Fri Dec 8 15:50:08 2017 from admin9 B8 ?3 [/ U4 c& P9 \/ j9 i# }
. E3 N9 V8 L- d p! P$ ssh node1' U S" N+ Q$ `/ ?9 T& O% F
Last login: Fri Dec 8 15:49:27 2017 from admin
' }) M6 k7 r2 K$ D测试没有问题，接下来，修改 ceph-deploy 管理节点上的 ~/.ssh/config 文件，这样无需每次执行 ceph-deploy 都要指定 –username cephd 。这样做同时也简化了 ssh 和 scp 的用法。
9 x2 |, T/ g5 w$ I! \$ cat ~/.ssh/config
- I9 D- g2 k. vHost node01 k w1 `0 ]' N/ t, h
Hostname node0; w! _. ~6 V ?3 c9 {' Q( R8 e
User cephd
`4 N; {1 B2 V7 j. WHost node1
5 Q, j `+ X3 R9 C) z+ T Hostname node1
5 F+ t# ^ x7 m; E9 X0 i& S User cephd
0 }3 V2 g) y/ z$ D$ L% v! R; R注意，此时再执行 ssh node0 会提示报错 Bad owner or permissions on /home/cephd/.ssh/config。原因是 config 文件权限问题，修改权限 sudo chmod 600 config 即可解决。
7 n. w8 M% O0 Y+ S9 z# \3 a3.5 其他网络配置+ [, f8 | N/ k/ D
官网文档中指定 Ceph 的各 OSD 进程通过网络互联并向 Monitors 上报自己的状态，所以要保证网络为开启状态，不过某些发行版（如 CentOS ）默认关闭网络接口。所以我们需要保证集群各个节点系统网络接口是开启的。
/ ~. C4 D4 O& m+ U! Y2 {! k# 在 Ceph 集群各节点进行如下操作+ H* o5 ~( Z9 h/ h: x, \2 D
$ _4 e7 ]+ B$ D6 l
$ sudo cat /etc/sysconfig/network-scripts/ifcfg-enp0s3
) G3 d1 V; E' P0 h( bTYPE="Ethernet"
$ D$ Q( e1 f6 ^BOOTPROTO="dhcp"# J W. h0 @ |$ J- k
DEFROUTE="yes"
7 }3 O/ X/ ~; {% n/ xPEERDNS="yes"% A8 Z- H9 Z* R$ t9 J" G4 Z) e0 c9 C
PEERROUTES="yes"$ E, q8 }/ y* M6 |
IPV4_FAILURE_FATAL="no"
9 O" ? o, F( b* N# C) o8 w, aIPV6INIT="yes"
# }8 Q0 l+ f: J- A7 `* c9 Q& ]8 LIPV6_AUTOCONF="yes"3 D8 Y1 @1 L% ?+ ?2 `; G. Q
IPV6_DEFROUTE="yes") R& o" R b3 h1 V+ V
IPV6_PEERDNS="yes"
: J9 s- Z6 {3 [% ~) ]! p3 N7 ZIPV6_PEERROUTES="yes"
/ \* ]5 G3 G2 Y0 U6 |6 ~* e( GIPV6_FAILURE_FATAL="no"& p, A% i( z( ^: a
IPV6_ADDR_GEN_MODE="stable-privacy"4 C0 E6 W5 S; @: B. X0 ]
NAME="enp0s3"# j+ W( O* q7 A* K3 O9 U
UUID="3e68d5a3-f9a6-4c83-9969-706f7e3b0bc2"
& Q. p2 {1 M! [, NDEVICE="enp0s3"9 U1 }% }' W5 C/ S0 h- G
ONBOOT="yes" # 这里要设置为 yes; l; E& c6 B9 Y$ q4 r5 O
注意：这里因为在我安装的虚拟机集群中网卡为 enp0s3，所以需要修改 /etc/sysconfig/network-scripts/ifcfg-enp0s3 文件，请根据自己系统网卡名去修改对应配置文件。) p6 I7 k: L& O1 O3 w. ]& G$ h7 G
SELINUX 设置，在 CentOS 系统上， SELinux 默认为 Enforcing 开启状态，为了方便安装，建议把 SELinux 设置为 Permissive 或者 disabled。
2 `! X( E4 O' f. C# 在 Ceph 集群各节点进行如下操作9 e# r9 w& O; M& H5 T2 H
, E" U$ n6 E4 ~
# 临时生效设置
' M) o9 A, c+ O; S$ sudo setenforce 0
, B/ S. W* r* o2 ?9 ]# k
* i* x) b& R5 f; }8 T# @# 永久生效设置. ~5 K) Q4 j1 k8 P2 ?
$ sudo cat /etc/selinux/config" t1 _) k5 q# \+ E
SELINUX=disabled # 这里设置为 Permissive | disabled; C/ R0 U4 Y- u% F
SELINUXTYPE=targeted ) J9 I0 a7 i) G s% ~ m7 L* n# {
开放所需端口设置，Ceph Monitors 之间默认使用 6789 端口通信， OSD 之间默认用 6800:7300 这个范围内的端口通信，所以我们需要调整防火墙设置，开放所需端口，允许相应的入站请求。
1 w) \( n2 P) B6 w4 {# 防火墙设置
* A$ v G% F3 I1 Q4 \. T9 l$ sudo firewall-cmd --zone=public --add-port=6789/tcp --permanent- {2 g9 x% z4 q$ G8 M& D
0 n5 J* t/ P8 v6 `) `# R, o
# 当然我们也可以关闭防火墙
+ g' I) l! J/ P" j, L$ sudo systemctl stop firewalld.service #停止 firewall
0 P% c" p8 `4 l0 R5 m$ sudo systemctl disable firewalld.service #禁止 firewall 开机启动$ x4 A% ^6 D2 m! }
4、Ceph 存储集群搭建
% p% {0 O0 Z, g好了，经过上边一系列的预检设置后，我们就可以开始 Ceph 存储集群搭建了，集群结构为 admin-node (ceph-deploy、Monitor)、node0（osd.0）、node1（osd.1）。首先要提一下的是，如果我们在安装过程中出现了问题，需要重新操作的时候，例如想清理我搭建的这个集群的话，可以使用以下命令。
. K0 b0 l4 g7 c/ e4 f" e; f# ceph-deploy (admin-node) 上执行7 Q) s$ @ d! G E$ z
1 O' g$ r8 }3 ~+ W0 f2 u
# 清理配置& }$ V1 U' M6 ^" o! Z% e9 f
ceph-deploy purgedata admin node0 node1
$ P4 _* p# D0 h+ t* ~ G; W' dceph-deploy forgetkeys& r7 \3 R# ?2 V" J L' L
) o5 o$ u$ P9 f9 X
# 清理 Ceph 安装包
( [. M" f1 a) r1 v! u5 H' }* Bceph-deploy purge admin node0 node1
- N4 _2 O# e7 i) k/ G好了，现在开始搭建。首先 Cephd 用户创建一个目录 ceph-cluster 并进入到该目录执行一系列操作。因为我们设计的 monitor 节点在 admin-node 节点上，所以，执行如下命令。' }: k3 d" {9 i8 C5 p
# 创建执行目录
, `) j$ J7 p) R0 V. Y9 x0 ?$ mkdir ~/ceph-cluster && cd ~/ceph-cluster
% E6 L1 }. _/ G, U7 V
. z+ b1 x/ o1 l# 创建集群9 k; q, V8 J: N- T
$ ceph-deploy new admin) K8 n: ?+ S2 O: q! l# [
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephd/.cephdeploy.conf: \' g: F n9 h" j. `1 U
[ceph_deploy.cli][INFO ] Invoked (1.5.39): /bin/ceph-deploy new admin3 A( _* k0 W L6 O
[ceph_deploy.cli][INFO ] ceph-deploy options:
: f2 v E! ^8 t, p[ceph_deploy.cli][INFO ] username : None% w: A6 ^% ]0 E2 q
[ceph_deploy.cli][INFO ] func : <function new at 0xf24938>
5 [0 V6 \3 c; m% T+ k! U* {: y) S[ceph_deploy.cli][INFO ] verbose : False6 B) [6 X3 t$ p; `9 ~6 Q
[ceph_deploy.cli][INFO ] overwrite_conf : False; P( l, q5 ~; Z: O4 [$ B0 Q
[ceph_deploy.cli][INFO ] quiet : False1 K5 D" F$ r- K: M0 k! p
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0xf1f6c8>
3 S1 C& g& t. Z7 ^% `0 ]! W[ceph_deploy.cli][INFO ] cluster : ceph* K4 e/ X3 B" ]; N
[ceph_deploy.cli][INFO ] ssh_copykey : True
( v* _, ^; S2 T# ]- c[ceph_deploy.cli][INFO ] mon : ['admin']
5 v( m% @, J) J3 E4 [$ k[ceph_deploy.cli][INFO ] public_network : None+ H7 d/ h: M# p$ ^( Y4 u( x! W2 H
[ceph_deploy.cli][INFO ] ceph_conf : None
2 l+ J1 A+ ?3 G[ceph_deploy.cli][INFO ] cluster_network : None1 M6 s& G$ n* p3 J/ x: A* x$ J
[ceph_deploy.cli][INFO ] default_release : False
- O" Y$ M: @/ a# d0 n' E+ c3 r& i6 z[ceph_deploy.cli][INFO ] fsid : None
0 `- r" y5 `( a[ceph_deploy.new][DEBUG ] Creating new cluster named ceph$ O: _2 {# U4 b% f% Q
[ceph_deploy.new][INFO ] making sure passwordless SSH succeeds
! C, R F* o$ `4 V...
% x" \7 u2 B$ I; _[ceph_deploy.new][DEBUG ] Resolving host admin
! ~+ `% U" J* z[ceph_deploy.new][DEBUG ] Monitor admin at 10.222.77.2136 h+ M: A X5 z \
[ceph_deploy.new][DEBUG ] Monitor initial members are ['admin']- n& n+ W8 j; V% f ]3 J
[ceph_deploy.new][DEBUG ] Monitor addrs are ['10.222.77.213']3 }. y# ]) F8 S, W
[ceph_deploy.new][DEBUG ] Creating a random mon key...
4 M7 o; j6 S8 `& s[ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring...
( }! u/ f5 Z( z- s, O[ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf...) [# a) T% I1 g0 Q
此时，我们会发现 ceph-deploy 会在 ceph-cluster 目录下生成几个文件，ceph.conf 为 ceph 配置文件，ceph-deploy-ceph.log 为 ceph-deploy 日志文件，ceph.mon.keyring 为 ceph monitor 的密钥环。2 \9 s/ S) y+ j+ m) D X$ D
$ ll ceph-cluster
) H5 G: @$ ]! X* C-rw-rw-r--. 1 cephd cephd 196 12月 7 14:46 ceph.conf q6 w/ w, ^+ W9 I* N0 w6 e( Q
-rw-rw-r--. 1 cephd cephd 3694 12月 7 14:46 ceph-deploy-ceph.log' A: l6 ?0 Q* H# h! `3 p$ C
-rw-------. 1 cephd cephd 73 12月 7 14:46 ceph.mon.keyring
" b |- W G0 b% S" G: ^
@4 ]1 ? M: H0 e/ l$ cat ceph.conf $ s9 x+ _$ A: @! E7 D- {
[global]
* p0 |% m8 i) f7 L; p% a7 kfsid = 363c3cf4-dba4-4f13-8ec5-ed93416f4e26/ | @2 F! l( d0 P. ?9 ~6 p
mon_initial_members = admin2 b( y4 h: l w. ?6 Z* C. ?
mon_host = 10.222.77.213
6 z% M0 f8 Y5 C Sauth_cluster_required = cephx! z0 r% e6 K/ W% A7 k: N
auth_service_required = cephx
3 ~* P8 R) c& q" `8 Q3 yauth_client_required = cephx$ r( t( [' ^1 W9 S, O7 e
接下来，我们需要修改下 ceph.conf 配置文件，增加副本数为 2，因为我们有两个 osd 节点。
( M7 N( T6 o( u' z7 B( U. b3 G& k9 l$ cat ceph.conf ! `0 |0 ^: R$ J, I% z
[global]" R1 h5 W6 {5 [& u+ r6 C6 |* ^
fsid = 363c3cf4-dba4-4f13-8ec5-ed93416f4e26
9 m/ ` C. ~3 e& i! V7 n _2 Q- wmon_initial_members = admin% T" ^) |& ]: I8 P( k5 I$ k0 ]4 W8 @
mon_host = 10.222.77.213
4 `, U, t' F, _; `% n" w3 ^! X3 hauth_cluster_required = cephx
( [6 z- |$ W; Eauth_service_required = cephx; K/ T% E+ ~1 f
auth_client_required = cephx( v2 O1 ~, B# P3 B; Y
osd pool default size = 2 #增加默认副本数为 2
' t3 S# D" M, o) T/ ]9 V! B+ A然后，我们需要通过 ceph-deploy 在各个节点安装 ceph。
$ a3 d6 P. {, T7 R) U$ ceph-deploy install admin node0 node1
6 G' L" V$ \) Q8 s; d: J[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephd/.cephdeploy.conf
! Y( Q- ~" M o, g, b8 n9 U[ceph_deploy.cli][INFO ] Invoked (1.5.39): /bin/ceph-deploy install admin node0 node1
. F S4 A7 ^# f2 ]% v[ceph_deploy.cli][INFO ] ceph-deploy options:
/ D# d: n( i% f: o2 ?3 u; X[ceph_deploy.cli][INFO ] verbose : False
$ R- I/ Z3 I7 m2 S$ G" k, w[ceph_deploy.cli][INFO ] testing : None' K/ m- \# l( i' }
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0xde1c68>
% T5 Z* h5 y; I: o, W[ceph_deploy.cli][INFO ] cluster : ceph) j9 w l- q) N3 ?/ ~3 {
[ceph_deploy.cli][INFO ] dev_commit : None
9 M) c; N3 @5 X, i[ceph_deploy.cli][INFO ] install_mds : False) s0 k; M$ i/ U
[ceph_deploy.cli][INFO ] stable : None* A. ?* y! I$ B; s+ k
[ceph_deploy.cli][INFO ] default_release : False9 } F" j5 U* y }
[ceph_deploy.cli][INFO ] username : None
$ D1 Z0 j- {0 v- p; _4 f[ceph_deploy.cli][INFO ] adjust_repos : True& D3 G# a4 n8 B% I
[ceph_deploy.cli][INFO ] func : <function install at 0xd5b140>
) W) Q% b4 ~! W6 |0 D C% z[ceph_deploy.cli][INFO ] install_mgr : False
' z( |, ^1 q! ^' H! J- n- O[ceph_deploy.cli][INFO ] install_all : False4 T9 }5 e) y6 K% b1 E" E+ n
[ceph_deploy.cli][INFO ] repo : False( {, L! D! {4 w& W9 e( b
[ceph_deploy.cli][INFO ] host : ['admin', 'node0', 'node1']
3 A2 {- Y+ r, A) k/ T6 N7 a. L[ceph_deploy.cli][INFO ] install_rgw : False
9 N) b+ V! d: c5 Z* c" ]) h$ D/ ~+ O[ceph_deploy.cli][INFO ] install_tests : False
7 P3 b/ {! a. e4 |4 Y2 C[ceph_deploy.cli][INFO ] repo_url : None
# u* P2 z4 m( D[ceph_deploy.cli][INFO ] ceph_conf : None
! n8 ~3 w! F$ U3 J9 w' h[ceph_deploy.cli][INFO ] install_osd : False$ c, S7 d- ~9 h# b, l1 Y B
[ceph_deploy.cli][INFO ] version_kind : stable* d7 u" ]1 D* O' U0 W
[ceph_deploy.cli][INFO ] install_common : False- V$ a' p# k; Y
[ceph_deploy.cli][INFO ] overwrite_conf : False" \! f% n& a3 d
[ceph_deploy.cli][INFO ] quiet : False$ e4 E, J8 P8 A( I- u
[ceph_deploy.cli][INFO ] dev : master8 x. F# r) ]4 H
[ceph_deploy.cli][INFO ] nogpgcheck : False
( @: P9 U+ G. `. y/ w( M: z/ K0 G[ceph_deploy.cli][INFO ] local_mirror : None
* _% D8 E) S5 z v- Y U[ceph_deploy.cli][INFO ] release : None
+ C s' U1 M/ V. m9 Y- x( c7 w[ceph_deploy.cli][INFO ] install_mon : False1 S" ~. ^7 H% ?6 {8 w3 Y
[ceph_deploy.cli][INFO ] gpg_url : None3 E" w8 V1 k6 R3 }* ^
[ceph_deploy.install][DEBUG ] Installing stable version jewel on cluster ceph hosts admin node0 node1! f( B6 a, z: D' C. X
[ceph_deploy.install][DEBUG ] Detecting platform for host admin ...
+ @2 g3 A& ]" ?: D...8 f& j$ D# N+ k9 v' w6 j
此过程需要等待一段时间，因为 ceph-deploy 会 SSH 登录到各 node 上去，依次执行安装 ceph 依赖的组件包。3 m& R# |/ ^. Z# J; h9 M- J
漫长的等待安装完毕之后，接下来需要初始化 monitor 节点并收集所有密钥。; y7 r! K% J& M. p+ {7 i/ M* b( m
$ ceph-deploy mon create-initial
0 l* b3 Z! l- B7 ~3 R+ _+ u...
. _ J7 D' ?1 x0 S7 bceph_deploy.mon][ERROR ] RuntimeError: config file /etc/ceph/ceph.conf exists with different content; use --overwrite-conf to overwrite! o- b. N0 j& ?* X% `+ j- r+ A
[ceph_deploy][ERROR ] GenericError: Failed to create 1 monitors, s0 h9 f1 ?) G- ~% Y3 J6 m7 Z7 [" \
...
, d* v0 ]8 {2 K: i7 g不过很遗憾，执行过程中报错了。查看原因应该是已经存在了 /etc/ceph/ceph.conf 配置文件了，解决方案就是加上 --overwrite-conf 参数，覆盖已存在的配置。
' d4 ]7 B3 m" _; t6 P% k0 M# k$ ceph-deploy --overwrite-conf mon create-initial
: R. j/ T) Y* m# Z0 A[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephd/.cephdeploy.conf% O% Y, i/ h) ?2 ^: f
[ceph_deploy.cli][INFO ] Invoked (1.5.39): /bin/ceph-deploy --overwrite-conf mon create-initial) c) p" M/ Q+ F% A* K8 g
[ceph_deploy.cli][INFO ] ceph-deploy options:
4 R7 Q/ j4 J3 h3 A! W& o: O1 N[ceph_deploy.cli][INFO ] username : None+ {- O/ ]: L1 o0 l
[ceph_deploy.cli][INFO ] verbose : False6 ?6 \! C8 B# C% T
[ceph_deploy.cli][INFO ] overwrite_conf : True
" h1 R' q8 z, p+ }/ t r6 L% @[ceph_deploy.cli][INFO ] subcommand : create-initial7 O1 T# `8 T$ V0 T5 C% B6 s
[ceph_deploy.cli][INFO ] quiet : False2 S/ n" |# e d) [6 ~
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x1d9e758>
& G6 P# ~; F4 S) A. i) j& Q0 p[ceph_deploy.cli][INFO ] cluster : ceph
D' q$ V: ]( K* Z- r) X$ u& V$ Y[ceph_deploy.cli][INFO ] func : <function mon at 0x1d8ea28>
9 y ^9 a8 |/ j' b! Y% j; ?& Q[ceph_deploy.cli][INFO ] ceph_conf : None
1 I) H7 ?/ u3 F: \[ceph_deploy.cli][INFO ] default_release : False
& c" a7 n" _6 E. M4 o[ceph_deploy.cli][INFO ] keyrings : None8 G# e* V% Q: j! U5 h; D3 n
[ceph_deploy.mon][DEBUG ] Deploying mon, cluster ceph hosts admin* a% G) V9 F! v+ V1 X, T
[ceph_deploy.mon][DEBUG ] detecting platform for host admin ...5 N) {: q" z* M: h/ E% U% n
.... {. K: P, C; i
[admin][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --admin-daemon=/var/run/ceph/ceph-mon.admin.asok mon_status
( D, N8 Z2 j) Y8 N, z2 Y[admin][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-admin/keyring auth get client.admin/ D0 ?+ q- ?0 p P; N
[admin][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-admin/keyring auth get client.bootstrap-mds
- H' q- R$ h+ u# w0 ]2 m% ^4 _[admin][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-admin/keyring auth get client.bootstrap-mgr" M0 [2 S0 ?* l; u0 x5 ^
[admin][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-admin/keyring auth get-or-create client.bootstrap-mgr mon allow profile bootstrap-mgr
6 O6 M/ ?* @: u( F; x6 W$ C% m[admin][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-admin/keyring auth get client.bootstrap-osd) N, v2 j/ W( _ P
[admin][INFO ] Running command: sudo /usr/bin/ceph --connect-timeout=25 --cluster=ceph --name mon. --keyring=/var/lib/ceph/mon/ceph-admin/keyring auth get client.bootstrap-rgw
5 c9 S' G0 h5 E4 \8 Q[ceph_deploy.gatherkeys][INFO ] Storing ceph.client.admin.keyring: F* D1 u: Y* E! `( ?) N2 ~0 A
[ceph_deploy.gatherkeys][INFO ] Storing ceph.bootstrap-mds.keyring
4 Z9 Q' ]* _- H6 |& C: Y[ceph_deploy.gatherkeys][INFO ] Storing ceph.bootstrap-mgr.keyring/ J* V/ I* _: p8 Y& Z- H- `8 V1 U
[ceph_deploy.gatherkeys][INFO ] keyring 'ceph.mon.keyring' already exists, k4 z7 P# B/ s* W
[ceph_deploy.gatherkeys][INFO ] Storing ceph.bootstrap-osd.keyring
/ D* F( ]8 m+ P5 c) V1 V) U9 w[ceph_deploy.gatherkeys][INFO ] Storing ceph.bootstrap-rgw.keyring
6 s9 K: ^2 u! G2 G. h8 W/ O[ceph_deploy.gatherkeys][INFO ] Destroy temp directory /tmp/tmpLv4mug8 R) J( |! |7 B" a9 Q# A, Y5 K
执行完毕后，会在当前目录下生成一系列的密钥环，应该是各组件之间访问所需要的认证信息吧。0 x& \6 N% K7 y5 A
$ ll ~/ceph-cluster: M% E3 [1 g0 ^" ]' S, {
-rw-------. 1 cephd cephd 113 12月 7 15:13 ceph.bootstrap-mds.keyring: B0 Z8 y" _/ ~- t$ K: Y
-rw-------. 1 cephd cephd 71 12月 7 15:13 ceph.bootstrap-mgr.keyring& y0 E- |' c J* j/ ^
-rw-------. 1 cephd cephd 113 12月 7 15:13 ceph.bootstrap-osd.keyring4 t+ ?7 l/ A9 C9 r* Q8 U
-rw-------. 1 cephd cephd 113 12月 7 15:13 ceph.bootstrap-rgw.keyring& T6 R% t) {! r# Z2 U- d1 X
-rw-------. 1 cephd cephd 129 12月 7 15:13 ceph.client.admin.keyring
0 A* N6 U& ^, Q8 F-rw-rw-r--. 1 cephd cephd 222 12月 7 14:47 ceph.conf# B7 g- n! e# K6 ]. z. p& u
-rw-rw-r--. 1 cephd cephd 120207 12月 7 15:13 ceph-deploy-ceph.log
' P1 l( W7 _- K; R' g6 J+ s) l-rw-------. 1 cephd cephd 73 12月 7 14:46 ceph.mon.keyring
* \, ], y+ }: t) e% g到此，ceph monitor 已经成功启动了。接下来需要创建 OSD 了，OSD 是最终数据存储的地方，这里我们准备了两个 OSD 节点，分别为 osd.0 和 osd.1。官方建议为 OSD 及其日志使用独立硬盘或分区作为存储空间，不过本机虚拟机上不具备条件，但是我们可以在虚拟机本地磁盘上创建目录，来作为 OSD 的存储空间。
) _- `$ a( F/ z0 P# ceph-deploy (admin-node) 上执行: K/ }( k' t3 @( Z+ Y/ C; \5 Q
1 r5 { T/ Z* T4 O7 k
$ ssh node0
. \- O8 x. U4 g1 _* N) @& y$ sudo mkdir /var/local/osd0. {; V, i/ ^& i L0 h
$ sudo chown -R ceph:ceph /var/local/osd0+ K1 q1 [$ R: ~. y6 |3 ]( I' o9 W
$ exit
; k$ L3 t7 ~! a
# M- Q; F# Q% Q# f) `$ ssh node1
& q _4 @% n2 q# f7 \% Z; `$ sudo mkdir /var/local/osd1! ^6 f( f9 S& o T! p7 A" Y
$ sudo chown -R ceph:ceph /var/local/osd1
4 A% @ H* I7 f+ Y+ ^2 Q. x$ exit
" ^6 ?4 g2 \3 a X1 x. I+ f注意：这里执行了 chown -R ceph:ceph 操作，将 osd0 和 osd1 目录的权限赋予 ceph:ceph，否则，接下来执行 ceph-deploy osd activate ... 时会报权限错误。' i5 ]& C6 j1 f# [3 N
接下来，我们需要 ceph-deploy 节点执行 prepare OSD 操作，目的是分别在各个 OSD 节点上创建一些后边激活 OSD 需要的信息。
: Y. t; e: g+ r$ ceph-deploy --overwrite-conf osd prepare node0:/var/local/osd0 node1:/var/local/osd1
_5 C1 }% {, m% {[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephd/.cephdeploy.conf
( d P8 b) l% U& J[ceph_deploy.cli][INFO ] Invoked (1.5.39): /bin/ceph-deploy --overwrite-conf osd prepare node0:/var/local/osd0 node1:/var/local/osd1
, r# ^4 i% G! |3 X[ceph_deploy.cli][INFO ] ceph-deploy options:
) @$ Q; z, M1 f; V1 q) v[ceph_deploy.cli][INFO ] username : None
B/ [% ~$ l/ e1 E1 Z[ceph_deploy.cli][INFO ] block_db : None7 f S8 F( M$ U: v1 e. n5 B/ A
[ceph_deploy.cli][INFO ] disk : [('node0', '/var/local/osd0', None), ('node1', '/var/local/osd1', None)]% A. g6 J' T0 w& `8 J* P3 ^
[ceph_deploy.cli][INFO ] dmcrypt : False
6 @( M" c- @4 n9 l) @: s$ A' P[ceph_deploy.cli][INFO ] verbose : False
8 l. |1 B7 ^( R d% d9 c[ceph_deploy.cli][INFO ] bluestore : None
9 O- A1 R7 }/ u0 H" B5 d; |: U[ceph_deploy.cli][INFO ] block_wal : None# r, H: E0 t) I, x2 R7 p
[ceph_deploy.cli][INFO ] overwrite_conf : True% |: x9 ^9 m4 {2 p9 v6 h0 c
[ceph_deploy.cli][INFO ] subcommand : prepare
* W8 N( Z5 I" `% U# }3 j* l* @[ceph_deploy.cli][INFO ] dmcrypt_key_dir : /etc/ceph/dmcrypt-keys
# d" _" D, M0 X8 N3 ?1 C, C. E2 ^[ceph_deploy.cli][INFO ] quiet : False& }9 O& W- q* s8 E
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x117d950>7 ^$ Y! R( V0 p( F+ @; ~
[ceph_deploy.cli][INFO ] cluster : ceph& H: W7 Y( P. [: L& \* Z2 M
[ceph_deploy.cli][INFO ] fs_type : xfs1 n5 p' P) V2 O `* P! j
[ceph_deploy.cli][INFO ] filestore : None2 S8 Q; y! J1 \6 Q' @& ]) l
[ceph_deploy.cli][INFO ] func : <function osd at 0x116a320>7 S; K- N# D/ E0 I) w# Q W5 s
[ceph_deploy.cli][INFO ] ceph_conf : None' G) T- M% W8 ~4 X0 t4 r5 {
[ceph_deploy.cli][INFO ] default_release : False
4 v& h9 P6 q* y" r[ceph_deploy.cli][INFO ] zap_disk : False; I% T J" b! R' j6 R
[ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks node0:/var/local/osd0: node1:/var/local/osd1:
) M' w7 z7 p4 m& j7 _! w; D. ?& d9 d[node0][DEBUG ] connection detected need for sudo
# s, s4 l7 X1 m, `6 C[node0][DEBUG ] connected to host: node0 6 u( @2 f7 {+ C0 k/ [ g0 Z. `" K
[node0][DEBUG ] detect platform information from remote host
0 T6 ~8 V( e$ v7 [3 T[node0][DEBUG ] detect machine type
0 t% N1 W) I* O: Q$ u7 d[node0][DEBUG ] find the location of an executable
+ i4 ?, s' z( y9 U! [ o! J[ceph_deploy.osd][INFO ] Distro info: CentOS Linux 7.3.1611 Core
+ m" x: f& ]& S% c" F' ~% D[ceph_deploy.osd][DEBUG ] Deploying osd to node09 B0 n( X4 ? F. a6 b5 e
...
5 F! D0 H5 A5 E; L- r[node0][INFO ] checking OSD status...
3 b: x7 ]/ c) @[node0][DEBUG ] find the location of an executable4 D/ @, ?2 f8 C
[node0][INFO ] Running command: sudo /bin/ceph --cluster=ceph osd stat --format=json9 t A3 R# V6 D+ C. s: S" y% Z
[ceph_deploy.osd][DEBUG ] Host node0 is now ready for osd use.
' K* d( D& f5 H6 W1 s. @7 ][node1][DEBUG ] connection detected need for sudo
7 }7 u% a" G6 ?6 y[node1][DEBUG ] connected to host: node1 6 b# w3 m0 N+ ~( u3 I2 k( U
[node1][DEBUG ] detect platform information from remote host- M% _: `' W8 t+ S
[node1][DEBUG ] detect machine type
; ~% D, O7 X* k0 w* n! b$ I[node1][DEBUG ] find the location of an executable0 v4 Z# Z% U5 q5 U3 s0 b+ H
[ceph_deploy.osd][INFO ] Distro info: CentOS Linux 7.3.1611 Core
4 v3 n$ i4 K9 n4 M[ceph_deploy.osd][DEBUG ] Deploying osd to node1
; f5 Q0 r+ W, ^, c9 S2 q...
3 Z! \: ^& n5 g[node1][INFO ] checking OSD status...( L4 v8 J, ~, \* X& F& |
[node1][DEBUG ] find the location of an executable
; y2 x+ Q3 e% G) ]8 G0 P, Y2 v; s[node1][INFO ] Running command: sudo /bin/ceph --cluster=ceph osd stat --format=json
8 b. m: p1 D3 H, _9 [8 C[ceph_deploy.osd][DEBUG ] Host node1 is now ready for osd use.9 Z4 {" V& C" B' C" S; G4 q
OK 接下来，我们需要激活 activate OSD。/ ]3 V. l. {' ~4 ~4 Z- W/ Q
$ ceph-deploy osd activate node0:/var/local/osd0 node1:/var/local/osd1; y4 S2 C f; j
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephd/.cephdeploy.conf) C3 x8 y& o+ ^- | P; H, R
[ceph_deploy.cli][INFO ] Invoked (1.5.39): /bin/ceph-deploy osd activate node0:/var/local/osd0 node1:/var/local/osd1
( b; ^/ C( V3 I( I3 v: S; u[ceph_deploy.cli][INFO ] ceph-deploy options:
" n$ D. e3 s- ?! n" O[ceph_deploy.cli][INFO ] username : None7 W. _ Z3 _; C* i0 P# P
[ceph_deploy.cli][INFO ] verbose : False4 W8 \3 g( V2 e8 l) s9 @* U
[ceph_deploy.cli][INFO ] overwrite_conf : False
: B7 H3 l( ?1 _[ceph_deploy.cli][INFO ] subcommand : activate _2 K0 Q) D9 C9 x; `
[ceph_deploy.cli][INFO ] quiet : False6 F, H, j8 l( ^7 M2 T( r- U; X0 O) X
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x2045950>
3 ~7 e8 D, k, t- v& W[ceph_deploy.cli][INFO ] cluster : ceph
3 D7 R m0 {; h+ P1 x4 m6 L0 D$ {[ceph_deploy.cli][INFO ] func : <function osd at 0x2032320>
, N9 H& N4 b$ B3 h[ceph_deploy.cli][INFO ] ceph_conf : None5 |7 H2 V; g7 @( ~) M3 Z$ w
[ceph_deploy.cli][INFO ] default_release : False
0 o; o9 s' t4 {, r* l3 u[ceph_deploy.cli][INFO ] disk : [('node0', '/var/local/osd0', None), ('node1', '/var/local/osd1', None)]
7 _1 p& m9 M5 s' E+ L[ceph_deploy.osd][DEBUG ] Activating cluster ceph disks node0:/var/local/osd0: node1:/var/local/osd1:
5 z+ J! |; l# q1 W, R[node0][DEBUG ] connection detected need for sudo
5 o7 d. K/ D) S m& K+ |6 K% M. \[node0][DEBUG ] connected to host: node0
7 O$ l% o0 h' h6 v+ N" k[node0][DEBUG ] detect platform information from remote host
2 ~4 P- f3 m- R4 X3 b, [[node0][DEBUG ] detect machine type! K) \+ u+ K$ n4 \7 V/ T
[node0][DEBUG ] find the location of an executable
; s; }$ M1 ^7 J/ H[ceph_deploy.osd][INFO ] Distro info: CentOS Linux 7.3.1611 Core" u( s5 s- H% a7 W) v& b
[ceph_deploy.osd][DEBUG ] activating host node0 disk /var/local/osd0/ q6 u* l. q y4 T0 Q }7 y! y
[ceph_deploy.osd][DEBUG ] will use init type: systemd
) o' n8 O6 }* x& K...6 U. C8 _- U0 G9 s0 o& M
[node0][INFO ] checking OSD status...* f8 g1 J: y) |& I- I$ ]
[node0][DEBUG ] find the location of an executable, p5 u8 Z; {% A. k9 G3 U
[node0][INFO ] Running command: sudo /bin/ceph --cluster=ceph osd stat --format=json
2 K) ]5 c& H) D[node0][INFO ] Running command: sudo systemctl enable ceph.target: ?* K6 Q9 w. T
[node1][DEBUG ] connection detected need for sudo7 P; N$ M! F. q5 k. I1 g
[node1][DEBUG ] connected to host: node1
% O g) D1 t$ ^[node1][DEBUG ] detect platform information from remote host
4 F0 u' P! v( C$ d+ \[node1][DEBUG ] detect machine type
% i5 J# M! e: L& J& B[node1][DEBUG ] find the location of an executable6 Q& i- N& V. C" v: f
[ceph_deploy.osd][INFO ] Distro info: CentOS Linux 7.3.1611 Core+ ~) n0 U$ O' z; |9 U4 D8 m
[ceph_deploy.osd][DEBUG ] activating host node1 disk /var/local/osd1
) u' }: p- j1 M' t3 T! r[ceph_deploy.osd][DEBUG ] will use init type: systemd
1 B- e6 U2 j$ Y2 M; {8 W...
f( ?2 H, I+ F! G% {0 @+ \[node1][INFO ] checking OSD status...& l7 Y' t: C6 v% b% h3 |( u4 t- K
[node1][DEBUG ] find the location of an executable6 c) f, T1 @* B% U% C" f
[node1][INFO ] Running command: sudo /bin/ceph --cluster=ceph osd stat --format=json
3 \0 J4 h) a$ r( `/ P" E0 l# O* U6 u[node1][INFO ] Running command: sudo systemctl enable ceph.target) y, R+ T$ Y5 G7 v
看日志，激活也没有问题，最后一步，通过 ceph-deploy admin 将配置文件和 admin 密钥同步到各个节点，以便在各个 Node 上使用 ceph 命令时，无需指定 monitor 地址和 ceph.client.admin.keyring 密钥。
2 Y H9 I1 `; y/ c. O! Z$ ceph-deploy admin admin node0 node1
2 m* q# b0 r5 _[ceph_deploy.conf][DEBUG ] found configuration file at: /home/cephd/.cephdeploy.conf: L3 |9 p- M& h8 A
[ceph_deploy.cli][INFO ] Invoked (1.5.39): /bin/ceph-deploy admin admin node0 node11 n$ ~# o9 ]7 Z0 a" u8 \ v3 X
[ceph_deploy.cli][INFO ] ceph-deploy options:
+ k% z' i: z' {[ceph_deploy.cli][INFO ] username : None5 J3 F; N; Z* f% l) J9 Y
[ceph_deploy.cli][INFO ] verbose : False0 h4 u6 m. W8 a$ G0 s4 V
[ceph_deploy.cli][INFO ] overwrite_conf : False! y. ^) i) V! J$ W
[ceph_deploy.cli][INFO ] quiet : False5 \+ V: z/ v$ w+ d! w7 \
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x15a93b0> J, Z1 U1 Q% _8 l1 K- v" a& n
[ceph_deploy.cli][INFO ] cluster : ceph
" \+ @0 Y, I& v' n) K+ t' V[ceph_deploy.cli][INFO ] client : ['admin', 'node0', 'node1']
. ~6 |0 {9 s" `! T2 i* {2 V! i7 H[ceph_deploy.cli][INFO ] func : <function admin at 0x1500cf8>
6 o( ]8 m+ | k! j[ceph_deploy.cli][INFO ] ceph_conf : None
- r5 b0 S' a. z; f! k[ceph_deploy.cli][INFO ] default_release : False
( G( I- b- @& V' K8 t- ]7 W2 b[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to admin
( ]' e( I% m/ v' Q9 ][admin][DEBUG ] connection detected need for sudo
# K! ]+ h6 }9 K9 H; A' R[admin][DEBUG ] connected to host: admin
0 a; e1 \% n9 R9 u[admin][DEBUG ] detect platform information from remote host8 C- P- u ~& P8 C4 j8 D! `
[admin][DEBUG ] detect machine type
! p: N* G- a8 p' K% k[admin][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
$ {2 D1 h0 D, h7 |) W3 M+ L; i[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node0
/ g5 P& |4 C, B; Z: Q/ U, |& }8 o[node0][DEBUG ] connection detected need for sudo
& Y1 U1 i# J J: L+ {+ m) \[node0][DEBUG ] connected to host: node0 7 h, Z% n$ n! l
[node0][DEBUG ] detect platform information from remote host
: p$ j$ K l- _2 A v[node0][DEBUG ] detect machine type
, C( w% {! T% Q4 a S, I[node0][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf% r! {+ |1 r) }' p8 t3 I; N
[ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to node1
$ H6 y! I5 v6 D6 ?: N9 R[node1][DEBUG ] connection detected need for sudo
7 m/ n5 @! f6 e2 ?7 e4 }, ~5 }[node1][DEBUG ] connected to host: node1 4 O9 o. N8 D8 p: w
[node1][DEBUG ] detect platform information from remote host/ ~, K2 T2 B" i7 F2 c& }- Y9 O! |
[node1][DEBUG ] detect machine type
* r# h9 s' L ^+ `1 ^1 h[node1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf+ \; v" L- T5 R8 C, K( ~0 w
同时为了确保对 ceph.client.admin.keyring 有正确的操作权限，所以还需要增加权限设置。
4 U- a1 W0 Q6 a" q. O7 ~* g$ sudo chmod +r /etc/ceph/ceph.client.admin.keyring) t5 v' ~1 f2 x! \9 z
至此，Ceph 存储集群已经搭建完毕了，我们可以查看那一下集群是否启动成功！
1 B7 _2 Q4 w. F7 L2 h# 查看集群状态. z& N/ T& B- W/ U* O8 x3 v
$ ceph -s
7 W2 x# |+ r0 j; @ B/ F$ g" Ucluster 363c3cf4-dba4-4f13-8ec5-ed93416f4e26
, K2 V1 s9 c( K2 I health HEALTH_OK
& Y5 }, Z4 V. |1 A1 u; ]9 Q monmap e1: 1 mons at {admin=10.222.77.213:6789/0}8 m* z& }# ^& Z) g; u1 e; _
election epoch 6, quorum 0 admin. u' O$ D g- q! ]/ [+ m. k+ V
osdmap e15: 2 osds: 2 up, 2 in5 v% j% t) \( P7 ]
flags sortbitwise,require_jewel_osds
/ t5 b/ F1 S" M1 ~' F' Y pgmap v9276: 64 pgs, 1 pools, 863 MB data, 226 objects
' m7 \" {& o, H: v ~ 33440 MB used, 33574 MB / 67015 MB avail
! y% M2 r' @/ A" G5 l 64 active+clean
8 D+ q* \2 Z% O+ h# E% b
M; {2 d6 S2 v! X8 |4 S. P# 或者查看集群健康状况
7 D) D- D5 M' i. K5 p0 g$ ceph health
& L% U& m4 X& |9 W# PHEALTH_OK' Q0 ~# F" e8 d+ Y7 T, h7 ~, Q$ s& g
: j2 o2 d" d; D, e6 x4 W" j
# 查看集群 OSD 信息( T) I3 J+ N7 h
$ ceph osd tree
9 K6 m5 v0 D" J( r" R$ FID WEIGHT TYPE NAME UP/DOWN REWEIGHT PRIMARY-AFFINITY
# }: u% M* Q+ F8 J-1 0.06400 root default ' @) x: ^9 v& l
-2 0.03200 host node0 & \- s$ o) k6 K( [! X0 @. A
0 0.03200 osd.0 up 1.00000 1.00000
8 N+ n8 q6 ~) u; g: M, ?- r-3 0.03200 host node1 + k5 C: a$ K4 ] T$ A5 @6 t
1 0.03200 osd.1 up 1.00000 1.00000
- l, r2 x# P9 J5 M好了，通过参考官方文档中的 Ceph 安装和集群搭建说明，一步一步的完成整个存储集群的搭建，文档写的很详细，非常容易上手操作。本次就先记录到这，下一篇我们继续研究学习 Ceph 存储体系中的对象存储、块设备和文件系统。 |
|
发表于 2018-7-18 10:20:46
