|
|
楼主 |
发表于 2018-9-28 12:35:27
|
显示全部楼层
添加mon节点$ }& M( q1 W- [/ Y$ y/ ]& p
ceph monitor的个数是2n+1(n>=0)个，在线上至少3个，只要正常的节点数>=n+1，ceph的paxos算法能保证系统的正常运行。所以，对于3个节点，同时只能挂掉一个。
+ K. C1 t: u, D 当前ceph cluster中仅含有1个mon节点，将其扩展至3个mon节点。
$ D, Y( K; J( p& ]6 O+ p 查看当前ceph cluster状态0 e9 t% `, E c/ _
[root@ceph-osd-1 ceph-cluster]# ceph -s @) m% P, c% u3 N' ?
cluster 9d717e10-a708-482d-b91c-4bd21f4ae36c
9 v. K' G1 e8 v! J9 ] health HEALTH_OK. C9 Z3 S' z$ C$ f0 s+ f
monmap e5: 1 mons at {ceph-osd-1=10.10.200.163:6789/0}, election epoch 69, quorum 0 ceph-osd-1
2 l7 m+ |& t% } osdmap e220: 7 osds: 7 up, 7 in
8 Z: {. x3 X. b! N pgmap v473: 256 pgs, 1 pools, 0 bytes data, 0 objects
2 B$ b/ }3 L6 G8 Z) O 36109 MB used, 11870 GB / 11905 GB avail
2 v6 L& h' N+ v& t" M: t8 u 256 active+clean
/ L2 ] H! s, }+ \1 m " y5 }! ? r+ k# |0 f; \1 V
此时要向ceph cluster中添加两个节点分别为ceph-osd-2，ceph-osd-3
9 k5 \( l% S/ {; J$ m( P 首先修改配置文件如下，添加public_network0 B( z) E0 k% s
[global]/ O# G4 Z5 q( R0 W: D+ P6 x" Y
auth_service_required = cephx9 q1 p- I/ W$ _9 L
filestore_xattr_use_omap = true8 c [: T7 h9 U. R8 F9 `5 {
auth_client_required = cephx1 \2 `5 g1 T: D( i0 P
auth_cluster_required = cephx
' t1 D. D; G" X. a, Q" Q) h$ g- m4 Nmon_host = 10.10.200.163, 10.10.200.164# p) F/ y, G- c: b. s( }9 Z0 Q. |; q
mon_initial_members = ceph-osd-1, ceph-osd-2* f, K ?' P4 w0 a ~1 A
fsid = 9d717e10-a708-482d-b91c-4bd21f4ae36c* v! ]7 z4 ^1 ?" O! u Y
public_network = 10.10.200.0/24& U" w1 u/ n5 B; y5 o2 W9 ?
0 C8 J9 W# v2 Y' V! b/ q: e
添加mon节点
) r6 i' h \9 x: O7 _7 ~" I. c[root@ceph-osd-1 ceph-cluster]# ceph-deploy mon create ceph-osd-2 ceph-osd-3</span>2 ]+ x$ m; w0 o% ]* J* h. M4 U, M
2 e) Z9 y$ K7 V4 ?
查看添加mon节点后，查看mon quorum状态信息
( w, [* e2 J5 M6 @[root@ceph-osd-1 ceph-cluster]# ceph quorum_status --format json-pretty A! ^0 R6 [; p5 ~. e- G( ]
5 F9 D! ?) X( }& Q/ B# V6 D{ "election_epoch": 72,4 r3 [( u! Y1 V0 a Y4 T
"quorum": [ H( J0 a) p( u% z
0,
; s7 c6 K F& i# c* I$ D- g) I: j 1,
, w9 \9 @0 K I9 M1 t( i- ^ 2],
7 y! a) ~1 T9 k$ @6 z9 \6 B "quorum_names": [
/ F5 |% Q% n! A- j" I+ j "ceph-osd-1",2 }+ B N2 s4 F$ l
"ceph-osd-2",2 K- L& _9 {5 ^% b
"ceph-osd-3"],
" f; Y3 V# K: d& J0 m# H% d "quorum_leader_name": "ceph-osd-1",
; G; n4 Y' S$ h. ` "monmap": { "epoch": 7,
( K0 [- G: n* g: \$ e "fsid": "9d717e10-a708-482d-b91c-4bd21f4ae36c",
2 X1 b6 c4 M s5 N+ }5 C "modified": "2014-11-14 09:10:28.111133",! F) ^, m$ b+ \; b+ q: N
"created": "0.000000",
) s' b; R& ^% ~. K "mons": [- B- F' T4 y8 i9 N
{ "rank": 0,8 Z5 G) ~5 j) h. I/ u5 z& a
"name": "ceph-osd-1",: j: t/ W9 J. S7 J# Y. H
"addr": "10.10.200.163:6789\/0"},2 i+ z0 f4 z7 ]9 R/ k
{ "rank": 1,' f0 d+ f3 @& x V; Q
"name": "ceph-osd-2",4 D& L/ e. V# v6 F
"addr": "10.10.200.164:6789\/0"},3 o. M5 d' Q# w# h @! C
{ "rank": 2,/ u) c+ C' C* I, J
"name": "ceph-osd-3",0 ^) q# n/ C% Q& ^+ g' r0 t
"addr": "10.10.200.165:6789\/0"}]}}
^" M3 U6 u/ \% M# s7 B# Y9 R( ^, B$ o- `
查看此时ceph cluster状态7 i& Q% S" i1 N
[root@ceph-osd-1 ceph-cluster]# ceph -s
+ b; t0 ]1 J5 B5 c7 l3 {% P cluster 9d717e10-a708-482d-b91c-4bd21f4ae36c& y; }9 d1 m1 p9 _) v
health HEALTH_WARN clock skew detected on mon.ceph-osd-3
3 D6 H0 z5 C5 ~" b2 ` e monmap e7: 3 mons at {ceph-osd-1=10.10.200.163:6789/0,ceph-osd-2=10.10.200.164:6789/0,ceph-osd-3=10.10.200.165:6789/0}, election epoch 72, quorum 0,1,2 ceph-osd-1,ceph-osd-2,ceph-osd-3# e' c4 ] E) N2 {
osdmap e220: 7 osds: 7 up, 7 in
0 z" h# q( a/ {4 D' A pgmap v475: 256 pgs, 1 pools, 0 bytes data, 0 objects
5 R, D% K* X9 B, ^6 {& `) H 36109 MB used, 11870 GB / 11905 GB avail9 `8 D# d/ X$ d! ?2 Y2 T
256 active+clean2 Y+ ~$ X8 i* Q# f
+ H6 w) L @( ^! M! n
可以发现mon.ceph-osd-3节点的时间与mon.ceph-osd-1的时间不同步，同步各mon节点的时间。
2 G& k) L9 l. `: U0 c 此时ceph mon节点已经添加完毕，模拟ceph-osd-1 mon节点故障，查看ceph cluster能否正常工作，查看此时ceph cluster信息
/ V2 y1 ?& F) R( v$ L* y[root@ceph-osd-2 ~]# ceph -s
5 g: e8 E/ g% {2 Y2014-11-14 09:27:28.582467 7f9cd8712700 0 -- :/1014338 >> 10.10.200.163:6789/0 pipe(0x7f9cd4024230 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f9cd40244c0).fault x& e0 @7 i: l' C
cluster 9d717e10-a708-482d-b91c-4bd21f4ae36c
/ E6 G" `; M" v0 X$ j) F health HEALTH_WARN 256 pgs degraded; 256 pgs stuck unclean; 256 pgs undersized; 1/7 in osds are down; 1 mons down, quorum 1,2 ceph-osd-2,ceph-osd-3" m. W8 f0 }. N: p: y: ]' I, _
monmap e7: 3 mons at {ceph-osd-1=10.10.200.163:6789/0,ceph-osd-2=10.10.200.164:6789/0,ceph-osd-3=10.10.200.165:6789/0}, election epoch 88, quorum 1,2 ceph-osd-2,ceph-osd-32 M. S/ m- L. w+ j3 A/ w! q
osdmap e263: 7 osds: 6 up, 7 in
% q" |* A" r& C/ z7 z% ^ pgmap v542: 256 pgs, 1 pools, 0 bytes data, 0 objects/ i8 @$ x' U2 k/ `: w- B# ]8 \
36112 MB used, 11870 GB / 11905 GB avail
+ X3 u) ~! o8 w0 I7 g/ x 256 active+undersized+degraded S6 G1 W Q6 Z, L' \
/ b6 p9 i5 Q( }$ [! L2 Y. ]4 p& p9 y* U 因为ceph-osd-1节点上面拥有1个mon节点以及1个osd节点，所以在osd cluster中，有个osd也处于down状态。
% E/ k: p: o9 ~( v8 K 本文的开头部分讲过，ceph mon规定在3个节点的状态下，只允许1个mon节点down，那么2个mon节点down会怎么样，继续down掉ceph-osd-2节点
- T; {+ ]$ d, Q. z' ]4 O 通过ceph -s查看此时ceph cluster状态信息5 c3 J; u; S* z4 K @& c# ^
[root@ceph-osd-3 ~]# ceph -s
0 u( d5 H& `! o9 V* e# Y; o2014-11-14 09:30:23.483264 7f677c28b700 0 -- :/1014680 >> 10.10.200.163:6789/0 pipe(0x7f6778023290 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f6778023520).fault
3 l% m& ^- C7 J4 [+ o5 M2014-11-14 09:30:26.483313 7f677c18a700 0 -- :/1014680 >> 10.10.200.164:6789/0 pipe(0x7f676c000c00 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f676c000e90).fault$ W7 a5 j6 t4 K
2014-11-14 09:30:29.483664 7f677c28b700 0 -- :/1014680 >> 10.10.200.163:6789/0 pipe(0x7f676c0030e0 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f676c003370).fault
3 [4 S* X1 X+ p( s. U' f2014-11-14 09:30:32.483904 7f677c18a700 0 -- :/1014680 >> 10.10.200.164:6789/0 pipe(0x7f676c003a00 sd=4 :0 s=1 pgs=0 cs=0 l=1 c=0x7f676c003c90).fault0 {7 V$ h7 }) N$ Z5 Q
2014-11-14 09:30:35.484221 7f677c28b700 0 -- :/1014680 >> 10.10.200.163:6789/0 pipe(0x7f676c0031b0 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f676c002570).fault
& k( J6 f% I4 X! ^) C5 O1 ]2014-11-14 09:30:38.484476 7f677c18a700 0 -- :/1014680 >> 10.10.200.164:6789/0 pipe(0x7f676c002a60 sd=3 :0 s=1 pgs=0 cs=0 l=1 c=0x7f676c002cf0).fault
1 w) d7 p. V/ A$ r: c5 ~' s3 S7 l. T6 Y; j6 u( a
通过以上信息，ceph cluster已经无法正常工作。所以在3节点的mon cluster中，仅允许1个mon节点down掉。
6 ~+ k/ A7 G r+ h0 D, E6 C, y- \* F9 b
|
|
发表于 2018-9-28 12:09:53