|
|
楼主 |
发表于 2021-6-9 15:00:17
|
显示全部楼层
3.1.1 说明: n1 j6 U, D) e0 p' Y2 h! n
降级：由上文可以得知，每个PG有三个副本，分别保存在不同的OSD中，在非故障情况下，这个PG是active+clean 状态，那么，如果PG 的 副本osd.4 挂掉了，这个 PG 是降级状态。+ ] z j0 s" W7 @+ N
3.1.2 故障模拟
) K7 v" `' {/ @( U* e$ Na. 停止osd.1
6 O' p3 S! k/ ]! I4 ~ $ systemctl stop ceph-osd@1
1 E" F5 T4 D W+ H6 Tb. 查看PG状态6 x1 M. K# j) M/ m* L9 {+ l
$ bin/ceph pg stat 20 pgs: 20 active+undersized+degraded; 14512 kB data, 302 GB used, 6388 GB / 6691 GB avail; 12/36 objects degraded (33.333%)
1 U2 f9 _! G, l7 O/ A' N- r zc. 查看集群监控状态
5 Q' [) ~- n# P7 ^ $ bin/ceph health detail
9 h, X% z( ?. H* R1 _) s$ \HEALTH_WARN 1 osds down; Degraded data redundancy: 12/36 objects degraded (33.333%), 20 pgs unclean, 20 pgs degraded; application not enabled on 1 pool(s)
9 N; }! ^6 ~( d1 a3 E# [( |OSD_DOWN 1 osds down
3 P5 ]3 v* v& M4 y+ H& ]. a osd.1 (root=default,host=ceph-xx-cc00) is down * i' J1 p& }" K5 i7 r
PG_DEGRADED Degraded data redundancy: 12/36 objects degraded (33.333%), 20 pgs unclean, 20 pgs degraded
8 V" `& G! A( N7 Y3 F* | pg 1.0 is active+undersized+degraded, acting [0,2] # ^/ a9 Z1 k& [) u2 D4 x+ R) H, e
pg 1.1 is active+undersized+degraded, acting [2,0] 4 M% L2 R j* h1 }& {+ u% D# q
d. 客户端IO操作$ [3 Z) ]! S& Y- c
#写入对象
, t) r. o# t8 ^* O9 Y$ bin/rados -p test_pool put myobject ceph.conf 8 d$ K: B3 c7 o
: p3 G s9 Z0 J2 O% d
#读取对象到文件
; J' _* R: F9 J5 i- s5 c$ bin/rados -p test_pool get myobject.old 0 t2 r3 k, g4 H* p* T
/ J4 O' y" i. E#查看文件 5 ^/ z5 y& j$ c' R3 h4 O" l
$ ll ceph.conf*
& z3 {) H9 K* _% l-rw-r--r-- 1 root root 6211 Jun 25 14:01 ceph.conf % \; L' y/ O9 k. q1 S, w- A6 S
-rw-r--r-- 1 root root 6211 Jul 3 19:57 ceph.conf.old # P5 d% [+ d; o/ i# j
故障总结：
1 f6 ]7 \8 {( f; u2 }为了模拟故障，(size = 3, min_size = 2) 我们手动停止了 osd.1，然后查看PG状态，可见，它此刻的状态是active+undersized+degraded,当一个 PG 所在的 OSD 挂掉之后，这个 PG 就会进入undersized+degraded 状态，而后面的[0,2]的意义就是还有两个副本存活在 osd.0 和 osd.2 上, 并且这个时候客户端可以正常读写IO。! I, g1 o' _8 f& i5 R+ H
3.1.3 总结. D0 Q4 l/ o4 `
降级就是在发生了一些故障比如OSD挂掉之后，Ceph 将这个 OSD 上的所有 PG 标记为 Degraded。
5 O S: e" {; i降级的集群可以正常读写数据，降级的 PG 只是相当于小毛病而已，并不是严重的问题。
" b: n! }' A$ C2 c( NUndersized的意思就是当前存活的PG 副本数为 2，小于副本数3，将其做此标记，表明存货副本数不足，也不是严重的问题。
9 d- `3 Y6 A5 w$ i- K, j/ t3.2 Peered0 ~3 O) U2 N' a) y1 H
3.2.1 说明9 |$ R& J& W4 B$ S9 z- f
Peering已经完成，但是PG当前Acting Set规模小于存储池规定的最小副本数(min_size)。
/ ~. ` O4 ?5 J: j3.2.2 故障模拟7 L7 p. h- \: S9 i9 V
a. 停掉两个副本osd.1,osd.0
* W2 C8 l& _8 M. F $ systemctl stop ceph-osd@1
! {6 U+ D( e' Z" [# i& n! n( j6 L4 R- t $ systemctl stop ceph-osd@0 9 u5 G- i% a- C* E) G( H
) L7 T N- ~6 ~: f# {
9 Z0 A6 t" m6 ~3.2.1 说明
; r5 K" ?9 m0 N1 x xPeering已经完成，但是PG当前Acting Set规模小于存储池规定的最小副本数(min_size)。7 h! d$ Q P% |5 q9 n; Y
3.2.2 故障模拟
$ S0 l7 D( w' U
, A7 ?9 \7 d' p9 w7 i3 e; Ga. 停掉两个副本osd.1,osd.0* v; e! ~) A- Q; n9 v: K
2 m1 o: M" J- p7 D2 U# x
$ systemctl stop ceph-osd@1 7 k; q O% n& h+ i, \
$ systemctl stop ceph-osd@0
* S$ a* L k q) c- m- y2 q2 C0 w; j' b* @, P! G
b. 查看集群健康状态
9 \$ L8 O# G, }0 s8 I% K5 k6 ~. I% I) W5 M) l
$ bin/ceph health detail 3 r) Z. ? f7 |
HEALTH_WARN 1 osds down; Reduced data availability: 4 pgs inactive; Degraded data redundancy: 26/39 objects degraded (66.667%), 20 pgs unclean, 20 pgs degraded; application not enabled on 1 pool(s)
9 ] Y- \+ k3 |; ~OSD_DOWN 1 osds down # k! N; Z* R. D
osd.0 (root=default,host=ceph-xx-cc00) is down
3 l' a1 B. C9 C/ \* N+ fPG_AVAILABILITY Reduced data availability: 4 pgs inactive " I! K. N: _' J% Z
pg 1.6 is stuck inactive for 516.741081, current state undersized+degraded+peered, last acting [2] ( H2 b- E' E' X+ k0 w
pg 1.10 is stuck inactive for 516.737888, current state undersized+degraded+peered, last acting [2] ; T1 h& y# T0 q. A
pg 1.11 is stuck inactive for 516.737408, current state undersized+degraded+peered, last acting [2] # y) ^' ?; }6 v( w
pg 1.12 is stuck inactive for 516.736955, current state undersized+degraded+peered, last acting [2] & e8 u$ G$ H" N) @0 S, N
PG_DEGRADED Degraded data redundancy: 26/39 objects degraded (66.667%), 20 pgs unclean, 20 pgs degraded
2 [8 Z, ]* D# O0 m0 \ pg 1.0 is undersized+degraded+peered, acting [2] - P7 x4 w0 E4 Z; W2 u8 r
pg 1.1 is undersized+degraded+peered, acting [2]
4 K9 _% g6 F* j4 T( Nc. 客户端IO操作(夯住)( B9 ]3 W* D3 s% t( l. ]9 c; `
: p, l) \5 D5 Z0 g2 B Q #读取对象到文件，夯住IO
) v4 M& @* ^2 ~9 W! W4 N) L& s# q; x$ bin/rados -p test_pool get myobject ceph.conf.bak
/ _! @8 C0 {2 m( X故障总结：+ @7 r* ~' s3 R$ f- @9 v
# [: M6 P/ p! J现在pg 只剩下osd.2上存活，并且 pg 还多了一个状态：peered，英文的意思是仔细看，这里我们可以理解成协商、搜索。. d4 ]9 C$ }# z6 T
这时候读取文件，会发现指令会卡在那个地方一直不动，为什么就不能读取内容了，因为我们设置的 min_size=2 ，如果存活数少于2，比如这里的 1 ，那么就不会响应外部的IO请求。; X z* @' `8 L. E. \
d. 调整min_size=1可以解决IO夯住问题' Y. M2 ?) h8 o. t1 R; w: ^+ S
5 u% {5 u/ x0 `, P/ a4 w
#设置min_size = 1 , _, @, r: w/ E5 r4 Z% \
$ bin/ceph osd pool set test_pool min_size 1 : _+ T2 W' e% V( j* H
set pool 1 min_size to 1 " [ C j N8 {4 J- L @1 e
e. 查看集群监控状态! p0 s5 w! ^2 f4 ?8 v- L A5 y
# y: l. ?6 W% i7 P1 b _# a9 B
$ bin/ceph health detail + U/ n3 P( d& `0 q* v6 k
HEALTH_WARN 1 osds down; Degraded data redundancy: 26/39 objects degraded (66.667%), 20 pgs unclean, 20 pgs degraded, 20 pgs undersized; application not enabled on 1 pool(s) ( ^# a: B4 h+ O/ t( P$ L" w! v8 F
OSD_DOWN 1 osds down / C* |5 Q9 i) \ T! _2 ~
osd.0 (root=default,host=ceph-xx-cc00) is down。8 O2 a" \; s( p" K( P9 g+ _
PG_DEGRADED Degraded data redundancy: 26/39 objects degraded (66.667%), 20 pgs unclean, 20 pgs degraded, 20 pgs undersized 3 p8 U8 f1 g. Q, v2 s- |
pg 1.0 is stuck undersized for 65.958983, current state active+undersized+degraded, last acting [2]
& q( i: ^5 t: n5 v0 x% H/ o' |pg 1.1 is stuck undersized for 65.960092, current state active+undersized+degraded, last acting [2]
/ a$ W1 L0 N% O: R2 Z* Zpg 1.2 is stuck undersized for 65.960974, current state active+undersized+degraded, last acting [2]
* c( v- h1 t- G9 p. |! {3 xf. 客户端IO操作 U$ [4 y" q% W! I+ _. C
F5 M- i7 G/ ~, `& v* s& W7 s
#读取对象到文件中
& K; F+ h# V, W+ O) J( j' q$ ll -lh ceph.conf* : K3 e, Y8 j5 n; X
-rw-r--r-- 1 root root 6.1K Jun 25 14:01 ceph.conf
4 }5 c$ R5 n3 n7 K& i3 s( G-rw-r--r-- 1 root root 6.1K Jul 3 20:11 ceph.conf.bak % ?6 H" F/ @7 {- X
-rw-r--r-- 1 root root 6.1K Jul 3 20:11 ceph.conf.bak.1
) B# Z- F8 g& x故障总结：
& H/ y L1 }0 F
2 b. n3 d0 P+ L% ]0 T: t% y可以看到，PG状态Peered没有了，并且客户端文件IO可以正常读写了。9 D+ l3 E4 u8 Z0 Q5 \/ J* |
当min_size=1时，只要集群里面有一份副本活着，那就可以响应外部的IO请求。$ b+ [, c w+ o6 u3 r+ D
9 G' ]" O# [2 Z# q
|
|
发表于 2021-6-9 15:00:16