找回密码
 注册
查看: 2122|回复: 1

HEALTH_WARN Degraded data redundancy ... has slow ops

[复制链接]

1

主题

0

回帖

12

积分

管理员

积分
12
QQ
发表于 2021-10-25 17:00:42 | 显示全部楼层 |阅读模式
HEALTH_WARN Degraded data redundancy: 822/429494 objects degraded (0.191%), 115 pgs degraded; 154 slow ops, oldest one blocked for 389 sec, mon.ceph1 has slow ops  ]9 K6 Z% V6 M, b6 x
PG_DEGRADED Degraded data redundancy: 822/429494 objects degraded (0.191%), 115 pgs degraded
9 w9 ]7 z- C) Q* Y' [* l7 |    pg 8.473 is active+undersized+degraded, acting [18,9]' _( U$ H; X1 O/ m  Q% z. ]
    pg 8.483 is active+undersized+degraded, acting [9,20]! p  E% O6 q; N1 r# l3 `0 `
    pg 8.48a is active+undersized+degraded, acting [11,21], x3 ~7 |: i) X
    pg 8.49d is active+undersized+degraded, acting [23,6]- H* \1 l2 g# G0 U4 @+ B5 k& n
    pg 8.4a7 is active+undersized+degraded, acting [11,20]
' V% N$ e' l1 b    pg 8.4a8 is active+undersized+degraded, acting [8,20]
" ]* M* [. Y1 N3 g: }/ h4 _+ n) O    pg 8.4a9 is active+undersized+degraded, acting [18,8]4 p. o* r) H* N( K; L6 Q$ Y3 z
    pg 8.4b2 is active+undersized+degraded, acting [11,21]
4 r2 r( @" S7 a) r9 T4 k# c9 a    pg 8.4c8 is active+undersized+degraded, acting [7,22]
# c! g. C% U1 U3 d3 s    pg 8.4e4 is active+undersized+degraded, acting [9,20]& e. l/ J6 G: I* ~
    pg 8.4e7 is active+undersized+degraded, acting [18,11]
* E( C/ |3 ?# J    pg 8.4f2 is active+undersized+degraded, acting [23,8]2 I4 N8 ?4 N5 O8 t
    pg 8.4fb is active+undersized+degraded, acting [18,7]
! R+ E6 F4 e5 P) W% R0 @    pg 8.505 is active+undersized+degraded, acting [8,21]
& w9 q5 s; v( d6 |) J6 M" C    pg 8.54e is active+undersized+degraded, acting [8,22]
, t9 L8 p% F' F2 O% ^    pg 8.566 is active+undersized+degraded, acting [23,10]) g' \  u4 w5 ~
    pg 8.567 is active+undersized+degraded, acting [8,19]
$ S) s: M2 z1 Z+ K1 v* K# W" A3 F    pg 8.57c is active+undersized+degraded, acting [23,9]
5 p! L7 |# [5 u( G1 |: J7 M# a    pg 8.57d is active+undersized+degraded, acting [8,22]5 W2 L, i& }9 d
    pg 8.596 is active+undersized+degraded, acting [23,8]
) N; n: F, X  P    pg 8.5b3 is active+undersized+degraded, acting [9,18]+ Q% O. s( F# A3 g( [' r/ G9 }' j" Q
    pg 8.5fc is active+undersized+degraded, acting [8,19]
7 `! K6 [- @9 }" x% D$ U3 c! {, {    pg 8.606 is active+undersized+degraded, acting [8,19]# R4 z& [! R+ [, K& |/ P
    pg 8.627 is active+undersized+degraded, acting [11,22], p% e, \2 ]( |- {% y8 O
    pg 8.636 is active+recovery_wait+degraded, acting [15,21,10]7 D# C- a& E" Z& U. B# q+ d; Y/ ]  a
    pg 8.651 is active+undersized+degraded, acting [23,6]+ O( C5 [/ R$ j: \, @, v" _
    pg 8.65b is active+undersized+degraded, acting [11,21]6 l0 w* n  V- c& g3 d. [; v* f6 u  s
    pg 8.666 is active+undersized+degraded, acting [23,6]
& w/ v! J/ t& T7 h" }5 e3 o% {2 z$ |    pg 8.679 is active+undersized+degraded, acting [11,21]
6 ]1 |2 B; D8 X% ]; ]9 W    pg 8.6a3 is active+undersized+degraded, acting [7,22]
+ [; a  i  \9 k2 j/ C, I; A    pg 8.6c3 is active+undersized+degraded, acting [23,6]
1 |4 V  e, l; ~4 h6 g1 t7 H$ C    pg 8.6e1 is active+undersized+degraded, acting [8,18]$ z" F. u6 c. L; F' q  Q, @1 \9 \) ~
    pg 8.6ed is active+undersized+degraded, acting [8,22]
' H) l" O) K6 P    pg 8.701 is active+undersized+degraded, acting [7,19]
0 P. g5 E% K9 I6 W    pg 8.724 is active+undersized+degraded, acting [23,10]: Z( k0 t6 b) ?' @; M0 D5 T' b
    pg 8.728 is active+undersized+degraded, acting [18,10]
% T4 v$ i/ q1 V' w  E    pg 8.72f is active+undersized+degraded, acting [7,23]! E2 w0 p6 s) @# J* T
    pg 8.732 is active+undersized+degraded, acting [9,18]& V2 F3 D7 o5 `4 S
    pg 8.739 is active+undersized+degraded, acting [9,22]; F3 @3 h& X# y) W) M; ?
    pg 8.76a is active+undersized+degraded, acting [11,21]
. T, P+ |) n, O( G! [+ C1 t2 B    pg 8.770 is active+undersized+degraded, acting [23,9]% B: C/ @2 k( w" c7 @% n9 u* u% Q
    pg 8.781 is active+undersized+degraded, acting [7,20]$ W; r/ R8 l, i$ r' q3 s
    pg 8.791 is active+undersized+degraded, acting [11,22]9 `; s& w6 R! s3 u* ~# m& Q
    pg 8.7b5 is active+undersized+degraded, acting [23,10]
8 N8 J# n/ k1 W) i$ ]    pg 8.7bb is active+undersized+degraded, acting [8,18]$ U+ C, \, |2 a$ n1 K$ ^
    pg 8.7be is active+recovery_wait+degraded, acting [15,19,6]
! M" z4 o. W2 `8 @    pg 8.7c3 is active+undersized+degraded, acting [23,9]
% ^, X& p, j) K/ [    pg 8.7d4 is active+undersized+degraded, acting [9,18]
4 t# N; \" I! p7 [# ~    pg 8.7da is active+undersized+degraded, acting [18,10]  n1 i/ Z% a8 S; D5 T% f8 [2 W
    pg 8.7e5 is active+undersized+degraded, acting [18,8]
' |3 H5 ]4 [3 x- `. N, g& o    pg 8.7fa is active+undersized+degraded, acting [8,21]
6 M& F  k" I9 ?8 Y  \SLOW_OPS 154 slow ops, oldest one blocked for 389 sec, mon.ceph1 has slow ops
8 {0 c) d- {* f/ A0 E) g2 u; S' U

1

主题

0

回帖

12

积分

管理员

积分
12
QQ
 楼主| 发表于 2021-10-25 17:04:32 | 显示全部楼层
解决办法:5 s, H3 R4 I' W* f# W
systemctl restart ceph-mon@`hostname -s`.service1 L6 v8 k& T% q; l
' F, k4 t8 s7 i& |  w& y2 q

: x, \  a2 t# i- q( l报mon的错误被清除了。* e* D; Q0 Y8 U5 X4 V( p

: h: ^  U  ^$ }6 i. R[root@ceph1 ~]# ceph health detail
6 H4 u* Z4 ?% ]1 \, _# ~! n# G) bHEALTH_WARN 6 osds down; Reduced data availability: 170 pgs inactive; Degraded data redundancy: 9370/429503 objects degraded (2.182%), 1102 pgs degraded
  {3 T5 d: o" ^5 X2 cOSD_DOWN 6 osds down' ^1 x  K+ U/ {5 l5 a4 T
    osd.20 (root=hdd,host=hdd-ceph3) is down
3 @; E; c- i6 y: E) g! W- r    osd.22 (root=hdd,host=hdd-ceph3) is down
) }/ m" |! Z8 {6 f' Y! {: ?    osd.23 (root=hdd,host=hdd-ceph3) is down! w6 `- N; w; ?: I( o( y0 j
    osd.25 (root=ssd-openstack,host=ssd-compute1) is down
: U7 N' `6 `" p/ Q) m1 h    osd.28 (root=ssd-openstack,host=ssd-compute2) is down
  l# f4 g4 B$ l0 q    osd.29 (root=ssd-openstack,host=ssd-compute2) is down# A( |3 z* m  ~: t0 E  n
PG_AVAILABILITY Reduced data availability: 170 pgs inactive1 U* s! L- J# l- W- g7 m$ n: t
    pg 7.13a is stuck inactive for 268.657552, current state undersized+peered, last acting [7]
: [7 w4 `0 N- g    pg 7.140 is stuck inactive for 173.462639, current state undersized+peered, last acting [6]) N- H' m, E3 i- p- a3 Q
    pg 7.149 is stuck inactive for 188.507770, current state undersized+peered, last acting [15]
2 z1 c* N4 d) h0 @. V    pg 7.14a is stuck inactive for 296.614537, current state undersized+peered, last acting [11]
  w) m0 U- m. `* u% y    pg 7.15a is stuck inactive for 230.993717, current state undersized+peered, last acting [7]  \+ p% l+ R) u3 \' q
    pg 7.160 is stuck inactive for 157.298697, current state undersized+peered, last acting [16]  W$ y- z" N/ f) ^. |0 x5 P; a
    pg 7.162 is stuck inactive for 168.378304, current state undersized+peered, last acting [8]
7 P) X- e4 e0 a: D/ ]( t& f    pg 7.163 is stuck inactive for 230.964708, current state undersized+peered, last acting [11]
' e0 i5 t/ x- M! G" p2 l. x    pg 7.164 is stuck inactive for 193.535188, current state undersized+peered, last acting [12]
; `' M* }" y1 H: l: g# K0 F    pg 7.166 is stuck inactive for 157.314779, current state undersized+peered, last acting [9]  E* C4 n! G7 E5 l% R1 q
    pg 7.167 is stuck inactive for 231.001296, current state undersized+peered, last acting [14]# \: }( _- c5 |! W/ }
    pg 7.16f is stuck inactive for 157.301826, current state undersized+peered, last acting [6]; ~9 o0 m; u9 P0 _+ I: g: W& ]; n
    pg 7.18e is stuck inactive for 230.971182, current state undersized+peered, last acting [17]& h0 G" i0 W4 ~' R" f6 V% L
    pg 7.196 is stuck inactive for 157.295252, current state undersized+peered, last acting [10]
% d0 Z) x% X3 A# \    pg 7.332 is stuck inactive for 188.514026, current state undersized+peered, last acting [15]
# P. ?# d- [4 g6 E/ g    pg 7.333 is stuck inactive for 230.989099, current state undersized+peered, last acting [17]
+ ]& R( M% l; _# p% V    pg 7.347 is stuck inactive for 243.608764, current state undersized+peered, last acting [14]1 e7 `, _6 ~- X0 L: K
    pg 7.352 is stuck inactive for 230.961993, current state undersized+peered, last acting [17]' u( I4 T4 G' Q
    pg 7.358 is stuck inactive for 296.668417, current state undersized+peered, last acting [13]
4 z: H8 s+ I4 W' k! z' s    pg 7.35b is stuck inactive for 296.611193, current state undersized+peered, last acting [9]0 U* h: X: [$ f9 _5 V' u
    pg 7.364 is stuck inactive for 231.014057, current state undersized+peered, last acting [7]
6 K0 C. I: E' ~  D. n    pg 7.372 is stuck inactive for 230.976751, current state undersized+peered, last acting [11]
% M' [+ {" m; W: _# r9 Z$ w    pg 7.374 is stuck inactive for 157.314500, current state undersized+peered, last acting [13]. `9 g3 y  B5 f, ~4 j! {
    pg 7.375 is stuck inactive for 268.647540, current state undersized+peered, last acting [7]( v) y# B9 Q, q4 ?+ x
    pg 7.376 is stuck inactive for 157.302772, current state undersized+peered, last acting [7]
5 z) g5 Z: j; a8 w    pg 7.37c is stuck inactive for 157.292778, current state undersized+peered, last acting [9]* W. A0 I& r$ F  a
    pg 7.381 is stuck inactive for 268.670664, current state undersized+peered, last acting [7]' c2 g+ W. q0 f" _: M
    pg 7.38a is stuck inactive for 157.311084, current state undersized+peered, last acting [6]
# u' ]. I/ k3 A9 ~" `' D    pg 7.38b is stuck inactive for 230.974616, current state undersized+peered, last acting [10]+ F0 {( X) T& w
    pg 7.391 is stuck inactive for 253.571528, current state undersized+peered, last acting [10]
* b! k/ q9 X8 B    pg 7.399 is stuck inactive for 231.014969, current state undersized+peered, last acting [7]
. K& [- V) p9 T- u) Q5 h4 G) z    pg 7.3a0 is stuck inactive for 157.314500, current state undersized+peered, last acting [17]
/ k2 F- \5 z" v3 x    pg 7.3a3 is stuck inactive for 157.310316, current state undersized+peered, last acting [16]# E: J( N. U/ m4 ]
    pg 7.3ad is stuck inactive for 243.592394, current state undersized+peered, last acting [14]
; j4 I+ ]6 L. X' g& B6 ]9 N    pg 7.3af is stuck inactive for 188.502747, current state undersized+peered, last acting [15]3 I5 c' R) l& h. A- K
    pg 7.3b2 is stuck inactive for 168.373475, current state undersized+peered, last acting [8]) n3 T+ u# f, j' g, k+ ?% |8 d% Y
    pg 7.3b4 is stuck inactive for 173.465040, current state undersized+peered, last acting [6]
" O* p# U# Q& g8 B0 I0 W    pg 7.3b5 is stuck inactive for 157.322812, current state undersized+peered, last acting [9]( F; H& I" A8 X: K, |, t2 d
    pg 7.3b6 is stuck inactive for 230.969301, current state undersized+peered, last acting [11]+ `: V8 \$ ^5 [1 r( |' A
    pg 7.3b9 is stuck inactive for 157.296625, current state undersized+peered, last acting [14]
2 M8 T2 q1 y9 U1 H4 e& i    pg 7.3ba is stuck inactive for 231.015066, current state undersized+peered, last acting [7]
3 j% K  k3 O0 q. R  J+ L1 L! x    pg 7.3be is stuck inactive for 157.311582, current state undersized+peered, last acting [8]
; g2 h( `; t: v# t. V/ _2 N; h9 `  P    pg 7.3c5 is stuck inactive for 168.378059, current state undersized+peered, last acting [8]
8 y* r+ J# _+ S+ G    pg 7.3cd is stuck inactive for 230.974691, current state undersized+peered, last acting [10]
4 ~; B/ p9 ~2 x0 ~: N7 @* ]    pg 7.3cf is stuck inactive for 168.383761, current state undersized+peered, last acting [8]
8 t6 r/ l- c4 i& j    pg 7.3df is stuck inactive for 157.314279, current state undersized+peered, last acting [10]
% {4 @  s. [( _- ]    pg 7.3e3 is stuck inactive for 296.651325, current state undersized+peered, last acting [16]
0 _3 o5 f- G: \! `    pg 7.3eb is stuck inactive for 173.465968, current state undersized+peered, last acting [6]& Y! N2 c0 a# v  [8 K/ |/ \
    pg 7.3f8 is stuck inactive for 230.983836, current state undersized+peered, last acting [11]
5 b5 F3 W9 g  j% V: m' @    pg 7.3fb is stuck inactive for 157.305951, current state undersized+peered, last acting [16]
; M) A6 K" y& o# D/ p1 U    pg 7.3ff is stuck inactive for 268.659557, current state undersized+peered, last acting [7]
9 H5 C/ M& B- C. x6 b" f; CPG_DEGRADED Degraded data redundancy: 9370/429503 objects degraded (2.182%), 1102 pgs degraded
% U( U3 |$ {1 c0 [5 q; K; g9 w& d    pg 8.795 is active+undersized+degraded, acting [11,17]
, I6 G) T# {+ U/ d9 }    pg 8.79b is active+undersized+degraded, acting [10,12]7 B* e+ h' o+ [: P7 ]5 W( F
    pg 8.79c is active+undersized+degraded, acting [13,10]" L5 g" m9 r4 m0 a
    pg 8.79f is active+undersized+degraded, acting [9,13]
# U" u9 Q; {' ]$ d, ~    pg 8.7a0 is active+undersized+degraded, acting [8,17], l! B2 t0 g* n) ^/ Z) V; ], }
    pg 8.7a1 is active+undersized+degraded, acting [16,9]
& q. t' w: g# ^8 s  w    pg 8.7a2 is active+undersized+degraded, acting [10,17], G* ^& u4 W& O( J
    pg 8.7a3 is active+undersized+degraded, acting [17,11]
  o5 ?7 a0 _, e+ @# l0 \    pg 8.7a5 is active+undersized+degraded, acting [11,17]
8 U9 s6 c# W3 _% z) x- h    pg 8.7a8 is active+undersized+degraded, acting [13,7]9 k6 E4 b9 U' y* [" ~
    pg 8.7a9 is active+undersized+degraded, acting [15,7]
+ q4 ~5 ]& |8 l9 Z, Z    pg 8.7ad is active+undersized+degraded, acting [15,9]
3 s9 }% t6 y6 L5 a8 u    pg 8.7ae is active+undersized+degraded, acting [14,10]
3 K  Z2 [; [* N7 h! s5 \7 d* q* s# I5 b    pg 8.7b0 is active+undersized+degraded, acting [12,6]0 `( R! g8 f8 L. h
    pg 8.7b5 is active+undersized+degraded, acting [10,15]7 V) b1 e0 O2 I7 u& M* q+ ]
    pg 8.7b6 is active+undersized+degraded, acting [15,9]0 G$ ~$ a( ~/ d7 Y* B
    pg 8.7b8 is active+undersized+degraded, acting [8,16]
8 C( l8 @* A8 ~) c    pg 8.7ba is active+undersized+degraded, acting [13,6]2 F: H8 R7 x: x9 u% M: I* U
    pg 8.7bd is active+undersized+degraded, acting [7,13]
7 m+ S# P; n$ w9 a6 [2 v! ?    pg 8.7bf is active+undersized+degraded, acting [17,11]2 v% y5 _- l7 n/ x% G
    pg 8.7c1 is active+undersized+degraded, acting [12,10]. l- C8 U7 `; [
    pg 8.7c2 is active+undersized+degraded, acting [6,12]$ J  w; U. {9 s2 O) B8 J* M
    pg 8.7c3 is active+undersized+degraded, acting [9,15]
( Y  R( e2 l2 [# m    pg 8.7c4 is active+undersized+degraded, acting [6,16]
- v1 i5 O4 {& p5 v* g( x5 d' ?, `2 _    pg 8.7c5 is active+undersized+degraded, acting [8,14]
! v  l, X6 \+ K2 }9 u    pg 8.7c6 is active+undersized+degraded, acting [17,7]: N" ~0 z4 ?% p; B
    pg 8.7c7 is active+undersized+degraded, acting [16,11]
6 w$ |/ c" Z5 V" V. \    pg 8.7ca is active+undersized+degraded, acting [16,11]2 Q/ n  b5 t+ z7 q9 p
    pg 8.7cd is active+undersized+degraded, acting [13,6]
7 }5 A' K: B' k% `  H( n    pg 8.7ce is active+undersized+degraded, acting [13,8]
: s) Q, d$ [3 L" Z+ e    pg 8.7d0 is active+undersized+degraded, acting [17,10]/ i0 @7 I0 I) N& h' U0 o
    pg 8.7d5 is active+undersized+degraded, acting [14,11]9 g! F: D+ {% X- S9 Y$ [! q) I
    pg 8.7d8 is active+undersized+degraded, acting [13,10]9 ^# J  W4 F! R8 |* s
    pg 8.7d9 is active+undersized+degraded, acting [9,13]
7 n8 F0 H5 S8 k6 k9 g  A9 c    pg 8.7dd is active+undersized+degraded, acting [12,11]
8 \* _) @% G7 A. C! l' |) c( z    pg 8.7df is active+undersized+degraded, acting [11,16]
- \2 O% Z' S- G6 j% d2 m6 G$ L; F    pg 8.7e0 is active+undersized+degraded, acting [17,6]
: B, ^) ]3 ]$ [, l; ~3 m    pg 8.7e1 is active+undersized+degraded, acting [14,10]
1 l7 N3 H$ D6 N  g  n+ B" Y& }( q! K    pg 8.7e2 is active+undersized+degraded, acting [9,13]
1 a2 ~9 x& z# F8 R% t2 h    pg 8.7e3 is active+undersized+degraded, acting [14,9]
# ~  W, ]+ j. g* d$ ~, B    pg 8.7e7 is active+undersized+degraded, acting [6,15]
( u  ]& O/ R' |- g' x    pg 8.7ea is active+undersized+degraded, acting [9,16]
& ^' ?6 u7 v4 ?8 [: ?    pg 8.7eb is active+undersized+degraded, acting [14,9]
( Z( [7 ^9 J# j% J0 t1 M0 Y: t    pg 8.7ec is active+undersized+degraded, acting [16,11]' Y6 s, b+ T, A* ?- ^
    pg 8.7ed is active+undersized+degraded, acting [14,10]
% L3 K! n& J  R7 T! C) r6 j- M    pg 8.7f0 is active+undersized+degraded, acting [17,6]% }: y' p/ E/ I' C1 q1 X
    pg 8.7f1 is active+undersized+degraded, acting [15,7]
2 R& D0 y/ ?' K9 O# L    pg 8.7f3 is active+undersized+degraded, acting [9,17]: G, p0 K8 }( g, Q
    pg 8.7f4 is active+undersized+degraded, acting [9,14]3 `, X( X# d  B3 D3 K) Q
    pg 8.7fb is active+undersized+degraded, acting [9,17]) w- G, B5 T2 b+ [+ c$ @8 z
    pg 8.7fc is active+undersized+degraded, acting [15,8]
* b1 j) k, a0 X
您需要登录后才可以回帖 登录 | 注册

本版积分规则

返回首页|Archiver|手机版|小黑屋|易陆发现技术论坛 ( 蜀ICP备2026014127号-1 )

GMT+8, 2026-6-12 01:01 , Processed in 0.019955 second(s), 23 queries .

Powered by Discuz! X5.0

© 2001-2026 Discuz! Team.

快速回复 返回顶部 返回列表