找回密码
 注册
查看: 2120|回复: 1

HEALTH_WARN Degraded data redundancy ... has slow ops

[复制链接]

1

主题

0

回帖

12

积分

管理员

积分
12
QQ
发表于 2021-10-25 17:00:42 | 显示全部楼层 |阅读模式
HEALTH_WARN Degraded data redundancy: 822/429494 objects degraded (0.191%), 115 pgs degraded; 154 slow ops, oldest one blocked for 389 sec, mon.ceph1 has slow ops. N: D& A) U  k
PG_DEGRADED Degraded data redundancy: 822/429494 objects degraded (0.191%), 115 pgs degraded
" b! x1 m8 y2 r" `# O    pg 8.473 is active+undersized+degraded, acting [18,9]
6 R: [0 k, m& c+ k2 P    pg 8.483 is active+undersized+degraded, acting [9,20]
! f9 V2 K1 {+ A3 G1 N    pg 8.48a is active+undersized+degraded, acting [11,21]7 f. ]% }- S  ^$ Z( _4 q$ H
    pg 8.49d is active+undersized+degraded, acting [23,6]7 i, `1 F, H7 ^% j8 ^0 u$ L
    pg 8.4a7 is active+undersized+degraded, acting [11,20]
( d6 b6 d( w! ~# i    pg 8.4a8 is active+undersized+degraded, acting [8,20]
; ^$ z2 S+ L! X    pg 8.4a9 is active+undersized+degraded, acting [18,8]
5 {% j4 o! M6 Z2 c" {    pg 8.4b2 is active+undersized+degraded, acting [11,21]/ ~; Y$ B, q; k0 I
    pg 8.4c8 is active+undersized+degraded, acting [7,22]- `& l: w- Z- B  c7 S2 f# w
    pg 8.4e4 is active+undersized+degraded, acting [9,20]
. m4 [0 Z) T. ]5 B6 q+ z    pg 8.4e7 is active+undersized+degraded, acting [18,11]
* n+ U) K! L2 S" @# Q& ^    pg 8.4f2 is active+undersized+degraded, acting [23,8]( n. X! t" x7 f9 l! Q
    pg 8.4fb is active+undersized+degraded, acting [18,7]
# E, X  d+ M' |+ v    pg 8.505 is active+undersized+degraded, acting [8,21]# k8 b1 W9 b+ j: W
    pg 8.54e is active+undersized+degraded, acting [8,22]3 {: y) o* k" a7 }2 c+ P/ Q
    pg 8.566 is active+undersized+degraded, acting [23,10]
1 j" F  {+ o. b: b! P    pg 8.567 is active+undersized+degraded, acting [8,19]- ^$ ^3 U( i+ q7 M
    pg 8.57c is active+undersized+degraded, acting [23,9]6 }  @$ ^$ d8 a3 k; g
    pg 8.57d is active+undersized+degraded, acting [8,22]
- {& ~. [" k7 @    pg 8.596 is active+undersized+degraded, acting [23,8]
6 G0 R' e% P1 g( O1 J8 A3 N3 T& |    pg 8.5b3 is active+undersized+degraded, acting [9,18], s, n% z& ~5 p; A6 D) S, y
    pg 8.5fc is active+undersized+degraded, acting [8,19]& \& Y+ [7 @- W- q# P  [( D
    pg 8.606 is active+undersized+degraded, acting [8,19]7 L4 g9 n  _8 B
    pg 8.627 is active+undersized+degraded, acting [11,22]
& }# t5 T! ~7 w5 G# e7 {# C    pg 8.636 is active+recovery_wait+degraded, acting [15,21,10]4 Q7 E2 z$ H5 t/ ]( @. N
    pg 8.651 is active+undersized+degraded, acting [23,6]
" z8 I+ I1 ^$ Q( ^/ {9 x2 ~    pg 8.65b is active+undersized+degraded, acting [11,21]
6 e; Y6 f# l; o+ N: p    pg 8.666 is active+undersized+degraded, acting [23,6]
- `! e$ B" H0 `    pg 8.679 is active+undersized+degraded, acting [11,21]+ K5 s, C# A, f4 W% w
    pg 8.6a3 is active+undersized+degraded, acting [7,22]
& ^: e& A; {' [, W    pg 8.6c3 is active+undersized+degraded, acting [23,6]4 C/ ]& N- y3 U" p9 z
    pg 8.6e1 is active+undersized+degraded, acting [8,18]
! X9 T5 T. u5 G( [, B4 S    pg 8.6ed is active+undersized+degraded, acting [8,22]
* s1 O# s, F; ^- X! }5 ]    pg 8.701 is active+undersized+degraded, acting [7,19]
3 p8 d, R* s! A2 E    pg 8.724 is active+undersized+degraded, acting [23,10]8 B3 a4 `/ A5 F9 |0 l+ E0 L; y9 A, G
    pg 8.728 is active+undersized+degraded, acting [18,10]
6 U4 O- I- \' @6 A    pg 8.72f is active+undersized+degraded, acting [7,23]
9 @. e9 Q' l' h, I    pg 8.732 is active+undersized+degraded, acting [9,18]
& }* Y6 u" M/ A- \0 }    pg 8.739 is active+undersized+degraded, acting [9,22]9 z2 K! C+ F' [
    pg 8.76a is active+undersized+degraded, acting [11,21]
9 |! U2 U- Y& q6 i" [! c    pg 8.770 is active+undersized+degraded, acting [23,9]
0 ]" x3 s, h/ h5 _7 B9 B    pg 8.781 is active+undersized+degraded, acting [7,20]0 m, t! a' c9 N. v. }) }
    pg 8.791 is active+undersized+degraded, acting [11,22]$ B0 L1 g+ X! _" d# P9 L
    pg 8.7b5 is active+undersized+degraded, acting [23,10]3 p  `; n+ S5 S( K! K; p$ q: d
    pg 8.7bb is active+undersized+degraded, acting [8,18]+ ~) `6 T) o: R# V. I
    pg 8.7be is active+recovery_wait+degraded, acting [15,19,6]
7 P. [2 U! x; B3 r' d% u( f    pg 8.7c3 is active+undersized+degraded, acting [23,9], ^# j" ~3 @: z3 I: y0 `
    pg 8.7d4 is active+undersized+degraded, acting [9,18]
% i$ d/ y  V5 v5 \3 ]8 n! _( A    pg 8.7da is active+undersized+degraded, acting [18,10]
2 S9 N# F( A  }# s    pg 8.7e5 is active+undersized+degraded, acting [18,8]8 L% |- n' |8 w) v
    pg 8.7fa is active+undersized+degraded, acting [8,21]3 W( `+ {2 I$ J+ G; u! I3 m) F
SLOW_OPS 154 slow ops, oldest one blocked for 389 sec, mon.ceph1 has slow ops9 e7 k$ r& j  d6 f7 g1 m* E6 V

; W2 c7 I/ I% q  g' k0 I

1

主题

0

回帖

12

积分

管理员

积分
12
QQ
 楼主| 发表于 2021-10-25 17:04:32 | 显示全部楼层
解决办法:: z4 ^) x% r1 X% T! p7 o. f
systemctl restart ceph-mon@`hostname -s`.service1 g8 N0 C7 J3 M! H% F

% G) z8 Y% v5 H$ C/ z  X
# Q' j8 _  I( p) ~3 {报mon的错误被清除了。
$ \7 j5 H8 \6 q; [# h6 J& z) M/ |) x  c& [$ R( U
[root@ceph1 ~]# ceph health detail
! `3 O9 O* z2 @. }/ hHEALTH_WARN 6 osds down; Reduced data availability: 170 pgs inactive; Degraded data redundancy: 9370/429503 objects degraded (2.182%), 1102 pgs degraded
4 V* Q2 @' f; t3 aOSD_DOWN 6 osds down
/ y: e, d. M2 E: e0 N    osd.20 (root=hdd,host=hdd-ceph3) is down
. b) I& ^) J& @$ J    osd.22 (root=hdd,host=hdd-ceph3) is down
, e5 \, F7 ]- ?  Y' x. a7 T    osd.23 (root=hdd,host=hdd-ceph3) is down$ n' d& J- p% h0 g5 k6 J6 ]' R
    osd.25 (root=ssd-openstack,host=ssd-compute1) is down/ Z- y- r- ]8 p- `$ G8 i
    osd.28 (root=ssd-openstack,host=ssd-compute2) is down
& k5 y! y7 I5 B+ ^' h" H0 K    osd.29 (root=ssd-openstack,host=ssd-compute2) is down+ d+ t) n3 Z  N  G. o) ^
PG_AVAILABILITY Reduced data availability: 170 pgs inactive
2 a6 S- N! a- n6 v. \    pg 7.13a is stuck inactive for 268.657552, current state undersized+peered, last acting [7]
7 \# @6 k1 v4 N$ t    pg 7.140 is stuck inactive for 173.462639, current state undersized+peered, last acting [6]
/ q8 @6 l' C/ q) c) v7 e; C    pg 7.149 is stuck inactive for 188.507770, current state undersized+peered, last acting [15]& v) G4 X, l$ f: x* \$ x# L* Y
    pg 7.14a is stuck inactive for 296.614537, current state undersized+peered, last acting [11]: }1 P0 {( I) Y7 `
    pg 7.15a is stuck inactive for 230.993717, current state undersized+peered, last acting [7]
/ L4 n& @: b7 e+ A2 E/ F    pg 7.160 is stuck inactive for 157.298697, current state undersized+peered, last acting [16]) j0 t  m! M$ e8 F( E% l" C$ h
    pg 7.162 is stuck inactive for 168.378304, current state undersized+peered, last acting [8]
+ T  Q% N; W! ]2 U2 t  X" g    pg 7.163 is stuck inactive for 230.964708, current state undersized+peered, last acting [11]
! s8 ]4 I1 \- Z+ Q$ x8 P    pg 7.164 is stuck inactive for 193.535188, current state undersized+peered, last acting [12]$ l+ Z* E( N: w0 i
    pg 7.166 is stuck inactive for 157.314779, current state undersized+peered, last acting [9]
2 M" w5 y- [+ K- A8 ?3 h% t    pg 7.167 is stuck inactive for 231.001296, current state undersized+peered, last acting [14]) ?8 R1 U9 _* C6 ~
    pg 7.16f is stuck inactive for 157.301826, current state undersized+peered, last acting [6]( o; w+ I& p- z
    pg 7.18e is stuck inactive for 230.971182, current state undersized+peered, last acting [17]
6 D# ~/ a& L, P! Y9 d+ d$ }2 q    pg 7.196 is stuck inactive for 157.295252, current state undersized+peered, last acting [10]
- _) e7 d0 z1 y6 w# `+ h: N- j/ B    pg 7.332 is stuck inactive for 188.514026, current state undersized+peered, last acting [15]% M4 M- s/ Q/ n* u' I% \8 u
    pg 7.333 is stuck inactive for 230.989099, current state undersized+peered, last acting [17]; c0 R4 u) E/ F2 e+ D
    pg 7.347 is stuck inactive for 243.608764, current state undersized+peered, last acting [14]* ?" v  c& a8 f% [$ v" J
    pg 7.352 is stuck inactive for 230.961993, current state undersized+peered, last acting [17]
" U2 P, M. u* q. c' G/ D    pg 7.358 is stuck inactive for 296.668417, current state undersized+peered, last acting [13]
- S: a' D+ N% A  M- A    pg 7.35b is stuck inactive for 296.611193, current state undersized+peered, last acting [9]) x9 [# ?& Z! d* x
    pg 7.364 is stuck inactive for 231.014057, current state undersized+peered, last acting [7]
- p) t( ~8 h! s$ ?5 S    pg 7.372 is stuck inactive for 230.976751, current state undersized+peered, last acting [11]
7 F# b3 S0 s: b4 c    pg 7.374 is stuck inactive for 157.314500, current state undersized+peered, last acting [13]
2 I1 W4 a2 z: Q    pg 7.375 is stuck inactive for 268.647540, current state undersized+peered, last acting [7]. y. Y- u7 T* _9 w4 Z! O  K3 V) }
    pg 7.376 is stuck inactive for 157.302772, current state undersized+peered, last acting [7]6 b6 D( V7 ~1 b" ?" b4 ]# A$ q: R& A, G
    pg 7.37c is stuck inactive for 157.292778, current state undersized+peered, last acting [9]4 ~4 Y+ Y. ]1 Q0 D# |8 V
    pg 7.381 is stuck inactive for 268.670664, current state undersized+peered, last acting [7]
6 m# s2 E& V9 p. X& A, P    pg 7.38a is stuck inactive for 157.311084, current state undersized+peered, last acting [6]
) P) F5 k2 \& N1 C1 L( x+ u    pg 7.38b is stuck inactive for 230.974616, current state undersized+peered, last acting [10]
" e5 _/ Q# x( O    pg 7.391 is stuck inactive for 253.571528, current state undersized+peered, last acting [10]
  Q( c- \* k; n& X    pg 7.399 is stuck inactive for 231.014969, current state undersized+peered, last acting [7]
- i8 k9 C0 Z+ U( M0 p    pg 7.3a0 is stuck inactive for 157.314500, current state undersized+peered, last acting [17]
( |* |- A6 o5 R: v0 `    pg 7.3a3 is stuck inactive for 157.310316, current state undersized+peered, last acting [16]
$ k0 @2 r# \( B+ b    pg 7.3ad is stuck inactive for 243.592394, current state undersized+peered, last acting [14]# _+ n# j  A' M% P% ^* ^/ \
    pg 7.3af is stuck inactive for 188.502747, current state undersized+peered, last acting [15]$ A+ F. a3 N$ q6 E* i. j
    pg 7.3b2 is stuck inactive for 168.373475, current state undersized+peered, last acting [8]
" V; U1 W, ^% K8 A/ p6 I    pg 7.3b4 is stuck inactive for 173.465040, current state undersized+peered, last acting [6]
% D2 }  C9 ~. ?/ O: @    pg 7.3b5 is stuck inactive for 157.322812, current state undersized+peered, last acting [9]
. A; }  k' i8 O* [% r- e! b    pg 7.3b6 is stuck inactive for 230.969301, current state undersized+peered, last acting [11]
6 U$ G3 x8 }- h$ n6 c1 e, Q8 D, [    pg 7.3b9 is stuck inactive for 157.296625, current state undersized+peered, last acting [14]1 C$ j+ r5 x4 h" j
    pg 7.3ba is stuck inactive for 231.015066, current state undersized+peered, last acting [7]
( Z: ]' j9 W8 g  j    pg 7.3be is stuck inactive for 157.311582, current state undersized+peered, last acting [8]
) F8 H' A+ u/ T  y: L) w    pg 7.3c5 is stuck inactive for 168.378059, current state undersized+peered, last acting [8]) G. B0 N, @; e' x. n1 u6 P) I! E
    pg 7.3cd is stuck inactive for 230.974691, current state undersized+peered, last acting [10]
* }8 c) r- F) a    pg 7.3cf is stuck inactive for 168.383761, current state undersized+peered, last acting [8]& Q9 |4 V5 p( h* s
    pg 7.3df is stuck inactive for 157.314279, current state undersized+peered, last acting [10]
5 k" v& a, H2 u) z" w* t    pg 7.3e3 is stuck inactive for 296.651325, current state undersized+peered, last acting [16]# ?3 i1 x; x3 k) i) p7 z9 n/ P
    pg 7.3eb is stuck inactive for 173.465968, current state undersized+peered, last acting [6]
& C" a0 r  |8 R+ R* N" d    pg 7.3f8 is stuck inactive for 230.983836, current state undersized+peered, last acting [11]1 r+ }3 r9 `" b& }
    pg 7.3fb is stuck inactive for 157.305951, current state undersized+peered, last acting [16]
! \: k0 H/ C. [; l" [9 J    pg 7.3ff is stuck inactive for 268.659557, current state undersized+peered, last acting [7]
( z4 L! ^# w& s& m3 XPG_DEGRADED Degraded data redundancy: 9370/429503 objects degraded (2.182%), 1102 pgs degraded) e7 X) X* y! E5 W9 h2 O
    pg 8.795 is active+undersized+degraded, acting [11,17]
) v/ }8 l/ G+ ~3 j& G3 Z( s    pg 8.79b is active+undersized+degraded, acting [10,12]
8 j) p+ M) F1 g1 |( s! R% x  r    pg 8.79c is active+undersized+degraded, acting [13,10]4 q" O7 h0 l7 W8 r1 N1 C
    pg 8.79f is active+undersized+degraded, acting [9,13]
  B" ^7 D: w9 ?! L7 I) G    pg 8.7a0 is active+undersized+degraded, acting [8,17]& x& W: s4 @1 U. ^1 ?& i
    pg 8.7a1 is active+undersized+degraded, acting [16,9], K; B: c0 r+ s
    pg 8.7a2 is active+undersized+degraded, acting [10,17]/ W3 g2 B2 ?* E0 N& E& {7 _( O
    pg 8.7a3 is active+undersized+degraded, acting [17,11]
# {, q7 O9 i3 P" @; g# b2 t    pg 8.7a5 is active+undersized+degraded, acting [11,17]5 ?8 i. Q- H2 ~
    pg 8.7a8 is active+undersized+degraded, acting [13,7]# l2 p2 E4 s9 E( o% [4 f$ E% D
    pg 8.7a9 is active+undersized+degraded, acting [15,7]
5 T4 N0 ]: T; b) R) H* F& t( ~$ s+ m    pg 8.7ad is active+undersized+degraded, acting [15,9]! o4 W3 T  X) w9 b
    pg 8.7ae is active+undersized+degraded, acting [14,10]
/ B+ X. q7 R! l    pg 8.7b0 is active+undersized+degraded, acting [12,6]
! b2 `8 [* w- J" P7 C: x  T    pg 8.7b5 is active+undersized+degraded, acting [10,15]. z9 p  Z. k1 g  G/ `% A. L
    pg 8.7b6 is active+undersized+degraded, acting [15,9]# Y0 R7 \/ ]+ i/ T7 @$ f5 [0 K
    pg 8.7b8 is active+undersized+degraded, acting [8,16]
/ {1 `7 N- N  l1 A* R) S    pg 8.7ba is active+undersized+degraded, acting [13,6]
4 r8 c& P1 |& V  M8 O    pg 8.7bd is active+undersized+degraded, acting [7,13]' Z2 H/ D) ]' Q/ I& h* I$ Q! x. g2 @
    pg 8.7bf is active+undersized+degraded, acting [17,11]
$ v% `! y  }' r' _    pg 8.7c1 is active+undersized+degraded, acting [12,10]# X4 U% e) W& B$ z$ d% L' K
    pg 8.7c2 is active+undersized+degraded, acting [6,12]  `! z6 U, p& o; `$ S
    pg 8.7c3 is active+undersized+degraded, acting [9,15]
% b& I' V9 l- D9 j6 V8 }, I    pg 8.7c4 is active+undersized+degraded, acting [6,16], |' u4 q& G2 {1 r' G2 ~
    pg 8.7c5 is active+undersized+degraded, acting [8,14]. s0 P( `2 {: Y& r- m7 h& x! o
    pg 8.7c6 is active+undersized+degraded, acting [17,7]. I( P& t$ {) r& D+ r( v
    pg 8.7c7 is active+undersized+degraded, acting [16,11]/ t' ~& G0 I( k1 n- f- W; j* u: }. d! u
    pg 8.7ca is active+undersized+degraded, acting [16,11]+ W7 V; V, N1 [, v6 V/ M2 V
    pg 8.7cd is active+undersized+degraded, acting [13,6]* V( n7 W, m, C2 z
    pg 8.7ce is active+undersized+degraded, acting [13,8]& g& Q: T# o( L! {( ]* G3 Q
    pg 8.7d0 is active+undersized+degraded, acting [17,10]' J" ^" I/ n( u7 ?
    pg 8.7d5 is active+undersized+degraded, acting [14,11]
0 W# I/ T1 ?7 T6 N. R9 o* [( z    pg 8.7d8 is active+undersized+degraded, acting [13,10]3 n1 ~  I2 L' A! m& t8 K
    pg 8.7d9 is active+undersized+degraded, acting [9,13]; K( v% K7 M+ m  C, r
    pg 8.7dd is active+undersized+degraded, acting [12,11]8 M, w6 d: r5 o+ R6 h; c- M
    pg 8.7df is active+undersized+degraded, acting [11,16]0 K) ^( V! V* c; X4 \: [
    pg 8.7e0 is active+undersized+degraded, acting [17,6]3 `* E- k! J. p) b- j* K
    pg 8.7e1 is active+undersized+degraded, acting [14,10]7 j8 X- G" T2 p6 k
    pg 8.7e2 is active+undersized+degraded, acting [9,13]
" J) T& _8 W2 G, R3 I- q    pg 8.7e3 is active+undersized+degraded, acting [14,9]
$ Q2 Y" m7 s- A2 o# M5 B7 J/ P( w) X    pg 8.7e7 is active+undersized+degraded, acting [6,15]
) O% s6 ~! w8 @. O' |    pg 8.7ea is active+undersized+degraded, acting [9,16]
# M" ]* V+ H0 d. ~+ T  {: s% ?    pg 8.7eb is active+undersized+degraded, acting [14,9]
) x, p5 c1 C- b) k    pg 8.7ec is active+undersized+degraded, acting [16,11]; Y" p0 E$ Y, e( ^$ \+ T
    pg 8.7ed is active+undersized+degraded, acting [14,10]7 B) {* j% ]1 U' m1 w
    pg 8.7f0 is active+undersized+degraded, acting [17,6]
' k) F! ]$ }* y7 F, r    pg 8.7f1 is active+undersized+degraded, acting [15,7]
/ {4 k& {6 B! \0 T4 b8 ?    pg 8.7f3 is active+undersized+degraded, acting [9,17]( c. z' d# v; Q; W! |# E- Z' M: l
    pg 8.7f4 is active+undersized+degraded, acting [9,14]
2 s. Z; A) M: W1 U9 S    pg 8.7fb is active+undersized+degraded, acting [9,17]( K' g! t6 l! ]
    pg 8.7fc is active+undersized+degraded, acting [15,8]- [- E9 }% S& D5 A  s$ D
您需要登录后才可以回帖 登录 | 注册

本版积分规则

返回首页|Archiver|手机版|小黑屋|易陆发现技术论坛 ( 蜀ICP备2026014127号-1 )

GMT+8, 2026-6-12 00:04 , Processed in 0.017638 second(s), 23 queries .

Powered by Discuz! X5.0

© 2001-2026 Discuz! Team.

快速回复 返回顶部 返回列表