记一次Possible data damage: 2 pgs recovery_unfound处理过程

admin

Possible data damage: 2 pgs recovery_unfound

) g8 l+ [! W2 w( p( y! ]
7 J9 Q& t$ |6 ^- A4 a记一次Possible data damage: 2 pgs recovery_unfound处理过程
[root@gm268-2 ~]# ceph health detail
HEALTH_ERR 1 clients failing to respond to capability release; 1 MDSs report slow metadata IOs; 1 MDSs report slow requests; 1 MDSs behind on trimming; 9/23985260 objects unfound (0.000%); Reduced data availability: 10 pgs inactive, 10 pgs down; Possible data damage: 2 pgs recovery_unfound; Degraded data redundancy: 5523/47970520 objects degraded (0.012%), 2 pgs undersized
' _5 ]6 z7 ~0 r, F. a# b6 y[WRN] MDS_CLIENT_LATE_RELEASE: 1 clients failing to respond to capability release
    mds.cephfs.gm268-3.ppyjrl(mds.0): Client gm268-2 failing to respond to capability release client_id: 205033
5 c8 h$ ?" C3 W) |& t. ?- q  f9 W[WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs
4 d+ d  x/ D8 F0 e! Z% @    mds.cephfs.gm268-3.ppyjrl(mds.0): 20 slow metadata IOs are blocked > 30 secs, oldest blocked for 237450 secs
" M( p3 K8 G# A* p2 p+ Z[WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests
& ~, ]6 d0 Z, W, J% j5 e& o! K. O    mds.cephfs.gm268-3.ppyjrl(mds.0): 11 slow requests are blocked > 30 secs
[WRN] MDS_TRIM: 1 MDSs behind on trimming
    mds.cephfs.gm268-3.ppyjrl(mds.0): Behind on trimming (268/128) max_segments: 128, num_segments: 268
[WRN] OBJECT_UNFOUND: 9/23985260 objects unfound (0.000%)
% `9 E5 D, |0 ^    pg 2.7fc has 5 unfound objects
    pg 2.f70 has 4 unfound objects
[WRN] PG_AVAILABILITY: Reduced data availability: 10 pgs inactive, 10 pgs down
    pg 2.512 is down, acting [20,0]
9 k* x, M, K" h: Z' P    pg 2.8d4 is down, acting [20,33]
    pg 2.905 is down, acting [20,10]
    pg 2.f62 is down, acting [20,19]
! k% D2 w( Y8 b2 R    pg 2.fb4 is down, acting [20,12]
    pg 2.1660 is down, acting [20,6]
    pg 2.1752 is down, acting [20,16]
5 C6 l, P; M$ b* O    pg 2.18b0 is down, acting [20,23]
+ }0 ]! D  M$ g$ ?' I    pg 2.1ba5 is down, acting [20,5]
    pg 3.a9 is down, acting [20,34]
[ERR] PG_DAMAGED: Possible data damage: 2 pgs recovery_unfound
    pg 2.7fc is active+recovery_unfound+undersized+remapped, acting [23], 5 unfound
    pg 2.f70 is active+recovery_unfound+undersized+remapped, acting [23], 4 unfound
[WRN] PG_DEGRADED: Degraded data redundancy: 5523/47970520 objects degraded (0.012%), 2 pgs undersized
7 t1 D6 m: b# C* v1 n+ H4 i    pg 2.7fc is stuck undersized for 21m, current sta


+ J& f0 s* Y9 F, P2 D

admin

ceph pg deep-scrub $pgid

6 n" ]2 I  \2 \7 f+ NAfter that you should try to repair the PGs

ceph pg repair $pgid

Maybe Ceph is able to find the missing objects again. But it could be that they are lost.

admin

1.查看集群状态
2 k7 A! l# T3 ^  B, `[root@k8snode001 ~]# ceph health detail
2 _8 W1 c# ?1 I# t
7 U, \- _% G1 T1 ~7 WHEALTH_ERR 1/973013 objects unfound (0.000%); 17 scrub errors; Possible data damage: 1 pg recovery_unfound, 8 pgs inconsistent, 1 pg repair; Degraded data redundancy: 1/2919039 objects degraded (0.000%), 1 pg degraded
- B, y- W" j. k3 n& v8 A9 C+ m
8 E  t7 \3 a1 n( JOBJECT_UNFOUND 1/973013 objects unfound (0.000%)

: R+ s' N& A9 W& e# ~' Z2 j    pg 2.2b has 1 unfound objects

( X% x3 ?, A. ?4 J3 a5 p/ Y6 \- dOSD_SCRUB_ERRORS 17 scrub errors

PG_DAMAGED Possible data damage: 1 pg recovery_unfound, 8 pgs inconsistent, 1 pg repair

    pg 2.2b is active+recovery_unfound+degraded, acting [14,22,4], 1 unfound
) q  s: c- \( L( H
; v: M' Z, B: m$ U    pg 2.44 is active+clean+inconsistent, acting [14,8,21]
: n) L* ?4 k' T5 _+ T8 _/ Q9 s) @
    pg 2.73 is active+clean+inconsistent, acting [25,14,8]
- r! g5 {; N& H6 c; t2 L. E
    pg 2.80 is active+clean+scrubbing+deep+inconsistent+repair, acting [4,8,14]
% D) h- I+ ^, t) s8 ]
    pg 2.83 is active+clean+inconsistent, acting [14,13,6]
/ {* q) \5 A1 r+ I
, b/ C8 R! j/ e) {! E* A5 S    pg 2.ae is active+clean+inconsistent, acting [14,3,2]

    pg 2.c4 is active+clean+inconsistent, acting [8,21,14]
9 v- \7 w$ ^4 y, V/ e3 x) [
    pg 2.da is active+clean+inconsistent, acting [23,14,15]
6 i" A7 q" Z( F3 e: ^- ?
    pg 2.fa is active+clean+inconsistent, acting [14,23,25]

! x5 T: L9 A0 K; ZPG_DEGRADED Degraded data redundancy: 1/2919039 objects degraded (0.000%), 1 pg degraded

4 o% o) `4 a' a8 p' k1 x    pg 2.2b is active+recovery_unfound+degraded, acting [14,22,4], 1 unfound
4 R6 p) [: T3 v6 m' n7 J* C: m! D1 |4 @
5 h2 E% k5 K' t$ r9 Q  `. m5 q  Z3 V
2 t9 L; s+ f& N6 Q4 U* a+ N
: ?: l; q( T" D4 V% l+ p1 V从输出发现pg 2.2b is active+recovery_unfound+degraded, acting [14,22,4], 1 unfound
& H, x( K( i$ g& K" S/ s# h! |
% E% e1 ^  H4 `; g$ M* v( q现在我们来查看pg 2.2b，看看这个pg得想想信息。

6 o& E# Z- E6 ?9 _- e  I) ^
, g  `2 i' ?+ @+ S7 z( Z; p9 x' y
( X* f& Z7 P9 m/ ?' c7 d8 W[root@k8snode001 ~]# ceph pg dump_json pools    |grep 2.2b

" N4 D& d5 B# M1 Fdumped all

2.2b       2487                  1        1         0       1  9533198403 3048     3048                active+recovery_unfound+degraded 2020-07-23 08:56:07.669903  10373'5448370  10373:7312614  [14,22,4]         14  [14,22,4]             14  10371'5437258 2020-07-23 08:56:06.637012   10371'5437258 2020-07-23 08:56:06.637012             0
2 b* n% _, o8 j1 J

- q$ p- e" z' t3 ^) q+ z
可以看到它现在只有一个副本

. M3 [' b& K+ T3 e2.查看pg map
# C  _( _) P0 o4 g4 }  H

: R# o0 |# g4 M5 @% r7 j1 L  J[root@k8snode001 ~]# ceph pg map 2.2b

3 l' U$ v% I( J) y) C7 Wosdmap e10373 pg 2.2b (2.2b) -> up [14,22,4] acting [14,22,4]

' _- R  _! {8 \" i9 M

# Z/ P/ _' [  Q. u; W. r3 r从pg map可以看出，pg 2.2b分布到osd [14,22,4]上
; G* Y9 g# H/ _
5 I3 p3 _- L/ b6 ?; i7 D5 I# \
* C. W6 [2 O! |, d7 M7 E$ q3.查看存储池状态

8 A% r  \0 ^2 |- q6 g9 q' q
! y. d/ }& m" y9 _[root@k8snode001 ~]# ceph osd pool stats k8s-1
7 z7 y$ b4 Q" C+ x" Z& J: n+ C
pool k8s-1 id 2
+ V( P/ `$ _0 O' L1 E
) A. f, b) G6 R  1/1955664 objects degraded (0.000%)

  1/651888 objects unfound (0.000%)
  `' u2 w- D8 c7 Z
  client io 271 KiB/s wr, 0 op/s rd, 52 op/s wr
6 ^2 m5 M- [5 v


[root@k8snode001 ~]# ceph osd pool ls detail|grep k8s-1

$ u% u5 N* ^1 k( ppool 2 'k8s-1' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 88 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd
2 N* n" t& S; A) N# i4 x) S7 Y. n, Q: ~5 U
8 b% S0 u  C/ q- V% Z
' k% |1 v: e* e6 m
0 t1 O) Q* {. L, P) l! I4.尝试恢复pg 2.2b丢失的块
. `. y" j) t  |4 r0 n0 h; M[root@k8snode001 ~]# ceph pg repair 2.2b


2 a3 B% X$ b0 d  q  m
如果一直修复不成功，可以查看卡住PG的具体信息，主要关注recovery_state，命令如下

/ Q' K6 b1 S# }( u5 g  Q

[root@k8snode001 ~]# ceph pg 2.2b  query
7 d) e3 r: r1 Z% {& U& z* t
2 \( f9 k2 q' ^! K3 s9 U  O6 k{
6 d6 |, P9 v/ v# v4 E4 ]; @/ _
) H) d7 b# K7 G; ~1 r    "......

    "recovery_state": [

0 l4 Y3 ]. r% q$ j8 Q( R        {
8 m/ Q) u9 X! o
4 d5 ^) F' J( E) v0 p            "name": "Started/Primary/Active",

9 x- k8 T: w7 F7 m# T            "enter_time": "2020-07-21 14:17:05.855923",
/ i7 S. s  w0 G5 r3 E4 d
" r+ I7 @7 y0 p# }            "might_have_unfound": [],
. a9 v% w( n' \, X9 W2 S) f
            "recovery_progress": {
! z, \& O. Q8 M3 g; ]
$ _+ J6 i/ l  _                "backfill_targets": [],
: p( f; ]- `( |  B8 b8 A! W
                "waiting_on_backfill": [],
% W9 n4 m; `  b
9 f) P* ^' N% B3 P5 [                "last_backfill_started": "MIN",

                "backfill_info": {

                    "begin": "MIN",
% d2 ?3 q6 a3 q. U+ p" b
+ i$ m% Y+ ]4 t+ D- V3 s# ]                    "end": "MIN",
" F$ @+ E. V3 P7 b. o0 s& O( b& }- o
                    "objects": []

9 _: ]  j+ n4 d2 G5 w! g' k0 ^: ~                },
5 R* d, A* G3 c. t6 k
                "peer_backfill_info": [],
4 J5 `8 B5 k+ r: u1 B
1 S) f, H, w3 e& ~' y2 {* N4 n                "backfills_in_flight": [],

9 b) r* @6 W. {+ i  ]- ~  l" ^                "recovering": [],

9 U* h# D5 \" [: a                "pg_backend": {

                    "pull_from_peer": [],

                    "pushing": []

                }

* p% P# {2 a' R5 G: ^            },

$ [" T  T+ ~0 I8 L# L/ |5 O            "scrub": {
" N# o5 G' `( I0 l; o: a- {* ^* t
                "scrubber.epoch_start": "10370",
+ W; L# p: T2 w& @
                "scrubber.active": false,

! q  P' G& U  }, i; `- Q; a$ ~                "scrubber.state": "INACTIVE",
! K: T" p/ j- w1 }9 k) }
' S! ]* M2 O( P5 j1 d4 g5 `                "scrubber.start": "MIN",

7 v& C( N1 G! [' t                "scrubber.end": "MIN",
: X5 b: P/ y" k# A
                "scrubber.max_end": "MIN",
5 i; }# ]+ `( S
+ p8 j1 n- N8 I% s/ w; O- d& C                "scrubber.subset_last_update": "0'0",

, X2 B- ?- F( y* e# G+ k' p0 {3 A                "scrubber.deep": false,

                "scrubber.waiting_on_whom": []
) j8 |1 O0 v3 B/ B$ C+ h+ l
7 T+ }" [1 ]4 i3 M6 s' B2 o            }
6 _) W* o3 Q. r- x, {
3 d/ H3 }5 F5 q/ u6 z        },

. r! {% F6 j) Q6 A; u$ q        {

            "name": "Started",

            "enter_time": "2020-07-21 14:17:04.814061"
9 k! G  A3 G5 J+ W1 l/ z% v! d! ?
        }

( q1 j1 |, i9 M! _! X    ],

    "agent_state": {}

. o9 v+ C8 ^. k& l4 n. @, g8 b1 J+ ^}

0 r+ A; x' G2 k3 V0 h, U6 p

如果repair修复不了；两种解决方案，回退旧版或者直接删除
! c& v. D( A0 l* _, K
5.解决方案
5 H: U3 O/ l8 h( @8 W, g回退旧版

[root@k8snode001 ~]# ceph pg  2.2b  mark_unfound_lost revert
0 w, h- ^+ q8 k3 z2 o, S
直接删除
) Q3 E% g9 Z4 y  G0 r  ?& w; F
/ q1 o: u- ~8 w- z# f  m" a[root@k8snode001 ~]# ceph pg  2.2b  mark_unfound_lost delete

5 G, j- _0 N7 g; G) b0 k
8 a* v6 W9 R) u# }
4 \3 {. `" \9 ?! |8 [( v6 [" S6.验证
我这里直接删除了，然后ceph集群重建pg,稍等会再看，pg状态变为active+clean
2 Y$ d4 s, q1 z' W
( |/ R2 y: z8 L6 I2 W% W' I[root@k8snode001 ~]#  ceph pg  2.2b query

) W7 ]4 X) u. P2 L{

    "state": "active+clean",

    "snap_trimq": "[]",
0 i2 }7 E$ Y6 \
    "snap_trimq_len": 0,
& m- N/ P7 o. ]  \6 ]$ T9 }- M! i
    "epoch": 11069,

. ^. H: L6 x8 d. N; R& }7 x/ _& A  G    "up": [
8 ~3 Z$ f1 w: O
        12,
$ L0 S" B# F0 `8 n, J
        22,
; a& F8 s; ]! K
        4

    ],

/ m) `; T) \2 F( S9 K4 U9 g" |
' Y% Q$ b0 g8 N/ e9 ]  j: Z7 [
再次查看集群状态
+ ]$ y% M9 T, |6 K  e

8 {5 w) e$ P9 c: V; |  v
. `0 _' ~4 i: |$ O+ U8 j* {[root@k8snode001 ~]# ceph health detail

HEALTH_OK