记一次Possible data damage: 2 pgs recovery_unfound处理过程

admin

Possible data damage: 2 pgs recovery_unfound

' h! c' M' q9 J: j. r/ ]* Y- u
记一次Possible data damage: 2 pgs recovery_unfound处理过程
% r, S% C+ v! G1 f[root@gm268-2 ~]# ceph health detail
HEALTH_ERR 1 clients failing to respond to capability release; 1 MDSs report slow metadata IOs; 1 MDSs report slow requests; 1 MDSs behind on trimming; 9/23985260 objects unfound (0.000%); Reduced data availability: 10 pgs inactive, 10 pgs down; Possible data damage: 2 pgs recovery_unfound; Degraded data redundancy: 5523/47970520 objects degraded (0.012%), 2 pgs undersized
: ~% E7 W) H5 q( ?3 V& u( {, F: @1 _[WRN] MDS_CLIENT_LATE_RELEASE: 1 clients failing to respond to capability release
    mds.cephfs.gm268-3.ppyjrl(mds.0): Client gm268-2 failing to respond to capability release client_id: 205033
[WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs
    mds.cephfs.gm268-3.ppyjrl(mds.0): 20 slow metadata IOs are blocked > 30 secs, oldest blocked for 237450 secs
[WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests
    mds.cephfs.gm268-3.ppyjrl(mds.0): 11 slow requests are blocked > 30 secs
3 E- {* \: a- D[WRN] MDS_TRIM: 1 MDSs behind on trimming
3 K" s" j( y, u% i$ \; z    mds.cephfs.gm268-3.ppyjrl(mds.0): Behind on trimming (268/128) max_segments: 128, num_segments: 268
[WRN] OBJECT_UNFOUND: 9/23985260 objects unfound (0.000%)
    pg 2.7fc has 5 unfound objects
    pg 2.f70 has 4 unfound objects
& |6 C7 ~2 ]0 I  S( Y% O1 q" v) |! j[WRN] PG_AVAILABILITY: Reduced data availability: 10 pgs inactive, 10 pgs down
    pg 2.512 is down, acting [20,0]
    pg 2.8d4 is down, acting [20,33]
    pg 2.905 is down, acting [20,10]
    pg 2.f62 is down, acting [20,19]
    pg 2.fb4 is down, acting [20,12]
" e3 o+ k- Z- B, C3 ?, ^    pg 2.1660 is down, acting [20,6]
8 a2 Y( _' {0 v    pg 2.1752 is down, acting [20,16]
    pg 2.18b0 is down, acting [20,23]
: Z* p* @# m" O) s. k    pg 2.1ba5 is down, acting [20,5]
1 a2 w0 y& R# H) ]    pg 3.a9 is down, acting [20,34]
[ERR] PG_DAMAGED: Possible data damage: 2 pgs recovery_unfound
    pg 2.7fc is active+recovery_unfound+undersized+remapped, acting [23], 5 unfound
# k6 F4 g4 U( M  D    pg 2.f70 is active+recovery_unfound+undersized+remapped, acting [23], 4 unfound
7 u: O! }( D5 T) D) s4 ?' q[WRN] PG_DEGRADED: Degraded data redundancy: 5523/47970520 objects degraded (0.012%), 2 pgs undersized
( W8 z; v8 _7 f# e* Z    pg 2.7fc is stuck undersized for 21m, current sta
/ ~# ?2 w# B9 A3 T+ T8 w% f0 e7 F
/ a, m3 @; C8 S. j# m9 N1 Z! r

admin

ceph pg deep-scrub $pgid
& ~1 {: x! |- g5 ]* x8 l( ?. Z
After that you should try to repair the PGs

9 H: P) B5 v( P0 F! V2 L0 \1 \ceph pg repair $pgid
/ W$ n0 v7 T, R$ o0 G
Maybe Ceph is able to find the missing objects again. But it could be that they are lost.

admin

1.查看集群状态
[root@k8snode001 ~]# ceph health detail

HEALTH_ERR 1/973013 objects unfound (0.000%); 17 scrub errors; Possible data damage: 1 pg recovery_unfound, 8 pgs inconsistent, 1 pg repair; Degraded data redundancy: 1/2919039 objects degraded (0.000%), 1 pg degraded

OBJECT_UNFOUND 1/973013 objects unfound (0.000%)
7 G( ~* W) I4 r  l$ @- L
7 q* `, h/ A6 q  X6 D    pg 2.2b has 1 unfound objects
9 f4 A" E5 v( h, x
OSD_SCRUB_ERRORS 17 scrub errors
8 j$ T% S! y$ Z0 h2 J, p" q6 ]; o
PG_DAMAGED Possible data damage: 1 pg recovery_unfound, 8 pgs inconsistent, 1 pg repair

    pg 2.2b is active+recovery_unfound+degraded, acting [14,22,4], 1 unfound
7 U% b) W& C4 j6 ^  c
9 x' k, ]) |( w1 U! q. O    pg 2.44 is active+clean+inconsistent, acting [14,8,21]
5 \/ U. x$ m' E8 M
    pg 2.73 is active+clean+inconsistent, acting [25,14,8]
. v5 C7 X; O5 `+ o
    pg 2.80 is active+clean+scrubbing+deep+inconsistent+repair, acting [4,8,14]

6 o. v: o+ ^% }$ T# k    pg 2.83 is active+clean+inconsistent, acting [14,13,6]
: U8 q1 Q& T$ P9 z, K
    pg 2.ae is active+clean+inconsistent, acting [14,3,2]
+ d) c& R2 _4 K. X
0 }1 p! B* c2 K8 I" K    pg 2.c4 is active+clean+inconsistent, acting [8,21,14]
8 a! R$ l. [7 a" y
    pg 2.da is active+clean+inconsistent, acting [23,14,15]
' w" O5 K% N7 U. D
% p' r, p5 E/ [2 t    pg 2.fa is active+clean+inconsistent, acting [14,23,25]
$ ]# [5 j% l% N* p
* b7 n7 l3 h; f2 T: M( NPG_DEGRADED Degraded data redundancy: 1/2919039 objects degraded (0.000%), 1 pg degraded

4 C/ G" }* w) ~1 g+ N: _0 z6 k    pg 2.2b is active+recovery_unfound+degraded, acting [14,22,4], 1 unfound
- c7 @, |2 F* b% _9 _% ?1 n( l: d2 z1 U
$ o6 Z" a" i" i0 V
) N" B. W& E- e1 Q( n- i
从输出发现pg 2.2b is active+recovery_unfound+degraded, acting [14,22,4], 1 unfound
1 i& H( V& b+ H- |
现在我们来查看pg 2.2b，看看这个pg得想想信息。


+ ?9 O' B9 S4 M
7 R1 e$ d' f! X/ w( f[root@k8snode001 ~]# ceph pg dump_json pools    |grep 2.2b

  c6 F8 |' x; k6 @dumped all
7 {+ J9 v' }- P( m% C: D: G
2.2b       2487                  1        1         0       1  9533198403 3048     3048                active+recovery_unfound+degraded 2020-07-23 08:56:07.669903  10373'5448370  10373:7312614  [14,22,4]         14  [14,22,4]             14  10371'5437258 2020-07-23 08:56:06.637012   10371'5437258 2020-07-23 08:56:06.637012             0

1 w0 t) T6 a8 t& |" q* R3 k8 t0 R
# `- g$ A/ y2 p& V7 `4 P7 f& D
9 ?7 x: _6 u; X. _可以看到它现在只有一个副本

2.查看pg map

+ \6 q& }5 s, Q4 _4 ^
[root@k8snode001 ~]# ceph pg map 2.2b
1 X, Q$ F9 b! C) ~6 R3 s
0 R8 {9 w+ V# H. [0 r& uosdmap e10373 pg 2.2b (2.2b) -> up [14,22,4] acting [14,22,4]
1 o2 o, @. ?/ n2 Q5 {" R
/ ]& o6 n4 ~! c" F- I
8 R3 A2 G0 C% C2 u+ H! i, k
. z1 ]) S! @% [6 T3 u从pg map可以看出，pg 2.2b分布到osd [14,22,4]上


' o3 `7 J( }8 n9 v# U: Q3.查看存储池状态
( `) r% s( s, z; ~# G

4 A# q" j( }2 F# ~& s. E! k[root@k8snode001 ~]# ceph osd pool stats k8s-1

% T$ P+ ?8 X! Ypool k8s-1 id 2

  1/1955664 objects degraded (0.000%)
, ^0 [# a" f1 ?5 U
  ^' C- S. n, A5 n4 e2 `  1/651888 objects unfound (0.000%)
; V- T2 N+ z" _
- `$ e2 C& M2 t1 h  client io 271 KiB/s wr, 0 op/s rd, 52 op/s wr
6 c1 r, C" ~. T3 O( w$ g


0 \. b( |6 L1 G* h5 Q[root@k8snode001 ~]# ceph osd pool ls detail|grep k8s-1

2 y9 G; o' I: T# p0 o- P: npool 2 'k8s-1' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 88 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd
/ e* a$ ?* D) R7 O
3 Z' l2 u) \: E) |; @# Y/ j  _$ K
$ v; k+ s! Y# c$ D- r3 a. c$ B
4.尝试恢复pg 2.2b丢失的块
4 g: f; N, D/ ~& ^[root@k8snode001 ~]# ceph pg repair 2.2b
8 c# f1 }. W) C0 ~( g% ]/ h  Y

8 X0 |3 L, I6 X+ l
如果一直修复不成功，可以查看卡住PG的具体信息，主要关注recovery_state，命令如下


& g8 g) i* w( F) G# V
! h* |, e4 E1 i) {+ }* p6 M/ m[root@k8snode001 ~]# ceph pg 2.2b  query
- _. R; b- h3 p/ w  H- o& n
( k0 x% k- @4 v* B5 v* x, W4 e{
& T- D! K9 i1 l- u: C
    "......

    "recovery_state": [

" q% I1 a7 S6 j+ r        {
) q% y& L: W% `! z: Z. Z4 k/ w
3 G4 ^. f: V% T4 O/ |# p            "name": "Started/Primary/Active",
% ]0 m8 g7 B9 f
7 w  j/ l" \3 g1 `  R4 G            "enter_time": "2020-07-21 14:17:05.855923",

            "might_have_unfound": [],

+ z9 l2 q6 G% i! A0 R            "recovery_progress": {
1 F9 o8 l) a# m8 u8 k# v- K
                "backfill_targets": [],

                "waiting_on_backfill": [],
2 ]. U' N; C  T0 G* t9 Q* S: [
2 K9 _5 ?. u" u3 k+ z                "last_backfill_started": "MIN",
% s# w1 z" P) U7 @6 ]1 K) m
  j( h1 T$ a( Z" B9 N                "backfill_info": {
# ^' A9 d* A* K% ?' D+ f! _$ o  B
1 x  C" y' G# ^  Y& F0 W                    "begin": "MIN",

                    "end": "MIN",
, T! y* a/ i6 a. h+ \
                    "objects": []
1 T- _  ?$ \% X1 r: U: P& U' K; W
$ d  p! S* L4 G/ c. p6 w1 s9 X/ u                },
; P  D9 E& z. L- c
" c5 j+ G. K3 z3 f! H$ f9 P- r                "peer_backfill_info": [],

                "backfills_in_flight": [],

4 v' D0 \8 i- B                "recovering": [],
8 A' v1 @$ t1 @3 ~4 Y3 z) @+ l& @
9 U1 u+ l" E4 [5 R, ?; \                "pg_backend": {
: t  k+ a7 O' s1 `
+ S/ M5 R5 v! U. U                    "pull_from_peer": [],
; f) ^2 n# j. ]- V( K/ ~9 z
1 ?5 z- J' y3 {0 o1 W                    "pushing": []

                }

            },

            "scrub": {

                "scrubber.epoch_start": "10370",

& n) ~2 E, O. p' t, E/ A7 ?8 \1 _& N                "scrubber.active": false,
  [) k( c6 M% j# w
! E5 E9 x7 M# d2 W: i, M                "scrubber.state": "INACTIVE",
5 _9 n/ a6 j8 a6 y
8 H) J6 H3 b( P2 |                "scrubber.start": "MIN",

                "scrubber.end": "MIN",

; N7 }3 `! v1 i* F                "scrubber.max_end": "MIN",

7 B" s8 J! B; g                "scrubber.subset_last_update": "0'0",
! Z; o& U4 J  ]+ @8 y$ p
6 l9 ^7 J; G( {( E0 R2 G8 G. G                "scrubber.deep": false,

* Z4 ?8 U& N- ?+ {                "scrubber.waiting_on_whom": []
0 }  S; V: ^/ j
; d$ ?: j. {3 Z' G4 ]: v2 a* B; H( p7 F! W            }

        },
! @6 X. _6 R1 ?+ u8 ^4 s
2 r2 t$ M% R& L$ N4 ~' ^+ {        {
& W/ g8 u+ K, d5 q/ X2 d
            "name": "Started",

8 g9 m- O& k3 t; X( y" e            "enter_time": "2020-07-21 14:17:04.814061"

& ?% N, J* d1 }2 {        }

0 {# e& t: Z; i: [& H8 c    ],

% |7 W( P( }: M& G5 ]    "agent_state": {}

6 \+ t3 h* P8 S}

7 z% E! D) p" n! r8 R
' I- \( J6 b% C% P
如果repair修复不了；两种解决方案，回退旧版或者直接删除

' W6 Z, _8 L- k% R& ^5 u6 W5.解决方案
! {: l7 W: v0 b+ {- r回退旧版

[root@k8snode001 ~]# ceph pg  2.2b  mark_unfound_lost revert
- U5 G, L: M7 A4 |' P- s  o$ b
- H5 g3 Y& q& h# q3 O$ [直接删除

- @8 b( H* i2 _: X1 Q2 s" ?9 K[root@k8snode001 ~]# ceph pg  2.2b  mark_unfound_lost delete

' `% z" v# n2 Z
' x/ g* {- \" I# P+ R8 z2 C
6.验证
5 x. u/ W8 R1 v我这里直接删除了，然后ceph集群重建pg,稍等会再看，pg状态变为active+clean

  u1 V% g% M* m[root@k8snode001 ~]#  ceph pg  2.2b query
; ?, g, h0 y2 v
9 w) V4 u& J% ?& z- `$ l+ w% ~{
4 `! A0 }7 @: q
    "state": "active+clean",

    "snap_trimq": "[]",

* d: I  U8 U; v' Q" H    "snap_trimq_len": 0,
% P; U" N; |0 H, N6 L
6 P0 G6 h( w3 W( L/ W* N    "epoch": 11069,
- y3 m# U6 G+ y! J
/ K8 ]$ [" a5 ]% h4 H/ V    "up": [
+ @7 ?  o5 m9 Q0 I
        12,

        22,

. {9 l4 q9 g1 {, o+ ~        4

    ],

- r  @5 x# d' u% B. }
  K" `( o, g, `7 _* k) a
再次查看集群状态
0 ^0 Q5 T- V* \1 O

  [0 d" |0 J) X& r0 j
[root@k8snode001 ~]# ceph health detail

HEALTH_OK