记一次Possible data damage: 2 pgs recovery_unfound处理过程

admin

Possible data damage: 2 pgs recovery_unfound
- N. X4 l. c6 C0 e6 ~1 g/ a3 n

# S4 j: z( F5 L! E0 B2 {3 X3 v记一次Possible data damage: 2 pgs recovery_unfound处理过程
[root@gm268-2 ~]# ceph health detail
HEALTH_ERR 1 clients failing to respond to capability release; 1 MDSs report slow metadata IOs; 1 MDSs report slow requests; 1 MDSs behind on trimming; 9/23985260 objects unfound (0.000%); Reduced data availability: 10 pgs inactive, 10 pgs down; Possible data damage: 2 pgs recovery_unfound; Degraded data redundancy: 5523/47970520 objects degraded (0.012%), 2 pgs undersized
2 h& U! i% N; ?& U[WRN] MDS_CLIENT_LATE_RELEASE: 1 clients failing to respond to capability release
    mds.cephfs.gm268-3.ppyjrl(mds.0): Client gm268-2 failing to respond to capability release client_id: 205033
[WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs
    mds.cephfs.gm268-3.ppyjrl(mds.0): 20 slow metadata IOs are blocked > 30 secs, oldest blocked for 237450 secs
[WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests
8 j# m+ K  h/ j! D  u: G" P    mds.cephfs.gm268-3.ppyjrl(mds.0): 11 slow requests are blocked > 30 secs
& r7 ]' G& K" b4 [[WRN] MDS_TRIM: 1 MDSs behind on trimming
1 n" K& e8 X( k2 u0 Q    mds.cephfs.gm268-3.ppyjrl(mds.0): Behind on trimming (268/128) max_segments: 128, num_segments: 268
* i5 a1 }$ M- U[WRN] OBJECT_UNFOUND: 9/23985260 objects unfound (0.000%)
    pg 2.7fc has 5 unfound objects
" m8 }; t; O/ A% `$ r    pg 2.f70 has 4 unfound objects
[WRN] PG_AVAILABILITY: Reduced data availability: 10 pgs inactive, 10 pgs down
    pg 2.512 is down, acting [20,0]
    pg 2.8d4 is down, acting [20,33]
    pg 2.905 is down, acting [20,10]
    pg 2.f62 is down, acting [20,19]
    pg 2.fb4 is down, acting [20,12]
9 ?7 ]! J# a0 s5 m/ C& T$ H( u    pg 2.1660 is down, acting [20,6]
: `, K3 K; U0 ~: O; b4 o( n" ]    pg 2.1752 is down, acting [20,16]
    pg 2.18b0 is down, acting [20,23]
    pg 2.1ba5 is down, acting [20,5]
    pg 3.a9 is down, acting [20,34]
[ERR] PG_DAMAGED: Possible data damage: 2 pgs recovery_unfound
    pg 2.7fc is active+recovery_unfound+undersized+remapped, acting [23], 5 unfound
    pg 2.f70 is active+recovery_unfound+undersized+remapped, acting [23], 4 unfound
[WRN] PG_DEGRADED: Degraded data redundancy: 5523/47970520 objects degraded (0.012%), 2 pgs undersized
$ i3 @  v& W' g6 ^1 T! y    pg 2.7fc is stuck undersized for 21m, current sta

! d4 w! S4 B. @4 D% H1 M- N

admin

ceph pg deep-scrub $pgid

After that you should try to repair the PGs
& ~2 L, ~! ^2 B
ceph pg repair $pgid

' E9 g' M% Z. ?+ ZMaybe Ceph is able to find the missing objects again. But it could be that they are lost.

admin

1.查看集群状态
[root@k8snode001 ~]# ceph health detail

/ R5 T: {: v4 @9 {8 e+ L. o* V' e3 UHEALTH_ERR 1/973013 objects unfound (0.000%); 17 scrub errors; Possible data damage: 1 pg recovery_unfound, 8 pgs inconsistent, 1 pg repair; Degraded data redundancy: 1/2919039 objects degraded (0.000%), 1 pg degraded
$ f0 F# f8 c) t0 i, s3 X; x7 B$ A. Y1 d
OBJECT_UNFOUND 1/973013 objects unfound (0.000%)

" i, [. I; w) |' q    pg 2.2b has 1 unfound objects
; O& k+ Z1 h0 i8 |6 B4 \
OSD_SCRUB_ERRORS 17 scrub errors
- ~' t' d0 }- h8 F
, C& S. z/ w4 y3 w  b2 d% vPG_DAMAGED Possible data damage: 1 pg recovery_unfound, 8 pgs inconsistent, 1 pg repair

    pg 2.2b is active+recovery_unfound+degraded, acting [14,22,4], 1 unfound
8 U2 H7 @, b6 b: m% V
    pg 2.44 is active+clean+inconsistent, acting [14,8,21]
6 r3 K& W4 D: y2 w6 Q+ N+ X
$ a1 W7 ]5 p# |" q% n2 ]    pg 2.73 is active+clean+inconsistent, acting [25,14,8]

    pg 2.80 is active+clean+scrubbing+deep+inconsistent+repair, acting [4,8,14]
  [% w! v: S2 k0 k
$ o" k9 m. r+ f5 F    pg 2.83 is active+clean+inconsistent, acting [14,13,6]

    pg 2.ae is active+clean+inconsistent, acting [14,3,2]
4 }5 m2 a" `% C) M! }  p! C
    pg 2.c4 is active+clean+inconsistent, acting [8,21,14]

    pg 2.da is active+clean+inconsistent, acting [23,14,15]

    pg 2.fa is active+clean+inconsistent, acting [14,23,25]

/ \/ E) d$ f4 w5 ~PG_DEGRADED Degraded data redundancy: 1/2919039 objects degraded (0.000%), 1 pg degraded

+ B3 D9 N, R% Y& U    pg 2.2b is active+recovery_unfound+degraded, acting [14,22,4], 1 unfound
, V+ J. u: y7 G. s5 S
6 F# ?* d1 ~: G$ h
) J9 _) g! k' D7 u& s7 ~
, c4 {6 H" k( p: q& w从输出发现pg 2.2b is active+recovery_unfound+degraded, acting [14,22,4], 1 unfound
" f0 ~; U- b4 N
现在我们来查看pg 2.2b，看看这个pg得想想信息。
1 G" M6 |4 k7 o: @

$ f$ m+ I' R: f, ]
! q9 e, x; X( R( F[root@k8snode001 ~]# ceph pg dump_json pools    |grep 2.2b

dumped all

2.2b       2487                  1        1         0       1  9533198403 3048     3048                active+recovery_unfound+degraded 2020-07-23 08:56:07.669903  10373'5448370  10373:7312614  [14,22,4]         14  [14,22,4]             14  10371'5437258 2020-07-23 08:56:06.637012   10371'5437258 2020-07-23 08:56:06.637012             0

$ q5 a5 E6 z* f9 }  {9 b% ]7 w4 X
% Y+ H9 H% u; c6 Q/ y  b  r/ C5 X
可以看到它现在只有一个副本
- W: u. ~. B" e$ M( Q
9 q; e6 |  V9 P- h' M: t2.查看pg map
' ~9 c& f4 m- S, R4 {* @# l; X, ]
/ g4 S' W$ ]1 x* U
( x* c: y, m( ~! k/ P, E[root@k8snode001 ~]# ceph pg map 2.2b

osdmap e10373 pg 2.2b (2.2b) -> up [14,22,4] acting [14,22,4]

. z# ]( z$ X3 y+ C! H

从pg map可以看出，pg 2.2b分布到osd [14,22,4]上


3.查看存储池状态


, b" @2 g$ b1 v2 I[root@k8snode001 ~]# ceph osd pool stats k8s-1

pool k8s-1 id 2

. }; h9 A* A1 a2 F5 ^6 b2 U8 a  1/1955664 objects degraded (0.000%)
+ k8 K' V; N2 L
  1/651888 objects unfound (0.000%)
( u; u* z  h7 f9 c+ [8 }
  client io 271 KiB/s wr, 0 op/s rd, 52 op/s wr
$ D! V4 i7 M6 D

' E9 s( u% A9 ^9 g! ?
[root@k8snode001 ~]# ceph osd pool ls detail|grep k8s-1
5 B8 Y; p2 y, i6 U
pool 2 'k8s-1' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 88 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd



2 |. f* v- ^1 j) y2 D$ s4.尝试恢复pg 2.2b丢失的块
[root@k8snode001 ~]# ceph pg repair 2.2b
; w; w7 R3 d. }% L) k4 \1 V% G
$ ]) f! }4 C- g+ k  h2 P, u* W) g
  x" b% P, Z& D4 J$ r# ~# p
/ s7 f4 u; ?; g$ w8 u- y+ a如果一直修复不成功，可以查看卡住PG的具体信息，主要关注recovery_state，命令如下
- Q9 C7 ^( P- H: A9 n

# W) x2 z) T3 [  x
[root@k8snode001 ~]# ceph pg 2.2b  query

{
5 V5 o0 J; T2 e1 i
    "......
3 Q) P3 k9 p6 a0 v$ U# {
    "recovery_state": [

        {

$ r2 I& D+ `* h2 F9 O            "name": "Started/Primary/Active",
2 C8 x6 r# y0 f1 l
8 E1 }6 E3 e5 }, V0 J+ h            "enter_time": "2020-07-21 14:17:05.855923",
# R! S1 _4 y+ g, O( m
2 B+ H( D0 p, c2 p            "might_have_unfound": [],

            "recovery_progress": {
: {4 m1 m& A9 X
9 z" |) K8 e8 e/ \8 ?2 n                "backfill_targets": [],

3 k- A! K3 T% }$ H; Q( v                "waiting_on_backfill": [],
4 `+ |0 q% \; D( l! s2 i
                "last_backfill_started": "MIN",
: a% Y& d! U  u) ~3 i
, Z; P' {" |6 ^% D) P                "backfill_info": {

2 I# X/ X1 B+ A7 r5 N! p' J- [                    "begin": "MIN",

                    "end": "MIN",
* [, N# {0 a1 M) K( |  L+ ?
                    "objects": []
  o9 g4 ^1 w" n4 O, W; e# P% |+ t
                },

- a2 p, V' }4 z                "peer_backfill_info": [],
, }3 P0 m" g" U. U% w% @+ I7 ?
/ u5 V' c& u6 F9 \! {% c                "backfills_in_flight": [],
" x1 s3 [" d3 h; o
1 T4 u% s/ [; B                "recovering": [],
* @3 @( o! S$ T4 b% q# U
                "pg_backend": {
  |( I' H$ E8 f% N& E" w+ r' [
                    "pull_from_peer": [],

                    "pushing": []
( ~: _1 y  D4 v$ {6 Q# l4 x; `
                }
0 Y7 b7 G$ N4 `% O
% W1 m- Z  S, u! R            },

            "scrub": {

                "scrubber.epoch_start": "10370",

* d- z1 |& B& I/ o+ b4 y                "scrubber.active": false,
0 H8 j, S& b# B+ \1 F# f: v
                "scrubber.state": "INACTIVE",

" M+ Y$ S$ |$ z9 V$ M                "scrubber.start": "MIN",
+ N  ]6 F5 Q, J% O9 y& \$ m
3 P# o2 |2 E- @2 r3 G0 M7 v# \' O                "scrubber.end": "MIN",

                "scrubber.max_end": "MIN",
( P! Q/ l% m: |3 p
7 Z9 j# V5 b* `4 j4 l                "scrubber.subset_last_update": "0'0",

6 G$ F# T/ `4 `. v                "scrubber.deep": false,
* J: X% m- l- I
                "scrubber.waiting_on_whom": []
& P5 Q  P0 O5 \- j/ ~
            }
3 a. U: ^8 ^- M# G) k
; H  ?5 l" A2 |        },

        {
, ?9 C1 K& `" p* a/ g6 h
0 D# Q4 g* r; @2 {* F7 n            "name": "Started",

            "enter_time": "2020-07-21 14:17:04.814061"
( Y3 S% \) ^1 `8 M- R6 b1 {
( M$ L& s* a2 J# {$ m        }
: [  G5 P9 o$ T$ t/ y
    ],
, q$ z- l6 E! r# S2 w% j8 W
, j/ M; b; u5 p* W: W    "agent_state": {}
3 n8 `; X) j' H  V
}

+ {8 Y  P3 M- w& x! U
: G; a& y4 j+ k3 g. ~% N: ?
如果repair修复不了；两种解决方案，回退旧版或者直接删除
( B! a: u2 e4 M3 @- |+ G+ h/ y: ?
) f# w/ r0 N( Y  [& S5.解决方案
回退旧版
  K3 b! n: _' `  k, j# ]
0 B: [( ]& }& L; t6 G0 n1 D) p[root@k8snode001 ~]# ceph pg  2.2b  mark_unfound_lost revert

' A8 \* N1 O/ K2 l# f直接删除

" C4 N7 O, [4 t3 B% m9 w* N[root@k8snode001 ~]# ceph pg  2.2b  mark_unfound_lost delete

5 I* c6 B- A; m
( y1 a% M1 t" G: [! ?
6.验证
7 W! E8 \7 @, h% K  [9 H我这里直接删除了，然后ceph集群重建pg,稍等会再看，pg状态变为active+clean
% J2 f( y- Y% o
; s- A; ^1 z  I6 @4 }[root@k8snode001 ~]#  ceph pg  2.2b query
; h/ R9 u: d, L7 L3 n
* C# i, Q: a3 n) a, _& w& O{
' R! H% D; v/ b
    "state": "active+clean",

. T. r( f' q" [& d    "snap_trimq": "[]",
. ]* }! y, Y$ `2 r& h
, w8 j$ Z1 X5 `" F1 n    "snap_trimq_len": 0,
! W. s" n" e* f9 q, l) L8 S! p
    "epoch": 11069,
3 B0 F2 i4 I0 u" I; V( P7 t! A( K
    "up": [
; a! V: V# @% I# f& X5 t1 j
1 J. F; [, U1 I2 E        12,
! O7 D, g; c1 Z0 s* M. ^+ x# ?
        22,
% y! G3 E6 Q: H1 R' ~
        4
) P  o  T, {, M7 U1 |! p( G, s
# A  _/ `5 b5 x" O9 h    ],
6 A7 o* X3 U  t4 c

1 i8 G  a( N2 N1 a; b
再次查看集群状态


8 }# A- D& d9 l7 F
[root@k8snode001 ~]# ceph health detail
$ @1 n, W0 `6 y; L$ M
$ \: ~- V" z* D1 u6 RHEALTH_OK