记一次Possible data damage: 2 pgs recovery_unfound处理过程

admin

Possible data damage: 2 pgs recovery_unfound
' d8 v5 t; z. J  B+ i

( R* o- m1 ^* q; ?4 c4 T8 {记一次Possible data damage: 2 pgs recovery_unfound处理过程
7 _# H3 x5 Y) d  p[root@gm268-2 ~]# ceph health detail
HEALTH_ERR 1 clients failing to respond to capability release; 1 MDSs report slow metadata IOs; 1 MDSs report slow requests; 1 MDSs behind on trimming; 9/23985260 objects unfound (0.000%); Reduced data availability: 10 pgs inactive, 10 pgs down; Possible data damage: 2 pgs recovery_unfound; Degraded data redundancy: 5523/47970520 objects degraded (0.012%), 2 pgs undersized
- B* [2 Q$ V! A) B[WRN] MDS_CLIENT_LATE_RELEASE: 1 clients failing to respond to capability release
    mds.cephfs.gm268-3.ppyjrl(mds.0): Client gm268-2 failing to respond to capability release client_id: 205033
[WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs
( {; Q0 v7 B1 g    mds.cephfs.gm268-3.ppyjrl(mds.0): 20 slow metadata IOs are blocked > 30 secs, oldest blocked for 237450 secs
[WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests
    mds.cephfs.gm268-3.ppyjrl(mds.0): 11 slow requests are blocked > 30 secs
[WRN] MDS_TRIM: 1 MDSs behind on trimming
6 f: M5 L4 O3 K4 x    mds.cephfs.gm268-3.ppyjrl(mds.0): Behind on trimming (268/128) max_segments: 128, num_segments: 268
/ Y0 h( x1 k( O[WRN] OBJECT_UNFOUND: 9/23985260 objects unfound (0.000%)
/ x: e7 K3 K9 n1 g- G- P    pg 2.7fc has 5 unfound objects
3 I: K- c4 m8 C# b    pg 2.f70 has 4 unfound objects
/ _% W9 ?) R4 D6 ^( f8 {1 G[WRN] PG_AVAILABILITY: Reduced data availability: 10 pgs inactive, 10 pgs down
    pg 2.512 is down, acting [20,0]
" q6 m1 A5 X' ]1 \9 O9 J) V( V    pg 2.8d4 is down, acting [20,33]
7 ?8 `3 L6 x+ ?. H5 ^# y( @9 W    pg 2.905 is down, acting [20,10]
2 M$ Q' c6 b$ A    pg 2.f62 is down, acting [20,19]
5 h6 q+ ]* K' I    pg 2.fb4 is down, acting [20,12]
    pg 2.1660 is down, acting [20,6]
) ?% [! Q# ?& ^9 {1 T    pg 2.1752 is down, acting [20,16]
    pg 2.18b0 is down, acting [20,23]
$ j8 I4 t4 Y2 x, f2 S  U3 x9 M    pg 2.1ba5 is down, acting [20,5]
    pg 3.a9 is down, acting [20,34]
3 m6 W3 `1 F9 e5 D: _0 n  ~[ERR] PG_DAMAGED: Possible data damage: 2 pgs recovery_unfound
    pg 2.7fc is active+recovery_unfound+undersized+remapped, acting [23], 5 unfound
    pg 2.f70 is active+recovery_unfound+undersized+remapped, acting [23], 4 unfound
2 h" l4 [- P8 Z" e[WRN] PG_DEGRADED: Degraded data redundancy: 5523/47970520 objects degraded (0.012%), 2 pgs undersized
2 T3 y' l2 k1 v7 d- J; w    pg 2.7fc is stuck undersized for 21m, current sta

, m+ p2 [4 v- @$ X3 r

admin

ceph pg deep-scrub $pgid

After that you should try to repair the PGs
, N2 n9 D2 d1 u8 p* K; m
% g5 G3 J- z0 G8 D# q6 Sceph pg repair $pgid
) o+ c4 u* e  |( w
1 d6 q# {: l, D: d: X& K$ h# QMaybe Ceph is able to find the missing objects again. But it could be that they are lost.

admin

1.查看集群状态
7 g: e2 T" O/ S[root@k8snode001 ~]# ceph health detail
- W% x6 U) ?4 A6 ~' i* V8 U% J! I
HEALTH_ERR 1/973013 objects unfound (0.000%); 17 scrub errors; Possible data damage: 1 pg recovery_unfound, 8 pgs inconsistent, 1 pg repair; Degraded data redundancy: 1/2919039 objects degraded (0.000%), 1 pg degraded
# T: o( s" }8 u% h6 h. p
OBJECT_UNFOUND 1/973013 objects unfound (0.000%)
% H4 y4 R4 _  F) K1 Z* i
' O! j% _3 O- V: ?0 O& ^    pg 2.2b has 1 unfound objects
1 x8 J/ U& G# l! P" D- @
4 q% `4 O. R0 p! J. MOSD_SCRUB_ERRORS 17 scrub errors
4 l  u: {4 h% N/ n! g
& I& o  X# R% EPG_DAMAGED Possible data damage: 1 pg recovery_unfound, 8 pgs inconsistent, 1 pg repair

( y2 f2 ?; G% N8 H6 O$ X+ |    pg 2.2b is active+recovery_unfound+degraded, acting [14,22,4], 1 unfound

6 w$ x) H2 \' T( E4 J    pg 2.44 is active+clean+inconsistent, acting [14,8,21]
  n" C. V9 V6 D' t) Z% C5 ^4 Z
    pg 2.73 is active+clean+inconsistent, acting [25,14,8]
8 \' r  [- l3 w5 `+ l5 i
1 [' x' `* |( G4 N    pg 2.80 is active+clean+scrubbing+deep+inconsistent+repair, acting [4,8,14]
6 N  e9 \! L9 ^! i: _  F( F
4 d% y; j9 n7 V% W3 O4 d    pg 2.83 is active+clean+inconsistent, acting [14,13,6]

    pg 2.ae is active+clean+inconsistent, acting [14,3,2]

    pg 2.c4 is active+clean+inconsistent, acting [8,21,14]

4 J! ^( O! {- Z5 C4 N, \: b    pg 2.da is active+clean+inconsistent, acting [23,14,15]
- R; d1 O! E, U# d- D7 V$ a9 P+ [
+ s7 G! ]  A; r9 }8 Y( J7 v    pg 2.fa is active+clean+inconsistent, acting [14,23,25]
  S  X% Y  k( n+ V
# P4 l# g, l/ O1 V: C( k2 gPG_DEGRADED Degraded data redundancy: 1/2919039 objects degraded (0.000%), 1 pg degraded

3 i! T" S" C* F+ a    pg 2.2b is active+recovery_unfound+degraded, acting [14,22,4], 1 unfound
, M' j0 h, Q3 s& V4 [  e& n

8 T6 \) w0 U  p# b
从输出发现pg 2.2b is active+recovery_unfound+degraded, acting [14,22,4], 1 unfound
) U- M& M0 j3 n
7 G: j' B& ~2 I6 N3 o2 {) z. w  M现在我们来查看pg 2.2b，看看这个pg得想想信息。


  u% s5 y: {: _3 R7 d
/ s3 f0 H  z% N6 d8 E3 x1 Q[root@k8snode001 ~]# ceph pg dump_json pools    |grep 2.2b

$ p. \3 d/ m, m, j2 fdumped all

4 h9 m" x% B1 N' O2 u2.2b       2487                  1        1         0       1  9533198403 3048     3048                active+recovery_unfound+degraded 2020-07-23 08:56:07.669903  10373'5448370  10373:7312614  [14,22,4]         14  [14,22,4]             14  10371'5437258 2020-07-23 08:56:06.637012   10371'5437258 2020-07-23 08:56:06.637012             0
! ~# @' u! x1 C+ L; p

' t2 W9 O& Q4 A7 S
可以看到它现在只有一个副本

* {" ?) O0 I+ D) |; W4 x, P1 M2.查看pg map
. w. [1 C/ A3 |8 t+ I4 {* j, T

[root@k8snode001 ~]# ceph pg map 2.2b

. E( r0 \' i7 s8 g% s' n! Mosdmap e10373 pg 2.2b (2.2b) -> up [14,22,4] acting [14,22,4]

2 z( [" `+ b' d) p5 V, X, ^* O
; y0 @- u" ~  L$ W
" O% _' {8 ?2 h6 k" Z- `7 P从pg map可以看出，pg 2.2b分布到osd [14,22,4]上


3.查看存储池状态

* d: s% L$ p7 D. Q7 v
[root@k8snode001 ~]# ceph osd pool stats k8s-1
4 R9 l) l& G8 }+ `; G6 w
pool k8s-1 id 2

  1/1955664 objects degraded (0.000%)
3 m6 g0 A  [! V; Y9 L
: P, M( u: M# T: ~# E  1/651888 objects unfound (0.000%)

. M5 Q" j* O4 ~5 W/ J  client io 271 KiB/s wr, 0 op/s rd, 52 op/s wr
5 H7 K: o/ J) j

; B4 G! j0 Q6 h: _) r6 A
[root@k8snode001 ~]# ceph osd pool ls detail|grep k8s-1
% x; U5 c. e* O2 U) G2 M4 @) l. M
pool 2 'k8s-1' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 88 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd



4.尝试恢复pg 2.2b丢失的块
5 ~6 B& ]. A; Z# @6 [[root@k8snode001 ~]# ceph pg repair 2.2b



6 k) v# B& r6 E% c( w( N* G如果一直修复不成功，可以查看卡住PG的具体信息，主要关注recovery_state，命令如下
# g$ B- N! P% B% q6 [: K; C
% M7 E! z, S3 A. H6 H/ q4 K$ s

3 l6 b% g' o3 w# w# G; X[root@k8snode001 ~]# ceph pg 2.2b  query

8 T$ j% t7 m" n/ y{
+ A$ m& T! l, F) h! z+ G
    "......

. X& U! G1 C7 Z- Z    "recovery_state": [
1 D9 o# |5 H/ m7 _' |( l' ~
: L% E  E1 H; V0 }& b5 A* \        {

0 s, w* v4 v1 x, E& E& o, C            "name": "Started/Primary/Active",

9 O( h6 _2 D9 q+ O" e! n! ~            "enter_time": "2020-07-21 14:17:05.855923",
/ V+ S4 g  Q2 n: p7 ?  y
# {% M9 p( E2 C+ ~; H- V            "might_have_unfound": [],

            "recovery_progress": {
1 c$ L" U1 t) b: J
. e  v! _: ^! U: l9 V/ K' ?7 }- \& S8 C                "backfill_targets": [],

                "waiting_on_backfill": [],

                "last_backfill_started": "MIN",
! Q% F0 Q, r2 `! x
! e2 I1 m% v- f  K2 J                "backfill_info": {

                    "begin": "MIN",

' ]7 N# }4 D8 o6 _# h* v: _" N                    "end": "MIN",

. L, ?. A: S$ v3 n3 l                    "objects": []
" \, o& X5 q+ ]" I7 P4 `
3 ?4 |9 J8 y7 K% b7 y                },
0 l# h0 ]; D8 {
, K( b, V0 ?5 v0 x; A# G+ l" g                "peer_backfill_info": [],
$ B- T, A  H5 [
. {' y3 X2 o8 A7 E3 Y                "backfills_in_flight": [],
' Q5 J+ z, ^; v! K- \& u1 h
                "recovering": [],
# ~! v9 p( D' M# P; L+ w2 Z7 w
                "pg_backend": {
: Z/ A( Z2 A& Y3 ~; S& s; X
  E: `0 |  Q( c2 L4 R, E2 O                    "pull_from_peer": [],

                    "pushing": []
# q8 y& d% y! n6 ]. D
                }

7 [4 O$ p0 S# d- t. I; l# N& L& S: f            },
) c1 i. G: D" b/ S5 O* X( [
$ N4 S# G" L/ {; q6 S$ V; c            "scrub": {

                "scrubber.epoch_start": "10370",

                "scrubber.active": false,

                "scrubber.state": "INACTIVE",
* Q9 v) m6 c* Y) ?0 q3 T* o' R
                "scrubber.start": "MIN",

/ K& G; J: \, x                "scrubber.end": "MIN",

                "scrubber.max_end": "MIN",
2 K' n+ \0 E) D
                "scrubber.subset_last_update": "0'0",

5 Y4 C+ C/ \. {& X                "scrubber.deep": false,

                "scrubber.waiting_on_whom": []

            }
: W4 [$ l2 r  j( B
        },

& y6 y7 {/ l/ N) q$ D' X9 }% `        {
4 j% m6 s- ~4 p1 K' _
: P1 f7 e# E5 G5 o# {            "name": "Started",

            "enter_time": "2020-07-21 14:17:04.814061"

        }
/ Z% x! R8 I- ]" @
; q  Q( M( a0 `8 \9 O1 |% D1 v    ],

3 w# ~* l2 S% g; f4 w    "agent_state": {}

# |% K5 S! W' x# ]: \8 l}
5 s! @2 b/ d, p, J

* k7 m; f. g2 g/ h" p. z6 b
如果repair修复不了；两种解决方案，回退旧版或者直接删除

5.解决方案
回退旧版

[root@k8snode001 ~]# ceph pg  2.2b  mark_unfound_lost revert
. |* `+ V4 V0 b( n4 `6 n
. b+ g$ j7 a* ]0 g1 Z) v直接删除

: K/ y& E% t- S* n+ M[root@k8snode001 ~]# ceph pg  2.2b  mark_unfound_lost delete
( \: \* I) M- o4 {+ n1 r


6.验证
我这里直接删除了，然后ceph集群重建pg,稍等会再看，pg状态变为active+clean
7 e6 a$ {2 E0 r( r, E
[root@k8snode001 ~]#  ceph pg  2.2b query
, |  x4 X2 d1 ]; m
3 L5 t6 x8 b6 J% \5 F{
# L# ^* g. Z3 u- O* d: X. Z: `
    "state": "active+clean",
6 U: C/ b* C6 \% K
    "snap_trimq": "[]",

    "snap_trimq_len": 0,

4 \, m& j1 L8 u    "epoch": 11069,
- }+ d; R( T* C0 F& l6 w( O
6 G0 O0 T5 @' e7 }! W* V+ [    "up": [

8 L/ y8 q0 j0 }$ ~. H5 S" J4 a        12,
" e1 I3 N$ ^. H/ c5 _
4 X+ r6 O' V" N6 k2 _        22,

7 c6 A+ _( W) _9 `        4
2 Y& M$ y; W+ J. m0 b0 U& M1 J* f0 m
, \: j( G9 ]! _- Q- D9 k: t    ],
( ^8 C" v1 C# G; I) s; y5 ?
2 ~: {3 u7 p" u; x( j( y$ I; u

) L( f6 F/ y7 p0 Z+ s再次查看集群状态

* M$ D, b1 E$ p  s: Z" x. S: @
* c, O2 ]1 u6 y
[root@k8snode001 ~]# ceph health detail
/ {( f: r. l& K
HEALTH_OK