记一次Possible data damage: 2 pgs recovery_unfound处理过程

admin

Possible data damage: 2 pgs recovery_unfound

- [! N" G; d: u( W8 X8 E8 T. z
记一次Possible data damage: 2 pgs recovery_unfound处理过程
[root@gm268-2 ~]# ceph health detail
HEALTH_ERR 1 clients failing to respond to capability release; 1 MDSs report slow metadata IOs; 1 MDSs report slow requests; 1 MDSs behind on trimming; 9/23985260 objects unfound (0.000%); Reduced data availability: 10 pgs inactive, 10 pgs down; Possible data damage: 2 pgs recovery_unfound; Degraded data redundancy: 5523/47970520 objects degraded (0.012%), 2 pgs undersized
5 h9 `: ?& m: y0 H4 a[WRN] MDS_CLIENT_LATE_RELEASE: 1 clients failing to respond to capability release
, u# T2 ~5 @) h+ I9 @8 y3 j    mds.cephfs.gm268-3.ppyjrl(mds.0): Client gm268-2 failing to respond to capability release client_id: 205033
: s$ u8 `5 ]1 B* U7 d- {, t[WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs
" v+ T0 F9 t3 J    mds.cephfs.gm268-3.ppyjrl(mds.0): 20 slow metadata IOs are blocked > 30 secs, oldest blocked for 237450 secs
8 m( h0 B0 n5 O! v2 T; J  R[WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests
    mds.cephfs.gm268-3.ppyjrl(mds.0): 11 slow requests are blocked > 30 secs
( J5 }/ V# Z+ E) U( p[WRN] MDS_TRIM: 1 MDSs behind on trimming
- [1 `/ N5 ~& H8 p, C6 b    mds.cephfs.gm268-3.ppyjrl(mds.0): Behind on trimming (268/128) max_segments: 128, num_segments: 268
[WRN] OBJECT_UNFOUND: 9/23985260 objects unfound (0.000%)
    pg 2.7fc has 5 unfound objects
) J! _* P5 b' c( h& a    pg 2.f70 has 4 unfound objects
' M2 K. G  K, T/ w8 f- N. Q& ?[WRN] PG_AVAILABILITY: Reduced data availability: 10 pgs inactive, 10 pgs down
    pg 2.512 is down, acting [20,0]
    pg 2.8d4 is down, acting [20,33]
    pg 2.905 is down, acting [20,10]
    pg 2.f62 is down, acting [20,19]
. \( C" q# t; o- A    pg 2.fb4 is down, acting [20,12]
    pg 2.1660 is down, acting [20,6]
    pg 2.1752 is down, acting [20,16]
4 g$ M9 P' X% Z3 U7 W% {! ?    pg 2.18b0 is down, acting [20,23]
9 ?% Y/ N# ~3 z6 J$ A" o% ~    pg 2.1ba5 is down, acting [20,5]
    pg 3.a9 is down, acting [20,34]
[ERR] PG_DAMAGED: Possible data damage: 2 pgs recovery_unfound
" u7 f- @- c; Y6 L+ I$ c$ S7 G" y    pg 2.7fc is active+recovery_unfound+undersized+remapped, acting [23], 5 unfound
    pg 2.f70 is active+recovery_unfound+undersized+remapped, acting [23], 4 unfound
[WRN] PG_DEGRADED: Degraded data redundancy: 5523/47970520 objects degraded (0.012%), 2 pgs undersized
    pg 2.7fc is stuck undersized for 21m, current sta
0 x% P+ l. W2 `8 A

admin

ceph pg deep-scrub $pgid
6 ~. q) o3 X! M% i! \7 D
After that you should try to repair the PGs

ceph pg repair $pgid
7 f/ c8 l+ h/ N+ r& g( i1 H
Maybe Ceph is able to find the missing objects again. But it could be that they are lost.

admin

1.查看集群状态
[root@k8snode001 ~]# ceph health detail

HEALTH_ERR 1/973013 objects unfound (0.000%); 17 scrub errors; Possible data damage: 1 pg recovery_unfound, 8 pgs inconsistent, 1 pg repair; Degraded data redundancy: 1/2919039 objects degraded (0.000%), 1 pg degraded
. U) j% a4 R( F0 k2 l% @! M6 Z1 C' D
) w2 D. Q9 o' q8 y$ a% I- rOBJECT_UNFOUND 1/973013 objects unfound (0.000%)
$ n4 t& Z% N) R  g# ^
    pg 2.2b has 1 unfound objects

7 A/ @/ ]7 E) t" ^OSD_SCRUB_ERRORS 17 scrub errors
% O0 {: @' p: O
0 l* F- G; e% O( D1 P1 b% @PG_DAMAGED Possible data damage: 1 pg recovery_unfound, 8 pgs inconsistent, 1 pg repair
6 X+ P) y$ }/ ]0 }. [% G2 R
. [" D4 u; F; i6 H    pg 2.2b is active+recovery_unfound+degraded, acting [14,22,4], 1 unfound
6 A  e( M% Q4 }7 b5 I0 I$ m
    pg 2.44 is active+clean+inconsistent, acting [14,8,21]
5 \0 E0 E$ T' A
    pg 2.73 is active+clean+inconsistent, acting [25,14,8]
9 ~, O, I" z: p3 Z3 S
    pg 2.80 is active+clean+scrubbing+deep+inconsistent+repair, acting [4,8,14]

    pg 2.83 is active+clean+inconsistent, acting [14,13,6]

    pg 2.ae is active+clean+inconsistent, acting [14,3,2]

8 W2 m3 T) y! u- ~    pg 2.c4 is active+clean+inconsistent, acting [8,21,14]
" x6 O  u" t* U! S3 l8 b
    pg 2.da is active+clean+inconsistent, acting [23,14,15]
& u( I7 v/ C0 I, v# ?8 K1 f
7 L2 M5 S( N- x  P    pg 2.fa is active+clean+inconsistent, acting [14,23,25]
# C/ D/ h/ [4 l/ |
PG_DEGRADED Degraded data redundancy: 1/2919039 objects degraded (0.000%), 1 pg degraded

; t5 m# w7 G  Z$ `3 v    pg 2.2b is active+recovery_unfound+degraded, acting [14,22,4], 1 unfound
5 ?+ s* \) y' P2 O


0 h0 T6 B- b% X8 l3 Z$ u5 K4 L从输出发现pg 2.2b is active+recovery_unfound+degraded, acting [14,22,4], 1 unfound

现在我们来查看pg 2.2b，看看这个pg得想想信息。
' p, r% M, d6 M- O3 w

9 W3 _4 r8 Q. I, b1 c( |
8 R& `6 x* B/ l/ |' d1 Q[root@k8snode001 ~]# ceph pg dump_json pools    |grep 2.2b

  r3 E- o1 e: \# _/ |' A8 gdumped all
# c- a7 ~: y" F, j
2 S' @8 a1 ~4 Y# m" [4 d, K2.2b       2487                  1        1         0       1  9533198403 3048     3048                active+recovery_unfound+degraded 2020-07-23 08:56:07.669903  10373'5448370  10373:7312614  [14,22,4]         14  [14,22,4]             14  10371'5437258 2020-07-23 08:56:06.637012   10371'5437258 2020-07-23 08:56:06.637012             0
( g( R) H" @$ y8 _' D& d& a0 U+ {


9 L' W( w* l% O0 g; z/ ?可以看到它现在只有一个副本

2.查看pg map


[root@k8snode001 ~]# ceph pg map 2.2b
7 u# k, N8 K1 t' D- a! [3 ~- E+ r( G
osdmap e10373 pg 2.2b (2.2b) -> up [14,22,4] acting [14,22,4]



从pg map可以看出，pg 2.2b分布到osd [14,22,4]上


. H" k; @! [1 z* x3.查看存储池状态


[root@k8snode001 ~]# ceph osd pool stats k8s-1
, U" b" U4 X( V8 S% G+ G# c
2 K8 r; z" \+ ]pool k8s-1 id 2

& @6 Q+ j* ^6 V" N1 k6 @9 f  1/1955664 objects degraded (0.000%)
6 Z8 ^- ?. s: k- U, R7 |" }
9 W4 a" o/ H7 t# K  1/651888 objects unfound (0.000%)

9 V/ D; ^8 P$ |5 C: ?( b7 Z  client io 271 KiB/s wr, 0 op/s rd, 52 op/s wr
3 W# f; S6 n( y3 b

9 D0 h4 ?. ~2 a' p6 _! t5 o
[root@k8snode001 ~]# ceph osd pool ls detail|grep k8s-1

2 D$ \( C0 H" G9 \pool 2 'k8s-1' replicated size 3 min_size 1 crush_rule 0 object_hash rjenkins pg_num 256 pgp_num 256 last_change 88 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd


* U- O9 X8 @/ s& {) {$ i
! V8 l: n3 J" _5 K4.尝试恢复pg 2.2b丢失的块
) b# u9 I0 C3 h1 A* ~6 d[root@k8snode001 ~]# ceph pg repair 2.2b
7 N0 `# l) Z/ e
# M  x% B( C+ O7 C! v) M8 N

& r+ [) j( c) n如果一直修复不成功，可以查看卡住PG的具体信息，主要关注recovery_state，命令如下
& y; }7 p9 p# i


[root@k8snode001 ~]# ceph pg 2.2b  query

# D' m. X, F/ r{
; ?& n' i( R, z% ^8 z3 |+ }( B6 R
* }7 u( [# N: `1 L& r    "......
* z, c7 P0 T. [) g$ |) B
) D/ R1 o# }1 k! _9 D* Y    "recovery_state": [

- D2 N2 ^) K4 J' b8 c4 ~. [: S        {
, D( o' {# j7 I& s
            "name": "Started/Primary/Active",
! H( g& N, I. W7 a4 p, B# E
            "enter_time": "2020-07-21 14:17:05.855923",
3 ?" s: ~% _. v& c/ C! `! w
: z4 a# `. _: n' E; D1 _5 S            "might_have_unfound": [],

  T' O: h6 E' r  W& f6 l" H6 F% k            "recovery_progress": {

, m6 t* r7 I- A                "backfill_targets": [],

5 C, @0 q" b2 n4 v( W8 I                "waiting_on_backfill": [],

! N  g& U' E% b& c                "last_backfill_started": "MIN",
" Y  ?5 k5 x, f* B- G
                "backfill_info": {
) G( Z5 t6 C8 ^( n
. n* `% q# b3 P                    "begin": "MIN",

                    "end": "MIN",

                    "objects": []

9 R5 m5 F* i8 S' v/ P+ s% d( K                },

                "peer_backfill_info": [],
1 _1 e7 j$ ]& c: p/ x
7 @& w6 ?. h0 }) i                "backfills_in_flight": [],

8 `3 B  m) T* J: y" L+ M! r                "recovering": [],
3 b& M7 X; X' |' T1 u- \% b% U# d
* E5 R' u4 {5 a4 O$ J                "pg_backend": {

% a' q+ M" E% O% ?                    "pull_from_peer": [],

8 w1 n; \* D0 L* q3 w' n; |9 v4 b                    "pushing": []

( Z7 |, b) O  |2 V8 P9 I1 d9 g1 f                }

7 H. @# B1 M* f& o! x3 u            },

            "scrub": {
% h9 W+ P, I: R* {
                "scrubber.epoch_start": "10370",
/ i  E6 ]$ H" W* k4 y" J, n) A
; D4 |/ m$ \. _6 D7 d0 s; z5 n                "scrubber.active": false,
" c) m! P$ _! Z  T0 z
                "scrubber.state": "INACTIVE",

                "scrubber.start": "MIN",

                "scrubber.end": "MIN",
3 y2 ^  B, h2 E$ k5 D5 f8 f
+ P0 J4 k. L9 w0 ]+ h5 p: |                "scrubber.max_end": "MIN",

) M9 G9 F$ a9 f& B/ n7 `                "scrubber.subset_last_update": "0'0",

                "scrubber.deep": false,
# W) ]/ Z3 I; C0 k3 n
" L  Y5 w1 L/ t                "scrubber.waiting_on_whom": []

7 b0 n( z9 G- E2 _7 {3 h- d( s            }
  Y5 m9 L9 Q  s: @' J
        },

        {
& |/ ~6 a9 L0 O5 H4 C. _* N
            "name": "Started",

/ q5 g$ ^2 Y/ e. @0 A9 w            "enter_time": "2020-07-21 14:17:04.814061"
: D' M7 |7 G* i# ]- ?
        }
9 }% I( d* ~0 s
+ H* k7 D; ^7 u; V    ],
- T) O8 t4 ?  D( k7 U, o& ]
9 X. W2 a6 w/ _1 y' C5 ^; t: S/ l" Q    "agent_state": {}
% T" W1 h4 G0 F1 m% [2 k8 L! |
+ H* v' N+ \, |% ~3 f  Y}
0 }9 ]8 p! x% Q4 Y) W
% r. g& i+ q6 f

. R* S/ a! }8 b+ C, N1 Q* X! V- g如果repair修复不了；两种解决方案，回退旧版或者直接删除

. d& Y3 Z6 z# y5.解决方案
回退旧版
: _1 q, g7 f1 j- ]. v. J3 H) l
[root@k8snode001 ~]# ceph pg  2.2b  mark_unfound_lost revert
( f$ d, H4 ]# S9 {
8 i0 T- x0 r1 V+ b  Y3 ^( v" D6 V: X$ x直接删除

7 u* m5 x6 F" E$ n# \3 s[root@k8snode001 ~]# ceph pg  2.2b  mark_unfound_lost delete
- a* G- `  c. V% k( }) f
1 ~" v; e! K/ t
' e" a2 R9 n9 {; V0 r8 h( a
! H) {0 O" X  m* M7 F) {/ ?6.验证
我这里直接删除了，然后ceph集群重建pg,稍等会再看，pg状态变为active+clean
9 S; D7 {$ ?: [' K7 G1 b. g
0 \# L0 n( t) n3 l2 \! q! m7 Y$ P[root@k8snode001 ~]#  ceph pg  2.2b query
& S4 _% V3 d6 G: R# U: T& l' {
{

    "state": "active+clean",

# \8 z1 x& f+ R# w. ]3 c9 L    "snap_trimq": "[]",
5 V# J0 c$ B$ H7 `
    "snap_trimq_len": 0,
3 \4 }& Q* l( M7 R
5 P3 n; B8 U1 L    "epoch": 11069,
8 i1 R2 K  s; m- K2 u1 l
9 r  V. s/ H1 n- T5 f+ B    "up": [
: L* {5 ]) P& z. M
        12,
, M9 n1 Z1 O0 o, Y1 g8 H8 B& S, M
        22,
3 t& }$ ^0 x7 d6 B
, r0 v# J: n" G- M( j3 J! G        4
" w$ z) m: w2 Q$ h6 Y
% x8 f1 f. Z; k+ y: R% e    ],

2 t$ }* D  P) N" [* j. |# h

1 u! _% e$ G; v" F0 N8 i1 E! L再次查看集群状态
! @' W, S" l: E3 E, |3 H3 m7 R. s% M

& X. F, W  v' V/ V0 g1 ]( _, B, A& Z
6 M* g2 L, B# W0 E% L& Y[root@k8snode001 ~]# ceph health detail
9 z. O! c4 f3 I2 D) {
HEALTH_OK

7 A( \- |% p2 ~